As machine learning has grown up and shifted from a research novelty to an industrial-strength technology, the tools and systems required to run machine learning at scale have developed as well. For startups, making use of these improvements creates both upside and danger – nearly all of them are using machine learning in some form as they fight for a share of their markets.
The path to this point started just over 9 years ago, when the deep learning wave was set in motion by a 2012 entry named AlexNet in the annual ImageNet LSVRC competition (a computer vision contest organized by the research community). In that entry, a three-person team (Alex Krizhevsky, Illya Sutskever, and Geoffrey Hinton) applied a method called a convolutional neural network to identify what was in photos. They took the competition easily – outperforming every other team by a wide margin – and did so using a system trained on a $700 computer graphics card made for video games.
Machine learning changed permanently. In less than a year, startups started appearing to copy AlexNet. My former company, AlchemyAPI (bought by IBM in 2015), shipped one of the earliest commercial implementations of this research with our AlchemyVision computer-vision API in 2013. Other startups created around that period include DeepMind (acquired by Google), MetaMind (acquired by Salesforce), and Clarifai, among many others. Academia also underwent a major shift, as many specialists moved very rapidly from doubting artificial intelligence to embracing deep learning wholeheartedly.
Fast-forward to 2022: neural networks now influence nearly every part of machine intelligence in the software we rely on every day, from speech recognition to the systems that choose what appears in our news feeds (for better or worse). Modern systems still use neural networks – but at an enormously larger scale. Recent systems for understanding and producing human language, such as OpenAI’s GPT-3, were trained using supercomputer-level resources: thousands of GPUs (each priced at $10,000 or more) linked together with a sophisticated mesh of high-speed networking and data-storage equipment. Whereas the leading systems in 2012 could be trained on a $700 gaming card, the leading systems today – often called foundation models – probably demand tens of millions of dollars in compute to train.
The rise of these huge, expensive foundation models creates opportunity, risk, and constraints for startups and others trying to push artificial intelligence and machine learning forward. Even if they probably can’t go head-to-head with Google, Facebook, or OpenAI at the frontier of research, smaller organizations can make use of the work produced by these giants, including foundation models, to jump-start the creation of their own machine-learning-driven products.
Pre-trained networks give smaller teams a leg up
Neural networks like AlexNet were first trained from the ground up for each job – feasible when networks took only a few weeks on a single gaming machine, but far harder once network size, computing power, and training-data volume began to grow by orders of magnitude. This helped make popular an approach called pre-training, in which a neural network is first trained on a large, general-purpose dataset using substantial compute, and is then fine-tuned for the specific task using far less data and computing power.
Pre-trained networks have become hugely popular in recent years as machine learning has become more industrialized across many areas (such as language and speech processing) and as the amount of training data has risen dramatically. For a startup, the use of pre-trained networks makes it possible, for example, to create a product with far less data and computing power than would be required if everything began from zero. This method is also gaining traction in academia, where researchers can rapidly fine-tune a pre-trained network for a new assignment and then publish the findings.
In some task areas – including understanding or producing written text, identifying what is in photos or videos, and audio processing – pre-training has kept advancing with the arrival of foundation models like BERT, GPT, DALL-E, CLIP, and others. These models are pre-trained on huge general-purpose datasets (often numbering in the billions of training examples) and are being released as open source by well-funded AI labs such as those at Google, Microsoft, and OpenAI.
The pace of innovation in commercial machine learning applications and the equalizing effect of these foundation models cannot be overstated. They have been a cure-all for people in the field who do not have an idle supercomputer sitting around. They let startups, researchers, and others quickly catch up with the latest machine learning techniques without needing to invest the time and resources required to train such models from nothing.
The risks of foundation models: size, cost, and outsourced innovation
Still, things are not entirely bright in the world of pre-trained foundation models, and their growing adoption carries several risks.
One risk tied to foundation models is their constantly growing size. Neural networks such as Google’s T5-11b (open sourced in 2019) already need a cluster of costly GPUs just to load and generate predictions. Fine-tuning these systems demands even more resources. More recent models built in 2021-2022 by Google/Microsoft/OpenAI are often so large that these companies are not releasing them as open source – they now cost tens of millions of dollars to build and are increasingly treated as major IP investments even by these large companies.
However, even if those newest models were open sourced, simply loading them for prediction making (“inference,” in machine learning parlance) requires bringing up more resources than many startups and academic researchers can easily obtain. OpenAI’s GPT-3, for instance, needs a substantial number of GPUs just to load. Even with modern compute clouds like Amazon Web Services, that would mean provisioning dozens of Amazon’s priciest GPU machines into a high-performance computing cluster.
Dataset alignment can also be difficult for those using foundation models. Pre-training on a large general-purpose dataset does not ensure that the network will handle a new task on proprietary data. The network may lack enough context, or be biased by its pre-training, so much so that even fine-tuning may not quickly fix the problem.
For instance, GPT-2, a widely used foundation model in natural language processing, was first announced in early 2019 and therefore trained on data collected on or before that time. Consider all that has happened since 2019 – pandemic, anyone? The original GPT-2 model will certainly understand what a pandemic is, but it will not have the detailed COVID-19 context and variants that have appeared in recent years.
To show this, here is GPT-2 attempting to finish the sentence “COVID-19 is a …”:
GPT-2 (2019): “COVID-19 is a high capacity LED-emitter that displays information about the size and state of the battery.”
In contrast, GPT-J, an open-source language model released in 2021, completes the sentence like this:
GPT-J (2021): “COVID-19 is a novel coronavirus that mainly affects the respiratory system resulting in a disease that has a wide variety of clinical manifestations.”
A pretty striking difference, isn’t it? Dataset alignment and the freshness of training data can be hugely important depending on the use case. Any startup using foundation models in its machine learning work should pay close attention to these kinds of concerns.
Cloud APIs are easier, but outsourcing isn’t free
Companies such as OpenAI, Microsoft, and Nvidia have recognized the scale problem and are addressing it with cloud APIs that allow inference and fine-tuning of large-scale models to run on their hosted infrastructure. And, naturally, every major cloud provider now supplies a set of machine learning services and, in some cases, custom processors built specifically for these workloads. This can act as a limited pressure-release valve for startups, researchers, and even solo hobbyists by moving the compute and infrastructure burden onto a larger company.
This strategy comes with its own dangers, though. If you cannot host your own model, you must depend on centralized entities for training as well as inference. That can create risks pushed outward in the process of building production-ready machine learning applications: Network outages, API concurrency or rate limits, or even simple policy changes by the hosting company could cause major operational harm. In addition, the possibility of IP leakage may feel troubling to some when sensitive labeled datasets (some of which may be subject to regulations such as HIPAA) need to be sent to cloud providers so that fine-tuning or inference can take place.
From a cost standpoint, the COGS (cost of goods sold) effect of invoking these APIs can also worry teams that rely on cloud vendors for their machine learning requirements. Pricing structures differ from provider to provider, but, needless to say, the expense of API requests, data storage, and cloud compute will rise in step with usage. Many companies that have depended on cloud APIs for machine learning may eventually try to move to self-hosted or self-trained models to obtain greater control over their machine learning pipelines and remove outsourced risks and costs.
The opportunities and risks tied to hosted and pre-trained models have pushed many companies to use cloud APIs during the “experimentation phase” to jump-start product development. This is the stage when a company is trying to achieve product-market fit for its offering. Using cloud APIs can let a company quickly bring its product online at scale without needing to pour money into costly infrastructure, model training, or data collection. Cloud machine learning services and hosted pre-trained models from providers such as Google, IBM, Microsoft, and OpenAI now support thousands of startups and academic research projects.
After a company decides it has product-market fit, it often moves to self-hosted or self-trained models so it can have more control over data, process, and intellectual property. That move can be hard, because the company must scale its infrastructure to meet the model’s demands and also handle the costs tied to data collection, annotation, and storage. Companies are raising ever larger amounts of investor capital to fund this transition.
My newest startup, Hyperia, recently went through that kind of transition. At the beginning, we tried cloud APIs while we were learning to interpret the content of business meetings and customer voice conversations. But later we dove in headfirst, launching large-scale data collection and model training work to create our own proprietary speech and language engines. For many business models, that kind of shift is simply inevitable if they are to reach positive unit economics and market differentiation.
Be strategic and keep an eye on the big AI labs
Foundation models are among the newest disruptive developments in machine learning, though they will not be the last. As companies keep building larger and larger machine learning supercomputers (Facebook’s latest includes more than 16,000 GPUs), researchers are actively creating new methods to cut the computational expense of training and hosting state-of-the-art neural networks. Google’s latest LaMDA model uses several innovations to train more efficiently than GPT-3, and techniques such as model distillation and noisy student training are being rapidly advanced by the research community to shrink model size.
These innovations and others mean startups can keep innovating – but it’s important to stay alert as the landscape keeps shifting. Things worth keeping in mind include:
- Cloud APIs can certainly speed a company’s route to product-market fit, but they often create their own long-term problems. It’s important to have a strategic exit plan so these APIs do not determine your product’s fate. Foundation models can greatly accelerate your machine learning work and lower total training and data collection costs, but being mindful of the limits of these systems (e.g., recency of training data) matters. Keep track of what is emerging from the big corporate AI labs (Google, Microsoft, IBM, Baidu, Facebook, OpenAI, etc). Machine learning is evolving at an extremely fast pace, with new techniques, models, and datasets being released every month. These releases can often arrive at unexpected moments and can have a major effect on your company’s machine learning efforts if you are able to adapt quickly.
In the end, the future of machine learning and its effect on startups and technology companies remains uncertain, but one thing is clear: Companies that know what is available and make wise choices about how to use it will be in a far stronger position to succeed than those merely chasing a quick AI fix.