A Decade of Deep Learning: How the AI Startup Experience Has Evolved

The artificial intelligence field has evolved dramatically since the deep learning revolution kicked off in 2012, and Richard Socher has been around for all of it. He earned his PhD from Stanford working on NLP (natural language processing) before co-founding an AI startup called MetaMind in 2013. He then spent several years leading the AI team at Salesforce (after it acquired MetaMind) before tackling the search space with his new startup, you.com, in 2021.

In this interview, Socher covers several subjects, including how the AI startup landscape has shifted over the past decade, how startups, enterprises, and academia differ when it comes to AI, and how newer machine learning methods such as transformer models let companies create advanced products with only a fraction of the resources they once needed to build them quickly today.

---

It looks like a common path is for AI researchers, students and professors alike, to leave academia for startups, as you did. What are the main differences between those two worlds today?

RICHARD SOCHER: In academia, people still keep pushing to make progress in new areas where AI can matter, and some hope to move toward AGI (artificial general intelligence). I think two especially exciting examples of new, high-impact areas are the protein domain - protein sequences or amino acid sequences - and economics. Economics matters enormously for the world, but AI has not had nearly as much effect there as I believe it should in practice, despite how consequential it may be.

At the same time, startups can do very well when they have lots of data and a workflow that depends mostly on the data they already see, because then you can basically say, 'We know how this works.' You have a radiology image and try to determine, 'Is this bone broken or not?' Or you have a head CT scan and try to determine, 'Is there intracranial hemorrhage or a brain bleed?' Or you classify different cancers from pathology images. All of these uses follow a fairly established pattern: define the problem, gather data for it, train a large neural network, and then optimize and automate parts of the whole process instead of reinventing the entire pipeline from scratch every single time in practice.

And with that proven approach, you can have a lot of impact. It is a bit like electricity: once the basics were understood, you could make a huge difference just by bringing it to a town that had only oil lamps and fire before for everyone living there at the time, profoundly too.

This is possible partly because many interesting and important ideas have emerged over the last 10 years. Things that once seemed impossible - like having an AI write a reasonably long text - are now possible. One major shift is that not only images, but all data, are basically vectors. Everything is a list of numbers, and that list can be fed into a large neural network that can learn whatever you want on top of it. There are plenty of interesting and important algorithmic advances too - not to mention more data and more compute power - but that core end-to-end learning idea was the big one that changed so much across research, products, and the way people build systems today compared with the past in practice overall.

Vectors in NLP

In natural language processing, word embeddings are a way to represent words for text analysis, usually as real-valued vectors that capture meaning, so that words that are closer in vector space are expected to be more similar in meaning. ~ Wikipedia and are often used as inputs for machine learning models in NLP tasks generally.

What about moving from startups to big enterprises? It sounds like a double-edged sword, with more budget but also more limits ... in practice too.

They differ along so many dimensions. I’ll just name two examples here today.

AI tooling startups succeed in B2B when they find one part of a workflow that every other company might need one or two developers to handle, and then they build a product around that job that costs, say, a quarter of a developer. So many startups in AI tooling now take the less pleasant, less fun pieces and help developers get those things done in a more efficient way for those teams right now.

The best way to do this is probably to create an experience where the companies using the product still feel as if they are building and controlling the AI, but in reality they have found a partner to label their data. They have also found partners to examine data bias; collect the data in the first place; implement the model through Huggingface; scale model analytics while training via Weights and Biases; and deploy the model through ZenML on their behalf.

In the end, they depend on 10 to 15 outside systems, but they were able to train AI much more quickly, much more scalably, and much more accurately than if they had to reinvent 95 percent of the tooling around a particular AI model. It has been really interesting for startups to spot these various things that already exist, but they do not exist in a super-professional way where a strong team is focused on that specific area there.

At a larger enterprise company like Salesforce, you are mostly thinking about what truly moves the needle for many different customers. How can you help those customers with the datasets already in your system, in a way where they still feel like - and actually do - have control? That is not trivial at Salesforce, for example, because trust was our No. 1 value. You could not simply take everyone’s data and train something on it, because they own their data and are paying for the storage. And so you need to work together with customers, too, to try to get their AI projects off the ground in a careful way.

So, for an enterprise software vendor, the worry is that customers are paying a lot of money, and you cannot throw a wrench into the works in the name of testing a new feature there?

That is part of it. But maybe more important is making sure that it is trusted, easy to use, able to scale across all these different use cases, and still relatively inexpensive. If you are a platform company like Salesforce, you also have to do more than build one classifier: you have to enable all your customers to build their own classifiers, which brings all kinds of interesting and difficult technical challenges too for everyone involved there in practice today.

How does having an enterprise budget change things?

The biggest difference is that the larger your company is, the farther into the future you can and should look, do more interesting research, and have a stronger overlap with academia. Because you might be disrupted in two or three years, and you have enough runway to think four or five years ahead. So you need to anticipate a little of what is going to happen then if you want to stay relevant and avoid surprises later there.

So as an AI researcher in a large company, you have more luxury to think long term and build something, whereas in a startup, of course, you need to build something people want right now. And it has to be really, really good. And you need to be able to ship it in a reasonable timeframe. That is the big difference: the vast majority of startups work on applications and applied AI, rather than basic research; larger companies can do both in general today.

You mentioned a lot of what we might call horizontal applications when you were talking about B2B startups. Why do you think those are succeeding today, when that was not always the case?

There are always very useful vertical AI applications, but then there was a short phase when we thought maybe horizontal platforms could work. However, the early AI platform startups tried to take on too many different tasks.

For example, at MetaMind, we built technology that let you drag and drop text or images into your web browser, and then you would have a fully scalable system that classifies those documents. In some ways, it felt pretty magical because this was all before TensorFlow and before PyTorch. You had to implement all of those neural networks and every gory detail from scratch, with very few abstractions or developer tools around them. That has changed a great deal.

We created all of these capabilities at MetaMind—labeling, error analysis, deployments, modeling, analytics about how it is training. What stands out now is that each of those separate pieces is worth more than MetaMind ever was, when you look at the companies in that area that just do one of those things.

I think most companies and developers want to feel as if they are in charge of the AI, but they are fine handing over a bunch of separate smaller parts of that stack that are not really all that interesting to build. So, in a strange way, there is a bit of a tradeoff between what is enjoyable to implement and what makes everyone feel in control. As a vendor in the machine-learning tooling space, you have to avoid taking away too much control.

Transformer Models

A transformer is a deep learning model that uses self-attention, assigning different weights to the importance of each portion of the input data. It is used mainly in natural language processing and computer vision. Transformers are built to work with sequential input data, like natural language, for tasks such as translation and text summarization. ~ Wikipedia

How has the development of networks and models altered the way someone might think about founding a company or creating an AI product?

I do not really think the specific model changes much about how people would start companies. But I do think there are certain models that are now more efficient because they fit the hardware we have better. We are not really brain-inspired or theory-inspired or principles-inspired—we are GPU-inspired. We are mostly inspired by what performs well on a GPU.

The current popular model, transformers, is extremely efficient on GPUs and can be trained very efficiently. And if we had different computing architectures, then it might still be LSTMs, or perhaps even recursive neural networks. There are all sorts of different models as encoders of vectors that will come and go.

That does shift things a little for hardware startups. They look at Nvidia, and some other bigger companies, and think, “Well, there has to be some way to get a piece of that pie.” So we are going to see some innovation. At the same time, scaling is really difficult for them because for most major use cases, they have to provide their special hardware inside one of the large cloud providers.

And then, of course, the entire AI development stack has matured so much over the last 8 years. Back then, if you wanted it to be fast, you had to code everything in C++ from scratch, which is just incredibly slow. It took people a long time to catch up and learn. And today, all of that complexity can be hidden away and you can use products like the ones we discussed earlier, which make it so much faster, more convenient, and easier to build high-quality AI systems.

But algorithmic progress does matter, right? For instance, you.com is very focused on privacy, and it seems like one reason you can make that a priority is the ability to accomplish more with less data.

That is a great question, and it is absolutely true. I think if we had tried to build a search engine company 5 or 10 years ago, it would have been insanely hard and basically impossible to compete with Google because we would have needed hundreds of people and enormous amounts of training data to create the ranking systems we are building. Now, with a very small—albeit extremely smart and capable, but very small—team, we are actually able to build a ranking system that ranks any arbitrary intent and query that you type into the search engine, and provides the right sets of apps and the right sets of sources for those.

And the only reason a small company like you.com can compete with a large company like Google is because of the progress we have seen in AI. In particular, when it comes to so-called unsupervised and transfer learning. The idea here is that you can train very large neural networks on unsupervised text—basically all of Wikipedia, Common Crawl, and as much web text as you can find, while keeping in mind that not everything on the web is great for training AI.

Unsupervised models are trained with very simple objectives, such as predicting the next word in a sentence. For example, “I went to Southern California and enjoyed the …” If you know a lot about language and the world, you will realize a good next word might be “beach,” “desert,” or any of the things you might enjoy in Southern California. But you need a lot of knowledge to be able to predict what that word is. By training a model to keep predicting the next word across these very long sequences of millions and billions of words, it actually begins to absorb all of that knowledge.

It is unsupervised because nobody has to sit there and label what the next word is. You just take Wikipedia, and you get a lot of words in the correct order.

That has been an incredibly powerful idea that has essentially enabled NLP models that are very large, but then can be adjusted just a little to do what you want them to do. And they will generalize far beyond the specific, small labeled data you have because they have a sense of world knowledge; they know things like “best Thai restaurants near me” is very similar to “best Southeast Asian restaurants in my area.” Even though we have never had that exact phrase in our training data, our neural networks and our ranking systems can actually do this because they know those phrases are similar.

Speaking of search: One major thing I noticed about you.com is how it summarizes results. How much of that is purely a UI/UX choice that could have been built by anyone at any time, and how much is also driven by advances in machine learning where you are able to handle results in a different way?

Although it does not sound that cool, summarization is actually one of the hardest AI tasks, especially in natural language processing. And it is hard for a lot of interesting reasons. One, it is very personalized. Like, if I know what you (the recipient of the summary) know, I can give you much better and more accurate results for that summary.

If, for example, you do not know what a word vector is, then it is very hard to understand transformers. So you would first need an introduction to word vectors in order to understand transformer networks for NLP. But if you already know what a transformer is, then a summary of a research paper might be very short. It could just say, “They are training it on language modeling instead of machine translation, and that is a better objective function.”

And I think summarization is an important technology trend that more and more people in the next couple of years will value as they watch their time disappear. When your time is valuable, you want simple tools to help you get things done. Instead, we get pulled into engagement loops from companies whose business model is often advertisements. They do not want to help you get things done; they want to help you look at more content in order to show you more ads.

We want to push back against that. Summarization is a big part of that, to help you search less and do more, or search less and code more. We have apps with code snippets you can just copy and paste, and that is often the right summary. If you are searching “How do I sort a dictionary in Python,” the right answer is not a long block of text. It is just a code snippet, and that is it. Or when we show you a paper, there is a link to a GitHub repo that implements an open-source version of that paper.

I think the next generation of search engines is fundamentally built on different values, but also different business models.

Given all the advances we’ve discussed, if you were advising someone who wants to enter the AI space now, what should they do or what skills should they learn to start well with it?

That depends a lot on their age, their skill set, their time commitment, and which part of the field they want to work in. If you’re young and really want to steer your career that way, you still need to learn the basics of programming, math, statistics, some probability, a lot of linear algebra, and similar things there now.

And if you’re a practitioner and want to get into it, there are a lot of exciting new online classes, videos, and platforms. There’s so much material available now. Even the Stanford CS224 NLP lessons are out there, so you can go very deep if you want to. That’s what I’d encourage people to do for your own path too.

Once you’ve done that, the next step is to get your hands dirty and build something, and experiment with these models. Think about the kinds of processes and tasks people are still doing by hand, or sometimes mechanically, but that still require a human to supervise. Could you automate them and create something original?

How far can you get just by using, say, cloud APIs and different layers of abstraction versus having to really understand how this stuff works in practice today?

It really depends on your background. If you studied math at some point in higher education, then you can very quickly grasp some of the fundamentals and jump straight to hacking on real models, and you don’t need to rebuild them all from scratch. But the more you depend on abstractions, the harder it can be to do something truly novel or to understand how to resolve performance issues and bugs when they arise in production later.

Still, there are many use cases where you don’t need to do anything novel. You might want to automate a sprinkler system, so you’re only trying to answer: “Is there a person standing here? Yes or no.” And if there isn’t, turn on the sprinkler system. You don’t need to invent anything new for that. You just need to take all the standard steps and use good tools for an image classifier in that case.

But those abstractions still leak, and they’re not perfect. So the more important the application is — the more important it is to your company, affected users, or your career — the more you still want experts who understand these systems deeply. Experts who know how to fix certain errors or performance issues, and also people who think through how that AI system might affect people. Only then can you really automate certain processes in a way that is safe and maximizes positive impact overall there.