Will It Scale? Applying Data, Science, and Economics to the Art of Ideas

John List is Walmart’s chief economist, a role he formerly had at Lyft and Uber, and he is also a professor of economics at the University of Chicago. He recently released a book called The Voltage Effect: How to Make Good Ideas Great and Great Ideas Scale.

In this interview, List talks through several of the book’s central ideas, including why knowing when to stop or change course matters, and how the science of scaling can help an idea succeed. He also offers his views on economics and technology, the current state of behavioral economics and data science, and the possibility of using AI to bring back promising ideas that did not work before.

---

What is the voltage effect?

Could you give a short explanation of “the voltage effect” — both the idea and the book — and how that differs from merely judging whether something was a success or a failure?

JOHN LIST: The Voltage Effect is a book about ideas. It starts from the idea that scaling is a science, not an art. And inside that science, there are scientific rules we ought to know about ideas that scale and ideas that do not.

The book’s first half covers the five signatures, or five vital signs, of ideas that have scalable traits. That begins with making sure your idea has voltage in the first place — that people want it — and it reaches all the way to the supply side of scaling. In other words: “What are the marginal cost features of your idea as you grow? Do you have economies of scale or diseconomies of scale?” So it covers both the demand side and the supply side, in a scientific way, giving the scholar or policy maker or VC or business planner a checklist for understanding the key features of ideas that have succeeded, historically, and ideas that have not.

It is not a book about execution. It is a book that says, “If you execute, these are ideas that will scale.” Of course, if you do not execute, it does not matter much. You can have a strong idea with excellent vital signs, but it still may not scale if you fail to execute.

Once your idea is launched, the second half of the book covers simple economic tools you can use to keep voltage high as you scale. So questions such as “What incentives are best to use?” and “What is the best way to think?” (And that, of course, means thinking at the margin instead of relying on averages, and it walks through a number of examples of that.) It also discusses the “optimal quitting rule,” by which I mean understanding, or having principles to follow, when deciding when and how to pivot.

And the last chapter is about creating a culture. Culture, from the outset, can be built in a way that is sustainable at scale, or you can create a culture that is very hard to grow alongside the company.

How would an organization go about setting itself up to make sure it can maximize voltage as much as possible for its ideas or initiatives, or to foresee that something is not going to scale or does not have voltage?

It is a multi-dimensional issue, in the sense that the world changes and what once was a great idea may no longer be an idea that is viable and capable of scaling.

I think the main point there is that you have to examine your opportunity set. A lot of the time people, when they quit or pivot, just look at their present situation and say, “Does it get soiled?” If it gets soiled, they quit. I think you want to pivot or move just as often when your opportunity set improves and when your opportunity set tells you that you should be pivoting.

It is not natural for people to think this way, because we tend to overlook the opportunity cost of time. Here is an example: I used to be the chief economist at Lyft and, most of the time, people think, “OK, chief economist at Lyft — that is really a lot of fun.” But I knew that every day I was chief economist at Lyft, that meant I could not be chief economist at another firm. I have recently taken a position at Walmart to be its chief economist, and I left Lyft. Not because it was soiled or it was a bad firm — it is not, the people at Lyft are wonderful — but because my opportunity set improved. We should be constantly scanning the opportunity set and asking, “Are there better opportunities out there for me?”

Academia vs ridesharing vs retail

How does the job change as you move across sectors between academia, ridesharing, and now retail?

I was chief economist at Uber for two years and then I went to Lyft for four. That is fairly similar — it is rideshare plus a mix of driver-side incentives, rider incentives, pricing, and strategic mission. It is about deciding which markets to expand into, which technologies to expand into, and which technologies to contract.

At a company like Walmart, it is more or less business as usual. Economics is life and life is economics, so economics is going to be everywhere. But there may be different priorities. For instance, I think Walmart is going to be very proactive in the tech area and that last-mile delivery, for example, will be relatively new there, whereas we sort of knew how to do that at Uber and Lyft. But you have similar economic incentives pushing both sides of that market.

The interesting thing about Walmart is that roughly 90% of Americans live within 10 miles of a Walmart, so you have this comparative advantage to use by applying economic thinking and big data through spatial considerations. Walmart just has a much larger sandbox to play in. It has international markets, it has new tech, it has 4,000-plus stores. If it were a country, Walmart would have one of the largest GDPs in the world — it would be like Belgium. It is the largest private employer in the United States. So you have a question around every corner that economics can help you answer.

I think the big question going into Walmart is: Where do you start? That was not really the case at Uber or Lyft; we sort of knew what we wanted to do there.

Wherever I am, my tool of choice is big data: studying the big data the firm already has, and also creating big data to deliver causal insight where naturally occurring data fall short. What I mean is using field experiments to produce huge amounts of data that are actionable. A lot of times I see people doing A/B testing and explorations, but they really do not have a plan for how they are going to use their data. You need to actually use a bit of game theory — backward induction — and say, “Look, if my data say this, here is what I am going to do. If instead the data show this particular causal pattern, then I am going to veer in this direction. And after I make those decisions, this is the next experiment I am going to run.” And you keep adapting and improving your data-generating process. The best firms do not even do this all the time yet, but it is what everyone needs to do as an embedded part of their fabric, their culture of experimentation.

The value, and risks, of big data

When big data first became fashionable, Walmart was often held up as a company with massive amounts of data and a clear sense of how to use it. But that belonged to an earlier era of data analysis, compared with how a company founded more recently thinks about data …

You’re correct. When you walk into Uber, you’ll see a sign reading, “Data is our DNA.” It’s not merely a slogan; they genuinely live it. Walmart is, in many respects, that kind of firm, but it wants to fully adopt it as part of its culture, and I think it will, though the starting point matters. Where firms or organizations begin culturally leaves a durable mark. How you establish that will shape your genetic makeup forever.

Of course, companies can change, but you can still see the contrast between businesses for which data is in the DNA and those for which it isn’t. Walmart has outstanding talent, and the executive team is world-class, so part of my job will be to embed a culture of experimentation, big data, and action.

Back to the voltage idea … How many ideas or ventures do you think succeed because companies planned carefully and examined the data, rather than simply got lucky?

When you enter a billiards hall, the best billiards players seem as if they’ve mastered all the physics. But, of course, they haven’t. Some people are naturally good at some of the physics, and some learn through doing – they make mistakes, test, and adapt. Is there a fair amount of luck combined with the billiards player who is naturally gifted? Absolutely.

And in business, of course, there are people who get everything right – they have all five vital signs. Did they know that from the start? I don’t know, but I don’t assume they did. They either discovered it through trial and error, or they came upon it by luck.

In that sense, I don’t think luck should be underestimated. Luck matters enormously in every venture. What I hope my book does is bring in more skill and science so you don’t have to depend on luck. Remember, historically, when we talk about scale, it’s “Move fast and break things.” “Throw spaghetti against the wall.” “Fake it til you make it.” And that’s art, that’s pure art.

So, never discount or belittle luck, but luck matters more when you’re using art and luck matters less if you use science. When it comes to the challenge of scaling, though, the science of using science has not really been at the forefront for people. Too often, people and organizations want to turn their great ideas into reality without first getting their hands dirty figuring out whether they’ll scale and how best to implement them. I want to change that.

Today, if you’re savvy enough or clever enough or inventive enough, you can generate a lot of data – especially digital data – to feed whatever experiments you want to run. How has this changed the way you think about behavioral economics?

A criminal is more dangerous if you let them buy weapons or you hand out weapons for free. We know that on the supply side. In the same way, a data analyst is really dangerous now because, at the drop of a hat or the push of a button, they can have mounds and mounds of data and they can make anything statistically significant. Because as the sample size goes to infinity, you can get statistical significance every time unless the estimate truly is zero.

So, on the one hand, it has made life easier for people who want to analyze data. We have statistical packages that are easy to use and data that is easy to download. But it has also made it easier to be careless and produce nonsensical results and correlations that people want to claim are causal. Remember the old adage: Once you have confidence and ignorance, success is ensured.

And we really need to be very careful, because this is still econometrics and I still want to figure out what causes what and I want a causal parameter to make a decision. If I have a correlation, it’s not necessarily going to change when I alter something, because it could’ve been a third variable that’s causing them both. Simply changing one variable doesn’t necessarily create any other changes when you do it yourself versus when nature did it. When nature does things, a lot of other variables are in play; those correlations will not always show up when you tweak the variable itself. A correlation in nature doesn’t amount to a causal relationship.

I see this mistake made all the time, and technology and big data being available at the drop of a hat make it much, much easier.

So more data isn’t always better?

That’s right. Most of the time, people take pride in gathering big data. But the real value lies in refining data. Oil is valuable, but it’s much more valuable because of the oil refinery. That’s what gives it value. Data is no different. Data, by itself, is useless unless you have a good refinery.

If the refinery is top notch then, of course, I would want more data because that lets me make more precise causal statements. But more data alone can be quite dangerous, as the reckless data refiner then has sharper messages. Think of it this way: a misguided missile is more dangerous the faster it travels.

How has the advent of data science as a discipline changed things?

Well, data science used to be called econometrics. Back in the last 40 or 50 years of the 20th century, economists were econometricians and they focused on trying to produce causal relationships. The new buzzword is trying to do some of the same things, but you call yourself a data scientist. Yet, data scientists begin with correlations often end there. Correlations are interesting, but they’re not as important as most people say they are. To me, they are the beginning, rather than the end of using science to explore data.

AI and understanding consumer behavior

You’ve suggested that artificial intelligence and/or automation, to some degree, could help bring back good ideas that didn’t or couldn’t scale the first time around. What types of scenarios are you thinking about?

I think, in the past, there may have been a series of ideas that a smart person, or a smart group of people, tried to grow into something big. But because of a constraint, whether it was human capital or the consumer base or some other infrastructural constraint, they threw their hands up and said, “We have to pivot. Let’s leave that idea along the wayside and let’s go for something else.” And I wonder, now that we have this fantastic explosion of technology – in particular, technology that is intelligent and can substitute for humans (in some ways) and other inputs that are very scarce – whether you begin to open up the box of ideas that we have discarded over the past several decades. Now that the world has changed and we have better and cheaper alternatives for certain tasks, are those old ideas we scrapped in the past now viable to scale?

I can see a fund doing something like that and I would bet there’s an expected value that’s positive – even an expected value that is greater than the opportunity cost of those funds.

With today’s focus on economics and all the data available to us, are we really any nearer to understanding consumer behavior? Or do our irrational tendencies keep changing as our cultures and technologies change?

Yes, I think we are greatly sharpening our understanding of consumers, and in one area especially. In the past, economists would say, “Well, if you buy a tee shirt for $10, that’s your contribution to the market demand curve.” Then they stopped. Now, we are starting to use the experimental method and field experiments, for example, to understand the foundations of why that specific consumer bought that tee shirt. Is it for yourself? Is it a gift? Is it for your son or your daughter?

This matters a great deal because now you not only know the contribution to market demand, but you also start to understand the reasons behind the purchase. And once you uncover the reasons, you begin to develop a much better set of incentives or policies that can help change the world for the positive or help change your firm.

As for irrational behavior, although irrationalities still exist, many of them are predictable. What I mean is that if a consumer has some non-standard preference – say, loss aversion – then it is likely the consumer across the street will have similar preferences. The same idea applies to many mistakes that can be anticipated. With field experiments, we can begin to understand who is making mistakes, how, and then why. Why are they making the mistake of choosing the wrong insurance program or the wrong 401k program?

I don’t think technology is making people commit brand-new kinds of mistakes (sure, at the margins a little bit if it makes things more confusing), but the larger mistakes are usually cognitive mistakes that humans have made for decades. The underlying elements are still more or less the same, but technology gives you new ways to make those errors. However, it also gives us new ways to find them and address them in markets.