As we saw in Part 1, you can begin addressing a range of important problems by constructing augmentation infrastructure around an artificial intelligence model’s strengths. Does the model produce text? Build around text. Can it predict 3D structures accurately? Build around 3D structures. But taking an artificial intelligence system entirely at face value also has its own limits.
Douglas Engelbart used the phrase co-evolution to describe how humanity’s tools and the methods for using them change and advance together. Models like GPT-3 and DALL-E mark a major step in the evolution of tools, but that is only one side of the equation. If you build around the model without also building new tools and processes for it, you are left with whatever it gives you. The model’s shortcomings become your shortcomings. If you dislike the outcome, you have to repair it yourself. And because training any of the large, complicated AI systems we have discussed so far demands enormous amounts of data and computation, you probably do not have the resources to alter the model very much.
This is somewhat of a dilemma: on the one hand, we lack the resources to meaningfully alter the model. On the other hand, to solve for our particular use case, we need to change the model, or at least find better ways to work with it. For prompt-based models like GPT-3 and DALL-E, the two simplest ways to address this fixed-model dilemma are prompt-hacking and fine-tuning — neither of which is especially efficient:
- Prompt-hacking: GPT-3 and DALL-E work from natural-language prompts — given a descriptive sentence (or even a set of input/output examples, in GPT-3’s case), the model produces one or more outputs. The quality of the model’s results depends heavily on that prompt, so what do you do when the output falls short of your expectations or your application’s needs? You can simply try another prompt! This iterative trial-and-error method is called prompt-hacking. Fine-tuning: Note that earlier I said, “You likely don’t have the resources to change the model all that much,” not, “You likely don’t have the resources to change the model at all.” In fact, you can adjust the model a little through a process known as fine-tuning. If you have data specific to your use case, you can provide it to the model, and the model becomes better suited to your application and more likely to be accurate (that is, it is fine-tuned to your application). While fine-tuning improves on the blind searching of prompt-hacking, it depends on data, which usually means locating enough high-quality data points and labeling them by hand.
The aim of augmented intelligence is to make manual workflows like these more efficient so humans can devote more time to the things they do well, such as reasoning and planning. The inefficiency of prompt-hacking and fine-tuning shows that the time is right for a reciprocal move in process evolution. So in this section, we will look at some examples of a new theme — building for the model — and the part it plays in producing more effective augmentation tools.
In-game assets, on demand
As a practical example, suppose you are an emerging game developer working on the next online gaming franchise. You have seen how games like Call of Duty and Fortnite have built massively successful (and profitable) marketplaces for custom skins and in-game assets, but you are a startup with limited resources. So rather than creating these assets yourself, you hand content generation off to DALL-E, which can produce any number of skins and asset styles at a fraction of the cost. That is a solid beginning, but using prompt-hacking to assemble a fully stocked asset store is inefficient.
To reduce the manual work, you can shift prompting to a text generation model like GPT-3. The secret to the virality of a game like Fortnite is the pairing of many core game assets — weapons, vehicles, armor — with a range of distinctive styles and references, such as striking patterns/colors, superheroes, and the latest pop culture trends. If you prime GPT-3 with your asset types, it can generate any number of these combinations as a prompt. Send that prompt to DALL-E, and your skin design appears.
This GPT-3-to-DALL-E transfer sounds promising, but it only really works if it yields engaging, high-quality skin designs for your users. Reviewing each design candidate by hand is not feasible, especially at scale. The key is to build tools that let the marketplace do the work for you. Users gravitate toward strong content and have no patience for weak content — apps like TikTok are built entirely on this idea. User engagement therefore becomes a strong signal for which DALL-E prompts are working (that is, leading to interesting skin designs) and which are not.
To let your users do the work for you, you will want to build a recursive loop that cross-checks user activity against each prompt and turns engagement metrics into a ranking of your active content prompts. Once you have that, standard A/B testing will automatically reveal prompt insights and you can promote good prompts, drop bad ones, and even compare how similar newly generated prompts are to ones you have tested before.
But that is not all — the same user engagement signal can also be used for fine-tuning.
Let’s step back one more time and concentrate on GPT-3’s performance. As long as you keep a record of the inputs you are sending to GPT-3 (asset types + candidate themes), you can combine that data with the quality rankings you just received from further down in your content pipeline to build a dataset of successful and unsuccessful input-output pairs. That dataset can then be used to fine-tune GPT-3 on game-design-focused prompt generation, making it even better at creating prompts for your application.
This user-driven cyclical pipeline helps DALL-E create better content for your users by surfacing the best prompts, and helps GPT-3 generate better prompts by fine-tuning on examples produced from your own user activity. With no need to worry about prompt-hacking and fine-tuning, you are free to focus on larger-ticket items, like which assets come next in the pipeline, and which new content themes might lead to even more interesting skins later on.
Personalized learning experiences
There is also a huge opportunity to build middleware that connects creative industries and creative, personalized content-generating models. AI models and the services they enable (for example, Copilot) could be useful in use cases that call for novel content creation. This, again, depends on using our understanding of the AI system and how it functions to imagine ways we can adjust its behavior ever so slightly to create new and better experiences.
Imagine you are building a service for learning to code that uses Copilot behind the scenes to generate programming exercises. Out of the box, Copilot will generate anything from a single line of code to an entire function, depending on the docstring it is given as input. That is great — you can build a bunch of exercises very quickly!
To make this educational experience more engaging, though, you will likely want to adapt the exercises generated by Copilot to the needs and interests of your users. For example, you might want to personalize across dimensions such as:
- Skill level: This is obvious. The exercises should sit in the Goldilocks zone of the user’s current abilities — not too easy, not too hard. (As a result, this is also what is known as the Flow Zone — meaning the blend of task difficulty and user ability most likely to help the user enter a state of flow.) Age range: If the user is a child, examples featuring superheroes and sports figures may be the most exciting. If the user is an adult, cute examples may not be as appealing as, say, current affairs or pop culture references. Intended occupation: Many people do learn to code in order to get a new job. Being able to work on programming problems tied to, say, finance or security might create a more satisfying learning experience for a future financial or security engineer.
Writing the docstrings yourself is tedious and manual, so personalizing Copilot’s outputs should be as automated as possible. Fortunately, we know of another AI system, GPT-3, that is excellent at generating virtually any kind of text — so perhaps we can outsource the docstring creation to GPT-3.
This can be done in one of two ways. One option is to ask GPT-3 to create generic docstrings that match a particular skill or concept (for example, looping, recursion, etc.). With a single prompt, you can produce any number of boilerplate docstrings. Then, using a curated list of target themes and keywords (a small amount of manual effort), you can swap variable names in the boilerplate to fit your target audience. Alternatively, you can try giving GPT-3 both the target skills/concepts and themes at the same time and let GPT-3 adapt the docstrings to your themes automatically.
Of course, the success of this idea depends on the quality of GPT-3’s content. For one thing, you will want to make sure the exercises generated by this GPT/Copilot combination are appropriate for the user’s age. Perhaps an aligned model like InstructGPT would be better here.
The path forward
We are now more than a decade into the latest AI summer. The surge of activity in the AI community has produced incredible breakthroughs that will have major effects across many industries and, possibly, on the course of humanity as a whole. Augmented intelligence is an opportunity to jump-start this progress, and all it requires is a slight reframing of our design principles for building AI systems. Beyond building models to solve problems, we can think of new ways to build infrastructure around models and for models; and even ways in which foundation models might work together (like GPT-3 and DALL-E or GPT-3 + CoPilot).
Maybe someday we will be able to hand off all of life’s “dirty work” to some artificial general intelligence and live hakuna-matata style, but until that day arrives we should think of Engelbart — paying less attention to machines that replace human intelligence and more to those that are clever enough to strengthen it.