This is a revised edition of a post that first appeared here.
Neuroscience and AI have a long, closely linked past. Early artificial intelligence researchers drew inspiration from how the brain is organized in order to create intelligent machines. Now, in a striking turnabout, AI is helping us investigate the very thing that inspired it: the human brain. This method of using AI to construct brain models is known as neuroAI. Over the coming decade, we’ll produce increasingly exact in silico brain models, especially for our two most dominant senses, vision and hearing. That will let us access and use sensory models on demand with the same ease as object recognition or natural language processing.
Many neuroscientists and artificial intelligence researchers are – understandably! – thrilled by this: brains on demand! Learning what it is to see, to feel, to be human! Less appreciated is the fact that there are broad practical uses in industry. I have spent much of my career as a researcher in this area, working since my PhD on how the brain turns vision into meaning. I have watched the field develop from the beginning, and I believe now is the moment to explore how neuroAI can foster greater creativity and better health.
I expect neuroAI to first become widely used in art and advertising, particularly when paired with new generative AI systems such as GPT-3 and DALL-E. Although today’s generative AI can create inventive art and media, it cannot tell you whether that media will actually convey a message to the intended audience – neuroAI could. For example, we could move beyond the trial and error of focus groups and A/B testing and directly produce media that says exactly what we intend. The huge market forces surrounding this use case will set off a virtuous cycle that strengthens neuroAI models.
Those improved models will open the door to health and medicine applications, from assisting people with neurological conditions to boosting the capabilities of healthy people. Picture generating the right images and sounds to help someone regain sight or hearing more rapidly after LASIK surgery or after receiving a cochlear implant, respectively.
These advances will become far more powerful thanks to other technologies arriving soon: augmented reality and brain-computer interfaces. Still, to fully capture the promise of downloadable sensory systems on demand, we’ll need to close existing gaps in tooling, talent and funding.
In this article I’ll describe what neuroAI is, how it may begin to develop and affect our lives, how it works alongside other innovations and technologies, and what is required to move it ahead.
What is neuroAI?
NeuroAI is an emerging field that aims to 1) examine the brain to learn how to build stronger artificial intelligence and 2) use artificial intelligence to gain a better understanding of the brain. One of neuroAI’s central tools is the use of artificial neural networks to build computer models of particular brain functions. This approach got a boost in 2014, when researchers at MIT and Columbia demonstrated that deep artificial neural networks could account for responses in a brain region involved in object recognition: the inferotemporal cortex (IT). They proposed a straightforward way to compare an artificial neural network with a brain. Using that recipe and iterating through testing across brain functions – shape recognition, motion processing, speech processing, arm control, spatial memory – scientists are assembling a patchwork of computer models for the brain.
A recipe for comparing brains to machines
So how do you create a NeuroAI model? Since its start in 2014, the field has used the same basic recipe:
1. Train artificial neural networks in silico to perform a task, such as object recognition. The resulting network is called task-optimized. Importantly, this usually means training on images, movies and sounds only, not on brain data.
2. Compare the intermediate activations of trained artificial neural networks with actual brain recordings. The comparison uses statistical methods such as linear regression or representational similarity analysis.
3. Select the highest-performing model as the current best model of these brain areas.
This recipe can be used with data gathered from inside the brain, from single neurons, or from non-invasive methods such as magneto-encephalography (MEG) or functional magnetic resonance imaging (fMRI).
A neuroAI model of a part of the brain has two essential properties. It’s computable: we can give this computer model a stimulus and it will predict how a brain area will respond. It’s also differentiable: it’s a deep neural net that we can optimize in the same way we optimize models for visual recognition and natural language processing. That means neuroscientists gain access to the powerful tools that have driven the deep learning revolution, including tensor algebra systems like PyTorch and TensorFlow.
What does this imply? In less than a decade, we moved from not understanding large portions of the brain to being able to download solid models of it. With the right investment, we’ll soon have excellent models of major parts of the brain. The visual system was modeled first; the auditory system followed soon after; and other regions will surely drop one by one as fearless neuroscientists hurry to solve the brain’s mysteries. Beyond satisfying our intellectual curiosity–a major motivation for scientists!– this advance will let any programmer download strong models of the brain and open up countless applications.
Application areas
Art and advertising
Let’s begin with this simple idea: 99% of the media we encounter reaches us through our eyes and ears. There are whole industries that can be reduced to delivering the right pixels and tones to these senses: visual art, design, movies, games, music and advertising are only a few. Now, it is not our eyes and ears themselves that interpret these experiences, since they are only sensors: it is our brains that make sense of the information. Media is made to inform, to entertain, to produce wanted emotions. But deciding whether the message in a painting, a professional headshot or an ad is understood as intended is a frustrating process of trial and error: humans have to stay in the loop to decide whether the message lands, which is costly and slow.
Large-scale online services have found ways around this by automating trial and error: A/B tests. Google famously tested which of 50 shades of blue to use for the links on the search engine results page. According to The Guardian, the winning option raised revenue relative to the baseline by 200M$ in 2009, or about 1% of Google’s revenue at that time. Netflix tailors thumbnails to each viewer to improve its user experience. These techniques are available to online giants with enormous traffic, which can absorb the noise built into human behavior.
What if we could forecast how people will respond to media before collecting any data? That would make it possible for small businesses to improve their written materials and websites even if they have little existing traction. NeuroAI is moving closer and closer to predicting how people will react to visual materials. For example, researchers at Adobe are developing tools to forecast and steer visual attention in illustrations.
Researchers have also shown that photos can be edited to make them more visually memorable or more aesthetically pleasing. It could be used, for example, to automatically choose a professional headshot most closely matched to the image people want to present of themselves–professional, serious, or creative. Artificial neural networks can even discover ways to communicate messages more effectively than realistic images. OpenAI’s CLIP can be probed to locate images that align with emotions. The image most closely aligned to the concept of shock would not look out of place beside Munch’s Scream.
Over the past year, OpenAI and Google have shown generative art networks with an impressive capacity to create photorealistic images from text prompts. We have not quite reached that point for music, but with the speed of progress in generative models, this will surely arrive in the next few years. By building machines that can hear like humans, we may be able to democratize music production, giving anyone the power to do what highly skilled music producers can do: communicate the right emotion during a chorus, whether melancholy or joy; make an earworm of a melody; or create a piece that is irresistibly danceable.
There are enormous market forces pushing us to improve audiovisual media, websites, and especially ads, and we are already folding neuroAI and algorithmic art into that workflow. That pressure should create a virtuous cycle in which neuroAI becomes stronger and more useful as more resources are invested in practical uses. One consequence is that we will end up with very strong models of the brain that will matter well beyond ads.
Accessibility and algorithmic design
One especially promising use of neuroAI is accessibility. Most media is built for the “average” person, even though visual and auditory processing varies widely across people. 8% of men, and 0.5% of women are red-green colorblind, and much media is not adjusted for them. Today there are several products that mimic color blindness, but they still depend on a person with normal color vision to read the output and decide what to change. Simple static color remapping is not enough either, because some content loses its meaning when colors are remapped (e.g. graphs that become hard to read). We could use neuroAI methods that preserve the semantics of existing graphics to automate the creation of color-blindness-safe materials and websites.
A further example is supporting people with learning disabilities, such as dyslexia, which affects up to 10% of people worldwide. One of the core problems in dyslexia is crowding sensitivity, meaning difficulty identifying shapes that share similar basic features, including mirror-symmetric letters like p and q. Anne Harrington and Arturo Deza at MIT are building neuroAI models of this effect and seeing very encouraging outcomes. Picture using models of the dyslexic visual system to create fonts that are both attractive and easier to read. With the right data on a particular person’s visual system, we can even tailor the font to that individual, which has already shown promise in improving reading performance. There are potentially huge quality-of-life gains here waiting to be realized.
Health
Many neuroscientists enter the discipline hoping their work will improve human health, especially for people living with neurological disorders or mental health problems. I’m very optimistic that neuroAI will open the door to new therapies: with a solid model of the brain, we can design the right stimuli so the right message reaches it, like a key fitting a lock. In that respect, neuroAI could be used much like algorithmic drug design, except that instead of small molecules, we deliver images and sounds.
The easiest problems to approach are the receptors of the eyes and ears, which are already fairly well understood. Hundreds of thousands of people have had cochlear implants, neuroprosthetics that electrically stimulate the cochlea of the ear, helping the deaf or hard-of-hearing hear again. These implants, which use only a few dozen electrodes, can be hard to use in noisy settings with multiple speakers. A brain model can tune the implant’s stimulation pattern to strengthen speech. What is striking is that this technology, built for implant users, could be repurposed to help people without implants understand speech better by adjusting sounds in realtime, whether they have an auditory processing disorder or are simply often in loud places.
Many people go through changes in their sensory systems over the course of life, whether from recovering after cataract surgery or becoming near-sighted with age. We know that after such a change, people can learn to reinterpret the world correctly through repetition, a process called perceptual learning. We may be able to optimize this perceptual learning so people recover their abilities more quickly and more effectively. A similar approach could aid people who have lost fluid limb movement after a stroke. If we could identify the best sequence of movements to strengthen the brain optimally, we may be able to help stroke survivors recover more function, such as walking more smoothly or simply holding a cup of coffee without spilling. Beyond helping people regain lost physical abilities, the same idea could help healthy people reach their highest sensory performance – whether they be baseball players, archers, or pathologists.
Finally, we could see these ideas applied to treating mood disorders. I visited many visual art shows to ease my boredom during the pandemic, and it improved my mood tremendously. Visual art and music can lift our spirits, and that offers a proof of concept that we may be able to deliver mood-disorder therapies through the senses. We know that controlling the activity of specific brain regions with electrical stimulation can ease treatment-resistant depression; perhaps indirectly controlling brain activity through the senses could produce similar effects. By using simple models – low-hanging fruit – that influence well-understood regions of the brain, we’ll start building momentum toward more complex models that can support human health.
Enabling technology trends
NeuroAI will take many years to be mastered and deployed in applications, and it will intersect with other emerging technology trends. Here I highlight two trends in particular that will make neuroAI much more powerful: augmented reality (AR), which can deliver stimuli very precisely; and brain-computer interfaces (BCI), which can measure brain activity to confirm that stimuli are working as intended.
Augmented reality
One trend that will make neuroAI applications far more powerful is the spread of augmented reality glasses. Augmented reality (AR) has the potential to become a ubiquitous computing platform, because AR is woven into everyday life.
Michael Abrash, chief scientist at Meta Reality Labs, has the hypothesis that if you create AR glasses that are good enough, everyone will want them. That means building glasses that understand the world and can make persistent world-locked virtual objects; thin and attractive frames, like a pair of Ray-Bans; and giving you real-world superpowers, like interacting naturally with people no matter the distance and improving your hearing. If you can achieve those things–a massive technical challenge–AR glasses could follow an iPhone-like path, so that everyone will have one (or a knockoff) 5 years after launch.
To make that happen, Meta spent 10 billion dollars last year on R&D for the metaverse. While we do not know exactly what Apple is doing, there are strong indications that they are developing AR glasses. So there is also a huge push on the supply side to bring AR about.
This would make a display device broadly available that is far more capable than today’s static screens. If it follows the path of VR, it will eventually include integrated eye tracking. That would create a widely available way to present stimuli that is much more controlled than what is possible now, which would be a dream for neuroscientists. And these devices are likely to have wide-ranging health uses, as Michael Abrash described in 2017, such as improving low-light vision, or letting people live normally despite macular degeneration.
The importance for neuroAI is obvious: we could deliver the right stimulus in a highly controlled manner on a continuous basis in everyday life. That is true for vision, and perhaps less obviously for hearing, since we can provide spatial audio. In practical terms, our tools for delivering neuroAI therapies to people with neurological conditions or for accessibility will become much more powerful.
BCI
With excellent displays and speakers, we can control the main inputs to the brain precisely. The next, more powerful step in delivering stimuli through the senses is to verify that the brain is responding in the expected way using a read-only brain-computer interface (BCI). In this way, we can measure how the stimuli affect the brain, and if the response is not what we expect, we can modify it in what is called closed-loop control.
To be clear, I am not referring here to BCI approaches like Neuralink’s chip or deep-brain stimulators that are implanted inside the skull; for these purposes, it is enough to measure brain activity non-invasively from outside the skull. There is no need to stimulate the brain directly either: glasses and headphones are sufficient to control most of the brain’s inputs.
A number of non-invasive read-only BCIs are already commercialized or in development that could be used for closed-loop control. Some examples include:
- EEG. Electroencephalography measures the brain’s electrical activity from outside the skull. Because the skull acts as a volume conductor, EEG offers high temporal resolution but low spatial resolution. That has limited consumer use to meditation products (Muse) and niche neuromarketing applications, but I’m optimistic about some of its uses in closed-loop control. EEG becomes much more powerful when you can control the stimulus, because you can correlate the presented stimulus with the EEG signal and decode what a person was attending to (evoked potential methods). Indeed, NextMind, which built an EEG-based “mind click” using evoked potentials, was acquired by Snap, which is now developing AR products. OpenBCI plans to release a headset that combines its EEG sensors with Varjo’s high-end Aero headset. I would not write EEG off. fMRI. Functional magnetic resonance imaging measures tiny changes in blood oxygenation linked to neural activity. It is slow, it is not portable, it needs its own room, and it is very expensive. However, fMRI is still the only technology that can non-invasively read activity deep in the brain with spatial precision. There are two paradigms that are fairly mature and relevant for closed-loop neural control. The first is fMRI-based biofeedback. A subfield of fMRI shows that people can change their brain activity when it is shown to them visually on a screen or through headphones. The second is cortical mapping, including methods such as population receptive fields and estimating voxel selectivity with movie clips or podcasts, which let one estimate how different brain regions respond to different visual and auditory stimuli. These two methods suggest that it should be possible to estimate how a neuroAI intervention changes the brain and guide it to work better. fNIRS. Functional near infrared spectroscopy uses diffuse light to estimate cerebral blood volume between a transmitter and a receptor. It depends on the fact that blood is opaque and increased neural activity causes a delayed influx of blood in a given brain volume (same principle as fMRI). Conventional NIRS has low spatial resolution, but with time gating (TD-NIRS) and massive oversampling (diffuse optical tomography), spatial resolution is much better. On the academic side, Joe Culver’s group at WUSTL has shown decoding of movies from the visual cortex. On the commercial side, Kernel is now making and shipping TD-NIRS headsets, which are impressive engineering achievements. And it is an area where people keep pushing and progress is rapid; my old group at Meta demonstrated a 32-fold improvement in signal-to-noise ratio (which could be scaled to >300) in a related technique. MEG. Magnetoencephalography measures tiny changes in magnetic fields, thereby localizing brain activity. MEG is similar to EEG in that it measures changes in the electromagnetic field, but it does not suffer from volume conduction and therefore has better spatial resolution. Portable MEG that does not require refrigeration would be a game changer for noninvasive BCI. People are making progress with optically pumped magnetometers, and it is possible to buy individual OPM sensors on the open market, from manufacturers such as QuSpin.
Alongside these more familiar methods, a few dark-horse technologies such as digital holography, photo-acoustic tomography, and functional ultrasound may trigger swift paradigm shifts in this area.
While consumer-grade non-invasive BCI is still in its infancy, there are a number of market pressures around AR use cases that will make the pie larger. Indeed, a significant problem for AR is controlling the device: you don’t want to have to walk around with a controller or muttering to your glasses if you can avoid it. Companies are quite serious about solving this problem, as evidenced by Facebook buying CTRL+Labs in 2019, Snap acquiring NextMind, and Valve teaming up with OpenBCI. Thus, we’re likely to see low-dimensional BCIs being rapidly developed. High-dimensional BCIs might follow the same trajectory if they find a killer app like AR. It’s possible that the kinds of neuroAI applications I advocate for here are precisely the right use case for this technology.
If we can govern what enters the eyes and ears while also measuring brain states with precision, we can provide neuroAI-based therapies in a supervised manner for maximum efficacy.
What’s missing from the field
The core science behind NeuroAI applications is advancing quickly, and several positive trends will broaden its overall usefulness. So what is still needed to bring neuroAI applications to market?
- Tooling. Other AI subfields have gained enormously from toolboxes that support rapid progress and the sharing of results. These include tensor algebra libraries such as Tensorflow and PyTorch, training environments like OpenAI Gym, and ecosystems for sharing data and models such as 🤗 HuggingFace. A centralized collection of models and methods, along with evaluation suites that could potentially make use of plentiful simulation data, would move the field ahead. There is already a strong community of open source neuroscience organizations, and they could be natural homes for these efforts. Talent. There are only vanishingly few places where research and development happens at the junction of neuroscience and AI. The Bay Area, with labs at Stanford and Berkeley, and the Boston metro area, with numerous labs at MIT and Harvard, will likely receive most of the investment from the existing venture capital ecosystem. A third likely hub is Montreal, Canada, boosted by major neuroscience departments at McGill and Universite de Montreal, together with the pull of Mila, the artificial intelligence institute founded by AI pioneer Yoshua Bengio. Our field would benefit from specialized PhD programs and centers of excellence in neuroAI to jump-start commercialization. New funding and commercialization models for medical applications. Medical applications often have a long path to commercialization, and protected intellectual property is usually required to secure funding and reduce risk for investment in the technology. AI-based innovations are notoriously hard to patent, and software-as-a-medical-device (SaMD) is only beginning to reach the market, which makes the road to commercialization uncertain. We will need funds focused on bringing together AI and medical technology expertise to nurture this emerging field.
Let’s build neuroAI
Scientists and philosophers have wondered about how brains work since time immemorial. How does a thin sheet of tissue, a square foot in area, allow us to see, hear, feel and think? NeuroAI is helping us tackle these deep questions by constructing models of neurological systems in computers. By satisfying that basic hunger for knowledge – what does it mean to be human? – neuroscientists are also creating tools that could help millions of people live fuller lives.