This is an edited version of a post that originally ran here.
For most queries, Google search feels pretty disappointing these days. Google is excellent at answering questions with an objective answer, like “# of billionaires in the world” or “What is the population of Iceland?” It’s pretty poor at handling questions that need judgment and context like “What do NFT collectors think about NFTs?”
The proof is all around us. Lately, I catch myself cutting through the junk Internet by Googling “Substack + future of learning” to surface the best thoughts on education. We game Twitter with the “what is the best” posts again and again. When I’m looking into a new product, I search “X item reddit” in Google. I get huge value from small, niche, often overlooked sites like Spaghetti Directory.
There’s a rise of tools like Notion, Airtable, and Readwise where people are collecting content and resources, bringing back the curated web. But right now these are mostly solo efforts — tucked away in private or semi-private corners of the Internet, scattered, badly indexed, and not open for public use. We still haven’t learned how to make them multiplayer. Where we have made them public and collaborative — here is a great example — these projects are often short-lived and poorly maintained.
The stated mission of a company worth almost two trillion dollars is to “organize the world’s information” and yet the Internet is still poorly organized. Or, put another way, in a world of infinite information, it’s no longer sufficient to organize the world’s information. It becomes necessary to organize the world’s trustworthy information.
How did we get here?
It’s hard to wrap your head around, but one of Google’s central problems, once it took off, was that there simply wasn’t much online to look at. A powerful search engine is pointless if someone searches for “how to grow an herb garden” and the answer doesn’t exist on the web. With the arrival of Google AdWords, it became lucrative to publish low-quality content that looked informative and crowded Google’s search engine results. The result is that the websites near the top of Google are not necessarily the best ones, but instead the ones that devoted the most effort to SEO. What began as a well-meaning attempt to organize the world’s information has become a business that spends most of its resources monetizing clicks to serve advertisers, rather than concentrating on delivering trusted search results for people.
The problem, now so sharply changed from a decade ago, is not what to read/buy/eat/watch/etc., but how to choose the best thing to read/buy/eat/watch/etc. with my limited time and attention.
Bold teams, like DuckDuckGo and Neeva, are trying to take on Google directly by constructing huge horizontal search engines. Instead of crawling and indexing things in their own way, they layer themselves over existing data sources and present themselves as privacy-focused alternatives to Google. But privacy protection is not a strong enough reason to quit Google. For most people, letting them “control their own data” is not a selling point, especially if it means paying for something they’re used to getting for free.
I believe the opportunity in search is not to go after Google directly with a giant, one-size-fits-all horizontal aggregator, but rather to build boutique search engines that index, curate, and organize things in fresh ways.
Vertical search aggregators
Google is a strong example of how the internet unlocked scale and speed: every page on the web returned in an instant. But, more and more, we’re seeing that this scale conflicts with a basic human need: relevance. Someone trying to find the best freelance designer, or the best sushi restaurant, or the best NFT to buy will not get the answer on Google.
There is no search architecture that can work universally across every category. It’s hard to picture using the same UX to search for recipes as to search for freelancers. While Google’s product starts and ends with a search bar, sacrificing functionality for simplicity, vertical search players like Yelp, Expedia, Zillow, and Behance arose to fill functionality and relevancy gaps with structured data tailored to their industries. With firm views on how to arrange information, shown in their choice of filters, vertical search aggregators have unique advantages that horizontal software can never reach.
But even here, relevance depends on the sociology of the current moment. For instance, on Behance, the online creative community, school and location are displayed prominently as filters — suggesting that where you live and where you went to school is an important signal of the quality of your design portfolio. In a world where talent is being separated from credentialism and geography, those filters are losing relevance.
As signals shift, new ways of indexing and areas for innovation appear. If Behance were created today, I’d argue neither “location” nor “school” would be filters.
On Yelp, a search for “electricians in Miami” opens with a page titled “The Best 10 Electricians in Miami, FL” with text underneath that shows most of these are sponsored results.
When you monetize with ads, curation gets pushed aside in favor of featuring advertisers because there is simply less digital real estate available to curate your own recommendations. So these platforms wind up making ethically dubious design choices that create huge trust gaps.
Additionally, across vertical search aggregators like Yelp, Zillow, LinkedIn, and Behance, anyone can set up a profile. A mix of irrelevant filters, ad-based business models, and unconstrained supply has overloaded consumers and made it difficult to find signals in these platforms.
Vertical search aggregators work when you know exactly what you want. But knowing what you want isn’t usually the beginning, which creates an opportunity to help the overwhelmed consumer with better discovery and curation along the funnel.
Curators, curators
It has become common to say we live in the information age, and we need “curation” to help us sort through the mess. But so far, the discussion around curation has focused too much on the content and not enough on the structure. We seem to have accepted the curator’s job as offering a product review, a list of links, or a song recommendation — all inside linear structures and chronological feeds built to surface the ideas of the last 24 hours, not to accumulate and surface knowledge as needed.
A daily email with the top five Alibaba products feels fun and gimmicky as a side project, but it doesn’t help when you’re trying to find the best crib for your baby. Eventually, you’ll want a way to search through a curator’s archives.
Curation, when understood in the context of sharing bite-sized, isolated pieces in feed-like architectures, is mostly about entertainment, not utility. It’s not wrong to say there is a market for this kind of curation. What people overlook, though, is that this market is already owned by Twitter, Facebook, and TikTok.
These entertainment giants deliver curation that grabs our attention, but they don’t deliver curation on demand. The opportunity is in shifting curated content feeds away from their never-ending-now orientation and toward more goal-oriented interfaces. People should be able to find whatever content they want on their terms and not be dependent on when the curator chooses to publish.
Boutique search engines are curation taken further
Every act of curation expands until it needs search, and every act of search expands until it needs curation.
—Ben Evans
Using Ben Evans’ framework, it is obvious that the vertical search players have grown too large and now require curation, while curation feeds have become too long to skim and call for search plus structured data. The answer is stronger search and stronger curation, all inside a better business model — a mix I refer to as boutique search engines.
Searchable, curated interfaces can move us from fleeting, time-based feeds to contextual, high-signal, trustworthy knowledge spaces. Because searchable interfaces are densely connected, an explorer can trace several paths through the content instead of being dropped into a “most recent” feed.
With such a close tie between curation and search, the real issue is not whether you need curation or search, but when it appears, and in what form:
- Spotify doesn’t curate which songs reach its platform. Instead, it takes the full universe of music and creates endless ways to discover and search its library, including a blend of manual curation (through playlists curated by its in-house team of curators and its users) and algorithms (such as Discover Weekly). Wirecutter does not review every product. It manually curates the best products, then uses search and other discovery tools to help you find what you need. Thingtesting is not automatically scraping every Consumer Packaged Goods brand on the Internet. Someone on its team or in its community deliberately added a brand to the database. If you’re searching the On Deck member database, you know everyone has applied, been vetted, and paid a fee to join the program. If you’re reading transcripts on Tegus, you know the content comes from experts handpicked by its team.
In all of these examples, value comes as much from what they leave out as from what they add. The friction on the supply side is what creates the signal.
Along with the signals, these businesses have developed robust search engines. OnDeck, for example, has built an opinionated graph that lets you discover talent in distinctive ways. For instance, you can filter for people with “software engineering” skills whose current status is “open to new ideas.” As a founder searching for engineering talent, I’ll choose this curated dataset over LinkedIn’s any day.
Compared with vertical search aggregators, boutique search engines feel less like the Yellow Pages and more like texting friends to ask for a recommendation. They have limited supply, which is the basis of their strongest moat: trust. Just as importantly, boutique search engines also create new business models that do not depend on advertising.
Questions remain, because search is difficult
Creating boutique search engines demands countless subtle product decisions. Finding the right balance of curation, search, algorithms, and business model will probably prove enormously hard and enormously valuable. Here is a non-exhaustive set of questions we are thinking through as we build Startupy, our effort to create a boutique search engine for qualitative insights.
If the value proposition is signal over noise, how can you scale the signal?
Again and again, curation sites run into an existential trap. They begin with high-quality, curated recommendations. As they expand, they scale through crowdsourcing, often making up the difference with scraping. Over time, the content shifts from great to good. Once that happens, vertical search aggregators like Yelp provide more utility. Yahoo, for instance, became too large to browse, lost its signaling power, and got to the point where Google was better. The boundary between curator, compiler, and cataloguer is thin, and there is a natural invisible asymptote — diminishing returns from more data over time.
What is the business model for this new wave of search engines?
At first glance, vertical search engines are straightforward — content is the supply, and eyeballs are the demand. But the previous wave of vertical search engines was built on ad-based business models, which complicated matters. In an ad-driven marketplace, the eyeballs sit on the supply side. Their attention is what the demand side — advertisers — seek. The drawback of this ad-driven model is that advertisers and content producers are competing for the same attention, which is why these sites end up feeling like marketing blogs.
This is why subscriptions offer an opening. Subscription reduces the network effects to two sides: content as supply, a paying audience as demand. But subscription itself is not a cure-all, especially when the use cases are not frequent enough. How often do you need to find a freelancer? An investor? If use cases are not frequent enough, the usefulness of a search engine will not turn into a sustainable business model and you will have to develop your own version of “come for the search, stay for something else.”
Moreover, as my friend Joey points out in this article, any product where you spend a lot of time using it in incognito mode has a pretty large UX problem. How many New York Times accounts did you create before finally surrendering to the paywall? With today’s subscription models, customers have no real incentive to help the platform grow. Early token-based business models show promising signs. By giving ownership to stakeholders and letting subscribers share in future upside, startups can get past the cold start problem.
Although appealing, a playbook for tokenized business models has not yet appeared. I suspect that will change in the coming years, and I’m eager to deepen my understanding of the topic.
Who curates the curators?
Platforms like Twitter hand this duty to their users, who must go through a long and arduous process of following a huge number of people to eventually reach a self-curated timeline that matches their interests. Some centralize their curation — at OnDeck, you trust that they are doing the work of selecting who can join their network. Others avoid curation in favor of more traditional crowdsourcing. The spectrum is broad.
How do you find the search engine in the first place?
I began this piece by arguing that Google needs to be unbundled. That is a catchy headline but, in reality, I think that until you can build habitual recall into your product, Google will remain an important part of how your engine gets discovered initially. Zillow and Airbnb are examples of search companies that receive a good amount of direct traffic, but SEO was a major part of their early strategy. By being among the first to create the definitive page for a home, they benefited from an SEO land grab that has been difficult to displace ever since.
We are still far from the grand vision of the Internet. The project of human knowledge, as it exists today, is a vast ocean of fleeting and fragmented information and ideas, with the best sources nearly impossible to find. We need more interfaces with a point of view on what information is missing, how it should be organized, and at what point in the value chain curation has to occur.