Building for the 99% Developers

Should you go serverless? Is GraphQL the fix for your API headaches? Should you adopt the newest DevOps playbook to boost system reliability? In the realm of tech tools, there is plenty of hype. But it does not always match the everyday life of programmers.

As the founder of a startup making developer tools, I’ve spoken with hundreds, if not thousands, of software developers over the past few years through ordinary user research. The recurring idea in these discussions, even larger than the need for the product we were building, was a broader need that is still not well served: building for real developers, or what I like to call the 99% Developers.

These are developers who are shipping work outside the trendy companies and frameworks, and who are often left out of talks about “what developers want.” There is a wide divide between what “developer-influencers” discuss and the day-to-day reality of most developers. When you examine what the tech media covers, or who speaks at major tech conferences, it is often people from fast-growing favorites like Airbnb or Stripe, or from long-established, very profitable companies like the FAANGs.

In reality, there has long been an idea that companies, apart from a small set of Silicon Valley unicorns, ought to aim for the processes of a “baby FAANG.” But that is more and more untrue. Our users would often admit, somewhat embarrassed, that their practices bear little resemblance to what they are “supposed to.” Yet for these “dark matter developers,” as Microsoft’s Scott Hanselman calls them, the ways of a Facebook or a Pinterest do not fit. Their user needs differ, and their team needs differ too.

It is important to discuss the 99% Developers because they are the developers creating the software that runs our lives — insurance, health care, retail, and banking, to name a few. It is not just small companies that cannot readily take on the processes of modern, tech-first companies; it is most companies that were not built around technology and that have decades of legacy software practices firmly embedded. Many of these companies move a great deal of money. Many of these companies manage a great deal of our personal data. If technology improvements are not helping these software teams, we are missing out on many meaningful gains to everyone’s quality of life.

In this article, I’ll lay out several truths that enterprise software buyers and builders alike can accept to clear away damaging myths and improve developer experience for everyone.

“Trickle-down” tooling is aspirational

The myth

Because so much writing and tooling comes from companies like Facebook, Netflix, LinkedIn, Google, and Amazon, many assume there is a trickle-down effect: great engineers at companies with money to spend invent solutions to problems everyone else will face one day. It is only a matter of time before your average small or medium business, or Fortune 500 company, runs into the same problems as Amazon or Facebook.

The reality

A FAANG-like company differs from an SMB or your usual Fortune 500 company in many ways, including scale requirements, attitudes toward build versus buy, and the composition of the engineering team. A small number of large, well-funded companies have whole teams of world experts devoted to observability, testing, developer productivity, and more. In addition, it is worth remembering that FAANGs are optimizing around a narrow set of products that are digital from day one, which is not true for most software shops out there.

Many non-FAANG teams have a small team of non-experts, or even a slice of one non-expert engineer, to handle things that FAANG-like companies have several expert teams to handle. These organizations depend mainly on outside tools and services, and they have little room to customize them.

What to do about it

First, the tech industry needs to accept that organizations working at different scales and with different engineering budgets will have different needs. A company handling low millions of user requests per day does not need to optimize its systems to the level of a Netflix or a Google. Most companies do not have the latency, data storage, and other issues that would push them to build their own bespoke infrastructure components and tools like, for example, Facebook did with its Tao data store and Hive data warehousing tool — and they likely do not have concerns that would justify using such tools either.

Accepting different demands will open room to discuss varied needs across a broad range of organizations. For example, companies with legacy systems that cannot afford to move to the newest architectures need to adopt new tools in different ways than newer companies, or companies that can assign a team to a major migration.

Recognizing that there is not one aspirational company profile can help builders look beyond the usual suspects when learning about users. This is essential for closing the gap between software needs and software tools in a practical way.

There is no gold standard development environment

The myth

If you watch enough conference talks or read enough blog posts, it can seem as if there are many software teams with spotless coding standards, non-flaky unit tests, staging environments that mirror production, and/or smooth people processes for handling incidents. Reaching that point only takes a firm mandate from above and discipline across engineering teams.

The reality

Like influencers in any other field, developer-influencers often describe a reality that is aspirational even for their own companies. It may be true that people writing about ideal processes live in an idealized setting where it is possible, in which case they are the exception that proves the rule. But most of the time — even if it is true in one part of an organization or at one point in time — this reality does not apply across their whole company and indefinitely.

For example, Spotify acknowledged that their celebrated DevOps approach did not scale once their team reached a certain size. We also see cases of companies adopting the new hot technology and then switching back when things did not work as well as expected — for instance, Segment moving from microservices back to a monolith.

What to do about it

As the audience that makes all of this possible, developers ought to be more discerning in demanding the truth. We should embrace posts about “real software process” just as eagerly as we crave idealized content. If someone works in an idealized process, with a world-class ops team and entire teams of people whose job is to support better software quality, the audience should know that! And we should welcome more talks, blog posts, and books that offer guidance for “real software environments”: what coding, testing, and shipping look like for understaffed teams, teams without dedicated devops experts, and teams where everyone who originally built the system has moved on.

The goal is progress, not perfection

The myth

Too many people think that pursuing good software quality means you must completely adopt that new technology, whether it’s microservices, GraphQL, or distributed tracing. You are not finished until you have moved over entirely to the ideal technology.

The reality

Right now, the gap between where “real developer” teams are and the mainstream guidance being handed out means many teams do not know where to begin when it comes to improving code quality or system reliability. For the 99% Developer, most of their code will never need to scale to an organization of thousands, or to billions of users. Many of these developers work on code bases older than the length of their entire careers. Most of these organizations do not have dedicated developer tools or developer-productivity teams internally.

Perfect code is not the objective — instead, the objective is code that is as reliable and secure as it reasonably can be, given other constraints. For example, if your company does not run across multiple clouds deploying hundreds of changes a day, a continuous delivery system like Netflix’s Spinnaker is probably not needed. Likewise, developers at a company with devops experts who know how to configure and maintain observability “power tools” will likely have a far better experience with those tools than a company without an expert team in place.

What to do about it

Acknowledge that there is progress to be made — there are lessons to learn from companies that have dedicated whole teams to refining their processes — but perfection is unrealistic in most situations. Instead of copying processes wholesale, identify which ones work well for teams with fewer resources and teams with different goals.

For example, this Google blog post offers the rule of thumb of 60% test coverage as “acceptable,” 75% as “commendable,” and 90% as “exemplary.” When you are a company of Google’s maturity, with the size and caliber of Google’s engineering team, that may make sense. But for most smaller, earlier-stage companies, the real test coverage is likely to be much lower despite what the company may hope for. And with the growth of service-oriented architectures and external APIs — practices that are far more common outside Google — testing in production is becoming a workable alternative to the traditional unit- and integration-testing techniques where “code coverage” makes sense as a concept.

In many of these modern systems, as Honeycomb co-founder Charity Majors wrote, “Once you deploy, you aren’t testing code anymore, you’re testing systems — complex systems made up of users, code, environment, infrastructure, and a point in time.”

A good demo doesn’t reveal the Day 2 snags

The myth

It is far too easy to slip into assuming that demos and onboarding tell you what it is like to use a product day after day. People have to decide whether to buy new products relatively fast, so we have come to judge a product by its demo.

The reality

Developer tool empires are constructed with polished gifs and video clips. Teams commit to tools — sometimes for years — after only a few minutes of demo. Investors invest based on demos. Builders are told to concentrate on the demo — and, after that, the first 60 seconds of use — as the decisive parts of the product. But while doing well in a demo may indicate that the team can deliver on developer experience in day-to-day use, the two are not necessarily correlated.

It should come as no surprise that for most tools worth their price tags, most of what a developer experiences — and experiences pain around — happens outside the first minute of use. First, integrations with developers’ day-to-day workflows (for instance, existing code review, CI/CD workflows, and modes of collaboration) are a better sign of whether a product will stick than initial delights are. It is well understood among developer tools creators, for example, that integrating with GitHub and GitLab will help make your tool much more useful and appealing.

Second, there are entire categories of tools where the real test of their effectiveness does not occur on Day 1. One example is debuggers. More than your development environment and more than your CI/CD environment, your debugging toolbox is the most important factor in your quality of life as a developer. When you solve an issue before you deploy, are you sure the issue will not show up in production? When you have to work late or on a day off because of a production incident, are you able to quickly pinpoint the root cause and think of possible fixes? Because of the complexity of these tools — and because they often do not reveal their true value until you have a major issue — debugging tools often get the least hype and the least reward for good developer experience.

What to do about it

The reason tool builders often focus far more on the Day 1 experience is the way users assess tools. So I’m going to give a special callout to users here. Users need to:

• Refrain from hyping tools based only on demo or Day 1 experience. Push back on tools that push you to sign big contracts before you or your developer team have spent time using the tool day in and day out for some period of time. Recommend tools to other people based on the less “sexy” dimensions that matter to your team’s productivity, such as how they fit with workflows, or how they lower collaboration friction within and across teams.

Doing these things will open up far more room for a better developer experience.

Heterogeneity is here to stay

The myth

There’s a common belief that the latest hot language or framework is the only thing running in someone’s environment. Developers and developer-influencers alike often promote new tools as though they are the sole tools in use: for example, microservice architectures, GraphQL, and OpenTelemetry-based tracing for observability. “One true framework” evangelism quietly presumes that organizations can move completely over to that new language, tool, or framework.

The reality

I’ve seen so many teams say migration will happen “next quarter.” In practice, even after they finally get going, migrations have turned into ongoing processes rather than one-time events. A 99% Developer team with legacy code and a lean team is probably never going to move its whole code base to microservices or GraphQL. For most organizations, tech stacks and tool chains are mixed, made up of layers of languages, frameworks, and tools accumulated over the years.

Many of the teams we meet will say they’re beginning to adopt microservices or GraphQL or OpenTelemetry. When I ask how much of their services are already in the new framework, the answer is often very small, especially at organizations more than a few years old. Some of these organizations will tell me they do not realistically expect to move their entire code base. (For example, organizations will expect to keep their legacy monolith alongside microservices, or REST and gRPC endpoints alongside GraphQL endpoints.) For many others, when I look again a few quarters later, they are often less far along in the intended migration than expected — and are living with the reality of maintaining software across a blend of frameworks and tools.

What to do about it

Software buyers, from individual developers to directors and CIOs, understand that heterogeneity is real. But truly accepting that means:

• Accepting slow migrations. I’ve run into many teams that think their issues will disappear once they finish moving from outdated tool X to hot new tool Y, with each one living in its own solution ecosystem. Sadly, the migration to tool X may not end until the hot new tool is Z — and then you are back to the two-ecosystem problem again. Accepting legacy subsystems. I’ve encountered many teams that concentrate on modernizing their toolbox around the newer pieces of their system. Unfortunately, the legacy subsystems are not disappearing — and having less tooling for them means longer triage and debugging times when something breaks.

If you accept that your APIs are unlikely to settle on GraphQL, you’ll be pushed toward more durable tooling for multiple API protocols. If you recognize that your org is probably not going to convert all of its legacy monolith to microservices, you can invest in tooling that doesn’t overlook monitoring and debugging code in either the monolith or the microservices.

On the builder side, the usual advice is to push hard for homogeneity for the “land” and accept heterogeneity for the “expand.” Whether you plan for heterogeneity will strongly affect how fast you can grow. Some kinds of developer tools need to be custom-built for every new language or framework. For example, a tool that offers insight only for GraphQL APIs may not extend easily to other kinds of APIs, especially because GraphQL includes richer information than REST or gRPC.

Other kinds of developer tools can scale across languages and frameworks with ease. A SaaS tool that only needs to be callable from different programming languages supports language heterogeneity, because major core components do not need to be translated to support each new language.