It was the second game of a double-header, and the Washington Nationals had a problem. Not on the field, naturally: the soon-to-be World Series champions were playing splendidly. But while they sat through a rain delay, something went wrong behind the curtain. A task scheduler deep in the team’s analytics infrastructure ceased operating.
That scheduler was responsible for gathering and combining game-time data for the Nationals’ analytics group. Like many tools of its type, it was built on cron, the decades-old workhorse for running jobs at regular intervals. Cron is especially effective when work must begin on a particular day, hour, or minute. It is especially ineffective — or fails entirely — when work needs to begin at the same moment as, say, a rain-delayed baseball game. Even though the data team worked hard to add custom logic to the basic scheduler, the odd circumstances of the double-header baffled it … and it simply stopped queuing new work.
Not until the following day did an analyst notice the mismatch when the data — essential figures that made up the very foundation of the team’s post-game analysis and advice — left out a particularly memorable play. There were no alerts or warning lights, because the process had never run at all. And so a new, time-draining task was added to the data analytics stack: manually inspecting the database each morning to confirm that everything had worked correctly.
This is not a tale of total collapse. In fact, I’m sure any engineer reading this can imagine many ways to address this specific problem. But few engineers would consider it a worthwhile use of time to sit and brainstorm every edge case beforehand — nor is it even possible to foresee, in advance, the billions of possible failures. In any case, engineers already have enough urgent problems to worry about without inventing new ones.
So the issue here was not that an error happened. Errors will always exist, even in the most advanced infrastructures. The true issue was how few ways the team had to deal with it. Confronted with an important business problem and a misleading cause, they were compelled to spend time, effort, and skill trying to ensure that this one surprising quirk would not show up again.
Negative engineering is “insurance as code”
So what would a better answer look like? I think it is something like risk management for code, or more simply, negative engineering. Negative engineering is the painstaking and sometimes irritating work engineers do to make sure their main goals succeed. If positive engineering means the everyday work engineers perform to produce useful, expected results, then negative engineering is the insurance that safeguards those results by protecting them from an endless range of possible failures.
After all, failure has to be planned for, even in a well-built system. Most modern software includes some level of major-error prediction or, at minimum, error tolerance. Negative engineering frameworks go a step beyond that: they let users work with failure instead of fighting it. Failure itself becomes a first-class element of the application.
You can think of negative engineering as similar to auto insurance. Buying auto insurance will not stop you from having an accident, but it can greatly lessen the burden if one happens. In the same way, having good instrumentation, observability, and even orchestration of code can offer comparable advantages when something breaks.
“Insurance as code” may sound unusual, but it is a perfectly fitting way to describe how negative engineering tools create value: they insure the outcomes that positive engineering tools are meant to deliver. That is why features like scheduling or retries, which can seem toy-like — meaning too simple or too rudimentary — can matter enormously: they are how users encode their expectations into an insurance framework. The simpler they are (in other words, the easier they are to use), the lower the cost of the insurance.
In applications, for instance, retrying failed code is a crucial step. Every action a user takes is mirrored somewhere in code; if execution of that code is interrupted, the user’s experience is essentially broken. Imagine how annoyed you would be if, every so often, an application just refused to add items to your cart, move you to another page, or charge your credit card. The reality is that these small refusals happen far more often than people realize, but users never notice because systems exist to catch those errors and run the faulty code again.
To engineers, these retry systems may look fairly straightforward: “just” isolate the code block that failed and run it again. To users, they are the difference between a product that does what it is supposed to do and one that never wins their confidence.
In mission-critical analytics pipelines, the importance of catching and retrying faulty code becomes even greater, as does the need for a similarly advanced approach to negative engineering. In this area, errors do not cause users to miss items in their carts, but instead cause businesses to build strategies from bad data. Ideally, these companies could rapidly change their code to detect and reduce failure cases. The harder it is to adopt the right tools or techniques, the greater the “integration tax” for engineering teams that want to use them. This tax is the same as paying a steep premium for insurance.
But what does it mean to go beyond a feature and provide insurance-like value? Think about the ordinary act of scheduling: a tool that schedules something to run at 9 a.m. is a cheap commodity, but a tool that tells you your 9 a.m. process failed to run is a vital piece of infrastructure. Raising commodity features into something that produces defensive insight is a major benefit of using a negative engineering framework. In a way, these “trivial” features become the channel for sending instructions to the insurance layer. By expressing more clearly what they expect to happen, engineers can better understand any departure from that plan.
To push this further, consider what it even means to “identify failure.” If a process is running on a machine that crashes, it may never get the chance to report its own failure before being erased from existence. A system that can only capture error messages will never even learn that it failed. By contrast, a framework with a clear expectation of success can infer that the process failed when that expectation is not met. This creates a new level of confidence by building logic around the absence of expected success instead of waiting for visible failures.
Why negative engineering? Because stuff happens
It is fashionable for big companies to boast about how sophisticated their data stacks are. But the truth is that most teams — even those doing sophisticated analytics — use relatively simple stacks that are the result of a sequence of practical decisions made under heavy resource constraints. These engineers do not have the time luxury to both meet business goals and think through every possible failure mode.
What is more, engineers dislike dealing with failure, and nobody truly expects their own code to break. Add to that the fact that negative engineering problems often come from the most ordinary features — retries, scheduling, and the like — and it becomes easy to see why engineering teams may choose to hide this sort of work or treat it as Someone Else’s Problem. It may not seem worth the time and effort.
When engineering teams do acknowledge the issue, one of the most common approaches I have seen in practice is to create a sculpture of bandages and duct tape: the accumulated total of a million tiny patches made without any attention to overall design. And beneath the weight of that monolith is a worn-out, underfunded team of data engineers who spend all their time watching and triaging their coworkers’ failed workflows.
FAANG-inspired universal data platforms have been offered as a fix for this problem, but they ignore the enormous cost of rolling out broad solutions at companies that are still trying to reach engineering stability. After all, none of them arrive bundled with FAANG-scale engineering teams. To avoid a high integration tax, companies should instead weigh the possible benefits of a given approach against the inconvenience of putting it into practice.
But here is the catch: the tasks tied to negative engineering often come from outside the software’s main purpose, or from interactions with external systems: rate-limited APIs, malformed data, unexpected nulls, worker crashes, missing dependencies, queries that time out, version mismatches, missed schedules, and so on. In fact, because engineers usually account for the most obvious sources of error in their own code, these problems are more likely to come from an unexpected or external source.
It is easy to brush off the harmful impact of small errors by failing to see how they can appear in puzzling ways, at inconvenient moments, or on the screen of someone unprepared to interpret them correctly. A small problem in one vendor’s API, for example, may set off a major crash in an internal database. A single row of malformed data could drastically distort the summary statistics that guide business decisions. Minor data problems can trigger “butterfly effect” chains of outsized damage.
Another story of simple fixes and cascading failures
The story that follows was first told to me as a challenge, as if to say, “Great, but how could a negative engineering system possibly help with this problem?” Here is the setup: another data team — this time at a high-growth startup — was running an advanced analytics stack when their entire infrastructure suddenly and completely failed. Someone noticed that a report was full of errors, and when the team of five engineers started investigating, a wave of error messages appeared at nearly every layer of their stack.
Beginning with the damaged dashboard and tracing things backward, the team uncovered one baffling error after another, as though every pipeline stage were not merely failing to do its work, but were also throwing up its hands in complete bewilderment. In the end, the team understood that each stage was handing its own failure to the next one as though it were valid input, causing erratic breakdowns as every step tried to handle something that simply could not be processed.
It took three days of digital archaeology before the team found the trigger: the credit card linked to one of its SaaS vendors had expired. That vendor’s API sat fairly early in the pipeline, and the billing error it produced spread aggressively through every later stage, eventually poisoning the dashboard. Within minutes of reaching that conclusion, the team fixed the issue.
Again, a trivial outside trigger caused chaos for a business, with remarkable consequences. Looking back, the matter was so simple that I was asked not to disclose the company’s name or the vendor involved. (And may any engineer who has never wrestled with an easy problem cast the first stone!) Once the root cause is known and can be addressed, nothing here is complicated or even especially hard. In fact, though it may seem odd, this is really a pretty ordinary negative engineering scenario.
A negative engineering framework can’t simply make a problem this idiosyncratic disappear — at least not by changing the credit card — but it can keep it from spreading. A well-instrumented workflow would have surfaced the root failure and stopped downstream jobs from running at all, since they could only have produced more errors. Beyond dependency management, the value of strong observability is similarly remarkable: altogether, the team lost 15 person-days to triaging this issue. With immediate visibility into the root error, the whole outage and its fix could have taken only a few minutes at most, a productivity improvement of more than 99 percent.
Remember: All they needed to do was enter a new credit card number.
Restore your productivity
“Negative engineering” under any label remains just as maddening — and it has gone by many labels. I recently talked with a former IBM engineer who said that, in the ‘90s, one of IBM’s Redbooks noted that the “happy path” for any software package made up less than 20 percent of its code; the remainder went to error handling and resilience. That matches the share of time modern engineers say they spend dealing with negative engineering problems — as much as an astonishing 90 percent of their working hours.
It almost sounds impossible: how can data scientists and engineers working on the most advanced analytics in the world be losing so much time to trivial problems? But that is exactly how this kind of issue behaves. Problems that seem simple can have unexpectedly time-draining effects when they spread without restraint.
For that reason, companies can gain tremendous leverage by concentrating on negative engineering. If given the option of cutting model development time by 5% or cutting the time spent hunting down errors by 5%, most companies would instinctively pick model development because it seems more valuable to the business. But in a world where engineers spend 90% of their time on negative engineering issues, reducing errors could have 10 times the effect. Think about a 10% reduction in those negative engineering hours — from 90% of time to 80% — which would double productivity from 10% to 20%. That is an exceptional return from a fairly small change, neatly reflecting how these frameworks operate.
Rather than allowing minor errors to grow into major obstacles, taking small steps to address negative engineering issues can produce enormous productivity gains.