← all posts

Why Most Reliability Problems Are Organizational Problems

You can buy better infrastructure. You can't buy clear ownership. The most resilient teams I've seen weren't the ones with the best technology, they were the ones where everyone knew exactly whose pager goes off.

Most reliability problems aren't technical.

They're organizational.

That's an uncomfortable claim, because it means you can't buy your way out with better tooling, and it means the postmortem should probably name a structure instead of a service. But I've sat in enough incident reviews to believe it. The technology was rarely the root cause. It was the messenger.

The postmortem says "bad deploy." The cause says something else.

We love a technical villain. The outage gets pinned on a bad deploy, a flaky dependency, a config that drifted. All true, all proximate, all satisfying because they're fixable with a code change.

But dig past the timeline into why the failure was possible, why it wasn't caught, and why it took so long to resolve, and the real cause usually isn't code. It's people structure:

  • Nobody actually owned the service that failed.
  • Two teams each assumed the other was watching it.
  • The on-call engineer had never seen the system before that night.
  • The fix had been known for months and never prioritized.
  • The alert fired into a channel nobody reads.

None of those are software bugs. You can't git revert any of them. And every one of them produces downtime as reliably as a null pointer does.

Reliability is a property of the org, not the binary

Here's the frame that reorganized how I think about this.

A system is only as reliable as the organization running it is honest about who's responsible for what.

A binary doesn't keep itself running. People keep it running: they notice it's unhealthy, they understand what it does, they have the authority to fix it, and they've decided in advance that it's theirs to fix. Take any of those away and the most beautifully engineered system in the world degrades to "works until it doesn't, and then we find out who's awake."

This is why two teams can run the same stack on the same cloud and have wildly different reliability. The infrastructure is identical. The org around it isn't. One team has clear ownership and rehearsed recovery. The other has diffusion of responsibility and a wiki page from 2023.

The specific ways structure becomes downtime

The organizational failures aren't vague. They show up in predictable shapes, and each one has a technical-looking symptom that hides a human cause.

  • Diffused ownership. When everyone owns it, no one does. The symptom is a service that's been degrading for weeks with no one assigned to notice.
  • The handoff gap. Team A built it, threw it over the wall, Team B operates it without understanding it. The symptom is a 3am page answered by someone reading the code for the first time, live, in production.
  • Known-but-unprioritized risk. The fix existed. It lost every sprint planning to feature work. The symptom is an outage whose postmortem action item was already in the backlog, stale, ignored.
  • Alerting into the void. The system tried to tell you. The page went somewhere no human watches. Observability is runtime architecture, and an alert with no owner is just a log line with extra steps.
  • No rehearsed recovery. "Restore from backup" was a sentence, not a practiced procedure. Recovery is the system, and a recovery path nobody has run is a recovery path that doesn't exist yet.

Notice that every symptom looks technical and every cause is structural. That's exactly why these problems survive. We fix the symptom, close the ticket, and leave the structure that generated it fully intact, ready to produce the next one.

You can't buy your way out

The seductive thing about reliability is that vendors will sell you the feeling of it. More monitoring. More replicas. More nines on a slide. And those help, right up until the failure is organizational, at which point they help not at all.

You can buy better infrastructure. You can't buy clear ownership. No tool decides whose pager goes off. No dashboard assigns accountability. No amount of redundancy compensates for two teams who each assumed the other was on it.

The most resilient teams I've worked with weren't the ones with the best technology. They were the ones where everyone knew exactly whose pager goes off, where ownership was explicit and uncomfortable rather than implied and comfortable, and where the recovery procedure had been run on a calm afternoon before it was ever needed at 3am.

Fixing the org, not just the bug

If reliability is organizational, then the highest-leverage reliability work isn't always in the codebase. Sometimes it's a sentence on a wiki that finally says who owns this. Sometimes it's a runbook that turns "restore from backup" into a checklist someone has actually executed. Sometimes it's killing the shared-ownership fiction and making one team uncomfortably, unambiguously responsible.

That work is unglamorous and it doesn't generate a satisfying diff. It also prevents more downtime than your next three infrastructure upgrades combined.

The one-line version

Most outages aren't caused by bad code. They're caused by unclear ownership, and you can't fix unclear ownership with better infrastructure. A system is only as reliable as the org running it is honest about who's responsible.