Every product has a latency budget.
Some teams write it down. Most let the architecture choose for them. That is where the product starts drifting before anyone notices.
The product team owns the latency budget
Latency is a product specification. How fast does the page need to load? How quickly does the search need to return? What's the maximum time between a user action and a system response? These are product questions. They define the user experience. They should be answered with numbers before engineering starts building.
The engineering team can meet many different targets. The product team has to choose the target that matters. When that division is clear, the conversation changes. "Can we make it faster?" turns into "what's the latency budget, and are we within it?" That's a better conversation because it forces the team to connect speed, user value, and cost.
A search that returns in 50ms is better than one that returns in 500ms. But if users tolerate 200ms, a 50ms search that costs 10x to operate may be the wrong product investment. The product team knows what the user tolerates. The engineering team knows what it costs to hit each target. The decision belongs to both, and it's a product decision first.
Tail latency is the real system
Average latency is a comforting metric.
It tells you how the system behaves when everything is fine, which is when you least need the answer.
The system is defined by its tail. p99, p99.9, the worst 1% of requests. That's where the engineering lives, because that's where the system breaks.
Average latency is dominated by the happy path. The cache hits. The fast queries. The uneventful requests.
You can have a clean 20ms average and an 8-second p99. The dashboard looks fine. The experience does not.
Tail latency is where distributed systems show their teeth. The tail is where retries happen, caches miss, garbage collectors pause, and the network has a bad day. The tail is the system under stress, and the system under stress is the system you're actually operating.
A team that optimizes average latency is usually polishing the part that already works.
A team that optimizes tail latency is doing the architectural work. That is where the trade-offs live and where the hard decisions get made.
The tail is an architecture problem
Tail latency rarely has a single fix. A faster query helps one path. A bigger cache helps one miss pattern. A tuned parameter buys room until the next load shape appears. The tail is an architecture problem, and it shows up in the same places every time.
Queueing. Under load, requests queue. The queue depth grows, and the wait time grows with it. Average latency barely moves because most requests still get through quickly. The tail, the requests that hit the back of the queue, explodes. This is the classic latency under load pattern, and it is an architecture problem. You need backpressure, admission control, or more capacity. All three are design decisions.
Retries. A retry is a second request. If 1% of requests fail and retry, your tail latency doubles for that 1%. If the retry also fails and retries again, it triples. Under load, retries amplify. They create a feedback loop where the system gets slower, which causes more timeouts, which cause more retries, which make the system slower. This is the same shape as a replay storm: amplification taking a quantity of work that was spread out and compressing it into a spike.
Dependency fan-out. A request that calls three services in parallel has the latency of the slowest one. A request that calls ten services in parallel has the latency of the slowest one, but the slowest one is now drawn from a larger sample, which means the tail is worse. Fan-out amplifies tail latency mathematically. The more dependencies, the worse the tail, even if every dependency is fast on average.
GC and pauses. Stop-the-world pauses hit the tail. They are rare enough to hide from the average, but the request in flight during the pause is the one that defines your p99. This is a runtime architecture problem for the same reason observability is: the runtime's behavior is part of the system's behavior.
The product decision that changes the architecture
The connection back to product is the latency budget. That number becomes an architecture constraint.
If the product requires p99 under 100ms, that's a different architecture than p99 under 500ms. The first requires aggressive caching, minimal fan-out, and probably a single-region deployment. The second allows for multi-region, more dependencies, and a simpler cache strategy. The product decision constrains the architecture, and the architecture constrains the cost.
This is why latency should be a product decision. The product team decides what the user needs. That decision flows into the architecture, which flows into the cost. When the product team doesn't participate, engineering guesses at the target, over-engineers for safety, and the system costs more than it should. Or they under-engineer, the tail is bad, and users feel the delay. Either way, the missing product decision creates waste.
The one-line version
Latency budgets are product decisions with architectural consequences.