Latency budgets are a budgeting problem

Every project starts with a number: “p99 under 300 ms.” It is a fine number. The trouble is that nobody spends it. The target sits in a document while each component quietly takes whatever it needs, and the first time anyone adds up the parts is during an incident.

A budget works the same way a financial budget does: you allocate it before the spending starts, and you make the allocation visible to everyone who can spend it.

Split the number first

For a request that crosses a browser, an edge proxy, one or two services, and a database, 300 ms is not one number — it is a stack of them. A division I have found workable on small systems:

browser → edge          60 ms   (connection setup, TLS reuse)
edge → app              40 ms   (queueing included)
app processing          80 ms   (CPU, serialization)
app → datastore ± RTT  100 ms   (one round trip, no N+1)
slack                   20 ms   (the part everyone forgets)

The slack line is the important one. If the components sum to exactly the target, the system fails its own SLO the moment anything is a little slower than the engineer imagined. Twenty percent of the budget should be unclaimed.

Budgets are per hop, not per team

The second mistake is assigning budgets to services instead of to hops. A service that is called twice per request gets half the budget per call, not the whole line item. Once a graph has fan-out, the arithmetic stops being obvious, which is exactly why it needs to be written down.

A p99 is not an average with a scarier name. It is a statement about the worst one percent of requests, and every hop contributes its own tail.

Tails add up in a way averages do not. If four independent hops each have a 1% chance of being slow, roughly 4% of requests touch at least one slow hop. This is the ordinary reason a chain of “fast” services produces a slow endpoint, and it is the reason budget reviews should look at per-hop percentiles rather than the end-to-end average.

Measure the budget, not just the total

A latency budget only stays honest if the measurement matches the split. I keep one latency histogram per hop, with the same bucket boundaries everywhere, and a small page that shows the budget line next to the current p50 and p99. When a hop goes over, it is visible before the end-to-end number moves — which is the entire point of budgeting early.

The dashboard is deliberately dull. If a hop is over budget for more than a day, the options are to give it more of the total (and take it from somewhere), to make it do less work, or to accept a lower target and update the document. What is not an option is pretending the number did not change.

Write it down where it can be argued with

I keep budgets in the same repository as the service, in a short markdown file next to the deploy config. It lists the target, the hops, the measurement source, and the date of the last review. It takes ten minutes to write and it turns “the API feels slow” into a specific conversation about a specific hop.

None of this is novel. It is just the difference between having a target and having a plan for spending it.