The Mean Is Not a Typical Request

Ask a team how fast their service is and you will usually get one number. Whatever that number is, it is hiding something.

Response time is not a value. It is a distribution — you sent a million requests and got a million different answers, and the shape of that spread is where all the interesting behaviour lives.

Why the average lies

Response-time distributions are right-skewed. There is a floor — nothing can be faster than the work actually required — and no ceiling. Most requests cluster near the floor; a few wander off to the right, sometimes very far.

The arithmetic mean is sensitive to those wanderers. Add one 30-second request to a thousand 100ms requests and the mean moves by 30ms — a 30% jump — while absolutely nothing changed for the 1,000 users who were served normally.

Sort your requests slowest-last and take the value at a position:

• p50 — the median. Half of requests were faster. This is your typical request, which is what people wrongly expect the mean to be.
• p95 — 1 in 20 requests was slower.
• p99 — 1 in 100 was slower.
• p99.9 — 1 in 1,000 was slower.

Feel it

Drag the dials. Watch what the stall rate does to the mean while the median barely moves — and note the percentage of requests that finish faster than the mean.

Then raise calls per page. That readout is the most important thing in this lesson.

The tail becomes the head

Here is the part that surprises people. Suppose a service’s slowest 1% of requests are painfully slow. Only 1% of users are affected — that seems tolerable.

Now suppose loading one page requires the frontend to call that service six times (a user record, their permissions, their notifications, a feature flag, a recommendation, an audit write). The page is not done until the slowest call is done.

The chance that all six land in the fast 99% is 0.99⁶ ≈ 0.941. So about 5.9% of page loads touch the slow tail — six times worse than the per-call figure suggests. At 40 calls it is 33%.

This is why teams that fan out heavily obsess over p99 and p99.9 in a way that looks pathological until you do this arithmetic. It is also an argument against gratuitous service decomposition: each hop you add multiplies your exposure to everybody else’s bad days.

Two ways benchmarks lie to you

Head-of-line blocking

A server handles a limited number of requests at once. If a few slow requests occupy those slots, everything queued behind them waits — even requests that would have been instant.

The consequence: the client’s measured time includes queueing, and a few genuinely slow operations inflate the measured latency of many fast ones. This is also why you must measure on the client side. A server that reports “I processed this in 4ms” is not lying, it is answering a narrower question than the one the user has.

Coordinated omission

The subtler one, and it is everywhere.

A naive load generator sends a request, waits for the response, then sends the next. When the system stalls for two seconds, the generator also stops sending — so the requests that would have been sent during the stall, and would have recorded terrible times, are never sent at all.

The system’s worst moment is precisely the moment your measurement politely declines to take samples.

Turning this into something you can hold people to

Percentiles are how a service-level objective becomes checkable:

99% of GET /api/orders requests complete in under 300ms, measured at the load balancer, over a rolling 28-day window.

Every clause is doing work. 99% names the percentile. 300ms is the threshold. At the load balancer says where it is measured, which excludes the client’s own network but includes your queueing. 28-day rolling window stops a single bad afternoon from being permanently damning, or a single good week from covering a bad one.

What to take away

Report percentiles, not means. Measure at the client, not the server. Make your load generator immune to coordinated omission. And before you split a service in two, multiply out what that does to your tail.