Why the average is a lie, why p99 is what your users actually feel, and why a system at 80% utilization is closer to collapse than you think.
Prerequisites: Latency Numbers Time to read: ~18 minutes
Your dashboard says: average response time 45 ms. Everything looks great. Meanwhile support is flooded with complaints that the app is unusable.
Both are true. The average is hiding the story:
900 requests at 20 ms
90 requests at 200 ms
10 requests at 3000 ms
────────────────────────
Average = (900×20 + 90×200 + 10×3000) / 1000 = 66 ms
66 ms looks fine. But 10 users out of every 1,000 waited three seconds, and at 10,000 QPS that’s 100 furious users per second. The average told you nothing about them.
This chapter is about measuring performance in a way that reflects reality.
Two different things that beginners conflate constantly.
Latency = how long one operation takes. Measured in ms. What a single user feels.
Throughput = how many operations complete per unit time. Measured in QPS/RPS. What determines how many users you can serve.
🧠 Mental model: a highway. Latency is how long your car takes to drive from A to B. Throughput is how many cars pass per hour. Adding lanes increases throughput without making any individual car faster. Raising the speed limit lowers latency for everyone.
They’re independent, and often in tension:
| Low latency | High latency | |
|---|---|---|
| High throughput | The goal (well-designed system) | Batch systems — a data pipeline processing 10 TB/hour where any single record takes minutes |
| Low throughput | A single fast server, underutilized | An overloaded system — the failure state |
📐 Batching is the classic trade: processing messages one at a time gives 2 ms latency and 500 msg/s. Batching 100 gives 50 ms latency and 20,000 msg/s. 40× throughput for 25× latency. Which is right depends entirely on whether a human is waiting.
🚨 A common interview mistake: “we’ll add servers to reduce latency.” Adding servers increases throughput. It reduces latency only indirectly, by relieving queueing. If a single request takes 200 ms because it does four sequential database calls, a hundred servers won’t change that.
Sort every request by duration. The p99 is the value at the 99th percentile — 99% of requests were faster than this.
| Metric | Means | Use it for |
|---|---|---|
| p50 (median) | Half of requests are faster | The typical experience |
| p90 | 9 in 10 faster | Broad health |
| p95 | 19 in 20 faster | Common SLO target |
| p99 | 99 in 100 faster | The one that matters most |
| p99.9 | 999 in 1000 faster | Large-scale systems, infrastructure |
| max | The worst | Useful for spotting timeouts; too noisy to alert on |
Why never use the average:
This is the insight that makes p99 matter far more than it first appears.
One user-facing page fans out to many backend services. If each has a p99 of 100 ms, what’s the chance the page is slow?
1 service: 99% chance all are fast → 1% of pages are slow
10 services: 0.99¹⁰ = 90% → 10% of pages are slow
100 services: 0.99¹⁰⁰ = 37% → 63% of pages are slow!
With 100 backend calls, your p99 becomes the typical experience. This is why Google, Amazon, and Netflix obsess over p99 and p99.9 — at their fan-out, the tail is the average.
Practical consequences: reduce fan-out, issue requests in parallel rather than sequentially, use hedged requests (send to two replicas, take the first response), and set tight timeouts with fallbacks.
You cannot average percentiles. Server A’s p99 is 100 ms, Server B’s p99 is 200 ms — the fleet p99 is not 150 ms. Percentiles are not linear.
To get a correct fleet-wide p99 you need the underlying distribution — which is why monitoring systems use histograms (Prometheus histograms, HDR histograms, t-digest) rather than storing precomputed percentiles per host.
This trips up real engineers regularly, and knowing it is a nice depth signal.
Here’s the counter-intuitive result that governs capacity planning.
As utilization approaches 100%, wait time doesn’t grow linearly — it explodes. For a simple queueing model:
Wait time ∝ utilization / (1 − utilization)
50% utilized → 1× the service time in queue
70% → 2.3×
80% → 4×
90% → 9×
95% → 19×
99% → 99×
🧠 Mental model: a supermarket checkout. With one shopper every few minutes, you walk straight up. When shoppers arrive at almost exactly the rate the cashier serves them, any small variation — someone with a full trolley, a price check — creates a queue that never fully drains. The queue isn’t caused by the average rate; it’s caused by variance meeting a lack of slack.
Design consequences:
🎙️ “I’d scale out at around 65% CPU rather than 85% — queueing delay grows non-linearly, so by the time we’re at 85% our p99 has already degraded badly.”
The most useful formula in capacity planning, and it’s one line:
L = λ × W
L = average number of requests in the system (concurrency)
λ = arrival rate (QPS)
W = average time in the system (latency)
Why it’s useful: it links the three quantities you care about, so knowing any two gives you the third.
📐 Example — how big should the connection pool be? Your service handles 1,000 QPS with an average latency of 50 ms.
L = 1,000 × 0.05 = 50 concurrent requests in flight
So you need ~50 database connections (plus headroom). A pool of 10 will queue and add latency; a pool of 500 wastes database resources and can overwhelm it.
📐 Example — how many threads? Each request holds a thread for 200 ms, and you need 500 QPS.
L = 500 × 0.2 = 100 concurrent requests → 100 threads
If your server has 50 threads, you cannot reach 500 QPS no matter what — you’ll cap at 250.
📐 Example — the reverse direction. You have a fixed 200-connection pool and each query takes 20 ms. Your maximum throughput is:
λ = L / W = 200 / 0.02 = 10,000 QPS
Beyond that, requests queue for the pool and latency climbs. This calculation explains a huge fraction of real production latency mysteries.
Error rate. Percentage of requests returning 5xx. Usually more important than latency — users tolerate slow more than broken. Track separately from 4xx (client errors).
Saturation. How full the constrained resource is: CPU, memory, disk IOPS, connection pool, queue depth. This is the leading indicator — saturation rises before latency does, giving you warning.
Apdex. A single 0–1 score: satisfied (< T), tolerating (< 4T), frustrated. Compresses the distribution into one number for non-engineers. Loses detail; fine for executive dashboards.
The Four Golden Signals (from Google’s SRE book) — if you instrument nothing else, instrument these:
| Signal | Question |
|---|---|
| Latency | How long do requests take? (split successful vs failed!) |
| Traffic | How much demand? |
| Errors | What fraction fail? |
| Saturation | How full is the constrained resource? |
🚨 Split latency by success/failure. A fast-failing service can look great on latency while being completely broken — 5 ms average, because everything is erroring instantly.
When you need to reduce latency, the causes in rough order of frequency:
🎙️ A good diagnostic sentence: “Is the latency increase across all requests, or is p50 flat while p99 grows? Flat p50 with a rising tail points to queueing, GC, or a hot shard — not to the code path generally being slower.”
| Choice | Gain | Cost |
|---|---|---|
| Batching | Much higher throughput | Higher per-item latency |
| Higher utilization | Lower cost | Non-linear latency growth; no spike headroom |
| Caching | Lower p50 dramatically | p99 may stay bad (misses); staleness |
| More replicas | Throughput and availability | Cost; replication lag |
| Hedged requests | Cuts tail latency substantially | ~5–10% extra load |
| Tight timeouts | Bounded latency, fail fast | Some requests that would have succeeded are killed |
Hedged requests deserve a mention: send the request to two replicas after a short delay (e.g. if no response by p95), take whichever answers first, cancel the other. Google reported this cutting p99.9 latency dramatically for only a few percent extra load. It’s a great thing to bring up when an interviewer asks how you’d fix the tail.
1. See the distribution. Load-test a local service and look at percentiles, not averages:
# with k6
k6 run --vus 50 --duration 30s script.js
# or with wrk (note it reports latency distribution)
wrk -t4 -c100 -d30s --latency http://localhost:8080/api/items
Look at how p50 and p99 diverge as you increase concurrency (-c). Push it until p99 goes vertical
— that’s your queueing cliff, and finding it is a genuinely useful exercise.
2. Verify Little’s Law. Note the throughput and average latency your load test reports. Multiply. Compare to the concurrency you configured. It’ll be close.
3. Break the average. Instrument an endpoint that’s fast 95% of the time and sleeps 2 seconds otherwise. Watch how healthy the average looks and how alarming the p99 looks. Then decide which one you’d want on your dashboard.