system-design

How to Find a Bottleneck

“The system is slow” is not a diagnosis. Before you optimize anything, find what is slow and why — because optimizing the wrong thing is worse than doing nothing.

Prerequisites: Performance Metrics, Computer Fundamentals Time to read: ~18 minutes

The cardinal rule: measure, don’t guess

🚨 The single most important principle in performance work: measure before you optimize. Engineers’ intuitions about what’s slow are wrong most of the time — the bottleneck is almost never where you think it is. Optimizing based on a guess wastes effort and often makes things worse (added complexity, new bugs) while the real problem persists.

🚨 Amdahl’s Law is why this matters: optimizing something that’s 5% of the total time gives you at most a 5% improvement, no matter how much you speed it up. If you spend a week making a rarely-used path 100× faster, you’ve achieved nothing. Find the part that dominates the time, and optimize that. → Scalability

🎙️ “First I’d measure to find where the time actually goes — intuition about bottlenecks is usually wrong, and by Amdahl’s Law, optimizing anything but the dominant cost is a waste.”

Step 1: which resource is saturated?

🚨 The first question is always: what is the system bound on? Every performance problem is one of four (from fundamentals):

Bound	Symptom	Look at
CPU-bound	CPU near 100%; slower as concurrency rises even with the DB idle	Compute: serialization, encryption, GC, hot loops
I/O-bound	Low CPU (20%) but can’t handle more; waiting	Disk IOPS, network, database, downstream calls
Memory-bound	Swapping (latency cliff) or OOM kills	Working set vs RAM; leaks
Network-bound	Bandwidth saturated (e.g. 1 Gbps cap)	Payload sizes, egress, cross-AZ

🚨 The most common mistake is mismatching the fix to the bound. Adding CPU to an I/O-bound service does nothing. Adding servers to a service bottlenecked on one database does nothing. The USE method (Utilization, Saturation, Errors — for every resource) is the systematic way to find the saturated resource. → Observability

Step 2: localize it with the observability tools

🚨 This is where the three pillars pay off:

Metrics — which service/endpoint is slow? (dashboards, golden signals)
Distributed tracing — 🚨 where in the request path does the time go? A trace waterfall shows you instantly whether it’s the database, a downstream service, or your own code. This is the fastest way to localize latency in a distributed system.
Logs — the exact detail of what’s slow.
Profiling — 🚨 within a process, which functions consume the CPU/time? (CPU profilers, flame graphs). Essential for CPU-bound problems.

The localization workflow: metric says checkout is slow → trace shows 80% of the time is in the database call → profile the query / check the query plan → find the missing index. Narrow from system to service to operation to line.

Step 3: read the distribution, not the average

🚨 A slow average and a slow tail are different problems with different causes (percentiles):

p50 (median) is high → the common path is slow. The whole operation is inefficient — a missing index, an N+1, slow code that runs every time.
p50 is fine but p99 is terrible → 🚨 a subset of requests hit a different path. Causes: cache misses (fast on hit, slow on miss), GC pauses (periodic spikes), lock contention (some requests wait), a hot shard, a slow downstream on some requests, one unhealthy instance.

🎙️ “Is p50 high, or is p50 fine and p99 terrible? Flat median with a bad tail points to queueing, GC, lock contention, or a hot shard — not to the code path generally being slow. Different diagnosis, different fix.”

This distinction guides the whole investigation and is a strong thing to say.

The usual suspects (in rough order of frequency)

🚨 When you localize a bottleneck, it’s usually one of these — worth knowing the common causes:

1. Missing index / bad query. 🚨 The #1 cause. A query doing a full table scan, or a plan that changed with data growth. Check the query plan. Often a 1000× fix from one line.

2. N+1 queries. A loop making one query per item — 1 + N round trips instead of 1. Batch it.

3. Missing cache / low hit rate. Recomputing or refetching the same thing. → Caching

4. Chatty / sequential calls. Services or queries called sequentially that could be batched or parallelized. sum() when it could be max().

5. Lock contention. Threads serializing on a shared resource — invisible in CPU graphs, shows as a bad tail. → Concurrency, Transactions

6. Connection pool exhaustion. Requests waiting for a database connection. → Connection Pooling

7. GC pauses. Periodic p99 spikes with no traffic change (JVM/Go/Node).

8. Serialization. JSON encoding is often the biggest CPU cost in a “simple” service. → Serialization

9. The working set doesn’t fit in RAM. The database fell off the page-cache cliff — reads went from ~100ns to ~100µs. A “sudden” slowdown at a data-size threshold.

The scaling ladder for reads vs writes

Once you know the bottleneck, the fix depends on whether it’s reads or writes — the next two chapters:

Read-bound? → cache, read replicas, denormalize, precompute. → Scaling Reads
Write-bound? → batch, async, shard, LSM storage. → Scaling Writes

🚨 Compute the read/write ratio first — it determines the whole strategy. Most systems are read-heavy, so caching and replicas do the most work.

Load testing: find the bottleneck before production does

🚨 You can find bottlenecks proactively, not just during incidents:

Load test to find where the system falls over and what saturates first.
Profile under load — the bottleneck at 10× traffic isn’t the same as at 1×.
Tools: k6, Gatling, Locust; plus continuous profiling (Pyroscope, Datadog) in production. → Capacity Planning

The optimization discipline

Once you’ve found the bottleneck:

Measure the baseline — you need a number to improve on.
Change one thing.
Measure again — did it actually help? By how much?
Repeat on the new bottleneck (fixing one often reveals the next).

🚨 Don’t optimize blind, and don’t over-optimize. Stop when it’s fast enough (meets the SLO) — further optimization has diminishing returns and costs engineer-time. And 🚨 the cheapest optimization is often “do less work” — add an index, cache, batch, delete data, or question whether the operation needs to be real-time at all — before scaling hardware. → Scalability

⚖️ Trade-offs

Approach	Gain	Cost
Measure first	Fix the real problem	Time to instrument/profile
Do-less-work fixes (index, cache, batch)	Often huge, cheap wins	May not exist for the bottleneck
Optimizing the dominant cost	Meaningful improvement	Finding it takes work
Optimizing a minor cost	—	Wasted effort (Amdahl)
Stop at “fast enough”	Time for other work	—
Over-optimizing	—	Diminishing returns; complexity

In the real world

“The bottleneck is never where you think” is a near-universal experience — engineers confidently optimize the wrong thing, and profiling reveals the real cost is somewhere unexpected (serialization, a lock, a single slow query). It’s why “measure first” is drilled so hard.
The missing-index fix is the most common production performance resolution — before scaling hardware or adding caches, the answer is frequently one index on a query doing a full scan, found by checking the query plan.
Flame graphs (Brendan Gregg’s technique) revolutionized CPU profiling by making the dominant cost visually obvious — the widest bar is where the time goes. Reading a flame graph is a genuinely useful skill.

🚨 Interview traps

Optimizing without measuring — you’ll fix the wrong thing.
Not identifying the bound (CPU/IO/memory/network) — the fix must match the bound.
Adding servers to an I/O- or database-bound problem — does nothing.
Not distinguishing p50 from p99 problems — different causes.
Ignoring the do-less-work fixes (index, cache, batch) in favour of scaling.
Optimizing a minor cost (Amdahl) instead of the dominant one.
Over-optimizing past “fast enough.”

🎙️ Soundbites

“First I’d measure — intuition about bottlenecks is usually wrong, and by Amdahl’s Law, optimizing anything but the dominant cost is wasted. Then I’d identify the bound: CPU, I/O, memory, or network, because the fix has to match it — adding CPU to an I/O-bound service does nothing.”
“Distributed tracing localizes it fastest — the waterfall shows immediately whether the time is in the database, a downstream service, or our own code. Then profile or check the query plan for the detail.”
“Is p50 high or is p50 fine and p99 terrible? A flat median with a bad tail points to queueing, GC, lock contention, or a hot shard — not to the code being generally slow. Different diagnosis.”
“The cheapest fix is usually to do less work — an index, a cache, batching, deleting data — before scaling hardware. A missing index is often a 1000× win from one line.”
“I’d stop at fast enough — meeting the SLO. Further optimization has diminishing returns and costs engineer-time that could build features.”

🛠️ Try it

1. Find a bottleneck end to end. Take a slow endpoint. Measure (baseline), trace it (where’s the time?), identify the bound (CPU? I/O?), and drill to the cause (query plan? profile?). Practice the narrow-from-system-to-line workflow — it’s the core skill.

2. Prove your intuition is wrong. Before profiling a program, guess where the time goes. Then profile it. Note how often the guess is wrong — that’s why “measure first” exists.

3. Distinguish a p50 vs p99 problem. Build one endpoint that’s uniformly slow (missing index) and one that’s fast-with-a-slow-tail (cache miss path). Look at the percentile distributions. See how the shape tells you the cause.

4. Read a flame graph. Profile a CPU-bound program and generate a flame graph. The widest bar is your bottleneck — find it, optimize it, and regenerate to see it shrink and reveal the next one.

Check yourself

1. Why must you measure before optimizing, and how does Amdahl's Law reinforce this?

Because engineers' intuitions about what's slow are wrong most of the time — the bottleneck is routinely somewhere unexpected (serialization, a lock, one slow query, GC), and optimizing based on a guess means you fix something that wasn't the problem, wasting effort and often adding complexity and bugs while the real bottleneck persists. Amdahl's Law makes this rigorous: the maximum improvement from optimizing a component is bounded by that component's fraction of the total time. If a function is 5% of the total, making it infinitely fast improves the whole by at most 5%; if you spend a week making a rarely-used path 100× faster, you've achieved essentially nothing on the overall latency. So you must first measure to find the part that *dominates* the time, and optimize that — the effort only pays off in proportion to how much of the total the target represents. Measure first, find the dominant cost, optimize it, then re-measure to find the new dominant cost.

2. What are the four resource bounds, and why does identifying the bound matter?

Every performance problem is bound on one of four resources. **CPU-bound**: CPU near 100%, gets slower with concurrency even when the database is idle — the compute (serialization, encryption, GC, hot loops) is the limit. **I/O-bound**: low CPU (say 20%) but can't handle more throughput — it's waiting on disk, network, the database, or downstream calls. **Memory-bound**: swapping (which causes a latency cliff) or OOM kills — the working set exceeds RAM, or there's a leak. **Network-bound**: bandwidth is saturated (e.g. hitting a 1 Gbps NIC cap) — payload sizes and data transfer dominate. Identifying the bound matters because the fix must match it, and mismatching is the most common performance mistake: adding CPU to an I/O-bound service does nothing (it's already idle, waiting); adding app servers to a service bottlenecked on a single database does nothing (the database is the constraint); adding RAM to a CPU-bound service does nothing. You can only fix what's actually saturated, so the first diagnostic question is always "what is this bound on?" — answered systematically by checking utilization, saturation, and errors for each resource.

3. What's the difference between a high-p50 problem and a high-p99-only problem?

They're different problems with different causes. A **high p50** (median) means the *common path* is slow — the typical request is inefficient, so the whole operation is affected. Causes: a missing index or a query doing a full scan (every request pays it), an N+1 query pattern, or generally slow code that runs on every request. The fix improves the main path. A **fine p50 but terrible p99** means most requests are fast but a *subset* hits a qualitatively different, slower path. Causes: cache misses (fast on a hit, slow on the miss that goes to the database), GC pauses (periodic stop-the-world spikes affecting whatever requests are in flight), lock contention (some requests wait on a contended resource), a hot shard or hot key (requests to it are slow), a slow downstream dependency on some requests, or one unhealthy instance in the fleet. The distinction guides the whole investigation: a high median says "the code path is slow, profile it"; a bad tail with a good median says "look for what's different about the slow subset — queueing, GC, contention, a hot spot," and crucially says it is *not* that the code is generally slow. Reading the distribution shape before diving in tells you which kind of problem you have.

4. How does distributed tracing accelerate finding a bottleneck?

By localizing *where in the request path* the time goes, instantly and visually, in a distributed system where a single request touches many services. Without tracing, "checkout is slow" means grepping logs across every service the request touched and manually correlating them by timestamp to reconstruct the sequence and find the slow step — hours of work, error-prone because clocks disagree. A trace renders the request as a waterfall of spans (one per service/operation, each with its duration), so you *see* at a glance that, say, 80% of a 2-second request was spent in the payment service, and within that, waiting on a third-party API. That collapses the "which service is slow?" question from hours of log archaeology to reading a picture. It's the fastest way to narrow from "the system is slow" to "this specific operation in this specific service is the problem," after which you profile that service or check that query's plan for the detailed cause. In the measure → localize → drill-down workflow, tracing is the localize step, and it's transformative for distributed systems.

5. Why is "do less work" often the best optimization, and when do you stop optimizing?

Because eliminating unnecessary work is usually cheaper, simpler, and higher-impact than scaling hardware to do the unnecessary work faster. The canonical example: a query doing a full table scan fixed with one index goes from seconds to milliseconds — a 1000× improvement from one line, versus throwing servers at it. Other do-less-work fixes: caching (serve from memory instead of recomputing or refetching), batching (one query instead of N — fixing an N+1), deleting or archiving data (shrink the working set so it fits in RAM), compressing (less to transfer), precomputing (move work from frequent read-time to rare write-time), and questioning whether the operation needs to be real-time at all. These attack the *amount* of work rather than the speed of doing it, and they're frequently available and dramatic. You stop optimizing when the system is *fast enough* — meaning it meets its SLO / performance requirement. Beyond that point, further optimization has sharply diminishing returns and costs engineer-time that could build features or fix real problems; over-optimizing (like premature optimization) is itself a waste. So the discipline is: measure, find the dominant cost, prefer do-less-work fixes, re-measure to confirm the win, and stop once you've met the requirement rather than chasing marginal gains.