system-design

Latency Numbers Every Engineer Should Know

Nine numbers, memorized, that let you reject bad designs in seconds without writing any code.

Prerequisites: Computer Fundamentals Time to read: ~12 minutes (then spend a week internalizing them)

Why memorize numbers?

Because most bad designs are arithmetically impossible, and you can only see that if you know the constants.

Someone proposes: “For each search result, we’ll fetch details from the database, then call the recommendation service, then enrich with user preferences.” Sounds reasonable. Now count: 20 results × 3 sequential calls × 1 ms = 60 ms of pure network, per search, before any work happens. Add a cross-region hop and it’s 15 seconds.

You don’t need a prototype to know that’s wrong. You need nine numbers.

The table

These are the canonical figures (Jeff Dean’s list, updated for modern hardware). Orders of magnitude are what matter — not precision.

Operation	Time	Relative to L1
L1 cache reference	1 ns	1×
Branch mispredict	3 ns	3×
L2 cache reference	4 ns	4×
Mutex lock/unlock	17 ns	17×
Main memory (RAM) reference	100 ns	100×
Compress 1 KB with Snappy	2 µs	2,000×
Read 1 MB sequentially from RAM	~10 µs	10,000×
Send 1 KB over 1 Gbps network	~10 µs	10,000×
SSD random read	~100 µs	100,000×
Read 1 MB sequentially from SSD	~200 µs	200,000×
Round trip within same datacenter	~500 µs (0.5 ms)	500,000×
HDD disk seek	~10 ms	10,000,000×
Read 1 MB sequentially from HDD	~20 ms	20,000,000×
Round trip CA → Netherlands → CA	~150 ms	150,000,000×

Just remember these nine

If you memorize nothing else:

L1 cache          ....  1 ns
RAM               ....  100 ns          (100× slower than L1)
SSD random read   ....  100 µs          (1,000× slower than RAM)
Datacenter RTT    ....  0.5 ms          (5× slower than SSD)
HDD seek          ....  10 ms           (100× slower than SSD)
Cross-continent   ....  150 ms          (300× slower than datacenter)

Plus three throughput figures:

1 Gbps network    ....  ~125 MB/s
SSD sequential    ....  ~1–3 GB/s
RAM bandwidth     ....  ~10–50 GB/s

🧠 Mental model: scale it to human time

Nanoseconds are meaningless to human intuition. Multiply everything by 1 billion, so 1 ns = 1 second:

Operation	Real	If 1 ns were 1 second
L1 cache	1 ns	1 second
RAM	100 ns	2 minutes
SSD random read	100 µs	1.5 days
Datacenter round trip	0.5 ms	6 days
HDD seek	10 ms	4 months
Cross-continent round trip	150 ms	5 years

Now the design implications become obvious:

Reading from RAM instead of SSD is the difference between 2 minutes and a day and a half. That’s what a cache buys you.
One cross-continent round trip is 5 years. Making 10 of them sequentially is a career.
A CPU waiting on a disk read is a person waiting four months for one piece of paper. This is why async I/O exists — you do not leave the worker standing there.

Once you’ve internalized this table, you stop needing to look things up. You just know that a design with a cross-region call inside a per-item loop is broken.

📐 Using them: worked examples

Example 1 — the N+1 query problem

Render a page listing 50 products, each with its seller’s name.

Naive:  1 query for products         =  0.5 ms
        50 queries for sellers       = 50 × 0.5 ms = 25 ms
        Total                        = 25.5 ms

Batched: 1 query for products        =  0.5 ms
         1 query: WHERE id IN (...)  =  0.5 ms
         Total                       =  1 ms

25× faster, same data. And this is the in-datacenter case. If that seller service is in another region: 50 × 150 ms = 7.5 seconds.

This is the single most common performance bug in the industry, and the numbers tell you why.

Example 2 — is this cache worth it?

You have a query taking 20 ms (a few disk reads plus work). You’re considering Redis.

Without cache:              20 ms
With cache, hit:            0.5 ms (network) + 0.1 ms (Redis) = 0.6 ms
With cache, miss:           0.6 ms + 20 ms = 20.6 ms   (slightly worse!)

At 90% hit rate:  0.9 × 0.6 + 0.1 × 20.6 = 0.54 + 2.06 = 2.6 ms average
At 99% hit rate:  0.99 × 0.6 + 0.01 × 20.6 = 0.59 + 0.21 = 0.8 ms average

7.7× better at 90% hit rate, 25× at 99%. And note the corollary: at a 20% hit rate you’d get ~16.7 ms — barely an improvement, for real added complexity and a new failure mode. Hit rate is the whole argument for a cache, which is why “what’s the expected hit rate?” is the right question to ask before adding one.

Example 3 — can one server serve this?

Serving 2 MB images from a machine with a 1 Gbps NIC:

1 Gbps = 125 MB/s
125 MB/s ÷ 2 MB per image = ~62 images/second

Sixty-two. Your CPU is idle, your RAM is fine, and you are completely saturated. This is why images and video go to a CDN — not for cleverness, but because the arithmetic leaves no choice.

Example 4 — does the working set fit in RAM?

A database with 500 GB of data and 64 GB of RAM. Users access ~10% of the data regularly.

Hot data = 50 GB.  Fits in 64 GB RAM.  →  most reads are page-cache hits ≈ 100 ns–1 µs

Now the product grows and hot data hits 200 GB:

Hot data = 200 GB.  Does NOT fit.  →  most reads hit SSD ≈ 100 µs

A 100–1,000× regression, from a change in data size alone. No code changed. This is why databases “suddenly” fall over at a certain scale, and why the first question about any database performance problem is “does the working set still fit in memory?”

Example 5 — a global user’s real experience

Your API responds in 5 ms. A user in Karachi hits your Virginia servers.

DNS (uncached)         ~50 ms
TCP handshake          ~230 ms  (1 RTT)
TLS 1.3 handshake      ~230 ms  (1 RTT)
Request + response     ~230 ms  (1 RTT) + 5 ms server time
                       ────────
First request:         ~745 ms
Subsequent (warm):     ~235 ms

Your 5 ms service is a 745 ms experience. 99.3% of the time is network. No amount of code optimization touches it — only moving closer (CDN/edge/region) or making fewer round trips does.

Reference: sizes and throughput

Data sizes to have in your head:

Thing	Size
A character (ASCII/UTF-8 basic)	1 byte
An integer / a timestamp	4–8 bytes
UUID	16 bytes (36 as a string)
A tweet	~300 bytes
A typical JSON API response	1–10 KB
A web page (HTML only)	~50–100 KB
A full page load (all assets)	2–5 MB
A photo (compressed)	1–5 MB
A minute of 1080p video	~50 MB
A 2-hour 1080p movie	~4 GB

Common conversions worth memorizing:


Seconds in a day	86,400 (≈ 10⁵)
Seconds in a month	~2.5 million
Seconds in a year	~31.5 million (≈ π × 10⁷)
1 million/day	≈ 12 per second
1 billion/day	≈ 11,600 per second
1 QPS sustained for a year	~31.5 million requests

🚨 86,400 ≈ 10⁵ is the single most useful approximation in estimation interviews. It turns “5 million requests per day” into “50 QPS” instantly, in your head, with no calculator.

What these numbers imply

Every one of these is a direct consequence of the table:

Because…	You should…
RAM is 1,000× faster than SSD	Cache hot data; size your database so the working set fits in memory
Sequential is ~100× faster than random on disk	Prefer append-only logs; that’s why Kafka and LSM-trees exist
A datacenter RTT is 0.5 ms but a cross-continent RTT is 150 ms	Colocate chatty services; replicate data to regions
Round trips dominate compute	Batch requests; avoid N+1; parallelize independent calls
Bandwidth is finite (125 MB/s per Gbps)	Serve large assets from a CDN; compress; paginate
A disk seek is 10 ms	Index everything you query; a full scan of a large table is a design error
Compression is fast (2 µs/KB) relative to network	Almost always compress payloads over the network

That last row is a nice one to know: compressing 1 KB costs ~2 µs of CPU and can save ~700 bytes of transfer. Over a 150 ms link, that trade is absurdly favourable. Compress by default.

🚨 Interview traps

Not knowing the datacenter RTT (~0.5 ms). It’s the number you use most, because every arrow in your architecture diagram costs one.
Treating “in-memory” as free. It’s 100 ns, not zero, and a network call to an in-memory cache is 0.5 ms — dominated by the network, not the memory.
Ignoring bandwidth. Latency questions get asked more, but bandwidth is what actually caps media systems.
Quoting the table without applying it. Reciting “SSD is 100 µs” scores nothing. Saying “that loop does 50 SSD reads, so 5 ms — fine in-region, but if we move that service it’s 7.5 seconds” scores a lot.

🎙️ Soundbites

“That’s an N+1 — 50 round trips at half a millisecond each is 25 milliseconds we can turn into one by batching.”
“A cross-region call is roughly 150 milliseconds, so we can afford one on this path, not ten. I’d replicate the data instead.”
“The whole question for this cache is hit rate. Above 90% it’s a 10× win; below 50% we’re adding a failure mode for very little.”
“At 2 MB per image and a 1 Gbps NIC, one server tops out around 60 requests per second on bandwidth alone — that’s a CDN problem, not an application problem.”

🛠️ Try it

1. Measure your own machine and compare with the table:

import time, os, random

# RAM sequential
data = bytearray(100 * 1024 * 1024)          # 100 MB
t = time.perf_counter(); sum(data[::4096]); print("RAM scan:", time.perf_counter() - t)

# Disk random reads (make the file larger than your RAM first, or the page cache lies)
#   dd if=/dev/urandom of=big.bin bs=1M count=20000
f = os.open("big.bin", os.O_RDONLY)
size = os.path.getsize("big.bin")
t = time.perf_counter()
for _ in range(1000):
    os.pread(f, 4096, random.randrange(0, size - 4096))
print("1000 random reads:", time.perf_counter() - t)

2. Feel the network. Ping servers at different distances and note the RTTs:

ping -c 5 1.1.1.1                 # nearby anycast, single-digit ms
curl -w '%{time_total}\n' -o /dev/null -s https://www.google.com

3. Make flashcards. Nine numbers. Five minutes a day for a week. This is genuinely worth rote-memorizing — you will use it in every design conversation for the rest of your career.

Check yourself

1. Roughly how much slower is a random SSD read than a RAM read? Why does it matter?

~1,000× (100 µs vs 100 ns). It matters because it's the entire justification for caching and for sizing your database's memory to hold the working set. A workload that fits in RAM and one that doesn't differ by three orders of magnitude with zero code changes.

2. A page makes 30 sequential API calls to services in the same datacenter. What's the network floor? What if one service moves to another continent?

30 × 0.5 ms = **15 ms** in-datacenter — noticeable but survivable. Move one call cross-continent and that single call adds ~150 ms, making it ~165 ms. If *all* 30 moved, it's 4.5 seconds. Fixes: parallelize independent calls (30 sequential becomes ~1 RTT), batch them, or keep services colocated.

3. 30 million requests per day. What's the average QPS, and roughly the peak?

30,000,000 / 86,400 ≈ **347 QPS** average. (Or the fast way: 30 M / 10⁵ ≈ 300.) Peak is typically 2–3× average for a consumer product, so **~700–1,000 QPS**. That's comfortably handled by a handful of servers — a useful reality check against the instinct that "30 million" sounds enormous.

4. Why does compressing an API response almost always pay off?

Compressing 1 KB costs ~2 µs of CPU. Transferring the ~700 bytes you save over a link with a 50–150 ms RTT and limited bandwidth costs vastly more. The CPU/network exchange rate is overwhelmingly in compression's favour for anything crossing a real network. (The exception is inside a datacenter for tiny payloads, or data that's already compressed like JPEG/video.)

5. Your database has 2 TB of data and 128 GB of RAM. What single question determines whether it's fast or slow?

"How large is the *working set* — the data actually accessed regularly?" If the hot 5% (100 GB) fits in RAM, most reads are page-cache hits at ~100 ns. If access is spread evenly across all 2 TB, nearly every read hits SSD at ~100 µs and you're limited by IOPS. Total data size barely matters; the ratio of *hot* data to RAM is everything.