system-design

How a Computer Actually Runs Your Server

Every scaling technique in this repo exists to work around one of four physical limits: CPU, memory, disk, and network. Meet them.

Prerequisites: What Is System Design? Time to read: ~20 minutes

The problem

Someone tells you “the server is slow.” That sentence is useless. A server can be slow for reasons that have nothing to do with each other:

The CPU is pegged at 100% doing JSON parsing.
The CPU is at 4% because every thread is waiting on the database.
RAM filled up and the OS is swapping to disk, making everything 1000× slower.
The disk is doing 300 random reads per second and physically cannot do more.
The network card is saturated at 1 Gbps.
There are 10,000 open connections and the process has run out of file descriptors.

Six completely different problems, six completely different fixes. Adding servers helps with two of them and makes one of them worse. You cannot design systems without knowing which is which.

So: what is a server actually made of?

🧠 Mental model: the workshop

A computer is a craftsman in a workshop.

Part	Workshop	Speed
CPU registers	The tool in their hand	Instant
L1/L2/L3 cache	The workbench in front of them	~1–40 ns
RAM	Shelves along the wall — a few steps away	~100 ns
SSD	The storeroom down the hall	~100 µs (1,000× slower than RAM)
HDD	A warehouse across town	~10 ms (100,000× slower than RAM)
Network	Ordering from a supplier in another country	1 ms–200 ms

The craftsman is extremely fast — billions of operations per second — but only when the material is already on the workbench. Every trip to the storeroom stalls them completely.

This one asymmetry explains almost everything. Caching, indexing, batching, CDNs, connection pooling, columnar storage, denormalization — all of them are the same idea: stop walking to the warehouse.

The four resources

1. CPU

The processor executes instructions. Modern server CPUs run at roughly 2–3 GHz, meaning ~3 billion cycles per second per core, with 8–128 cores.

What “CPU-bound” looks like: CPU usage near 100%, and requests get slower as concurrency increases even though the database is idle.

What actually burns CPU in a web service (in rough order of how often it’s the culprit):

Serialization — JSON encode/decode. It is astonishing how often this is the #1 cost in a service that “just returns data.”
Encryption — TLS handshakes especially. (Session resumption exists for this reason.)
Compression — gzip/brotli on responses.
Application logic — loops over large collections, sorting, template rendering.
Garbage collection — in JVM/Go/Node, GC pauses show up as latency spikes, not average slowness.

📐 Numbers: a typical Python/Node service handles ~500–2,000 requests/second/core for simple work. A Go or Java service, 5,000–20,000. A tuned Nginx serving static files, 50,000+. These numbers vary by an order of magnitude — the point is the ratio, not the absolute.

Cores and parallelism. Eight cores can do eight things simultaneously. If your process is single-threaded (classic Node.js, Python with the GIL), you use one core and the other seven idle. That’s why you run multiple processes — cluster in Node, Gunicorn workers in Python, one per core. → Concurrency Basics

Context switching. The OS juggles more threads than cores by switching between them. Each switch costs ~1–5 µs and trashes the CPU cache. 100 threads on 8 cores: fine. 10,000 threads: the machine spends its life switching instead of working. This is precisely why async/event-loop servers (Nginx, Node, Netty) exist — they handle 100,000 connections with a handful of threads.

2. Memory (RAM)

Fast, volatile, limited. Server RAM ranges from 8 GB to 24 TB, but 16–128 GB is typical.

What it’s used for: your program’s heap, the OS page cache (a copy of recently read disk data — this is why the second read of a file is 1000× faster), connection buffers, and your in-process caches.

🚨 The page cache is the most under-appreciated thing in system design. When people say “Postgres is fast,” a large part of what they mean is “the working set fits in RAM, so most reads never touch the disk.” A database with 64 GB of RAM and 50 GB of hot data behaves completely differently from the same database with 8 GB of RAM. This is why the first question about any database performance problem is “does the working set fit in memory?”

When you run out:

The OS starts swapping — moving memory pages to disk. A memory access silently becomes a disk access. Latency goes from 100 ns to 100 µs. Systems don’t get 20% slower; they fall off a cliff.
Or the OOM killer shoots your process dead. In Kubernetes, the pod restarts, load shifts to the remaining pods, they OOM too, and you get a cascading outage from one memory leak.

⚖️ RAM is roughly 100× more expensive per byte than SSD, and 1000× more than object storage. “Just keep it all in memory” is a real strategy (Redis is exactly this) but you pay for it.

3. Disk

Persistent, slow, cheap, and where correctness lives — data isn’t durable until it’s on disk (and usually, on several disks).

HDD vs SSD — this distinction matters more than beginners expect:

	HDD	SSD (NVMe)
Random read	~10 ms	~100 µs
Sequential read	~200 MB/s	2–7 GB/s
Random IOPS	~100–200	100,000–1,000,000
Cost/TB	Cheapest	~4–8× HDD

Sequential vs random is the big one. On an HDD, sequential reads are ~100× faster than random ones, because the physical head doesn’t have to move. Even on SSDs, sequential is several times faster.

This single fact shaped modern data systems:

Append-only logs (Kafka, write-ahead logs) are fast because appending is purely sequential. Kafka gets enormous throughput on cheap disks for exactly this reason.
LSM-trees (Cassandra, RocksDB) buffer writes in memory and flush them sequentially, instead of updating rows in place. → Storage Engines
B-trees are designed so that a lookup touches ~3–4 disk pages instead of scanning. → Indexing

IOPS is a hard limit. A disk that does 20,000 IOPS does 20,000 IOPS. If each request needs 5 random reads, that disk supports 4,000 requests/second. No amount of application optimization changes this — you add caching, add replicas, or buy faster disks. In the cloud, IOPS is a number you literally purchase (AWS gp3, io2), and running out of provisioned IOPS is a very common, very confusing outage.

Durability isn’t free. write() puts data in the OS page cache and returns — fast, but the data is gone if the machine loses power. fsync() forces it to physical storage: 0.1–10 ms. Every database chooses where on this spectrum to sit, and that choice is exactly the durability/latency trade-off.

4. Network

The one you’ll think about most, because in a distributed system nearly all the time is spent here.

Bandwidth — how much data per second. A typical cloud VM has 1–25 Gbps.

📐 At 1 Gbps you can push ~125 MB/s. Serving a 2 MB image means at most ~62 images/second from that machine. That number surprises people, and it’s why you serve images from a CDN, not from your app servers.

Latency — how long one round trip takes. And this is dominated by physics, not by your code:

Route	Round-trip
Same machine (localhost)	~0.05 ms
Same datacenter	~0.5 ms
Same region, different AZ	~1–2 ms
Karachi → Dubai	~30 ms
Karachi → Frankfurt	~110 ms
Karachi → Virginia (US East)	~230 ms
Anywhere → anywhere, worst case	~300 ms

Light travels 300,000 km/s in vacuum, ~200,000 km/s in fibre, and cables don’t run in straight lines. You cannot optimize your way past the speed of light. If your user is in Karachi and your server is in Virginia, every round trip costs ~230 ms, forever. The only fixes are: make fewer round trips, or move the server closer (→ CDN, Multi-Region).

🚨 This is why “chatty” APIs are a design smell. An endpoint that makes 10 sequential internal calls of 1 ms each costs 10 ms. The same 10 calls across regions cost 2.3 seconds.

Connections cost resources. Each TCP connection uses a file descriptor and kernel buffers (~10 KB of memory). Default file descriptor limits (often 1024) will stop your server long before RAM does. Establishing a connection costs a round trip; TLS costs one or two more. This is why connection pooling and keep-alive exist. → Connection Pooling

Putting it together: where does a request’s time actually go?

A “simple” API request — GET /users/42 — on a well-behaved service:

Client → Load Balancer          0.5 ms   network
LB → App server                 0.5 ms   network
App: parse request              0.1 ms   CPU
App → Cache (Redis) lookup      0.5 ms   network + ~0.1 ms Redis work
   ↳ MISS
App → Database query            1.0 ms   network
   Database: index lookup       0.2 ms   CPU + page cache
   Database: read row           0.1 ms   page cache hit (or 0.1 ms SSD)
   Database: serialize result   0.3 ms   CPU
App: build JSON response        0.5 ms   CPU  ← often the biggest single item!
App → Client                    1.0 ms   network
                               ────────
                               ~4.8 ms

Two lessons that show up constantly:

Network round trips dominate. Five hops at ~1 ms each is 5 ms before any real work happens. Cutting round trips beats micro-optimizing code almost every time.
Serialization is real CPU work. In many services, JSON encoding is the single largest CPU consumer. This is a big part of why internal services use protobuf/gRPC.

Now add a user in Karachi hitting a server in Virginia: add 230 ms to the first and last lines. Your 4.8 ms service is a 235 ms experience. Your code was never the problem.

⚖️ Trade-offs this chapter sets up

Technique	Buys you	Costs you
Cache in RAM	1000× faster reads	Staleness, memory cost, invalidation complexity
Sequential writes (append log)	100× write throughput	Reads need indexes/compaction
Batching	Fewer round trips, better throughput	Higher latency for the individual item
More threads	More concurrency	Context-switch overhead, memory per thread
Async I/O	100k connections on few threads	Harder code, one blocking call ruins everything
`fsync` on every write	Real durability	0.1–10 ms per write

In the real world

Kafka achieves millions of messages/second on ordinary disks by only ever writing sequentially and letting the OS page cache serve reads. No exotic hardware — just working with the physics instead of against it.
Redis is single-threaded for its core command loop. That sounds like a limitation until you realize it’s memory-bound, not CPU-bound: no locks, no context switches, ~100,000 ops/second on one core.
Netflix doesn’t stream from its own datacenters. It puts appliances full of SSDs (Open Connect) inside ISPs, because bandwidth and latency are physical, and the only real fix is proximity.

🚨 Interview traps

Saying “the server is slow” without localizing it. Always ask/state: CPU-bound, memory-bound, I/O-bound, or network-bound? Different answers → different designs.
Ignoring the speed of light. If the interviewer says “users are global,” and you put one database in one region and claim 50 ms latency, you’ve made a claim physics disagrees with.
Assuming disk = slow, memory = fast, end of story. The real distinction is random vs sequential and does it fit in the page cache. Sequential SSD reads are within an order of magnitude of RAM bandwidth.
Forgetting that adding threads has a limit. “We’ll just use more threads” fails at a few thousand. Beyond that you need async I/O or more machines.

🎙️ Soundbites

“Before optimizing, I’d want to know whether we’re CPU-bound or I/O-bound — the fixes are completely different.”
“The p99 here will be dominated by network round trips, not compute, so I’d focus on cutting the number of hops rather than the work per hop.”
“That design does five sequential internal calls. In-region that’s 5 ms, but cross-region it’s over a second, so I’d either parallelize them or colocate the services.”
“The key question for this database is whether the working set fits in RAM. If it does, reads are page-cache hits at ~100 µs. If not, we’re at SSD latency and IOPS becomes our ceiling.”

🛠️ Try it

Measure your own machine. Write a small script that times:

Reading 1 million integers from an in-memory array.
Reading 1,000 random 4 KB blocks from a large file (make the file bigger than your RAM, or the page cache will lie to you — use dd to create a 20 GB file).
Reading 4 MB sequentially from that same file.
100 HTTP requests to https://example.com.

Then compare your numbers to Latency Numbers. Two things will surprise you: how enormous the random-vs-sequential gap is, and how completely the network dominates everything else.

Bonus: run htop while your machine does something heavy. Watch which resource actually saturates.

Check yourself

1. Your service shows 100% CPU. Someone suggests adding a bigger database. Why is that probably wrong?

100% CPU on the *app server* means the app is doing the work — parsing, serializing, computing, or GC. A bigger database doesn't touch that. If the app were waiting on the database, you'd see *low* CPU and high latency. Match the fix to the saturated resource.

2. Why is appending to a log so much faster than updating rows in place?

Appending is sequential — the write goes to the end of a file with no seeking, and can be batched into large contiguous I/Os. In-place updates are random writes: find the right page, read it, modify it, write it back, and update every index that referenced it. On an HDD that's ~100× worse; on an SSD it also causes write amplification and wear. This is why write-ahead logs, Kafka, and LSM-trees all look the way they do.

3. A single server has a 1 Gbps NIC. Your API returns 500 KB responses. What's your ceiling in requests/second, ignoring everything else?

1 Gbps = 125 MB/s. 125 MB / 0.5 MB = **250 requests/second**. Notice you can be nowhere near CPU or memory limits and still be capped. Fixes: shrink the payload (do clients need all 500 KB?), compress it, paginate it, or serve it from a CDN.

4. Why does a user in Karachi see 250 ms latency from a US server even though the server responds in 5 ms?

Speed of light plus routing. ~230 ms of round-trip network time is physics and cannot be optimized away in software. Fixes are structural: CDN/edge for static and cacheable content, a regional replica for reads, or full multi-region deployment. Also: reduce round trips, since each one costs the full 230 ms.

5. Your service works fine at 1,000 concurrent connections and collapses at 5,000, with CPU at only 30%. Name two likely causes.

(a) File descriptor limit — the default `ulimit -n` of 1024 (or a low container limit) means new connections are simply refused. (b) Thread-per-connection exhaustion — 5,000 OS threads means huge context-switching overhead and gigabytes of stack memory, so the machine thrashes while CPU utilization looks unimpressive. Also plausible: connection pool exhaustion to a downstream database, or memory pressure from per-connection buffers.