Why one server can handle 100,000 simultaneous connections with four threads, and why adding threads eventually makes things slower.
Prerequisites: Computer Fundamentals Time to read: ~18 minutes
Your server handles one request at a time. A request takes 100 ms, of which 95 ms is waiting for the database.
Request 1: [work 5ms][──────── waiting 95ms ────────]
Request 2: [work 5ms][──── waiting ────]
Throughput: 10 requests/second. Meanwhile your CPU is 95% idle, doing nothing but waiting.
The fix is obvious — handle other requests during the wait. How you do that is one of the most consequential decisions in server design, and the options have very different scaling properties.
People use these interchangeably. They’re different.
Concurrency = dealing with many things at once. A structure where multiple tasks are in progress, interleaved.
Parallelism = doing many things at once. Requires multiple CPU cores.
🧠 Rob Pike’s framing: concurrency is one barista taking orders, starting a drink, taking the next order while the espresso brews, and juggling five drinks in progress. Parallelism is hiring four baristas.
You can have concurrency without parallelism (Node.js on one core juggles thousands of connections). You can have parallelism without concurrency (a numeric computation split across 8 cores, no interleaving). Most real servers want both.
Which one you need depends on your bottleneck:
| Workload | Bottleneck | You need |
|---|---|---|
| I/O-bound — waiting on database, network, disk | Waiting | Concurrency. More cores don’t help; you’re idle |
| CPU-bound — encoding video, ML inference, compression | Computation | Parallelism. More cores directly help |
🚨 Most web services are overwhelmingly I/O-bound. This is why Node.js — single-threaded — can outperform a naively threaded server, and why “we’ll add more CPU” often changes nothing.
The traditional approach. Each request gets an OS thread. When it blocks on I/O, the OS parks it and runs another.
Thread 1: [work][──── blocked on DB ────][work]
Thread 2: [work][──── blocked ────][work]
Thread 3: [work][──── blocked ────][work]
Why it’s nice: the code is completely straightforward. Sequential, readable, easy to debug, and stack traces make sense.
def handle_request(req):
user = db.get_user(req.user_id) # blocks — that's fine
orders = db.get_orders(user.id) # blocks
return render(user, orders)
Where it breaks:
📐 This is the C10K problem — the historical challenge of serving 10,000 concurrent connections on one machine. Thread-per-connection couldn’t do it. That constraint produced Nginx, Node.js, and every event-loop server since.
Used by: classic Java servlet containers (pre-virtual-threads), Python WSGI (Gunicorn sync workers), Ruby, PHP-FPM, Go (but see below — goroutines change the math entirely).
One thread. A loop. Non-blocking I/O.
Event loop:
├─ start DB query for req 1 → register callback, move on
├─ start DB query for req 2 → register callback, move on
├─ req 1's data arrived → run its callback
├─ start DB query for req 3 → register callback, move on
└─ req 2's data arrived → run its callback
The thread never blocks. It asks the OS “which of these thousands of sockets is ready?”
(epoll/kqueue/io_uring) and processes whichever are.
async function handleRequest(req) {
const user = await db.getUser(req.userId); // yields to the loop
const orders = await db.getOrders(user.id); // yields to the loop
return render(user, orders);
}
Note the code looks sequential — async/await gives you readability without threads. That’s why
it won.
What you gain: 100,000+ concurrent connections on one thread. Minimal memory per connection (kilobytes, not megabytes). No context-switch overhead.
What it costs:
🚨 One blocking call ruins everything. In a thread model, a slow operation blocks one thread. In an event loop, it blocks every request on that loop:
// Catastrophic in an event loop
const hash = bcrypt.hashSync(password); // 100 ms of CPU, synchronous
// Every other request — all 5,000 of them — waits 100 ms.
The rules that follow: never do synchronous I/O, never do heavy CPU work on the loop (offload to a worker thread or a separate service), and audit every library you use for hidden blocking calls.
Also: one event loop uses one core. Run one process per core (Node’s cluster, or several
container replicas) to use the whole machine.
Used by: Node.js, Nginx, Redis, Python asyncio, Netty, Rust Tokio.
The best of both — and where the industry is heading.
The runtime provides threads that are not OS threads: they’re cheap (a few KB), created in microseconds, and multiplexed by the runtime onto a small pool of OS threads. When one blocks, the runtime swaps in another with no kernel involvement.
// Go — looks like blocking code, scales like an event loop
func handleRequest(w http.ResponseWriter, r *http.Request) {
user := db.GetUser(r.UserID) // "blocks" the goroutine, not the OS thread
orders := db.GetOrders(user.ID)
render(w, user, orders)
}
// Go spawns a goroutine per request automatically. Millions are fine.
Why this is the sweet spot: you write simple sequential code, and get event-loop scaling. No callback complexity, no “don’t block the loop” discipline, and it uses all cores automatically.
Costs: the runtime must control the I/O layer (calling into blocking C libraries can still pin an OS thread), and stack traces / debugging can be less obvious.
Used by: Go (goroutines), Erlang/Elixir (processes), Java 21+ (virtual threads — a very big deal for the JVM ecosystem), Kotlin coroutines, Rust async tasks.
| Thread per request | Event loop | Lightweight threads | |
|---|---|---|---|
| Max concurrency | ~1,000s | ~100,000s | ~1,000,000s |
| Memory per unit | ~1 MB | ~KB | ~2–8 KB |
| Code style | Simple, sequential | async/await or callbacks | Simple, sequential |
| Uses all cores | ✅ | ❌ (one process per core) | ✅ |
| A blocking call | Hurts one request | Hurts everything | Hurts one goroutine |
| Debugging | Easy | Harder | Medium |
| Examples | Classic Java, Python WSGI, PHP | Node, Nginx, Redis | Go, Elixir, Java 21+ |
Concurrency’s real difficulty isn’t scheduling. It’s that two things touching the same data at the same time produces bugs that are non-deterministic, unreproducible, and appear only under load.
Race condition — the canonical example:
Thread A: read balance (100)
Thread B: read balance (100)
Thread A: write 100 − 30 = 70
Thread B: write 100 − 50 = 50 ← A's withdrawal vanished. Balance should be 20.
This is exactly the same problem as a distributed lost update — which is why understanding it locally makes transactions and isolation levels click immediately.
The tools:
| Tool | What it does | Watch out for |
|---|---|---|
| Mutex / lock | Only one thread in the critical section | Contention (serializes your parallelism); deadlock |
| Read-write lock | Many readers or one writer | Writer starvation |
| Atomic operations | Lock-free single-variable updates (CAS) | Only works for simple operations |
| Immutability | Nothing to race on | Allocation cost |
| Message passing | Don’t share memory; send copies | Erlang/Go channels; requires structural change |
| Single-threaded | Sidestep it entirely | Redis’s approach — no locks, one core |
🧠 The best strategy is avoidance. Redis is single-threaded for its command loop and gets ~100,000 ops/sec precisely because it has no locks, no contention, and no context switches. Go’s motto — “don’t communicate by sharing memory; share memory by communicating” — is the same idea.
Deadlock — two threads each holding what the other needs, both waiting forever. Prevention: always acquire locks in a globally consistent order, and use timeouts on lock acquisition. This reappears verbatim in distributed locking, where it’s even harder because you can’t inspect the other party.
Amdahl again: every lock is a serial section. → Scalability
Sizing thread and connection pools. Use Little’s Law (Performance Metrics):
concurrency = throughput × latency
1,000 QPS × 50 ms = 50 concurrent requests → ~50–100 pool slots
🚨 A classic production failure: 50 app servers × 100 connections each = 5,000 connections to a database that performs best at ~200. The database thrashes. Fix: smaller per-instance pools, or a connection pooler like PgBouncer in front. → Connection Pooling
Parallelize independent calls. If a request needs three unrelated services, calling them sequentially costs the sum; in parallel it costs the max.
Sequential: 30 + 40 + 50 = 120 ms
Parallel: max(30, 40, 50) = 50 ms
This is often the single largest latency win available in a service, and interviewers look for it.
Bound your concurrency. Unbounded parallelism is a way to DDoS your own dependencies. Use a semaphore or worker pool with a fixed size. “Spawn a goroutine per item” over a million items will melt something downstream.
Separate CPU-bound work. Video transcoding, image processing, and PDF generation don’t belong in your request path. Push them to a queue and a worker fleet you can scale independently.
| Choice | Gain | Cost |
|---|---|---|
| Thread-per-request | Simplest code | Memory and context-switch ceiling at ~1,000s |
| Event loop | Enormous connection counts, tiny memory | One blocking call stalls everything; one core per loop |
| Lightweight threads | Simple code + high scale | Runtime must own I/O; some debugging friction |
| Locks | Correctness on shared state | Contention, deadlock, serialization |
| Immutability / message passing | No races by construction | Copying cost, structural change |
| Larger pools | More concurrency | Downstream overload; more memory |
1. Feel the difference. Write the same “fetch from a slow API and return” endpoint twice: once blocking (Flask/sync), once async (FastAPI/Node). Load-test both with 500 concurrent connections. The gap will be dramatic, and it’ll be memory as much as throughput.
2. Block the event loop. In a Node server, add crypto.pbkdf2Sync(...) with high iterations to
one endpoint. Hit a different endpoint while it runs. Watch unrelated requests stall. This is the
single most important thing to internalize about async runtimes.
3. Create a race condition. Two threads incrementing a shared counter a million times. Run it — the result won’t be 2,000,000. Now add a lock. Then measure how much slower it got. That slowdown is the cost of coordination, and it’s the same cost that shows up in distributed systems.