The box that turns “a server” into “a fleet.” Also the box that’s usually the first single point of failure people forget to make redundant.
Prerequisites: Networking 101, Scalability Time to read: ~22 minutes
One server handles 5,000 QPS. You need 50,000. So you run ten servers.
Now: how does a request find one? You could give clients ten IP addresses and let them choose — but then what happens when server 4 dies? Every client needs to know. What about when you add server 11? What if server 7 is slow but not dead?
You need something in front that knows the current state of the fleet and routes accordingly. That’s a load balancer, and it does far more than “spread the traffic evenly.”
Distribution is only the first item on this list:
| Function | Why it matters |
|---|---|
| Distribute requests | The obvious one |
| Health checking | Automatically stops sending traffic to dead or sick instances — this is most of your MTTR improvement |
| Zero-downtime deploys | Drain a server, update it, return it, repeat |
| TLS termination | Handle HTTPS once, centrally, instead of per-service |
| A single stable address | Clients see one endpoint; the fleet behind it changes freely |
| Retries and timeouts | Retry a failed request on a different server, invisibly |
| Rate limiting / DDoS absorption | First line of defence, before your app burns CPU |
| Observability | One place that sees every request — traffic, latency, error rates |
🎙️ If asked “why not just use DNS round-robin?”, the answer is that whole second row: DNS has no idea whether a server is alive. → DNS
The most common load-balancer interview question, and it’s just “which layer does it make decisions at?”
Sees IP addresses and ports. Does not read the payload. It picks a backend when the connection is established and forwards packets for the life of that connection.
/api/* one way and /images/* another, because it
can’t see the URL.Examples: AWS NLB, HAProxy in TCP mode, IPVS, Google Cloud’s TCP load balancer.
Terminates the connection, parses HTTP, and makes decisions per request.
Examples: AWS ALB, Nginx, HAProxy in HTTP mode, Envoy, Traefik, Cloudflare.
| Use L4 when | Use L7 when |
|---|---|
| Non-HTTP protocols (databases, gRPC-over-raw-TCP, game servers) | HTTP/HTTPS (i.e. most systems) |
| Extreme throughput, minimal latency | You need path/host routing |
| End-to-end encryption required (passthrough) | You want TLS termination, retries, or per-request balancing |
| Preserving the client IP matters and you can’t use proxy headers | You need observability per endpoint |
🎙️ “L7 for the public API, because we need path-based routing and per-request balancing across keep-alive connections. If we were fronting the database or a raw TCP protocol, L4.”
In practice you often have both: an L4 balancer at the edge for raw throughput and DDoS absorption, feeding an L7 layer that does the intelligent routing. That’s essentially what AWS NLB → ALB, or Cloudflare → your Nginx, looks like.
How does it choose which backend?
| Algorithm | How | Use when |
|---|---|---|
| Round robin | Next server in rotation | Servers are identical and requests are uniform |
| Weighted round robin | Bigger servers get more | Heterogeneous fleet; also for canary rollouts |
| Least connections | Fewest active connections wins | Request durations vary a lot — usually the best default |
| Least response time | Fewest connections + lowest latency | Latency-sensitive; naturally routes around sick servers |
| IP hash | hash(client IP) % N |
Crude session affinity without cookies |
| Consistent hash | Hash onto a ring | Cache locality — same key → same server. → Consistent Hashing |
| Random with two choices | Pick 2 at random, take the less loaded | Nearly as good as least-connections at a fraction of the coordination cost — used a lot at scale |
🚨 Round robin’s failure mode is worth knowing. With uniform requests it’s fine. But if 1% of requests take 10 seconds and the rest take 10 ms, round robin will happily pile slow requests onto a server that’s already stuck with three of them. Least connections naturally avoids this, which is why it’s the better default for real workloads.
“Power of two random choices” deserves a mention because it’s elegant and it appears in real systems (Nginx, Envoy, and academic load-balancing literature): picking the better of two random servers gets you almost all the benefit of full least-connections tracking, without any global state. Good depth signal in an interview.
The mechanism that turns a load balancer into a fault-tolerance device.
Passive health checks: watch real traffic. If a backend returns errors or times out, mark it down. Free, but only detects problems by failing real user requests.
Active health checks: the LB periodically probes each backend.
GET /health every 5s
2 consecutive failures → mark unhealthy, stop sending traffic
3 consecutive successes → mark healthy, resume
🚨 Shallow vs deep health checks — this is the interesting design decision.
# Shallow: is the process alive?
@app.get("/health")
def health():
return {"status": "ok"}
# Deep: can it actually serve?
@app.get("/health/ready")
def ready():
db.execute("SELECT 1") # can we reach the database?
cache.ping() # can we reach the cache?
return {"status": "ok"}
Shallow checks miss the most common real failure: the process is running fine but its database
connection pool is exhausted, so every request fails. It passes /health and keeps receiving
traffic — a gray failure.
But deep checks have a serious failure mode of their own: if the shared database has a brief hiccup, every backend fails its health check simultaneously, the load balancer marks the entire fleet unhealthy, and you’ve turned a degraded database into a total outage. A momentary blip becomes a hard down.
⚖️ The resolution — and this is the answer that scores well: use both, for different purposes.
This liveness/readiness split is exactly how Kubernetes models it. → Kubernetes
Graceful shutdown is the other half. On deploy, the sequence should be: stop passing readiness → LB drains you (a few seconds) → finish in-flight requests → exit. Skipping the drain means every deploy drops requests.
Route a given user to the same backend every time, usually via a cookie the LB sets.
When you need it: WebSocket connections (they live on a specific server), in-progress multi-part uploads, and legacy apps with in-memory session state.
⚖️ What it costs you:
🎙️ “I’d avoid sticky sessions by putting session state in Redis, which keeps every instance interchangeable. The one place we can’t avoid affinity is the WebSocket layer — connections have to live somewhere — so there I’d pair it with a registry mapping user → gateway node, and route messages through pub/sub.” → Scalability
🚨 A single load balancer is a single point of failure that defeats the entire purpose. Candidates draw one box labelled “LB” in front of ten servers and don’t notice.
How it’s actually done:
Active–passive with a floating IP. Two LBs; one holds a virtual IP. If it dies, the standby claims the IP (via VRRP/keepalived). Failover in seconds. Simple; half your capacity idles.
Active–active via DNS. Multiple LB IPs in DNS; clients spread across them. Every LB works, but failover is bounded by DNS TTL — which, as covered in DNS, is slow and unreliable.
Anycast. The same IP announced from many locations; BGP routes each client to the nearest healthy one. Failover at network speed, no DNS involvement. This is how every large CDN and every serious global service does it.
Managed cloud LBs (AWS ALB/NLB, GCP LB, Azure LB) handle this for you — they’re internally redundant across availability zones. For most systems, “use the managed one” is the correct and boring answer, and saying so is not a weakness.
🎙️ “The load balancer itself needs to be redundant — I’d use a managed multi-AZ balancer, or an active-passive pair with a floating IP if we were self-hosting.”
More places than beginners draw:
flowchart TB
U[Users] --> DNS[GeoDNS / Anycast]
DNS --> E[Edge LB / CDN<br/>L4 + DDoS]
E --> A[Application LB<br/>L7, path routing]
A --> S1[Service A]
A --> S2[Service B]
S1 --> IL[Internal LB / service mesh]
S2 --> IL
IL --> S3[Service C]
S3 --> DBP[Connection pooler<br/>PgBouncer]
DBP --> DB[(Database)]
Client-side load balancing is worth knowing as the alternative: the client fetches the list of backends from service discovery and picks one itself. No extra network hop, no central bottleneck — but every client needs the logic, and updates propagate slowly. gRPC and service meshes do this.
| Choice | Gain | Cost |
|---|---|---|
| L7 over L4 | Content routing, retries, per-request balancing | More CPU, higher latency, sees plaintext |
| Least connections over round robin | Handles variable request duration | Requires connection tracking |
| Deep health checks | Detects gray failures | A shared dependency blip can mark the whole fleet down |
| Sticky sessions | In-memory state works | Uneven load, state loss, disruptive deploys |
| Managed cloud LB | Redundancy handled for you | Less control, vendor lock-in, per-request cost |
| Client-side LB | No extra hop, no central bottleneck | Logic in every client; slow config propagation |
1. Build one. Two toy HTTP servers that print their own name, and Nginx in front:
upstream backend {
least_conn;
server 127.0.0.1:3001 max_fails=2 fail_timeout=10s;
server 127.0.0.1:3002 max_fails=2 fail_timeout=10s;
}
server {
listen 8080;
location / { proxy_pass http://backend; }
}
Then: while true; do curl -s localhost:8080; sleep 0.2; done
Kill one backend. Watch how many requests fail before Nginx notices. Now add a proper health check and measure again. That delta is your MTTR, and seeing it as a real number changes how you think about health check intervals.
2. Break round robin. Make one endpoint sleep 3 seconds and another return instantly. Send mixed
traffic through round robin, then through least_conn. Compare p99. The difference is the whole
argument for least-connections.
3. Watch a bad deploy. Kill a backend with in-flight requests, without draining. Count the 502s.
Then add nginx -s reload with graceful shutdown and count again.