system-design

Failure Modes and Fault Tolerance

Everything fails. At scale, everything is failing right now. The design question isn’t whether — it’s what happens next.

Prerequisites: The 8 Fallacies, Availability Time to read: ~24 minutes

The problem

📐 Rare events are constant at scale. A disk with a 3-year mean time between failures, in a fleet of 10,000:

10,000 disks ÷ (3 × 365 days) ≈ 9 disk failures per day

Add servers, switches, power supplies, and software bugs, and something in your system is broken right now. Designing for “everything works” is designing for a fleet of one.

The useful skill is naming how things fail, because different failure modes need completely different defences.

The taxonomy of failures

Ordered from easiest to hardest to handle.

1. Crash-stop (fail-stop)

The node halts and stays halted. It stops responding entirely.

✅ The easiest failure, because it’s unambiguous — well, almost (see failure detection below). Handled by redundancy and health checks.

2. Crash-recovery

The node crashes and comes back, possibly with stale state.

🚨 The subtle danger: it comes back believing things that are no longer true. It thinks it’s still the leader. It holds a lock it lost. Its cached view of the cluster is from before the reconfiguration.

Defences: fencing tokens (distributed locking), epoch numbers on leadership (leader election), and re-validating all state on startup rather than trusting what’s in memory or on disk.

3. Omission

Messages are lost. The node is fine; the network drops the packet, or a buffer overflows.

Defences: timeouts, retries, sequence numbers, acknowledgments.

4. Timing / performance

🚨 The one that causes the most real damage. The node responds — eventually. Too slowly to be useful.

Why it’s worse than crashing: a crashed node is removed from the pool automatically. A slow node keeps receiving traffic, and every request to it consumes a caller’s thread, connection, and timeout budget. One slow node can take down healthy callers.

📐 The mechanism: caller has 100 threads, calls the slow node, each call blocks for 30 seconds. Within seconds all 100 threads are stuck and the caller stops serving everything, including requests that don’t touch the slow node.

Defences: aggressive timeouts, circuit breakers, bulkheads, load shedding, and health checks that measure latency, not just liveness. → Resilience Patterns

5. Byzantine

A node behaves arbitrarily — returns wrong data, lies, sends contradictory messages to different peers. Caused by corrupted memory, disk bit-rot, malicious actors, or severe bugs.

⚖️ Byzantine fault tolerance is expensive (needs 3f+1 nodes to tolerate f faults, versus 2f+1 for crash faults) and almost never used in ordinary systems. It matters for blockchains, aerospace, and adversarial multi-party settings.

🚨 But partial Byzantine behaviour is common and worth defending against: checksums on data at rest and in transit, validating inputs from other services rather than trusting them, and not assuming a peer’s response is well-formed. Silent disk corruption is real and happens at scale.

The failure detection problem

🚨 This is the deepest idea in the chapter.

You cannot distinguish a crashed node from a slow node, or from a network partition.

All three look identical: no response.

Node A → Node B: "are you alive?"
(silence)

Is B dead?  Is B in a 30-second GC pause?  Is the network between us down?
A cannot tell. Ever.

This is a theoretical impossibility, not an engineering gap. Chandra and Toueg proved that perfect failure detection is impossible in an asynchronous network. The FLP result showed you can’t even guarantee consensus with one faulty process in a fully asynchronous system.

So all real failure detection is a heuristic, and it forces a trade-off with no correct answer:

Short timeout (1 s)   → detect failures fast, but a GC pause triggers a false positive
Long timeout (30 s)   → few false positives, but 30 seconds of downtime for a real failure

And false positives are genuinely harmful. Declaring a healthy leader dead triggers an unnecessary election, disrupts traffic, and can cause split brain if the “dead” node doesn’t agree.

Better approaches:

Phi-accrual failure detectors (Cassandra, Akka) output a suspicion level rather than a binary verdict, adapting to observed network variability instead of using a fixed threshold.
Indirect probes (SWIM protocol) — before declaring a node dead, ask other nodes to probe it. If three peers can reach it, the problem is your link, not the node.
Quorum-based decisions — a majority must agree the node is unreachable.

🎙️ “Failure detection is fundamentally a heuristic — we can’t distinguish slow from dead. I’d use indirect probes so we don’t evict a node because of one bad network link, and accept that our timeout is a trade between detection speed and false positives.”

Cascading failure

The most common way distributed systems die completely, and it’s worth knowing the mechanism precisely.

5 servers, each at 60% capacity
   ↓
1 fails
   ↓
Its traffic redistributes: the other 4 are now at 75%
   ↓
Latency rises (queueing is non-linear near saturation)
   ↓
Clients time out and RETRY → effective load rises further
   ↓
A second server saturates and fails
   ↓
3 servers now handle 100% → they fail
   ↓
Total outage, ~90 seconds after the first failure

🚨 Note that retries actively accelerate it. A degraded service receives more load precisely when it can least handle it. This is why retry policy is a load-bearing design decision, not a detail.

Defences:

Defence	Effect
Headroom (run at 60–70%, not 90%)	Survivors can absorb a failure → Utilization
Circuit breakers	Stop calling a failing dependency; fail fast
Retry budgets	Cap retries at ~10% of traffic, globally
Backoff + jitter	Retries spread out instead of synchronizing
Load shedding	Reject some traffic to keep serving the rest
Bulkheads	One dependency can’t consume all your threads
Autoscaling	Add capacity — though too slowly to save you in a 90-second cascade

🚨 Autoscaling does not prevent cascading failure. Instance startup takes 30–120 seconds, and the cascade completes in under two minutes. Autoscaling handles gradual growth; only headroom and load shedding handle cascades.

Metastable failure

A subtler and increasingly-discussed mode: the system stays broken after the trigger is gone.

Normal: 10k QPS, cache hit rate 95%, database handles 500 QPS of misses

Trigger: cache restarts → hit rate drops to 0%
         database gets 10k QPS → saturates → requests time out
         clients retry → 20k QPS → worse

Trigger removed (cache is back), but:
         requests time out before they can populate the cache
         so the hit rate stays 0%
         so the database stays saturated
         → the system is stuck in the broken state, permanently

🚨 The defining feature: removing the original cause doesn’t fix it. The system found a stable bad equilibrium, sustained by its own retry and cache-miss dynamics.

Escaping requires breaking the loop from outside: shed load aggressively until the cache warms, drop traffic to a fraction, or restart in a controlled way with a warm-up phase.

Prevention: retry budgets, admission control, cache warming before accepting traffic, and explicit degraded modes.

Gray failure

The node is partially broken. It passes health checks and returns errors for 5% of requests, or is slow for one endpoint, or fails only for one shard.

🚨 The hardest failures to detect and the longest outages, because none of your automation triggers: health checks pass, the load balancer keeps routing, no failover happens, and no alert fires. You find out from users.

Why it happens: health checks measure the wrong thing. GET /health returning 200 tells you the process is running — not that its database connection pool is exhausted, or that one dependency is timing out.

Defences:

Health checks that exercise a real code path, not just liveness.
Per-endpoint, per-shard, per-instance metrics — fleet averages hide a single bad instance.
Alert on symptoms users feel (error rate, latency) rather than on process liveness.
Outlier detection in the load balancer — automatically eject instances whose error rate or latency deviates from the fleet.

Correlated failure

🚨 The assumption that breaks all your availability arithmetic.

Three replicas at 99% each should give 99.9999% (the math). You will not get that, because failures are not independent:

All three run the same buggy code you just deployed.
All three are in the same rack, sharing a power supply and a top-of-rack switch.
All three use the same certificate, expiring at the same instant.
All three depend on the same DNS provider, the same config service, the same region.
All three are hit by the same traffic spike.

Defences: spread across availability zones and regions; staggered canary deploys so a bad release hits 5% not 100%; no shared fate in credentials or certificate expiry; avoid a single config service everything blocks on at startup; and diversity in critical dependencies (two DNS providers).

🎙️ “Three replicas gives us six nines on paper, but failures correlate — same code, same rack, same certificate. Realistically we’re at three or four nines, so I’d spread across AZs and stagger deploys.”

Designing for failure

Assume every dependency will fail. For each one, decide: retry, fail fast, degrade, or queue.

Bulkheads. Separate resource pools per dependency, so a slow one can’t consume everything.

# ❌ One shared pool — the slow service starves everything
pool = ThreadPool(100)

# ✅ Bounded per dependency
payments = ThreadPool(20)
recommendations = ThreadPool(10)   # can only ever consume 10

Graceful degradation. Rank features by criticality. Recommendations down → hide the widget. Payments down → that’s an outage. Amazon’s product page renders and sells even when half its services are unavailable.

Fail fast, and fail loudly. A request that will fail should fail in 50 ms, not 30 seconds — the caller can then try an alternative or degrade. Silent failures (a swallowed exception, an empty result treated as “no data”) are far worse than loud ones.

Test failure deliberately. Kill instances, inject latency, partition the network. → Chaos Engineering

⚖️ Trade-offs

Decision	Gain	Cost
Short failure-detection timeout	Fast recovery	False positives from GC pauses; unnecessary failovers
Long timeout	Stable, few false alarms	Extended downtime for real failures
Aggressive retries	Recover from transient failures	Amplify load during degradation; cause cascades
Retry budgets	Prevent retry storms	Some recoverable requests fail
Headroom (60–70% utilization)	Survives instance loss; good p99	~40% more infrastructure cost
Deep health checks	Catch gray failures	A shared dependency blip can eject the whole fleet
Byzantine tolerance	Survives arbitrary behaviour	3f+1 nodes; complex; rarely justified

In the real world

AWS S3’s 2017 outage was triggered by a typo removing more capacity than intended, and lasted hours because the subsystems hadn’t been fully restarted in years — recovery was untested at that scale. The lesson is that your recovery path must be exercised, or you don’t know if it works.
Google’s “The Tail at Scale” documents how slow nodes (timing failures) dominate user experience at scale, and introduces hedged requests as a mitigation. → Performance Metrics
The metastable failure paper (Bronson et al., 2021) named a class of outages practitioners had been seeing for years without a vocabulary for it. Worth reading — it changes how you think about retry policy.
Netflix’s Chaos Monkey exists because untested failure handling is broken failure handling. Randomly killing production instances during business hours means you discover the problem with everyone available.

🚨 Interview traps

Only designing the happy path. Interviewers specifically probe failure. Volunteer it.
Not knowing that slow is worse than dead. A crashed node is ejected; a slow one poisons its callers.
Assuming independent failures when computing availability.
Retries without budgets or backoff. They cause the outage they’re meant to prevent.
Shallow health checks that miss gray failures.
Claiming autoscaling handles a cascade. It’s far too slow.
No degraded mode. Every non-critical dependency needs a timeout and a fallback.

🎙️ Soundbites

“A slow node is more dangerous than a dead one — a dead node gets ejected automatically, while a slow one keeps taking traffic and ties up every caller’s threads.”
“We can’t distinguish crashed from slow from partitioned. So detection is a heuristic, and I’d use indirect probes through peers rather than evicting a node because of one bad link.”
“I’d run at around 65% utilization. At 90% there’s no headroom to absorb a failed instance, and the redistribution triggers a cascade — which autoscaling can’t fix, because instances take a minute to start and the cascade completes in ninety seconds.”
“Retries need a global budget, capped at maybe 10% of traffic. Otherwise a degraded service receives more load exactly when it’s least able to handle it.”
“Three replicas is six nines only if failures are independent — same code, same rack, same certificate expiry. I’d spread across AZs and stagger the deploy so a bad release hits 5%.”

🛠️ Try it

1. Prove that slow is worse than dead. Three backends behind a load balancer. Kill one — measure the error rate and recovery time. Now instead make one respond in 30 seconds. Watch the entire system degrade far worse than when it was dead. This is the most valuable ten minutes in the chapter.

2. Cause a cascading failure. Five instances at 70% load. Kill one. Watch latency rise on the rest. Add client retries with no backoff. Watch it collapse. Then add a circuit breaker and retry budget and watch it survive.

3. Reproduce a metastable failure. App + cache + database. Warm the cache, then flush it while under load with clients that retry on timeout. Observe that restoring the cache doesn’t recover the system — requests time out before they can repopulate it. Then add load shedding and watch it escape.

4. Build a gray failure. Make one instance return errors for 5% of requests while its /health endpoint returns 200. Confirm your load balancer keeps sending it traffic and no alert fires. Then add outlier ejection based on error rate.

Check yourself

1. Why is a slow node more dangerous than a crashed one?

A crashed node stops responding entirely, so health checks fail and the load balancer ejects it within seconds — the failure is contained. A slow node keeps passing health checks and keeps receiving traffic, and every request to it occupies a caller's thread, connection, and memory for the full timeout duration. With enough concurrency, all of the caller's resources become tied up waiting, so the caller stops serving *all* requests — including ones that never touch the slow node. One degraded dependency takes down healthy services. This is why bulkheads, aggressive timeouts, and circuit breakers exist.

2. Why is perfect failure detection impossible?

Because in an asynchronous network, a crashed node, a very slow node, and an unreachable node all produce the same observation: no response within the time you waited. There's no message that says "I have crashed," and no bound on how long a healthy node might take. This is a proven theoretical result, not an engineering limitation. Practically, it means every failure detector is a heuristic with a timeout, and the timeout is a trade-off: short means fast detection but false positives from GC pauses (which cause disruptive unnecessary failovers), long means stability but extended downtime. Better detectors use adaptive suspicion levels (phi-accrual) and indirect probes through peers.

3. Walk through a cascading failure and name three defences.

One of five servers at 70% load fails. Its traffic redistributes, pushing the remaining four to ~88%. Queueing delay grows non-linearly, so latency spikes. Clients hit their timeouts and retry, adding *more* load. A second server saturates and fails. The remaining three now handle 100% of traffic at well over capacity and fail too — total outage in roughly ninety seconds. Defences: **headroom** (run at 60–70% so survivors can absorb the redistribution); **circuit breakers and retry budgets** so retries don't amplify load during degradation; and **load shedding** so the system serves a subset of traffic successfully rather than failing all of it. Note that autoscaling does *not* help — instances take longer to start than the cascade takes to complete.

4. What is a metastable failure and why is it particularly nasty?

A failure state that persists after its trigger is removed, because the system has found a stable bad equilibrium sustained by its own dynamics. Classic example: a cache restart drops the hit rate to zero, the database saturates under the miss load, requests time out *before they can populate the cache*, so the hit rate stays at zero and the database stays saturated — even though the cache is now healthy. It's nasty because normal remediation (fix the trigger, wait) doesn't work; you must break the feedback loop externally by shedding load aggressively until the system can re-establish its good state. Prevention: retry budgets, admission control, and warming caches before accepting traffic.

5. Why do three replicas not give you six nines of availability?

Because the multiplication assumes *independent* failures, and real failures are correlated. All three replicas typically run the same code (so a bad deploy kills all three simultaneously), may share a rack, power supply, or availability zone, often share a TLS certificate that expires at one instant, depend on the same DNS provider and config service, and are hit by the same traffic spike. There's also load-induced correlation: when one fails, its traffic goes to the others, which may then fail too. Realistic multi-instance services land around three to four nines. Mitigations: spread across AZs and regions, stagger deploys with canaries, avoid shared certificate/credential expiry, and use independent providers for critical dependencies.