system-design

Part 4 — Distributed Systems

What happens when the machines have to agree with each other, and none of them can tell whether the others are alive.

Time for this part: Week 8 on the Standard track, and it’s the part that most distinguishes senior candidates from mid-level ones.

Parts 2 and 3 gave you components. This part is about the properties that emerge when those components are spread across machines — and every one of them is a consequence of a single fact: you cannot distinguish a slow node from a dead one.

Chapters

Why it’s hard

# Chapter Time The problem it solves
1 The 8 Fallacies 20 min Eight assumptions that are true locally and false everywhere else
2 Failure Modes and Fault Tolerance 24 min Slow is worse than dead, and why cascades happen
3 Time, Clocks, and Ordering 22 min “Which happened first?” has no easy answer

Agreeing on things

# Chapter Time The problem it solves
4 Leader Election 20 min One node in charge, and split brain when there are two
5 Consensus: Paxos, Raft, ZAB 24 min Permanent agreement despite failures
6 Quorums 18 min W + R > N, and what it does not guarantee
7 Gossip Protocols & Anti-Entropy 18 min 1,000 nodes learning about each other without a coordinator
8 Conflict Resolution & CRDTs 24 min Two valid writes, one record

Doing work across machines

# Chapter Time The problem it solves
9 Distributed Transactions: 2PC, 3PC 20 min Atomicity across databases, and why 2PC blocks
10 The Saga Pattern 24 min Business transactions across services
11 Idempotency ⭐ 26 min The most valuable thing to raise unprompted in an interview
12 Distributed Locking 22 min Mutual exclusion — and why a lock isn’t a guarantee

Surviving failure

# Chapter Time The problem it solves
13 Resilience Patterns 24 min Circuit breakers, bulkheads, backpressure, load shedding
14 Retries, Timeouts, and Jitter 22 min Retries that recover instead of causing the outage

The through-line

The network is unreliable and you can't detect failure (ch 1, 2)
        ↓
So a timeout is ambiguous — you can't tell what happened
        ↓
And you can't order events by clock time either (ch 3)
        ↓
So agreement requires an explicit protocol (ch 4–7)
        ↓
And when you allow concurrent writes, conflicts are inevitable (ch 8)
        ↓
Multi-system atomicity is either blocking or eventually consistent (ch 9, 10)
        ↓
All of which is survivable only because operations can be made repeatable (ch 11, 12)
        ↓
And because you contain failures rather than inheriting them (ch 13, 14)

The seven things to remember

  1. A timeout is ambiguous. Never arrived, still running, or succeeded with a lost response — all look identical. This single fact drives most of the rest.
  2. A slow node is more dangerous than a dead one. Dead nodes get ejected; slow ones poison their callers.
  3. Perfect failure detection is impossible. Every detector is a heuristic with a timeout, and the timeout is a trade-off with no correct answer.
  4. Never order distributed events by wall-clock timestamp. Use logical clocks or a single sequencing authority.
  5. Exactly-once delivery is impossible. Build at-least-once delivery plus idempotent processing.
  6. A distributed lock is not a guarantee. A GC pause longer than the TTL means two holders. Only fencing tokens, enforced by the resource, provide correctness.
  7. Retries need backoff, jitter, and a budget — otherwise they cause the outage they were meant to prevent.

Before moving on

You should be able to answer these without notes:

If several of those are shaky, revisit before Part 12 — the senior-level case studies assume all of it.

Next: Part 5 — Architecture Patterns — how to arrange services, and when not to.

This site is open source. Improve this page.