system-design

Quorums and Read/Write Tradeoffs

One inequality — W + R > N — that lets you dial consistency, latency, and availability per query.

Prerequisites: Replication, Consistency Models Time to read: ~18 minutes

The problem

Data is on 3 replicas. A client writes.

Wait for all 3? Any single replica being slow or down blocks every write. You made availability worse by adding replicas.
Wait for 1? Fast and available — but a subsequent read might hit one of the two replicas that don’t have it yet, returning stale data.

You want a knob, not a binary. Quorums are that knob.

The inequality

N = number of replicas
W = replicas that must acknowledge a WRITE before it's considered successful
R = replicas that must respond to a READ before you return an answer

If  W + R > N   →  the read set and the write set MUST overlap
                →  at least one responding replica has the latest write

🧠 The intuition, which is worth being able to draw: with N=3, W=2, R=2 — you wrote to 2 of 3 and read from 2 of 3. By the pigeonhole principle, those sets share at least one replica. That replica has the new value, so the read sees it (using version numbers to pick the newest of the responses).

Replicas:   [A] [B] [C]
Write to:    ✓   ✓        (W=2)
Read from:       ✓   ✓    (R=2)
                 ↑ B is in both sets — it has the latest value

📐 Common configurations:

N	W	R	Property
3	2	2	✅ Strong-ish; survives 1 failure. The standard default.
3	1	1	⚡ Fastest, most available; eventually consistent
3	3	1	Fast reads, but any node down blocks all writes
3	1	3	Fast writes, but any node down blocks all reads
5	3	3	Strong; survives 2 failures
5	1	1	Maximum availability, weakest guarantee

🚨 Note rows 3 and 4. W=N or R=N means a single unavailable replica breaks that operation entirely. Extremes are almost always wrong — you gave up the fault tolerance that replication existed to provide.

Choosing W and R

The point is that you tune them per workload — and often per query.

Read-heavy (most systems): W=2, R=1 with N=3 gives fast reads at the cost of possible staleness. Or W=3, R=1 if you must have fresh reads and can tolerate fragile writes.

Write-heavy (logging, telemetry): W=1, R=3 — writes never block, reads pay the cost. Or just W=1, R=1 if eventual consistency is fine, which for telemetry it usually is.

Balanced: W=2, R=2 with N=3. This is the right default, and the one to state unless you have a reason otherwise.

Per-query tuning is the real advantage:

Payment record:     QUORUM write, QUORUM read      → correctness
Product catalogue:  ONE write, ONE read            → speed, availability
User session:       QUORUM write, ONE read         → durable, fast to check

🎙️ “I’d default to quorum on both sides, then relax to ONE for the catalogue reads where staleness is harmless and we want the lower latency and better availability.”

🚨 What `W + R > N` does NOT give you

This is the most important section in the chapter, and the most common overclaim in interviews.

Quorums do not give you linearizability. They give you “you’ll probably see the latest committed write,” which is meaningfully weaker. The gaps:

1. Concurrent writes. Two clients write simultaneously to overlapping quorums. Which is “latest”? You need conflict resolution — vector clocks, or last-write-wins with all its clock problems.

2. Partially-failed writes. A write reaches 1 of 3 replicas and then the client crashes. It never succeeded (W=2 not met), so the client got an error — but that value now exists on one replica and may surface in a later read. A write that “failed” can still become visible.

3. Sloppy quorums. See below — the overlap guarantee doesn’t hold.

4. Read repair races. Concurrent reads and repairs can interleave in ways that violate ordering.

5. No ordering between operations. Quorums say nothing about the relative order of operations on different keys.

🚨 The correct framing: quorums give you a probabilistic freshness guarantee that’s dramatically better than nothing and dramatically weaker than consensus. If you need real linearizability, you need consensus, not quorums.

🎙️ “Quorum reads and writes get us ‘very likely the latest value,’ not linearizability — concurrent writes and partially-failed writes both break the guarantee. For the operations that genuinely need linearizability, I’d use a consensus-backed store.”

That distinction is a strong senior signal, because most candidates say “W+R>N gives strong consistency” and stop.

Sloppy quorums and hinted handoff

Dynamo-style systems add an availability trick with a real cost.

Strict quorum: the W acknowledgments must come from the W nodes that own that key (its “home” nodes on the hash ring).

Sloppy quorum: if some home nodes are unreachable, accept writes on any W reachable nodes, which hold the data temporarily as a hint. When the home node returns, the hint is handed off to it.

⚖️ The trade:

✅ Writes succeed even when most home nodes are down — much higher write availability.
❌ 🚨 The overlap guarantee is gone. A read from the home nodes may miss a write that’s sitting as a hint elsewhere. W + R > N no longer implies you’ll see the latest value.

Hinted handoff is the recovery mechanism: the temporary holder delivers the data when the real owner returns. Combined with read repair (fix stale replicas discovered during a read) and anti-entropy (background comparison using Merkle trees), replicas converge. → Gossip Protocols

🚨 Sloppy quorums are the reason Cassandra and Dynamo are AP rather than CP, and knowing that they break the quorum guarantee — rather than just knowing the term — is the depth signal here.

Quorums vs consensus

Both use majorities. They are not the same thing.

	Quorum (Dynamo-style)	Consensus (Raft/Paxos)
Leader	None — any node accepts writes	One leader accepts all writes
Guarantee	“Probably the latest value”	Linearizable
Concurrent writes	Both accepted → conflict to resolve	Serialized by the leader → no conflict
Ordering	None across operations	Total order via the log
Write availability	High — no leader to fail	Blocked during elections
Throughput	High — no coordination bottleneck	Limited by the leader
Complexity	Lower	Higher
Examples	Cassandra, DynamoDB, Riak	etcd, ZooKeeper, Spanner, CockroachDB

🎙️ The one-line distinction: “Quorums tell you how many replicas must respond. Consensus tells you what order things happened in. If you need ordering or linearizability, quorums aren’t enough.”

Practical considerations

Latency is the slowest of the W (or R) responses, not the average.

📐 With N=3 and W=2, you wait for the second-fastest replica. With W=3, you wait for the slowest — which means your write latency is now the p99 of your slowest replica, and it’s much worse than you’d expect. This is a strong practical argument against W=N.

Multi-datacenter quorums need care. A quorum spanning regions means every write pays a cross-region round trip (~150 ms). Cassandra’s LOCAL_QUORUM is the standard answer — require a quorum within the local datacenter only, and replicate asynchronously across regions. You get local latency and cross-region durability, at the cost of losing recent writes if the whole region fails.

Replica count and durability. N=3 across availability zones is the common default. N=5 for data you really can’t lose. Note that N also determines your storage cost multiplier — N=3 means 3× the disk.

Monitor repair health. Read repair and anti-entropy are what make eventual consistency actually converge. If Cassandra repairs aren’t running (they’re notoriously easy to neglect), replicas drift apart indefinitely and deleted data can resurrect when a stale replica is read.

⚖️ Trade-offs

Setting	Gain	Cost
`W + R > N`	Very likely to read the latest write	Higher latency on both paths
`W=1`	Fastest writes; survives most failures	Stale reads; write may be lost if that node dies
`R=1`	Fastest, most available reads	May return stale data
`W=N` or `R=N`	Strongest for that operation	One node down breaks it entirely
Larger N	More durability and fault tolerance	More storage; higher quorum latency
Sloppy quorum	Writes succeed during failures	Breaks the overlap guarantee
`LOCAL_QUORUM`	Local latency in multi-region	Recent writes lost if the region fails

In the real world

Amazon’s Dynamo paper introduced this model, along with the observation that different data needs different settings — the shopping cart was configured for availability, deliberately accepting that a cart could temporarily diverge.
Cassandra’s consistency levels (ONE, QUORUM, LOCAL_QUORUM, ALL, EACH_QUORUM) are this chapter exposed as a per-query parameter, which is why the same cluster can serve strongly-consistent payments and eventually-consistent analytics.
DynamoDB simplifies it to two options: eventually consistent reads (the default, cheaper) and strongly consistent reads (2× the read capacity units, and routed to the leader replica). That pricing difference is a nice concrete illustration that consistency costs real money.

🚨 Interview traps

Claiming W + R > N gives linearizability. It doesn’t. Name the gaps: concurrent writes, partially-failed writes, sloppy quorums.
Choosing W=N or R=N without noticing it destroys fault tolerance.
Not mentioning per-query tuning. It’s the main advantage of the model.
Forgetting that latency is the slowest required response, not the average.
Quorums spanning regions without noting the cross-region round trip, or without mentioning LOCAL_QUORUM.
Not knowing sloppy quorums break the guarantee.
Ignoring repair processes — without them, eventual consistency isn’t eventual.

🎙️ Soundbites

“N=3, W=2, R=2 — the read and write sets overlap, so we see the latest committed write, and we still survive one node being down.”
“That gives us ‘probably the latest value,’ not linearizability. Concurrent writes still conflict, and a partially-failed write can surface later. For operations that genuinely need linearizability I’d use a consensus-backed store.”
“I wouldn’t set W=3 — write latency becomes the p99 of the slowest replica, and one node down blocks all writes. We’d have added replicas and reduced availability.”
“Across regions I’d use LOCAL_QUORUM so writes don’t pay a 150 ms cross-region round trip, and accept that a full region loss could lose the most recent writes.”
“Sloppy quorums keep writes succeeding when home nodes are down, but they break the overlap guarantee — a read can miss a write sitting as a hint on another node.”

🛠️ Try it

1. See consistency levels in Cassandra. Three-node cluster in Docker. Then:

CONSISTENCY ONE;
INSERT INTO users (id, name) VALUES (1, 'Bilal');

-- Stop a node, then:
CONSISTENCY QUORUM;
SELECT * FROM users WHERE id = 1;      -- still works (2 of 3)

-- Stop a second node:
SELECT * FROM users WHERE id = 1;      -- fails — no quorum
CONSISTENCY ONE;
SELECT * FROM users WHERE id = 1;      -- works again, possibly stale

Watching the same query succeed or fail purely based on consistency level is the clearest possible demonstration of the trade-off.

2. Prove W=1/R=1 is stale. With one node paused, write at ONE, then start the node and immediately read at ONE in a loop. Count how often you get the old value before read repair fixes it.

3. Measure the latency cost. Benchmark writes at ONE, QUORUM, and ALL. Plot them. Then introduce artificial latency on one node (tc netem) and re-measure — watch ALL degrade catastrophically while QUORUM barely moves. That’s the argument against W=N in one graph.

4. Create a conflict. Partition the cluster, write different values to the same key on each side at ONE, then heal it. Observe which value survives and work out why — that’s last-write-wins, and its dependence on clocks.

Check yourself

1. Why does W + R > N guarantee you read the latest write?

Because two sets drawn from N replicas, of sizes W and R, must overlap when W + R > N — the pigeonhole principle. With N=3, W=2, R=2: the write touched 2 replicas and the read queries 2, so at least one replica is in both sets and therefore holds the new value. The client compares version numbers or timestamps across the R responses and returns the newest. If W + R ≤ N, the sets can be disjoint — you could write to A and B while reading only from C — and the read would legitimately return stale data.

2. Why doesn't a quorum give you linearizability?

Several gaps. **Concurrent writes:** two clients writing simultaneously both meet W, so both succeed and you have a genuine conflict that quorums don't resolve — you need vector clocks or LWW. **Partially-failed writes:** a write reaching 1 of 3 replicas before the client dies never met W (so it "failed"), but that value exists on a replica and can surface in a later read — a failed write becoming visible. **Sloppy quorums:** if enabled, writes can be accepted by nodes that don't own the key, so the overlap guarantee doesn't hold at all. **No cross-operation ordering:** quorums say nothing about the relative order of operations on different keys. Real linearizability requires consensus.

3. What's wrong with setting W = N?

It eliminates fault tolerance for writes: every single replica must acknowledge, so one slow or dead node blocks all writes entirely. You added replicas for availability and made write availability *worse* than a single node. It also makes write latency the latency of the *slowest* replica — so your write p99 becomes the p99 of your worst-performing node, which is far worse than the median. The same reasoning applies to R = N for reads. Quorum values strictly between 1 and N are what preserve the fault tolerance replication was meant to provide.

4. What are sloppy quorums and hinted handoff, and what do they cost?

In a strict quorum, the W acknowledgments must come from the nodes that own that key on the hash ring. A **sloppy quorum** relaxes this: if home nodes are unreachable, any W reachable nodes accept the write and store it as a **hint**, delivering it to the rightful owner via **hinted handoff** when that node returns. The gain is much higher write availability — writes succeed even during substantial outages. The cost is that the overlap guarantee is broken: a read from the home nodes can miss a write parked as a hint elsewhere, so `W + R > N` no longer implies freshness. This is a principal reason Dynamo-style systems are classified AP rather than CP.

5. How would you configure quorums for a multi-region deployment?

Use a local quorum rather than a global one. Requiring a majority across regions means every write pays a cross-region round trip — 100–300 ms — which is unacceptable for user-facing writes. Cassandra's `LOCAL_QUORUM` requires a quorum only within the client's local datacenter, with asynchronous replication to other regions. Writes get local latency; data still reaches other regions for durability and disaster recovery. The accepted cost is that if an entire region is lost abruptly, the most recent writes that hadn't yet replicated are lost — so for data where that's unacceptable (payments), use `EACH_QUORUM` or a consensus-backed store, and pay the latency deliberately.