system-design

Replication

Keeping copies of your data on multiple machines. It buys you read throughput, availability, and disaster recovery — and hands you a consistency problem in exchange.

Prerequisites: Consistency Models, CAP & PACELC Time to read: ~25 minutes


The problem

One database. It works, until:

Replication answers all four with the same mechanism: keep the data on more than one machine.

The moment you do, those machines can disagree. That disagreement is the entire subject.


Leader–follower (primary–replica)

The dominant model. One node accepts writes; others copy from it and serve reads.

flowchart LR
    C[Clients] -->|writes| L[(Leader)]
    C -->|reads| F1[(Follower 1)]
    C -->|reads| F2[(Follower 2)]
    L -->|replication log| F1
    L -->|replication log| F2

How it works: the leader writes every change to a log (Postgres WAL, MySQL binlog). Followers stream that log and apply the same changes in the same order. Same operations, same order, same result.

What it gives you:

What it doesn’t give you: write scaling. All writes still go to one machine. That’s what sharding is for, and confusing the two is a very common interview error.

🚨 Replication scales reads. Sharding scales writes. Say it that way.


Synchronous vs asynchronous — the core trade-off

Asynchronous: the leader acknowledges the write immediately and ships it to followers afterward.

Client → Leader: WRITE
Leader → Client: OK ✅          (fast — one local write)
Leader → Follower: here's the change   (later)

✅ Fast writes, and a slow or dead follower doesn’t affect the leader. ❌ Data loss on failover. If the leader dies before shipping the last writes, they’re gone. ❌ Replication lag — followers serve stale data.

Synchronous: the leader waits for follower acknowledgment before confirming.

Client → Leader: WRITE
Leader → Follower: here's the change
Follower → Leader: got it
Leader → Client: OK ✅          (slower — includes a network round trip)

✅ No data loss on failover; followers are current. ❌ Writes are slower by at least one round trip (fine in-datacenter at ~1 ms; brutal cross-region at ~150 ms). ❌ A dead follower blocks all writes. Your availability just got worse, not better.

Semi-synchronous — the practical middle. One follower synchronous, the rest async. You get durability (at least one other copy has it) without a slow follower stalling everything. This is what most production systems actually run.

📐 The arithmetic that decides it:

Setup Write latency Data loss on leader failure
Async, same DC ~1 ms Up to the lag (ms to seconds)
Semi-sync, same DC ~2 ms None
Sync, same DC ~2 ms None
Sync, cross-region ~150 ms None
Async, cross-region ~1 ms Up to seconds of writes

🎙️ “I’d use semi-synchronous replication with one local synchronous replica for durability, and asynchronous cross-region replicas for disaster recovery — I don’t want a 150 ms round trip on every write.”


Replication lag: the bug factory

Followers are behind. Usually milliseconds. Under load, seconds or worse. Three user-visible bugs follow, and knowing their names is worth real points.

1. Read-your-writes violation

User posts a comment    → leader
Page reloads            → follower (200 ms behind)
Comment isn't there     → "the site ate my comment"
User posts again        → now there are two

Fixes: route a user’s reads to the leader for a few seconds after their write; track the write’s log position (LSN) and only read from a follower that has caught up past it; or read user-modifiable data from the leader always.

2. Monotonic reads violation

Two successive requests hit followers with different lag. A comment appears, then disappears, then reappears. Fix: route a given user consistently to the same follower (hash on user ID).

3. Causal violation

Ayesha’s reply replicates faster than Bilal’s original question, so a reader sees an answer to nothing. Fix: causal tracking, or keep causally-related writes on the same partition.

→ All three, in depth: Consistency Models

🚨 Lag is worst exactly when it hurts most: during a traffic spike, the leader takes more writes, followers fall further behind, and more users hit stale data. Always monitor and alert on replication lag — it’s a leading indicator of trouble and it’s often the missing metric on otherwise good dashboards.

-- Postgres: lag in seconds on the replica
SELECT EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()));

Failover: harder than it looks

The leader dies. Something must promote a follower.

The sequence: detect the failure → choose a new leader (usually the most up-to-date follower) → reconfigure clients and remaining followers to point at it.

Everything that goes wrong:

Data loss. With async replication, writes the old leader accepted but never shipped are gone. If those writes had external side effects — you charged a card and lost the record — you have a reconciliation problem, not just a data problem.

Split brain. 🚨 The most dangerous outcome. The old leader wasn’t actually dead — it was partitioned, or in a long GC pause. It comes back still believing it’s the leader. Now two nodes accept writes and diverge.

Prevention: fencing. Use a monotonically increasing epoch/term number; storage and clients reject writes from an older term. Or STONITH (“shoot the other node in the head”) — forcibly power off the old leader before promoting. Any failover design without fencing has a split-brain bug in it. → Leader Election

Bad timeouts. Too short: a GC pause triggers an unnecessary failover, and failovers are disruptive. Too long: extended downtime. There is no correct value, only a trade — 10–30 seconds is typical.

Cascading load. The new leader has cold caches and now serves all traffic. It can fall over too.

Auto-generated ID collisions. The classic real-world horror story: MySQL auto-increment IDs issued by the old leader but not replicated get reissued by the new leader. Now two different rows share an ID, and any external system (a cache, a search index, an analytics warehouse) that recorded the old meaning is silently wrong.

⚖️ Automatic vs manual failover: automatic minimizes MTTR but risks unnecessary and dangerous failovers. Manual is safer but means a human at 3 a.m. Most teams automate it and invest heavily in making detection reliable. Managed databases (RDS Multi-AZ, Cloud SQL) do this for you, which is another strong argument for using them.


Multi-leader replication

Multiple nodes accept writes and replicate to each other.

When it’s used: multi-datacenter deployments where each region writes locally (low latency, survives inter-region partition), offline-capable clients (your phone’s notes app is a leader), and collaborative editing.

🚨 The cost: write conflicts are now possible and inevitable. Two regions modify the same record simultaneously. Both accept. Now what?

Resolution strategies:

Strategy How Problem
Last write wins (LWW) Highest timestamp survives Silently discards data. And clocks disagree — → Time & Clocks
Application-defined merge Your code decides Correct, but you must write it for every conflict type
CRDTs Data structures that merge deterministically Limited to structures that can be modelled this way
Keep both, ask the user Present the conflict Only viable for user-facing documents
Avoid conflicts Route all writes for a given record to one region Loses some of the benefit, but is often the right answer

Conflict Resolution

🎙️ “Multi-leader would give us local write latency in both regions, but it introduces write conflicts. Unless we can partition writes so each record has a home region, I’d rather have a single leader and accept cross-region write latency.”


Leaderless replication

No leader. Clients write to several replicas directly, and read from several.

The Dynamo model. Correctness comes from quorums:

N = number of replicas
W = replicas that must acknowledge a write
R = replicas that must respond to a read

W + R > N  →  the read set overlaps the write set, so reads see the latest write

N=3, W=2, R=2  →  strong-ish, survives one node down          ← the usual choice
N=3, W=1, R=1  →  fastest, most available, weakest guarantee
N=3, W=3, R=1  →  fast reads, but any node down blocks writes

Repair mechanisms keep replicas converging:

Systems: Cassandra, DynamoDB, Riak. → Quorums

🚨 A subtlety worth knowing: W + R > N does not give you linearizability. Concurrent writes, failed writes that partially succeeded, and sloppy quorums with hinted handoff all create edge cases. It gives you “probably the latest value,” which is much better than nothing and not the same as a guarantee.


The comparison

  Leader–follower Multi-leader Leaderless
Write scaling ❌ One node ✅ Per region
Write conflicts None Yes Yes
Failover needed Yes Less critical No leader to fail
Consistency Strong at leader Eventual Tunable via quorum
Complexity Low High Medium
Examples Postgres, MySQL, MongoDB CouchDB, multi-region MySQL Cassandra, DynamoDB, Riak

What actually gets replicated

The mechanism matters more than people expect:

🎙️ Knowing that logical replication is what enables CDC and zero-downtime upgrades is a nice depth signal — most candidates don’t distinguish the mechanisms at all.


⚖️ Trade-offs

Decision Gain Cost
Add read replicas Read scaling, cheap Stale reads; replication lag bugs
Synchronous replication No data loss Slower writes; a dead replica blocks writes
Asynchronous Fast writes; replica failures are harmless Data loss window on failover
Cross-region replicas Local read latency; DR Large lag; expensive; sync writes impossible
Multi-leader Local writes everywhere Conflicts, and conflict resolution is genuinely hard
Leaderless quorums No failover; tunable No true linearizability; more client-side complexity
Automatic failover Low MTTR Unnecessary failovers; split-brain risk without fencing

In the real world


🚨 Interview traps


🎙️ Soundbites


🛠️ Try it

1. Build a replica. Postgres primary + streaming replica in Docker (about 20 minutes with the official image and pg_basebackup). Then:

-- On the primary
INSERT INTO t VALUES (1, 'hello');
-- Immediately on the replica
SELECT * FROM t;   -- sometimes it's there, sometimes not

2. Measure lag under load. Run a write-heavy benchmark (pgbench) against the primary while polling replication lag on the replica. Watch lag grow from milliseconds to seconds as write volume rises. That curve is the argument for monitoring it.

3. Cause the read-your-writes bug. Write to the primary, read from the replica, in a loop. Count how often the read misses the write. Then implement “read from primary for 5 seconds after a write” and watch it go to zero.

4. Kill the primary. With repmgr or patroni, trigger a failover and time it. How many writes were lost? How long were you unavailable? Those two numbers are your RPO and RTO, and having measured them yourself is worth more than reading about them.


Check yourself

1. What's the difference between replication and sharding, and which solves which problem? Replication makes **copies** of the same data on multiple machines — it scales *reads*, provides availability through failover, and gives you disaster recovery. Sharding **splits** different data across machines — it scales *writes* and lets you exceed one machine's storage capacity. They're orthogonal and usually combined: shard the data, then replicate each shard. If writes are your bottleneck, replicas don't help at all.
2. Why can synchronous replication make availability worse? Because the leader must wait for the follower's acknowledgment before confirming a write. If the follower is slow, overloaded, or dead, writes block or fail — so a *replica* failure becomes a *primary* outage. You added a machine and increased the number of things whose failure takes you down. Semi-synchronous replication (wait for any one of N followers) avoids this while keeping the durability guarantee.
3. What is split brain and how do you prevent it? Two nodes both believe they're the leader and both accept writes, so the data diverges — usually because the original leader was partitioned or in a long GC pause rather than actually dead, and came back after a follower was promoted. Prevention: **fencing**. Assign each leadership term a monotonically increasing epoch number and have storage/clients reject writes from an older epoch; or require a quorum for promotion so a minority-side leader can't commit; or STONITH — forcibly power off the old node before promoting. Any failover design without fencing has this bug latent in it.
4. Your replicas are 30 seconds behind during peak traffic. What are the consequences and what would you do? Consequences: users see 30-second-old data, read-your-writes is badly broken (posts appear to vanish), any logic reading from replicas makes decisions on stale state, and your disaster-recovery RPO is 30 seconds of writes. Causes: write volume exceeding the replica's apply rate (replay is often single-threaded), a long-running query on the replica blocking apply, undersized replica hardware, or network saturation. Actions: alert on lag; route critical reads to the leader; enable parallel apply if available; scale up replica hardware; reduce write amplification (fewer indexes); and shard if the leader's write volume genuinely exceeds what a replica can keep up with.
5. When would you choose multi-leader replication, and what must you build alongside it? When you need low-latency writes in multiple regions and can tolerate eventual consistency — or when clients must work offline (mobile apps, collaborative editors). Alongside it you *must* build conflict detection and resolution: either partition writes so each record has a single home region (avoiding conflicts entirely — usually the best answer), use CRDTs for data that can be modelled that way, implement application-specific merge logic, or surface conflicts to users. Last-write-wins is the tempting default and it silently discards data, made worse by the fact that clocks on different machines disagree.

Further reading