Keeping copies of your data on multiple machines. It buys you read throughput, availability, and disaster recovery — and hands you a consistency problem in exchange.
Prerequisites: Consistency Models, CAP & PACELC Time to read: ~25 minutes
One database. It works, until:
Replication answers all four with the same mechanism: keep the data on more than one machine.
The moment you do, those machines can disagree. That disagreement is the entire subject.
The dominant model. One node accepts writes; others copy from it and serve reads.
flowchart LR
C[Clients] -->|writes| L[(Leader)]
C -->|reads| F1[(Follower 1)]
C -->|reads| F2[(Follower 2)]
L -->|replication log| F1
L -->|replication log| F2
How it works: the leader writes every change to a log (Postgres WAL, MySQL binlog). Followers stream that log and apply the same changes in the same order. Same operations, same order, same result.
What it gives you:
What it doesn’t give you: write scaling. All writes still go to one machine. That’s what sharding is for, and confusing the two is a very common interview error.
🚨 Replication scales reads. Sharding scales writes. Say it that way.
Asynchronous: the leader acknowledges the write immediately and ships it to followers afterward.
Client → Leader: WRITE
Leader → Client: OK ✅ (fast — one local write)
Leader → Follower: here's the change (later)
✅ Fast writes, and a slow or dead follower doesn’t affect the leader. ❌ Data loss on failover. If the leader dies before shipping the last writes, they’re gone. ❌ Replication lag — followers serve stale data.
Synchronous: the leader waits for follower acknowledgment before confirming.
Client → Leader: WRITE
Leader → Follower: here's the change
Follower → Leader: got it
Leader → Client: OK ✅ (slower — includes a network round trip)
✅ No data loss on failover; followers are current. ❌ Writes are slower by at least one round trip (fine in-datacenter at ~1 ms; brutal cross-region at ~150 ms). ❌ A dead follower blocks all writes. Your availability just got worse, not better.
Semi-synchronous — the practical middle. One follower synchronous, the rest async. You get durability (at least one other copy has it) without a slow follower stalling everything. This is what most production systems actually run.
📐 The arithmetic that decides it:
| Setup | Write latency | Data loss on leader failure |
|---|---|---|
| Async, same DC | ~1 ms | Up to the lag (ms to seconds) |
| Semi-sync, same DC | ~2 ms | None |
| Sync, same DC | ~2 ms | None |
| Sync, cross-region | ~150 ms | None |
| Async, cross-region | ~1 ms | Up to seconds of writes |
🎙️ “I’d use semi-synchronous replication with one local synchronous replica for durability, and asynchronous cross-region replicas for disaster recovery — I don’t want a 150 ms round trip on every write.”
Followers are behind. Usually milliseconds. Under load, seconds or worse. Three user-visible bugs follow, and knowing their names is worth real points.
User posts a comment → leader
Page reloads → follower (200 ms behind)
Comment isn't there → "the site ate my comment"
User posts again → now there are two
Fixes: route a user’s reads to the leader for a few seconds after their write; track the write’s log position (LSN) and only read from a follower that has caught up past it; or read user-modifiable data from the leader always.
Two successive requests hit followers with different lag. A comment appears, then disappears, then reappears. Fix: route a given user consistently to the same follower (hash on user ID).
Ayesha’s reply replicates faster than Bilal’s original question, so a reader sees an answer to nothing. Fix: causal tracking, or keep causally-related writes on the same partition.
→ All three, in depth: Consistency Models
🚨 Lag is worst exactly when it hurts most: during a traffic spike, the leader takes more writes, followers fall further behind, and more users hit stale data. Always monitor and alert on replication lag — it’s a leading indicator of trouble and it’s often the missing metric on otherwise good dashboards.
-- Postgres: lag in seconds on the replica
SELECT EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()));
The leader dies. Something must promote a follower.
The sequence: detect the failure → choose a new leader (usually the most up-to-date follower) → reconfigure clients and remaining followers to point at it.
Everything that goes wrong:
Data loss. With async replication, writes the old leader accepted but never shipped are gone. If those writes had external side effects — you charged a card and lost the record — you have a reconciliation problem, not just a data problem.
Split brain. 🚨 The most dangerous outcome. The old leader wasn’t actually dead — it was partitioned, or in a long GC pause. It comes back still believing it’s the leader. Now two nodes accept writes and diverge.
Prevention: fencing. Use a monotonically increasing epoch/term number; storage and clients reject writes from an older term. Or STONITH (“shoot the other node in the head”) — forcibly power off the old leader before promoting. Any failover design without fencing has a split-brain bug in it. → Leader Election
Bad timeouts. Too short: a GC pause triggers an unnecessary failover, and failovers are disruptive. Too long: extended downtime. There is no correct value, only a trade — 10–30 seconds is typical.
Cascading load. The new leader has cold caches and now serves all traffic. It can fall over too.
Auto-generated ID collisions. The classic real-world horror story: MySQL auto-increment IDs issued by the old leader but not replicated get reissued by the new leader. Now two different rows share an ID, and any external system (a cache, a search index, an analytics warehouse) that recorded the old meaning is silently wrong.
⚖️ Automatic vs manual failover: automatic minimizes MTTR but risks unnecessary and dangerous failovers. Manual is safer but means a human at 3 a.m. Most teams automate it and invest heavily in making detection reliable. Managed databases (RDS Multi-AZ, Cloud SQL) do this for you, which is another strong argument for using them.
Multiple nodes accept writes and replicate to each other.
When it’s used: multi-datacenter deployments where each region writes locally (low latency, survives inter-region partition), offline-capable clients (your phone’s notes app is a leader), and collaborative editing.
🚨 The cost: write conflicts are now possible and inevitable. Two regions modify the same record simultaneously. Both accept. Now what?
Resolution strategies:
| Strategy | How | Problem |
|---|---|---|
| Last write wins (LWW) | Highest timestamp survives | Silently discards data. And clocks disagree — → Time & Clocks |
| Application-defined merge | Your code decides | Correct, but you must write it for every conflict type |
| CRDTs | Data structures that merge deterministically | Limited to structures that can be modelled this way |
| Keep both, ask the user | Present the conflict | Only viable for user-facing documents |
| Avoid conflicts | Route all writes for a given record to one region | Loses some of the benefit, but is often the right answer |
🎙️ “Multi-leader would give us local write latency in both regions, but it introduces write conflicts. Unless we can partition writes so each record has a home region, I’d rather have a single leader and accept cross-region write latency.”
No leader. Clients write to several replicas directly, and read from several.
The Dynamo model. Correctness comes from quorums:
N = number of replicas
W = replicas that must acknowledge a write
R = replicas that must respond to a read
W + R > N → the read set overlaps the write set, so reads see the latest write
N=3, W=2, R=2 → strong-ish, survives one node down ← the usual choice
N=3, W=1, R=1 → fastest, most available, weakest guarantee
N=3, W=3, R=1 → fast reads, but any node down blocks writes
Repair mechanisms keep replicas converging:
Systems: Cassandra, DynamoDB, Riak. → Quorums
🚨 A subtlety worth knowing: W + R > N does not give you linearizability. Concurrent writes,
failed writes that partially succeeded, and sloppy quorums with hinted handoff all create edge cases.
It gives you “probably the latest value,” which is much better than nothing and not the same as a
guarantee.
| Leader–follower | Multi-leader | Leaderless | |
|---|---|---|---|
| Write scaling | ❌ One node | ✅ Per region | ✅ |
| Write conflicts | None | Yes | Yes |
| Failover needed | Yes | Less critical | No leader to fail |
| Consistency | Strong at leader | Eventual | Tunable via quorum |
| Complexity | Low | High | Medium |
| Examples | Postgres, MySQL, MongoDB | CouchDB, multi-region MySQL | Cassandra, DynamoDB, Riak |
The mechanism matters more than people expect:
NOW(), RANDOM(), auto-increment). Largely abandoned for this reason.🎙️ Knowing that logical replication is what enables CDC and zero-downtime upgrades is a nice depth signal — most candidates don’t distinguish the mechanisms at all.
| Decision | Gain | Cost |
|---|---|---|
| Add read replicas | Read scaling, cheap | Stale reads; replication lag bugs |
| Synchronous replication | No data loss | Slower writes; a dead replica blocks writes |
| Asynchronous | Fast writes; replica failures are harmless | Data loss window on failover |
| Cross-region replicas | Local read latency; DR | Large lag; expensive; sync writes impossible |
| Multi-leader | Local writes everywhere | Conflicts, and conflict resolution is genuinely hard |
| Leaderless quorums | No failover; tunable | No true linearizability; more client-side complexity |
| Automatic failover | Low MTTR | Unnecessary failovers; split-brain risk without fencing |
1. Build a replica. Postgres primary + streaming replica in Docker (about 20 minutes with the
official image and pg_basebackup). Then:
-- On the primary
INSERT INTO t VALUES (1, 'hello');
-- Immediately on the replica
SELECT * FROM t; -- sometimes it's there, sometimes not
2. Measure lag under load. Run a write-heavy benchmark (pgbench) against the primary while
polling replication lag on the replica. Watch lag grow from milliseconds to seconds as write volume
rises. That curve is the argument for monitoring it.
3. Cause the read-your-writes bug. Write to the primary, read from the replica, in a loop. Count how often the read misses the write. Then implement “read from primary for 5 seconds after a write” and watch it go to zero.
4. Kill the primary. With repmgr or patroni, trigger a failover and time it. How many writes
were lost? How long were you unavailable? Those two numbers are your RPO and RTO, and having measured
them yourself is worth more than reading about them.