system-design

Multi-Region and Disaster Recovery

Surviving the loss of an entire datacenter or region, and serving users across the globe. The most expensive, complex thing in this book — and the place where CAP stops being theory.

Prerequisites: CAP & PACELC, Replication Time to read: ~22 minutes

Two problems, often conflated

Multi-region deployment serves two distinct goals, and conflating them causes confused designs:

  1. Disaster recovery (DR) — survive a whole-region failure (fire, flood, a cloud provider’s region outage, a botched deploy that takes down a region). Availability.
  2. Geographic performance — serve users near them, avoiding the ~150ms cross-continent latency. Latency. And sometimes data residency (compliance) forces it.

🚨 These have different solutions. DR can be a passive standby region; geo-performance needs active regions serving traffic. Know which problem you’re solving.

And a prior question: 🚨 do you even need multi-region? Multi-AZ already survives a single-datacenter failure cheaply. Multi-region is a large step up in cost and complexity, justified by whole-region-failure protection, global latency, or compliance — not by default.

RTO and RPO: the DR vocabulary

🚨 The two numbers that define your DR requirements, and they come up:

RTO = downtime tolerance     (how fast must you recover)
RPO = data-loss tolerance    (how much recent data can vanish)

📐 A payment system might need RTO of minutes and RPO of zero (lose no transactions). A blog might tolerate RTO of hours and RPO of a day. 🚨 These numbers drive the DR strategy and its cost — tighter RTO/RPO means more expensive architecture. Ask for them.

DR strategies (the spectrum)

From cheapest/slowest to most expensive/fastest:

1. Backup & restore. Back up to another region; on disaster, restore.

2. Pilot light. A minimal version always running in the standby region (core data replicated, but compute scaled to zero); scale it up on disaster.

3. Warm standby. A scaled-down but fully functional copy always running; scale up and shift traffic on disaster.

4. Hot standby / active-active. Full capacity in multiple regions, all serving traffic.

🎙️ “The strategy follows the RTO/RPO. For an RTO of hours and RPO of a day, backup-and-restore. For an RTO of minutes and RPO of seconds — a critical service — warm standby or active-active. The tighter the numbers, the more you pay.”

Active-passive vs active-active

The core architectural choice for a running multi-region system:

Active-passive — one region serves all traffic; the other(s) stand by (warm/hot). On failure, promote the passive region.

Active-active — multiple regions serve traffic simultaneously.

🚨 The central challenge of active-active is data. Reads are easy (replicate to every region, read locally). Writes are hard, because you’re now doing multi-master replication across a 150ms-latency link.

The data problem in active-active

This is the crux, and it’s where the distributed-systems theory pays off:

Option A — single-region writes (write-forwarding). All writes go to one “home” region (per record or globally); reads are local everywhere. Avoids conflicts, but cross-region writes pay the latency, and losing the write region needs failover.

Option B — regional write ownership (sharding by region). 🚨 Often the best answer: each record has a home region (e.g. by user location), and its writes always go there. EU users’ data is written in the EU, US users’ in the US — no two regions write the same record, so no conflicts. This also satisfies data residency. Cross-region access is slower but rare.

Option C — accept conflicts and resolve them. True multi-master; use CRDTs, vector clocks, or LWW to resolve concurrent writes to the same record. Powerful but complex, and LWW silently loses data.

Option D — a globally-consistent database. Spanner, CockroachDB, or DynamoDB Global Tables handle multi-region consistency for you — at the cost of latency (a globally-consistent write pays cross-region round trips) and money.

🎙️ “For active-active, the hard part is writes. I’d shard by region — each user’s data has a home region where its writes happen — so no two regions write the same record and there are no conflicts. That also handles data residency. Truly global writes to one record would need either write-forwarding, conflict resolution, or a globally-consistent database like Spanner, each with a latency or complexity cost.”

This region-sharding answer is strong — it sidesteps the conflict problem rather than solving it, which is the mature move.

Routing traffic to regions

How users reach the right region:

The things people forget

🚨 DR is only real if it’s tested:

⚖️ Trade-offs

Strategy RTO RPO Cost Complexity
Backup & restore Hours Hours 💚 Lowest Low
Pilot light ~30 min Minutes 💛 Low Medium
Warm standby Minutes Seconds 🧡 Medium Medium
Active-active ~Zero ~Zero 🔴 Highest High (write conflicts)
Choice Gain Cost
Multi-AZ Cheap datacenter-failure resilience Doesn’t survive region failure
Active-passive Simpler (single write region, no conflicts) Idle standby capacity; failover window
Active-active Geo-performance, instant failover Write conflicts, cost, complexity
Region-sharded writes No conflicts, residency handled Cross-region access is slow/rare
Global DB (Spanner) Consistency handled for you Latency and cost

In the real world

🚨 Interview traps

🎙️ Soundbites

🛠️ Try it

1. Define RTO/RPO and pick a strategy. For three systems (a bank, a blog, a game), define RTO and RPO, then pick the DR strategy each needs. Notice how the numbers drive the choice — and how much more the tight ones cost.

2. Build active-passive replication. Set up a primary database in one region replicating to a standby in another. Then simulate a region failure: promote the standby, shift traffic. Time the failover (your RTO) and check how much data was lost (your RPO). These are your real numbers.

3. Hit the write-conflict problem. In an active-active setup, write to the same record in two regions simultaneously (during a simulated partition), then heal. Watch the conflict — which write wins, and does LWW silently lose one? Then implement region-sharded writes and confirm the conflict disappears (each record has one home region).

4. Test a backup by restoring it. Take a database backup, then actually restore it into a fresh environment and verify the data is intact and usable. A surprising number of backups fail this — and finding out in a drill beats finding out during a disaster.

Check yourself

1. What are RTO and RPO, and why do they matter? They're the two numbers that define your disaster-recovery requirements. **RTO (Recovery Time Objective)** is how long you can be down — your downtime tolerance, "we must recover within X minutes." **RPO (Recovery Point Objective)** is how much data you can afford to lose — your data-loss tolerance, "we can lose at most the last X of data." They matter because together they drive the entire DR strategy and its cost: a tight RTO (recover in minutes) and RPO of zero (lose no data) demand an expensive architecture — a hot standby or active-active with synchronous replication — while a loose RTO (hours) and RPO of a day allow cheap backup-and-restore. A payment system needs RTO of minutes and RPO of zero (never lose a transaction); a blog tolerates RTO of hours and RPO of a day. Asking for these numbers is the first step in any DR design, because everything — the strategy, the replication approach, the cost — follows from them. Building a zero-RPO system for a blog wastes money; building a day-RPO system for payments loses transactions.
2. Why is multi-AZ often sufficient, and when do you actually need multi-region? Multi-AZ deployment (spreading across isolated datacenters within one region) already provides cheap resilience against the most common infrastructure failures — a single datacenter losing power, cooling, or network — because AZs are separate failure domains close enough for fast synchronous replication, so an AZ failure leaves your service running in the others with minimal cost or latency impact. Multi-region is a large step up in cost and complexity (cross-region latency, data replication challenges, potential write conflicts, doubled infrastructure), so you adopt it only when multi-AZ isn't enough: to survive a *whole-region* failure (a cloud provider's entire region going down, which does happen — AWS us-east-1 outages are notorious), to serve users in distant geographies with low latency (avoiding the ~150ms cross-continent penalty), or because data residency laws require data to stay in a specific region. If none of those apply, multi-AZ within one region is the right default, and reaching for multi-region "to be safe" imports enormous cost and complexity for protection you may not need.
3. Why is active-active fundamentally harder than active-passive? Because of writes. In **active-passive**, only one region accepts writes; the passive region(s) receive replicated data and stand by. Since there's a single writer, there are no conflicts — the data model is straightforward multi-master-free replication. In **active-active**, multiple regions serve traffic simultaneously, which means writes happen in multiple regions — and now you have multi-master replication across a high-latency (~150ms) link, where two regions can modify the *same record* concurrently before they've synchronized. This produces **write conflicts** with no obvious resolution: which write wins? This is exactly where the CAP theorem stops being theory and becomes an unavoidable design decision — during a network partition between regions, you must choose between consistency (reject writes on one side) and availability (accept writes on both and reconcile). Reads in active-active are easy (replicate everywhere, read locally); it's the writes that force you into conflict resolution (CRDTs, vector clocks, last-write-wins), write-forwarding to a single region, region-sharded ownership, or a globally-consistent database — each with real latency, complexity, or data-loss trade-offs.
4. How does sharding writes by region sidestep the active-active conflict problem? By ensuring no two regions ever write the *same* record. Instead of allowing any region to write any record (which creates conflicts when two regions modify the same one concurrently), you assign each record a *home region* — typically based on the user's location — and route all writes for that record to its home region. EU users' data is written only in the EU region, US users' only in the US region. Since a given record has exactly one writer region, concurrent conflicting writes to the same record can't happen, so there are no conflicts to resolve — you've sidestepped the hard problem rather than solving it. Reads can still happen locally in any region (data is replicated everywhere for reading), and the only cost is that *cross-region* access to a record not in your home region is slower — but that's rare when the sharding matches usage patterns. As a bonus, this naturally satisfies data residency compliance (EU data physically lives and is written in the EU). It's the mature answer because it converts a hard distributed-systems problem (conflict resolution across regions) into a simpler routing problem (send writes to the home region), which is why it's often the best approach for active-active — you get geo-distributed writes without multi-master conflicts.
5. Why is an untested disaster-recovery plan not actually a disaster-recovery plan? Because recovery paths that aren't exercised regularly reliably fail when you finally need them, and a DR plan's entire value is being able to execute it during an actual disaster. The failure modes are numerous and only surface when tested: backups that have been silently failing for months (GitLab's 2017 data loss happened because *five* separate backup mechanisms had all quietly broken, discovered only when they urgently needed to restore); failover procedures with wrong or outdated steps; standby regions missing a critical dependency that's still single-region (so failing over the app is useless because its database or config service isn't there); recovery tooling that depends on the very system that's down (Facebook's 2021 outage was worsened because the tools to fix it, and even the building's badge readers, depended on the failed infrastructure); and capacity in the standby that turns out insufficient under real load. None of these are visible on paper — the plan looks complete — and all of them are catastrophic during a real disaster when there's no time to debug them. The only way to know a DR plan works is to *run* it: game days that actually fail over a region, and restore drills that actually restore backups into a working environment and verify the data. An unexercised plan is a hopeful document, not a capability.

Further reading

This site is open source. Improve this page.