system-design

Scaling Story: 1 User → 1 Billion Users ⭐

Every technique in this repo, in the order you’d actually apply them, told as one continuous story. The single best way to understand when each tool earns its place.

Prerequisites: most of Parts 1-2; this chapter ties them together Time to read: ~25 minutes

Why this chapter

🚨 The most common way beginners fail is applying advanced techniques before they’re needed — sharding at 200 QPS, microservices for a five-person team, multi-region before product-market fit. This chapter is the antidote: it shows the ordering, so you understand that each technique solves a problem the previous stage created, and you only reach for it when the numbers demand it.

🎙️ In an interview, this ordering is exactly what scores. Starting simple and evolving under pressure (“this handles it for now; here’s what breaks first at 10× and what I’d do then”) beats drawing the final complex architecture immediately. Scaling is a journey, and knowing the stops in order is the skill.

Stage 0: One server (1 → 1,000 users)

[Users] → [One server: app + database + everything]

Everything on one machine — app, database, static files. It works. This is correct at this stage — don’t add complexity you don’t need. A monolith on one server serves thousands of users comfortably.

What breaks next: the single server becomes a single point of failure, and eventually one machine can’t hold both the app load and the database load.

Stage 1: Separate the database (1K → 10K)

[Users] → [App server] → [Database server]

🚨 The first split: move the database to its own machine. App and database have different resource profiles (app is CPU, database is memory/IO) and compete on one box. Separating them lets each be sized and scaled independently. → Databases

What breaks next: the one app server can’t handle the request volume, and it’s still a SPOF.

Stage 2: Multiple app servers + load balancer (10K → 100K)

[Users] → [Load Balancer] → [App server 1..N] → [Database]

🚨 Scale the app tier horizontally behind a load balancer. This requires making the app stateless — sessions move to a shared store, not server memory — which is the enabling change. Now you add app servers freely and survive one dying.

What breaks next: all those app servers hammer one database, and reads dominate.

Stage 3: Add a cache (100K → 500K)

[Users] → [LB] → [App servers] → [Cache (Redis)] → [Database]

🚨 Add caching — the biggest single lever. Most systems are read-heavy, and a cache with a 90%+ hit rate removes most of the database’s read load. Cheaper and higher-leverage than scaling the database. → Scaling Reads

What breaks next: even with caching, database reads (cache misses, uncacheable queries) grow; and static assets/media clog the app servers.

Stage 4: CDN + object storage (500K → 1M)

[Users] → [CDN] → [LB] → [App servers] → [Cache] → [Database]
                                       ↘ [Object Storage (S3)] for media

🚨 Serve static content and media from a CDN + object storage. Images/video don’t belong on app servers (a 1 Gbps NIC caps at ~60 images/sec); the CDN serves them near users and offloads the origin. This is a cost and latency decision as much as scale.

What breaks next: the single database is now the read bottleneck.

Stage 5: Read replicas (1M → 5M)

[Database Primary] ← writes
     ↓ replicates
[Read Replica 1..N] ← reads

🚨 Add read replicas: route writes to the primary, reads to replicas. Scales read throughput and gives failover. Handle replication lag / read-your-writes.

What breaks next: writes now bottleneck on the one primary (replicas only help reads), and slow synchronous work bogs down request latency.

Stage 6: Async processing with queues (5M → 10M)

[App] → [Queue] → [Workers]   (email, image processing, analytics, notifications)

🚨 Move slow, non-critical work off the request path onto a queue. The user doesn’t wait for the welcome email; workers process asynchronously, which also levels traffic spikes. → Scaling Writes

What breaks next: the data grows beyond one database, and write volume exceeds the primary’s capacity.

Stage 7: Shard the database (10M → 100M)

[Shard 1: users A-F] [Shard 2: G-M] [Shard 3: N-Z] ...

🚨 Shard — the biggest, most irreversible step. When one primary genuinely can’t handle the write volume or the data doesn’t fit, split it. This costs you cross-shard joins and transactions (→ sagas), forces a shard-key decision, and introduces hot shards. Only do this when the cheaper options are exhausted. → Scaling Writes

What breaks next: the monolith is now a coordination bottleneck across many teams, and different parts need different scaling.

Stage 8: Split into services (100M → 500M)

[API Gateway] → [Users svc] [Orders svc] [Search svc] [Notifications svc] ...
                each with its own datastore

🚨 Split into microservices — but note why: this is now an organizational need (many teams deploying independently), not just a scale need. You also add specialized stores per workload (search, time-series, analytics) — polyglot persistence. And now you need observability (tracing), resilience patterns, and async event-driven communication.

What breaks next: users are global, one region has high latency for distant users, and a region failure would be catastrophic.

Stage 9: Multi-region (500M → 1B+)

[Region: US] ⟷ [Region: EU] ⟷ [Region: APAC]
each: full stack, data replicated/sharded by region

🚨 Go multi-region: serve users near them (geo-latency), survive a region failure (DR), and satisfy data residency. This forces the CAP decisions to become concrete — active-active means write conflicts, best handled by sharding writes by region. The most expensive, complex stage.

And at every stage: caching layers deepen, hot keys get special handling (hybrid fan-out for celebrities), cost becomes a first-order concern, and monitoring/observability is essential throughout.

The ladder, at a glance

Stage 0: One server                          (1K)      — start simple
Stage 1: Separate the database               (10K)
Stage 2: Load balancer + app servers         (100K)    — requires statelessness
Stage 3: Cache                               (500K)    — biggest read lever
Stage 4: CDN + object storage                (1M)      — media off app servers
Stage 5: Read replicas                       (5M)      — scale reads
Stage 6: Queues + async workers              (10M)     — work off critical path
Stage 7: Shard the database                  (100M)    — the big, irreversible step
Stage 8: Microservices + polyglot            (500M)    — organizational need
Stage 9: Multi-region                        (1B+)     — geo + DR

🚨 Two crucial observations:

Almost every step is about the database (2, 3, 5, 6, 7) — the stateless app tier is easy; the stateful data layer is where scaling is hard.
Each step buys ~10× and adds complexity. You don’t skip ahead — a system at Stage 3 that jumps to Stage 9 has bought enormous cost for capacity it doesn’t need.

The interview application

🎙️ This is how to drive a design interview:

“I’d start with a monolith, one database, and a cache — that handles our estimated scale for a while. As we grow, the first bottleneck is database reads, so read replicas and heavier caching. Writes bottleneck next, so async processing via queues, and eventually sharding — though I’d exhaust vertical scaling and caching first because sharding is irreversible. I’d only split into services when team coordination becomes the bottleneck, and multi-region only for geo-latency or DR. At each stage I’d point out what breaks first and what I’d do about it.”

🚨 This — starting simple and evolving under stated pressure, naming what breaks at each 10× — is the single most effective interview technique. It shows judgment, not just tool knowledge.

In the real world

Every large system went through roughly this progression — Twitter, Facebook, Instagram, Airbnb all started as monoliths on modest infrastructure and evolved through these stages under growth pressure. None started at Stage 9. The public engineering-blog histories of these companies trace exactly this ladder.
Instagram scaled to tens of millions of users on a small team with a famously simple stack (Django, Postgres, Redis, a handful of stages), demonstrating how far the early stages go before you need the complex ones — and how premature complexity is the bigger risk than under-engineering.
The premature-scaling failure — startups building Stage 9 architecture for Stage 2 traffic — is a recognized way teams waste their runway on operational complexity instead of finding product-market fit. The ladder is as much about not skipping ahead as about knowing the steps.

🚨 Interview traps

Jumping to the final complex architecture instead of starting simple and evolving.
Sharding/microservices/multi-region too early — the premature-scaling mistake.
Not knowing the ordering — each stage solves the previous stage’s bottleneck.
Forgetting statelessness enables Stage 2 (horizontal app scaling).
Not noticing most stages are about the database.
Not naming what breaks next — the thing that makes you sound like you’ve operated systems.

🎙️ Soundbites

“Scaling is a journey — I’d start with a monolith, one database, and a cache, and evolve under pressure. Read replicas when reads bottleneck, queues when slow work bogs down requests, sharding only when one primary genuinely can’t keep up — because sharding’s irreversible.”
“Almost every scaling step is about the database — the stateless app tier is easy to scale horizontally; the stateful data layer is where it gets hard, which is why caching, replicas, and sharding dominate the ladder.”
“Each step buys about 10× and adds complexity, so I wouldn’t skip ahead — jumping to multi-region microservices for Stage 2 traffic is enormous cost for capacity we don’t need. Premature scaling is the bigger risk than under-engineering.”
“Microservices here would be an organizational decision — many teams deploying independently — not a scale one. A monolith scales horizontally fine; I’d split when team coordination is the bottleneck.”
“At each stage I’d name what breaks first at 10× and what I’d do — that’s how I’d drive the whole design rather than drawing the final architecture up front.”

🛠️ Try it

1. Walk a real product up the ladder. Pick an app (Instagram, Uber). For each stage, write what their bottleneck was and which technique solved it. Notice how each stage’s solution created the next stage’s problem — that causal chain is the whole point.

2. Practice the interview narration. Take any case study and narrate the scaling journey out loud: start simple, then “at 10× this breaks, so I’d add X; then Y breaks, so Z.” This narration is the highest-value interview skill — practice it until it’s automatic.

3. Find the premature-scaling smell. Look at an over-engineered design (real or a case study) and identify which techniques were applied before their stage. Recognizing ‘this is Stage 7 complexity for Stage 2 traffic’ is a real judgment skill.

4. Build stages 0-3. Actually build a tiny app and evolve it: one server → separate database → load balancer + two app servers (feel the statelessness requirement) → add a cache. Experiencing the statelessness change when you add the second app server makes Stage 2 permanent.

Check yourself

1. Why is knowing the ordering of scaling techniques more valuable than knowing the techniques themselves?

Because the most common and costly mistake isn't ignorance of the techniques — it's applying them at the wrong time, and specifically *too early*. Sharding a database serving 200 QPS, adopting microservices for a five-person team, or going multi-region before finding product-market fit all import enormous complexity, cost, and operational burden to solve problems the system doesn't have yet — wasting the scarce resources (engineer-time, runway) that should go toward the actual current problem. Each scaling technique solves a specific bottleneck that a *previous* stage created, and applying it before that bottleneck exists means paying its full price (sharding's lost joins and transactions, microservices' distributed-systems tax, multi-region's write-conflict complexity) for no benefit. Knowing the ordering means you understand *when* each technique earns its place — you apply caching before sharding because it's cheaper and higher-leverage, you exhaust vertical scaling before splitting the database because sharding is irreversible, you split into services when team coordination (not throughput) becomes the bottleneck. This judgment — recognizing which stage a system is at and applying only the appropriate level of complexity — is what distinguishes an engineer who's operated systems from one who's memorized architectures, and in an interview it's what lets you drive a design by starting simple and evolving under stated pressure rather than drawing an over-engineered final state.

2. Why is making the app tier stateless the enabling change for horizontal scaling (Stage 2)?

Because horizontal scaling means putting multiple app servers behind a load balancer that distributes requests across them, and that only works if *any* server can handle *any* request. If a server holds client-specific state in its memory — a user's session, a shopping cart, an in-progress workflow — then that user's requests must all return to that *specific* server (via sticky sessions), which breaks the free load distribution, loses the state when that server dies or is replaced, and disrupts users on every deploy. Making the app tier stateless — moving session and any per-client state out to a shared store (Redis) or into a token the client carries (JWT), and putting files in object storage rather than local disk — means the servers hold nothing client-specific, so the load balancer can send any request to any server, you can add and remove servers freely (autoscaling), a server crash affects only its in-flight requests, and deploys don't disrupt users. Statelessness is therefore the *precondition* for Stage 2: you can't just add app servers behind a load balancer if they're stateful, because the state ties requests to specific machines. This is why "push state out of the app tier" is the first move when scaling horizontally, and why it recurs throughout the ladder (stateless containers, stateless services).

3. Why is caching (Stage 3) added before scaling the database with replicas (Stage 5)?

Because caching is a bigger, cheaper lever for the read-heavy load that dominates most systems. Most systems are heavily read-dominated (often 100:1), so the database's read load is usually the first scaling pressure — and a cache attacks it more effectively and cheaply than adding database replicas. A cache with a 90-95% hit rate removes 90-95% of the database's read load by serving repeated reads from memory (nanoseconds to sub-millisecond) rather than querying the database at all, using a single Redis instance or even in-process memory — a 10-20× reduction in database read load for minimal cost. Read replicas, by contrast, still involve full database queries (just distributed across more machines), cost more (running additional database instances), and add replication lag and its consistency complications. So caching gives a larger reduction in database load for less infrastructure, which is why it's applied first — it may push the need for replicas out significantly or eliminate it for a while. Replicas come *later* (Stage 5) for what still misses the cache and requires a real database query, and for read failover. The ordering reflects the general "do less work before doing more work faster" principle: caching eliminates database reads entirely (do less), while replicas spread the reads you can't eliminate across more machines (do the remaining work in parallel) — you do the elimination first because it's cheaper and higher-impact.

4. Why is sharding (Stage 7) described as the biggest, most irreversible step, and why do it late?

Because sharding — splitting the data across multiple independent databases so each handles a subset — fundamentally and permanently changes what your data layer can do, and the change is extremely hard to undo. It's the biggest step because it removes capabilities the rest of the system depends on: cross-shard *joins* become impossible (you must denormalize or join in application code), cross-shard *transactions* become impossible (you need sagas with compensating actions and eventual consistency), global uniqueness constraints and auto-increment IDs break (you need distributed ID generation), aggregations become scatter-gather across all shards, and you inherit hot-shard problems and rebalancing complexity. It's the most irreversible because un-sharding means merging data back together and rewriting all the application logic that was adapted to the sharded model — a massive migration that few attempt. And the shard-key choice is a decision you effectively can't change without re-sharding everything. You do it late — only after exhausting vertical scaling (one primary handles tens of thousands of writes/second), batching, async write buffering, an LSM storage engine, and dropping unused indexes — precisely *because* it's so costly and irreversible: those cheaper options often buy years of runway, and committing to sharding's permanent loss of joins and transactions before you genuinely need it means paying an enormous, unrecoverable price for capacity you could have gotten more cheaply. The rule is: shard only when one primary genuinely cannot handle the write volume or the data truly doesn't fit, and never as a first resort.

5. Why does the chapter note that most scaling stages are "about the database," and what does that imply?

Because when you trace the ladder, the majority of stages exist to address the *stateful data layer* rather than the stateless application tier: separating the database (Stage 1), adding a cache to offload it (Stage 3), read replicas to scale its reads (Stage 5), queues to buffer and batch its writes (Stage 6), and sharding to split it when one machine can't hold it (Stage 7) — five of the nine stages are directly about keeping the database from being the bottleneck. This reflects the fundamental asymmetry of scaling: the stateless app tier is *easy* to scale — make it stateless and add machines behind a load balancer, essentially a solved problem — while the stateful data layer is *hard*, because data has a single source of truth that can't be freely duplicated (reads can be copied to caches and replicas, but writes must be coordinated at one place, and eventually the data itself must be split, sacrificing joins and transactions). The implication is that when you design or scale a system, the database is almost always where the real difficulty and the consequential decisions lie — the app tier is largely a non-issue, and effort spent scaling it is usually misplaced compared to effort spent on the data layer's read scaling (caching, replicas), write scaling (batching, async, sharding), and consistency trade-offs. It's why "what's the database strategy?" is often the crux of a design, why the read/write ratio is the first thing to compute (it dictates the data-layer approach), and why in an interview, spending your time on the data layer rather than the app tier reflects correct prioritization. The stateless parts scale almost for free; the stateful parts are the whole game.