system-design

Scalability: Vertical vs Horizontal

Adding capacity sounds simple. The reason it isn’t is that some things can be duplicated and some things can’t — and knowing which is which is most of system design.

Prerequisites: Performance Metrics Time to read: ~20 minutes

The problem

Your service handles 1,000 QPS. You need 100,000. What do you do?

There are exactly two options — get a bigger machine, or get more machines — and a third thing people forget: make the work smaller.

The interesting part isn’t the choice. It’s the consequences. Going from one machine to many changes the nature of your system: state has to live somewhere shared, failures become partial rather than total, and problems that were impossible (two servers disagreeing) become routine.

Vertical scaling (scale up)

Buy a bigger machine. 4 cores → 64 cores. 16 GB → 1 TB of RAM.

Why it’s better than beginners think:

Zero code changes. Often a single line in a config file, or an instance-type change and a reboot.
No distributed systems problems. No consistency issues, no network partitions, no coordination. Everything a single process does is atomic and ordered for free.
Modern machines are enormous. AWS offers instances with 448 vCPUs and 24 TB of RAM. A single well-tuned Postgres on good hardware handles tens of thousands of QPS and several TB of data.

📐 Reality check that changes people’s instincts: Stack Overflow served ~2 billion page views a year from nine web servers and one SQL Server with plenty of headroom. Most systems you’ll ever build will never outgrow a single large database.

Where it stops working:

A hard ceiling. You eventually reach the biggest machine that exists.
Cost is super-linear. Doubling the machine typically more than doubles the price. The top of the range is brutally expensive per unit of compute.
Single point of failure. One machine, one power supply, one fate. No redundancy at all.
Downtime to resize. Usually a reboot, sometimes a migration.
Diminishing returns. 64 cores don’t give 64× the throughput if your code has a global lock, a single-threaded hot path, or is I/O-bound.

Horizontal scaling (scale out)

Add more machines and spread the load.

What you gain:

No ceiling. Need more? Add more.
Fault tolerance. One machine dies, the rest serve.
Commodity hardware. Ten cheap machines usually cost less than one huge one.
Elasticity. Scale up for the evening peak, down overnight. This is where the cloud earns its price.
Rolling deploys. Update a few at a time with no downtime.

What it costs you — and this is the real content of this chapter:

State becomes a problem. Covered below; it’s the whole game.
The network is now in your critical path, with all its latency and failure modes. → Fallacies
Partial failure. Not “up or down” but “3 of 10 nodes are behaving strangely.” Much harder to reason about and to debug.
Coordination costs. Anything requiring agreement between machines gets slow and complicated. → Consensus
Operational overhead. Service discovery, config management, distributed tracing, and a deployment pipeline that handles fleets.

The comparison

	Vertical	Horizontal
Complexity	Low	High
Ceiling	Hard limit	Effectively none
Cost curve	Super-linear	Roughly linear
Fault tolerance	None	Built in
Code changes	Usually none	Statelessness required
Failure mode	Total	Partial
Deployment	Downtime	Rolling, zero-downtime
Best for	Databases, early-stage products, stateful systems	Stateless app tiers, caches, workers

🎙️ The answer interviewers want to hear: “I’d scale vertically first because it’s simpler and we’re nowhere near the ceiling, and design the app tier to be stateless so we can scale horizontally the moment we need to. The database is the piece that will force the harder conversation.”

That sentence shows you know both, know the ordering, and aren’t over-engineering.

The real dividing line: stateless vs stateful

This is the most important idea in the chapter.

Stateless = the server keeps nothing about a client between requests. Every request carries what it needs. Any instance can serve any request.

Stateful = the server holds something (a session, a connection, a file being uploaded, a game room) that only exists there.

Stateless things scale horizontally almost for free. Stateful things do not.

❌ STATEFUL APP SERVER
   User logs in → session stored in server A's memory
   Next request hits server B → "who are you?" → logged out
   Server A dies → session gone
   Add a server → it has no sessions → uneven load

✅ STATELESS APP SERVER
   User logs in → session stored in Redis (shared)
   Any request, any server → looks up session in Redis → works
   Any server dies → no user impact
   Add a server → it works immediately

The universal move: push state out of the app tier.

State	Move it to
Sessions	Redis, or a signed token the client carries (JWT)
Uploaded files	Object storage (S3), never local disk
Background jobs	A queue
Cached computation	A shared cache
Application data	The database (and then you scale that, which is the hard part)

You haven’t eliminated the state problem — you’ve concentrated it into systems built to handle it. That’s the win: one hard problem in one place, instead of a hard problem in every service.

Sticky sessions: the tempting wrong answer

Sticky sessions (session affinity) route a user always to the same server, so in-memory state works.

⚖️ It’s a real tool, but understand what it costs: uneven load distribution (some servers get the heavy users), lost state when a server dies, disruption on every deploy, and it makes autoscaling much less effective (you can’t freely remove a machine).

Use it when you genuinely must — WebSocket connections have to live somewhere — and pair it with a shared registry so you can still route messages to the right node. Don’t use it as a substitute for externalizing session data.

The scaling ladder

Systems grow through predictable stages. Knowing the order lets you say “we’re at stage 4, so the next thing that breaks is X.”

flowchart TB
    S1["1 · Single server<br/>app + DB together"] --> S2["2 · Separate the database"]
    S2 --> S3["3 · Add a cache"]
    S3 --> S4["4 · Multiple app servers + load balancer<br/>(requires stateless app tier)"]
    S4 --> S5["5 · Read replicas"]
    S5 --> S6["6 · CDN + object storage for static/media"]
    S6 --> S7["7 · Async work → queues + workers"]
    S7 --> S8["8 · Shard the database"]
    S8 --> S9["9 · Split into services"]
    S9 --> S10["10 · Multi-region"]

Two things to notice:

Almost every step is about the database. The app tier is the easy part. Steps 2, 3, 5, 6, 7, and 8 are all ways of keeping the database from being the bottleneck. That’s not a coincidence: the stateful component is always where scaling gets hard.
Each step buys roughly 10× and adds complexity. Don’t skip ahead. A system at stage 3 that jumps to stage 9 has bought enormous operational cost for capacity it doesn’t need.

→ Full walkthrough with numbers: 1 user → 1 billion users

Why scaling isn’t linear: Amdahl and Universal Scalability

Amdahl’s Law. If 5% of your work is inherently serial, then even with infinite parallelism your maximum speedup is 20×. The serial fraction dominates.

Speedup = 1 / (S + P/N)      S = serial fraction, P = parallel fraction, N = workers

S = 0.05:   N=10 → 6.9×    N=100 → 16.8×    N=∞ → 20×

In practice, it’s worse than Amdahl — because adding machines adds coordination, which is work that didn’t exist before. The Universal Scalability Law adds a crosstalk term, and the result is a curve that rises, flattens, and then goes back down.

Throughput
    │       ╭─────╮
    │     ╭─╯      ╰──╮        ← adding more nodes now makes it SLOWER
    │   ╭─╯            ╰───
    │ ╭─╯
    │╭╯
    └──────────────────────► Nodes

🚨 This is real and people hit it. Add nodes to a database cluster and throughput can drop, because every node now coordinates with every other node. If an interviewer asks “what if we just add more servers?”, the sophisticated answer is: “up to a point — beyond that, coordination overhead grows faster than capacity, and we’d need to reduce coordination instead, by sharding or partitioning the work.”

Where the serial/coordination fraction hides: a shared database, a distributed lock, a leader node, a global sequence generator, cache invalidation broadcasts, and cross-shard transactions.

The third option nobody mentions: do less work

Before scaling anything, consider making the work smaller. This is frequently the cheapest win and almost nobody proposes it in interviews.

Add an index. A query going from 2,000 ms to 2 ms is a 1,000× capacity increase for free.
Cache. Serve from RAM instead of recomputing.
Batch. One query instead of 50.
Compress. Less bandwidth, less I/O.
Precompute. Move work from read time (frequent) to write time (rare).
Delete data. Retention policies. Nobody queries the 2019 logs.
Say no. Does this endpoint need to be real-time? Would a 5-minute-stale answer do?

🎙️ “Before adding capacity I’d check whether we’re doing unnecessary work — an N+1 or a missing index can cost more than the entire traffic increase we’re worried about.”

⚖️ Trade-offs

Decision	Gain	Cost
Vertical first	Simplicity, no distributed problems	Ceiling, SPOF, expensive at the top
Horizontal	Unlimited capacity, fault tolerance	Statelessness required; network in the path; partial failure
Stateless app tier	Free horizontal scaling	State moves to Redis/DB, which you now must scale
Sticky sessions	Keeps in-memory state working	Uneven load, state loss on failure, painful deploys
Sharding	Unlimited data scale	No cross-shard joins or transactions; rebalancing pain
Autoscaling	Cost efficiency	Scale-up lag (cold starts), thrashing, harder capacity reasoning

In the real world

Stack Overflow ran for years on a handful of servers and one very large SQL Server, serving billions of page views. Their published architecture is a standing argument for vertical scaling plus caching, and a useful counterweight to microservices enthusiasm.
Shopify scales horizontally by pods — self-contained units each with their own database shard, serving a subset of merchants. A pod failure affects only its merchants. This is horizontal scaling applied to the whole stack, not just the app tier.
Discord moved message storage from MongoDB to Cassandra to ScyllaDB as their write volume outgrew each. Each migration is an instance of hitting a scaling ceiling and choosing a system with a different set of trade-offs.

🚨 Interview traps

Jumping straight to horizontal scaling and microservices. Estimate first. If the answer is 200 QPS, one server is correct, and saying so is a strength.
Not mentioning statelessness when you propose adding app servers. It’s the enabling precondition; state it.
Forgetting the database. “We’ll add 50 app servers” — pointing at the same database, which now gets 50× the connections and falls over. The database is almost always the real constraint.
Assuming linear scaling. 10 servers ≠ 10× throughput, and beyond a point more can be worse.
Ignoring “do less work.” Often the cheapest, fastest fix.

🎙️ Soundbites

“I’d scale up first — we’re far from the ceiling and it costs us nothing in complexity — but I’d keep the app tier stateless so scaling out is a config change when we need it.”
“Adding app servers doesn’t help if they all hit one database. The real question is how we scale the stateful layer, and that’s where I’d spend our design time.”
“Sticky sessions would work, but they cost us even load distribution and make deploys disruptive. I’d rather put sessions in Redis and keep every instance interchangeable.”
“There’s a coordination cost to adding nodes — past a certain point throughput flattens or drops. If we get there, the fix is reducing coordination through partitioning, not adding hardware.”

🛠️ Try it

1. Break statefulness, then fix it. Build a tiny app that stores the login session in an in-process dictionary. Run two instances behind Nginx round-robin. Log in and refresh a few times — watch yourself get randomly logged out. Now move sessions to Redis and watch it work. This takes 30 minutes and makes the concept permanent.

2. Find the non-linearity. Load-test a service with 1, 2, 4, and 8 worker processes against a shared database. Plot throughput vs workers. You will not get a straight line — find where it bends and work out what the shared bottleneck is.

3. Do less work. Take a slow query in any toy database with a million rows. Time it. Add the right index. Time it again. Compute how many servers that index just saved you.

Check yourself

1. Why is "stateless" the precondition for horizontal scaling?

Because if a server holds client-specific state, requests from that client must return to that specific server. That breaks free load balancing, makes instance death user-visible, makes adding capacity ineffective (new servers have no state), and makes deploys disruptive. Statelessness means any instance can serve any request, which is what lets you add, remove, and lose machines freely.

2. You add 50 app servers and performance gets worse, not better. Give two plausible explanations.

(1) **Database connection exhaustion** — 50 servers × a 100-connection pool each = 5,000 connections to a database that handles a few hundred well. The database spends its time context-switching and everything slows. (2) **Coordination overhead** — more nodes contending on a shared lock, a hot row, or cache-invalidation chatter (the USL downturn). Also plausible: the load balancer or a shared downstream dependency is now the bottleneck, or cold caches on the new instances are hammering the origin.

3. When is vertical scaling the right answer, despite its ceiling?

When you're far from the ceiling and simplicity has real value: early-stage products, the database tier (which is the hardest thing to scale out), workloads that are genuinely hard to partition, and any situation where the engineering cost of distribution exceeds the hardware cost. Given that single machines now reach hundreds of cores and terabytes of RAM, this covers far more real systems than most engineers assume.

4. What does Amdahl's Law tell you about a system where 10% of the work is serial?

Maximum speedup is 1/0.10 = **10×**, no matter how many machines you add. At 100 machines you get ~9.2×; at 1,000 you get ~9.9×. The lesson is that you should hunt for and eliminate the serial fraction — the global lock, the single sequence generator, the one shared table — rather than adding hardware, because after a point hardware does essentially nothing.

5. Your app tier autoscales beautifully, but every deploy causes a latency spike. Why?

Cold caches and cold runtimes. New instances start with empty in-process caches, unwarmed connection pools, and (for JVM/JIT languages) uncompiled hot paths, so their first requests are slow and they also hammer the database with cache misses. Fixes: gradual traffic ramp for new instances (slow start on the load balancer), cache warming before accepting traffic, readiness probes that only pass once warm, and deploying a smaller fraction at a time.