system-design

Capacity Planning and Autoscaling

How much do you need, and how do you get more automatically? The answers involve arithmetic you already know and a few traps that catch people who trust autoscaling too much.

Prerequisites: Back-of-the-Envelope Estimation, Performance Metrics Time to read: ~16 minutes

The two questions

Capacity planning — how much infrastructure do you need? This is back-of-the-envelope estimation applied to provisioning: QPS × cost-per-request, storage growth, bandwidth. You already have the skill.

Autoscaling — how do you automatically add and remove capacity as load changes, so you’re not manually provisioning for a peak you hit twice a day?

🚨 The tension: provision too little and you fall over at peak; provision too much and you pay for idle capacity. Autoscaling promises to solve this by matching capacity to demand — but it has real limits that make capacity planning still necessary.

Capacity planning: the arithmetic

Same recipe as estimation:

Peak QPS         = daily requests / 86,400 × peak-multiplier (2-10×)
Servers needed   = peak QPS / per-server capacity
Add headroom     = servers × 1.5 (run at ~65%, not 100%)
Add redundancy   = survive N failures (N+2)
Storage          = data/day × retention × replication (×3) × index overhead
Bandwidth        = QPS × payload size

🚨 Two things people get wrong:

Peak, not average. Provisioning for average means being overloaded every peak. The peak multiplier depends on the workload — 2-3× for a globally-spread consumer app, 10×+ for a regionally-concentrated one (food delivery at dinner), 100×+ for event-driven spikes (ticket sales).

Headroom for queueing. 🚨 Run at ~65% utilization, not 100% — because queueing delay grows non-linearly as you approach saturation, and you need slack to absorb variance and survive a lost instance. Provisioning to exactly 100% means terrible p99 and no failure tolerance.

Autoscaling: how it works

Automatically adjust capacity based on metrics:

Horizontal scaling — add/remove instances (the usual and preferred way; requires stateless services). → Kubernetes HPA

Vertical scaling — resize instances (bigger machines). Limited, usually needs a restart.

What to scale on:

CPU / memory — the common default. 🚨 Scale at ~60-70%, not 90% (queueing).
Custom metrics — 🚨 often better: queue depth (scale workers by backlog), request rate, p99 latency, or a business metric. CPU isn’t always the right signal — a service bottlenecked on a downstream dependency has low CPU but needs more instances.
Scheduled — scale up before a known peak (business hours, a sale). Predictable patterns don’t need reactive scaling.

Scale-out vs scale-in asymmetry — 🚨 scale out aggressively (add capacity fast when load rises), scale in conservatively (remove slowly, to avoid thrashing and to have headroom if load returns). Removing capacity too eagerly means you scale out again moments later — flapping.

The traps (why autoscaling isn’t magic)

🚨 These are the interview-relevant limits, and trusting autoscaling blindly causes outages:

1. Lag. Autoscaling isn’t instant — new instances take seconds to start (image pull, boot, warm-up), new nodes take minutes. So autoscaling handles gradual load changes but 🚨 cannot handle sudden spikes or cascading failures (which complete in under 2 minutes — faster than instances start). For spikes you need headroom, load shedding, and queues — not autoscaling.

2. Cold starts / warm-up. 🚨 New instances start cold — empty caches, unwarmed connection pools, uncompiled hot paths (JIT). Their first requests are slow, and they hammer the database with cache misses. Scaling up under load can briefly make things worse. Mitigate with slow-start ramping, readiness probes that only pass when warm, and cache warming. → Scalability

3. The database doesn’t autoscale (easily). 🚨 You autoscale the app tier to 50 instances, and they all hammer one database with 50× the connections — which falls over. The stateful tier is the constraint, and it doesn’t scale as easily. Autoscaling the app tier without considering the database is a classic mistake. → Connection Pooling

4. Downstream dependencies. Scaling up means more load on everything downstream (databases, third-party APIs, other services). Scaling your service can just move the bottleneck — or overwhelm a dependency that can’t scale with you.

5. Cost runaway. 🚨 Autoscaling + a bug (a retry storm, a traffic flood) = a huge bill, because the system scales up to serve the bad load rather than falling over. Set maximum limits and cost alarms. → DDoS / denial of wallet

6. Flapping. Aggressive scale-in causes oscillation — scale down, load returns, scale up, repeat. Use cooldowns and conservative scale-in.

🎙️ “Autoscaling handles gradual load, but it has real limits — it lags (instances take seconds, nodes minutes), so it can’t absorb sudden spikes or cascades; new instances are cold and can briefly make things worse; and the database tier doesn’t scale with the app tier, so I’d watch connection exhaustion. I’d keep headroom for spikes and set maximum limits so a bug doesn’t scale us into a huge bill.”

This nuanced “autoscaling isn’t magic” answer is a strong signal — many candidates propose autoscaling as a cure-all.

Load testing: know your numbers

🚨 You can’t plan capacity or configure autoscaling without knowing your actual per-instance capacity and failure point. Load testing gives you these:

Find the per-instance capacity — how many QPS one instance handles before p99 degrades.
Find the breaking point — where and how it falls over (CPU? connections? memory?).
Validate autoscaling — does it trigger correctly and in time?

Types: load testing (expected load), stress testing (beyond capacity, to find the breaking point), spike testing (sudden surge, to test autoscaling lag), soak testing (sustained load, to find leaks). Tools: k6, Gatling, Locust, JMeter, wrk.

⚖️ Trade-offs

Choice	Gain	Cost
Autoscaling	Match capacity to demand; cost efficiency	Lag; cold starts; can’t handle spikes; complexity
Over-provisioning	Handles spikes instantly; simple	Pay for idle capacity
Scale on CPU	Simple default	Not always the right signal (downstream-bound services)
Scale on custom metrics	Accurate to the real bottleneck	More setup
Aggressive scale-in	Cost savings	Flapping; no headroom if load returns
Reserved capacity	Cheaper for baseline	Less flexible

In the real world

Autoscaling failing to handle a spike is a recurring incident pattern — a flash sale or a viral event spikes traffic in seconds, autoscaling can’t provision fast enough, and the system falls over before capacity arrives. The fix is always the same: pre-scale for known events, keep headroom, and use queues to absorb the burst rather than relying on reactive scaling.
The “cold instances made it worse” effect is well-documented — scaling up during a load-induced slowdown adds instances that start with cold caches and empty pools, hammering the already-stressed database, so the intervention backfires. Slow-start ramping and cache warming are the standard mitigations.
Serverless denial-of-wallet (from DDoS) is the extreme of cost runaway — a system that scales infinitely bills infinitely, so maximum concurrency and budget limits are essential.

🚨 Interview traps

Provisioning for average, not peak.
Provisioning to 100% utilization — no headroom for queueing or failures.
Treating autoscaling as instant/magic — it lags and can’t handle spikes.
Autoscaling the app tier while ignoring the database — connection exhaustion.
Ignoring cold-start / warm-up effects.
No maximum scaling limit — a bug scales you into a huge bill.
Scaling on CPU when the bottleneck is downstream.
No load testing — you don’t know your real capacity or breaking point.

🎙️ Soundbites

“Provision for peak, not average — a globally-spread app is 2-3× average, a regional one 10×, an event-driven spike 100×. And run at ~65% utilization, because queueing delay grows non-linearly near saturation and I need slack to absorb a lost instance.”
“Autoscaling handles gradual load but lags — instances take seconds, nodes minutes — so it can’t absorb sudden spikes or cascades that complete in under two minutes. For those I’d keep headroom, shed load, and use queues.”
“New instances are cold — empty caches, unwarmed pools — so scaling up under load can briefly make things worse by hammering the database with misses. I’d use slow-start ramping and readiness probes that only pass when warm.”
“The app tier autoscales; the database doesn’t. Fifty app instances hitting one database with 50× the connections is a classic outage — I’d watch connection exhaustion and put a pooler in front.”
“I’d set maximum scaling limits and cost alarms — otherwise a retry storm or a traffic flood scales us up to serve the bad load and produces a huge bill.”

🛠️ Try it

1. Find your per-instance capacity. Load-test a single instance with k6, ramping up QPS until p99 degrades. That number is what all your capacity planning divides by — and it’s usually lower than people guess.

2. Find the breaking point. Keep pushing past capacity. What breaks first — CPU, memory, connections? Knowing how it fails tells you what to scale on and where the real bottleneck is.

3. Test autoscaling lag with a spike. Configure autoscaling, then hit the service with a sudden 10× spike. Watch requests fail while new instances start up — the lag is real, and it shows why autoscaling alone can’t handle spikes.

4. Cause the cold-start problem. Scale up under load and measure the new instances’ first-request latency and the database’s cache-miss rate. Watch the scale-up briefly stress the system before the new instances warm up. Then add cache warming and compare.

Check yourself

1. Why must you provision for peak rather than average, and why not to 100%?

**For peak** because you must be able to serve your busiest moment — provisioning for average traffic means being overloaded and dropping requests during every peak, which for a consumer app is a few hours every day. The peak multiplier over average depends on the workload: a globally-distributed consumer app whose users span time zones is relatively flat (2-3× average), a regionally-concentrated app has a sharp daily peak (10×+ — food delivery at dinner), and event-driven systems spike enormously (100×+ for ticket sales or a viral moment). **Not to 100%** because utilization and latency have a non-linear relationship: as a system approaches saturation, queueing delay grows explosively (the u/(1-u) relationship — at 90% utilization, wait times are ~9× the service time), so a system running at 100% has terrible tail latency even before it fails. You also need slack to absorb normal variance and, critically, to survive losing an instance — if you're at 100% across N instances and one dies, the rest are instantly over capacity, triggering a cascade. Running at ~65% keeps latency healthy and leaves room to absorb spikes and failures.

2. Why can't autoscaling handle sudden spikes or cascading failures?

Because it's too slow to react. Autoscaling observes a metric (CPU, queue depth), decides to add capacity, and then that capacity takes time to become available: new instances take *seconds* to start (pull the image, boot, warm caches and connection pools before serving), and if the existing nodes are full, provisioning new *nodes* (machines) takes *minutes* (cloud VM creation, node join, scheduling). A sudden spike (a flash sale, a viral event) arrives in seconds and overwhelms current capacity long before new capacity comes online. A cascading failure unfolds even faster — failing instances shift their load to survivors, which then also fail, completing the collapse in under two minutes, far quicker than autoscaling can respond. Autoscaling is designed for *gradual* load changes (traffic growing over minutes to hours), where its reaction time is adequate. For spikes and cascades you need mechanisms that work instantly: pre-provisioned headroom to absorb the surge, load shedding to reject excess and protect the core, rate limiting, circuit breakers, and queues to buffer bursts. Relying on autoscaling to catch a spike is a classic cause of outages.

3. Why can scaling up under load briefly make things worse?

Because new instances start *cold* and need warming up before they're fully effective, and during that warm-up they can add stress rather than relieve it. A freshly-started instance has: empty in-process caches (so its requests miss and go to the database), unwarmed connection pools (establishing connections adds latency), and for JIT-compiled runtimes (JVM, Node), uncompiled hot code paths that run slowly until the compiler optimizes them. So when you scale up during a load-induced slowdown, the new instances serve their first requests slowly *and* hammer the already-stressed database with a burst of cache misses (since their caches are empty), which can push the database further toward the edge — the intervention meant to help briefly hurts. This is counterintuitive and catches people out: adding capacity during an incident can deepen it before it helps. Mitigations: slow-start ramping (route a gradually-increasing fraction of traffic to new instances so they warm before taking full load), readiness probes that only pass once the instance is genuinely warm (so the load balancer doesn't send traffic prematurely), pre-warming caches and connection pools on startup, and keeping enough headroom that you rarely need to scale reactively under stress in the first place.

4. Why is autoscaling the app tier while ignoring the database a classic mistake?

Because the app tier is easy to scale but the database — the stateful tier — isn't, so scaling the app tier just moves the bottleneck to the database and can take it down. When autoscaling grows your stateless app tier from 5 to 50 instances under load, each instance opens its own pool of database connections, so the database suddenly faces 50× the connection count — and databases handle a limited number of connections well (Postgres allocates a backend process per connection, thrashing beyond a few hundred), so it exhausts connections, refuses new ones, and slows to a crawl, taking down the very service you scaled. More broadly, the app tier's throughput is often ultimately limited by the database's capacity, so adding app instances past that point yields no benefit and only adds pressure. The database doesn't autoscale the way stateless instances do — you can't just add replicas instantly, sharding is a major undertaking, and vertical scaling has limits. So capacity planning must account for the whole chain: the app tier's scaling is bounded by what the database (and other downstream dependencies) can absorb. Mitigations: a connection pooler (PgBouncer) between the app tier and database to multiplex many app connections onto few database ones, read replicas to offload reads, caching to reduce database load, and recognizing that the stateful tier is usually the real scaling constraint.

5. Why do you need maximum scaling limits and cost alarms on autoscaling?

Because autoscaling responds to load by adding capacity, and it can't distinguish *legitimate* load from *bad* load — so a bug or an attack that generates excessive requests causes the system to scale up to serve it, converting what would have been an outage into a massive bill. Examples: a retry storm where a failing dependency causes clients to retry aggressively, multiplying request volume; a recursive trigger (a function that invokes itself); a traffic flood or DDoS; or a runaway loop. In a fixed-capacity system, excess load causes degradation or an outage — bad, but bounded. In an autoscaling (especially serverless) system, the platform dutifully scales up to meet the demand and bills you for all of it, with no natural ceiling — there are well-documented cases of accidental five- and six-figure bills from a single misconfiguration. This is "denial of wallet" in the extreme. So you set a **maximum** number of instances (or concurrency) so scaling can't run away, and **cost alarms/budgets** that alert (or auto-throttle) when spend spikes abnormally, so a runaway is caught in minutes rather than discovered on the invoice. The maximum limit means that under a genuine overwhelming spike you degrade gracefully (shed load) rather than scaling infinitely — which is usually the correct trade-off.