What “99.99% uptime” actually costs, why more components means less availability, and why recovering fast beats failing rarely.
Prerequisites: Scalability Time to read: ~18 minutes
“The system should be highly available.” Every requirements document says this. It means nothing until you attach a number, because the difference between 99% and 99.99% is the difference between a weekend project and a team of engineers with a pager rotation.
And there’s a subtlety that surprises people: adding components to a system usually makes it less available, not more — unless you add them in a specific way. Understanding why is most of this chapter.
Availability = the fraction of time the system is usable.
Availability = uptime / (uptime + downtime)
| Nines | Availability | Downtime/year | Downtime/month | Downtime/week |
|---|---|---|---|---|
| One nine | 90% | 36.5 days | 3 days | 16.8 hours |
| Two nines | 99% | 3.65 days | 7.2 hours | 1.7 hours |
| Three nines | 99.9% | 8.76 hours | 43.8 minutes | 10.1 minutes |
| Four nines | 99.99% | 52.6 minutes | 4.4 minutes | 1 minute |
| Five nines | 99.999% | 5.26 minutes | 26 seconds | 6 seconds |
| Six nines | 99.9999% | 31.5 seconds | 2.6 seconds | 0.6 seconds |
Memorize the shape: each nine is 10× less downtime and roughly 10× more expensive.
📐 What each tier actually means in practice:
🚨 Interview trap: when someone says “highly available,” ask which nines, and then push back if the number is unreasonable. Proposing 99.99% for an internal admin dashboard is over-engineering; saying “99.9% seems right here — do we agree a 45-minute monthly outage is acceptable?” shows judgment.
Three different words that get used interchangeably and shouldn’t be.
| Term | Question it answers | Failure looks like |
|---|---|---|
| Availability | Can I reach it right now? | 503 errors, timeouts |
| Reliability | Does it behave correctly over time? | Wrong answers, corrupted data, dropped messages |
| Durability | Will my data still be there? | Permanent data loss |
A system can be available and unreliable: it responds to every request, quickly, with wrong answers. Arguably worse than being down, because nobody notices.
Durability is measured separately and in more nines. S3 advertises 99.999999999% (eleven nines) of durability — meaning if you store 10 million objects, you’d expect to lose one every 10,000 years — while offering 99.99% availability. The distinction matters: your data is safe, but you might not be able to read it for a few minutes.
MTBF and MTTR:
MTBF = Mean Time Between Failures (how often it breaks)
MTTR = Mean Time To Recovery (how long to fix it)
Availability = MTBF / (MTBF + MTTR)
🚨 The key insight: you can improve availability by increasing MTBF or by decreasing MTTR — and decreasing MTTR is almost always cheaper.
📐 A system failing once a month and taking 4 hours to fix: 99.4%. The same system, still failing once a month, but recovering in 2 minutes: 99.995%. You made it four and a half nines without preventing a single failure.
This is why the industry invests so heavily in automated rollback, health checks, circuit breakers, feature flags, and good runbooks. Preventing all failures is impossible. Recovering in seconds is achievable.
🎙️ “Rather than trying to prevent this failure mode entirely, I’d focus on detecting it in seconds and failing over automatically — MTTR is the cheaper lever.”
This is the part most people get wrong.
If your request needs the load balancer AND the app server AND the database, all must be up:
A_total = A₁ × A₂ × A₃
Three components at 99.9% each:
0.999³ = 0.997 = 99.7% ← worse than any individual component!
Ten components at 99.9% each:
0.999¹⁰ = 0.990 = 99.0% ← 3.65 days of downtime per year
Every dependency you add makes you less available. A microservice architecture with 30 services in a request path, each at 99.9%, gives you 97% — 11 days of downtime a year.
🚨 This is the single strongest argument against gratuitous microservices, and bringing it up unprompted is a real senior signal. It’s also why circuit breakers, timeouts, and graceful degradation exist: they let you survive a dependency being down instead of inheriting its failure.
If any one of N redundant replicas can serve the request:
A_total = 1 − (1 − A)ᴺ
Two servers at 99% each: 1 − 0.01² = 99.99%
Three servers at 99%: 1 − 0.01³ = 99.9999%
Two servers at 99.9%: 1 − 0.001² = 99.9999%
Redundancy is extraordinarily powerful. Two mediocre machines beat one excellent one.
flowchart LR
C[Client] --> LB[Load Balancer<br/>2× redundant<br/>99.99%]
LB --> A1[App 1]
LB --> A2[App 2]
LB --> A3[App 3]
A1 --> D[(Database<br/>primary + replica<br/>99.95%)]
A2 --> D
A3 --> D
App tier (3 in parallel at 99% each): 1 − 0.01³ = 99.9999%
LB (redundant pair): 99.99%
Database (primary + replica failover): 99.95%
Total (series): 0.999999 × 0.9999 × 0.9995 = 99.94%
Notice where the availability went. The app tier is essentially perfect; the database dominates. That’s the general pattern: your availability is set by your least-redundant stateful component. Optimizing anything else is wasted effort.
🎙️ “The app tier is easy to make redundant. Our availability ceiling is the database, so that’s where the design attention should go — failover time, replica lag, and whether we can serve reads during a primary failure.”
The arithmetic above assumes independent failures. In reality they aren’t:
📐 So “three replicas at 99% gives 99.9999%” is a ceiling, not a promise. Real multi-instance services land closer to 99.9–99.99%, because correlated failure dominates.
Design implications:
Single point of failure (SPOF). Any component whose failure takes everything down. Walk your architecture diagram and ask of every box: “what if this dies?” Common ones people miss: the load balancer itself, DNS, the config service, the message broker, the auth service, and the shared Redis holding all sessions.
Cascading failure. One component fails → its load shifts to others → they fail → everything is down. The classic sequence: one of five servers dies, the other four now get 25% more traffic, they saturate, and the whole fleet collapses in ninety seconds. Mitigation: run with headroom (see utilization), rate limit, shed load, and use circuit breakers.
Retry storms. A service gets slow → clients retry → 3× the load → it gets slower → more retries → total collapse. Retries turn a degradation into an outage. Mitigation: exponential backoff with jitter, retry budgets (cap retries at ~10% of traffic), and circuit breakers. → Retries & Timeouts
Gray failure. Not down, not up — slow, or wrong for 5% of requests, or working for some users
only. Health checks pass. Monitoring looks fine. These are the hardest and longest outages.
Mitigation: health checks that exercise a real code path (not /ping returning 200), and
per-endpoint/per-shard metrics rather than fleet aggregates.
Thundering herd. Everything retries or refreshes at the same instant. → Thundering Herd
| Technique | Effect |
|---|---|
| Redundancy (N+1, N+2) | The foundation. Always have more capacity than you need |
| Health checks + automatic removal | Cuts MTTR from minutes to seconds |
| Multi-AZ deployment | Survives a datacenter failure. Cheap. Do it by default |
| Multi-region | Survives a region failure. Expensive and complicated |
| Graceful degradation | Serve a reduced experience rather than an error page |
| Circuit breakers | Contain a failing dependency instead of inheriting its failure |
| Load shedding | Reject some traffic to keep serving the rest |
| Canary / rolling deploys | Bad code affects 5% of users, not 100% |
| Feature flags | Turn off a broken feature in seconds without a deploy |
| Chaos engineering | Find the failure modes before they find you |
Graceful degradation deserves emphasis because it’s cheap and interviewers love it. Amazon’s product page: if the recommendations service is down, the page still renders and sells the product — minus a widget. If reviews are down, same. The core transaction never depends on the periphery.
🎙️ “I’d make the recommendation call non-blocking with a 50 ms timeout and an empty fallback. A degraded page beats a 500, and it means recommendations being down doesn’t count as an outage.”
Error budget: if your SLO is 99.9%, you may be unavailable 0.1% of the time — about 43 minutes a month. That’s not failure, it’s budget. Spend it on shipping. If you’re well under budget, you’re being too conservative and should ship faster; if you’ve blown it, freeze features and fix reliability.
This reframes reliability from “never break” into a resource-allocation decision, which is why it works as an organizational tool. → SLIs, SLOs, SLAs
1. Find the SPOFs. Take any architecture you’ve built or seen. List every component. For each, write what happens when it dies and how long recovery takes. The ones with no answer are your real risks.
2. Do the arithmetic. For a system with a load balancer (99.99%), three app servers (99% each), a cache (99.9%), and a database (99.95%), compute total availability. Then compute it again if the cache is optional (the app works without it, just slower). Notice how much making one dependency optional buys you.
3. Kill something. Run three service instances behind a load balancer with health checks. Kill one mid-load-test. Measure exactly how many requests failed and how long until traffic rerouted. That number is your MTTR for that failure mode. Then tune the health check interval and measure again.