Three patterns that stop one failing component from taking down everything that depends on it. They’re what separates a degraded system from an outage.
Prerequisites: Failure Modes, Performance Metrics Time to read: ~24 minutes
Your service calls a recommendations service. It starts taking 30 seconds instead of 30 milliseconds.
Your service has 200 worker threads.
Requests arrive at 500/second.
Each one calls recommendations and waits 30 seconds.
Within 0.4 seconds, all 200 threads are blocked.
Your service now returns nothing — including for endpoints
that don't touch recommendations at all.
🚨 A non-critical dependency took down your entire service. And notice: your service is healthy, your database is healthy, and 95% of your functionality doesn’t need recommendations. You failed because you had no mechanism to contain the damage.
The three patterns below address this from different angles.
🚨 Before any pattern below: every network call must have a timeout. A call without one waits forever, and “forever” is how threads leak.
📐 Setting them properly:
Timeout ≈ p99.9 of the dependency + a small margin
Not the average — that fails most slow-but-succeeding requests. Not 30 seconds “to be safe” — that’s long enough to exhaust your thread pool.
Timeouts must decrease as you go inward:
Client 30s → Gateway 25s → Service A 20s → Service B 15s → Database 10s
Otherwise an inner component keeps working on a request nobody is waiting for, burning resources for a response that will be discarded.
🧠 Mental model: an electrical circuit breaker. When current exceeds a threshold, it trips and cuts the circuit — protecting the wiring and giving you a chance to fix the fault. It doesn’t try to keep supplying power to a short circuit.
The insight: when a dependency is clearly broken, calling it is worse than not calling it. Each call consumes a thread for the full timeout, and adds load to something already struggling.
stateDiagram-v2
[*] --> Closed
Closed --> Open: failure rate > threshold
Open --> HalfOpen: after the reset timeout
HalfOpen --> Closed: trial requests succeed
HalfOpen --> Open: a trial request fails
Closed (normal) — requests pass through; failures are counted.
Open (tripped) — 🚨 requests fail immediately without being attempted. No thread is consumed, no timeout is waited out, and the struggling dependency gets breathing room. The caller returns an error or a fallback in microseconds instead of 30 seconds.
Half-open (testing) — after a reset timeout, allow a small number of trial requests. Success → close. Failure → open again.
class CircuitBreaker:
def __init__(self, failure_threshold=0.5, min_requests=20, reset_timeout=30):
self.state = "closed"
self.failures = self.total = 0
self.opened_at = None
def call(self, fn, fallback=None):
if self.state == "open":
if monotonic() - self.opened_at > self.reset_timeout:
self.state = "half_open"
else:
return fallback() # fail fast — no call made
try:
result = fn()
self._on_success()
return result
except Exception:
self._on_failure()
if fallback:
return fallback()
raise
Fail on a rate, not a count. “5 failures” trips on 5 failures out of 5 (broken) and 5 out of 50,000 (fine). Use a percentage over a rolling window, with a minimum request threshold so low-traffic periods don’t trip on noise.
Count slow responses as failures. A dependency at 25 seconds under a 30-second timeout is “succeeding” and destroying you. Trip on latency, not just errors.
Don’t trip on 4xx. A client sending malformed requests isn’t a dependency failure. Count 5xx, timeouts, and connection errors.
Half-open must be limited. Letting full traffic through on the first success re-kills a recovering dependency instantly. Allow a trickle.
Per-dependency, and often per-endpoint. One breaker for a whole service means a broken endpoint trips the healthy ones too.
Have a fallback. A tripped breaker with no fallback just fails faster. The value comes from
return cached_results() or return [].
Tools: Resilience4j (Java), Polly (.NET), gobreaker (Go), or built into Envoy / service mesh, which is increasingly where this lives.
🧠 Mental model: ship compartments. A hull breach floods one compartment, not the vessel.
Isolate resources per dependency, so one saturated dependency can’t consume everything.
# ❌ One shared pool — the slow dependency starves everything
executor = ThreadPoolExecutor(max_workers=200)
# ✅ Bounded per dependency
payments = ThreadPoolExecutor(max_workers=50) # critical — more capacity
recommendations = ThreadPoolExecutor(max_workers=10) # can NEVER take more than 10
search = ThreadPoolExecutor(max_workers=20)
Now recommendations being down consumes at most 10 threads. The other 190 keep serving.
Levels of isolation, from cheap to expensive:
| Level | Isolation | Cost |
|---|---|---|
| Semaphore (concurrency limit) | Caps in-flight calls | Nearly free |
| Separate thread pools | True isolation, including from slow calls | Thread overhead |
| Separate connection pools | Per-dependency connection limits | Config only |
| Separate service instances | Full isolation | 2× infrastructure |
| Separate clusters per tenant | Complete blast-radius isolation | Expensive |
⚖️ Semaphores are usually enough and are much cheaper than thread pools. They limit concurrency without a separate thread for each call, which matters in async runtimes where a thread pool per dependency is wasteful.
🚨 The connection pool version is the one people miss. If your service has one 100-connection pool to Postgres and one slow query type consumes them all, every other query queues. Separate pools for OLTP and reporting queries is a cheap, effective bulkhead. → Connection Pooling
The idea: when you can’t keep up, tell someone, rather than silently queueing until you die.
🚨 The failure this prevents:
Requests arrive at 1,000/s. You process 500/s.
Unbounded queue → grows forever → memory exhausted → crash
Or: the queue reaches 60 seconds deep, and every response
arrives after the client already gave up.
You're doing 100% of the work and delivering 0% of the value.
Bounded queues plus an explicit rejection policy is the fix, and the crucial realization is that rejecting work quickly is a feature, not a failure.
# ❌ Unbounded — memory grows until the process dies
queue = Queue()
# ✅ Bounded, with an explicit policy
queue = Queue(maxsize=1000)
try:
queue.put_nowait(request)
except QueueFull:
return 503, {"error": "overloaded", "retry_after": 5} # shed it
How backpressure is signalled:
| Layer | Mechanism |
|---|---|
| TCP | Receive window shrinks — built in |
| HTTP | 429 Too Many Requests or 503 with Retry-After |
| Reactive streams | The consumer requests N items; the producer sends at most N |
| Message queues | The consumer stops polling; the queue depth grows visibly |
| gRPC | Flow control per stream |
🚨 The message queue case is subtle. A queue is a buffer, so it absorbs bursts — that’s its job. But if the consumer is structurally too slow, the queue grows without bound and you’ve just moved the problem. Monitor queue depth and consumer lag, and distinguish a bounded burst (fine) from sustained under-capacity (needs a fix).
Backpressure’s aggressive cousin: deliberately reject some requests to keep serving the rest.
📐 The arithmetic that justifies it:
Capacity: 1,000 rps. Arriving: 2,000 rps.
Without shedding: all 2,000 queue, latency climbs to 10 s,
everyone times out → 0 successful requests.
With shedding: serve 1,000 successfully, reject 1,000 immediately
→ 1,000 happy users instead of zero.
Prioritized shedding is the sophisticated version, and it’s a strong thing to mention:
Shed first: prefetch, analytics, recommendations, batch
Shed last: checkout, payments, login, health checks
Never shed: anything already in flight (finish what you started)
🎙️ “Under overload I’d shed by priority — drop analytics and prefetch requests before anything on the checkout path. Serving 60% of traffic correctly beats failing 100% of it slowly.”
Adaptive concurrency limits (Netflix’s concurrency-limits, Google’s approach) are the modern
version: instead of a fixed limit, use TCP-congestion-control-style algorithms to continuously
discover the current capacity and adjust. Better than a static number that’s wrong at 3 a.m.
Rank features by criticality and design what “partially working” looks like.
| Dependency down | Response |
|---|---|
| Recommendations | Hide the widget. Page renders and sells. |
| Reviews | Show the product without them. |
| Search | Fall back to browse-by-category. |
| Cache | Serve from the database, slower, with a concurrency limit so it survives. |
| Payments | This is an outage. |
🎙️ “I’d make the recommendation call non-blocking with a 50 ms timeout and an empty-list fallback. A page missing a widget beats a 500, and it means recommendations being down doesn’t count as an outage.”
Amazon’s product page is the canonical example: it renders and completes a purchase even when several peripheral services are unavailable. The core transaction never depends on the periphery.
The layers compose, and a good answer names several:
flowchart LR
R[Request] --> LS[Load shedding<br/>overloaded? reject now]
LS --> BH[Bulkhead<br/>bounded concurrency<br/>for this dependency]
BH --> CB[Circuit breaker<br/>dependency broken?<br/>fail fast]
CB --> TO[Timeout<br/>bounded wait]
TO --> D[Dependency]
CB -.tripped.-> FB[Fallback<br/>cache / default / degraded]
Order matters: shed before you allocate resources, bulkhead before you attempt, break the circuit before you wait out a timeout. Each layer is cheaper than the one after it.
| Pattern | Gain | Cost |
|---|---|---|
| Timeouts | Bounded resource use | Some slow-but-successful requests are killed |
| Circuit breaker | Fail fast; dependency gets breathing room | Rejects requests that might have succeeded; tuning is fiddly |
| Bulkhead | One dependency can’t consume everything | Lower peak utilization; more configuration |
| Backpressure | No unbounded queues; no memory exhaustion | Requests are rejected |
| Load shedding | Serve some traffic well | Some users get errors, deliberately |
| Graceful degradation | Partial service beats none | Fallback paths to build and test |
🚨 The unifying trade: all of these deliberately fail some requests to keep the system alive. That’s the point, and it’s often the hardest thing for stakeholders to accept — “why are we returning errors on purpose?” The answer is that the alternative is returning errors to everyone, slowly.
1. Cause the thread pool exhaustion. Service A calls service B. Give B a 30-second sleep. Load test A with 100 concurrent requests, hitting both the B-dependent endpoint and an unrelated one. Watch the unrelated endpoint stop responding. That’s the whole problem, in five minutes.
Now add a bulkhead (limit B calls to 10 concurrent) and re-run. The unrelated endpoint stays healthy.
2. Watch a circuit breaker work. Add Resilience4j or gobreaker. Make B fail. Observe: response time drops from 30 s to microseconds once the breaker opens; B’s load drops to zero; the breaker half-opens and recovers when B comes back. Graph it — the recovery curve is satisfying and instructive.
3. Prove load shedding wins. Under 2× overload, measure successful requests per second with an unbounded queue versus a bounded queue that returns 503. The bounded version serves more successful requests, which is counter-intuitive until you see it.
4. Break the breaker. Configure it to trip on a fixed failure count rather than a rate. Send high traffic with a 0.1% error rate. Watch it trip on a perfectly healthy dependency.