Retries recover from transient failures and cause outages. The difference is entirely in the details — and the details are backoff, jitter, and budgets.
Prerequisites: Resilience Patterns, Idempotency Time to read: ~22 minutes
A call fails. Networks drop packets, instances restart, a leader election takes two seconds. Most failures are transient — retrying works.
So you retry. And then this happens:
Service B degrades and starts failing 50% of requests.
Every client retries 3 times.
B's effective load goes from 1,000 rps to ~2,500 rps.
B degrades further → failure rate rises → more retries → 4,000 rps
B collapses completely.
🚨 The retries caused the outage. B was struggling and would have recovered; the retry storm killed it. This is one of the most common patterns in real incident reports.
The rule to internalize: a retry is not free. It is additional load applied to a system that is already failing.
You cannot retry without a timeout — you’d wait forever on the first attempt.
📐 Setting the value:
Timeout ≈ p99.9 latency of the dependency + margin
🚨 Measure, don’t guess. If your dependency’s p99.9 is 200 ms, a 30-second timeout means a degraded dependency ties up a thread 150× longer than it should.
Timeouts must shorten as you go inward:
Client 30s → Gateway 25s → Service A 20s → Service B 15s → Database 10s
Otherwise the database is still executing a query for a request the client abandoned twenty seconds ago — burning resources for a result that will be discarded. This is a real and common source of wasted capacity.
🎙️ The deadline propagation refinement, which is a strong detail to mention: instead of each hop having a fixed timeout, pass the remaining budget with the request. gRPC does this natively.
Client sends: deadline = now + 30s
Gateway (5s elapsed) forwards: deadline = now + 25s
Service A (3s elapsed) forwards: deadline = now + 22s
→ every hop knows the real remaining budget, and can refuse work it can't finish in time
attempt 1 → fail → wait 1s → attempt 2 → fail → wait 1s → attempt 3
🚨 The problem: if the dependency is overloaded, you’re hammering it at a constant rate and giving it no room to recover.
attempt 1 → wait 1s → attempt 2 → wait 2s → attempt 3 → wait 4s → attempt 4 → wait 8s
✅ Gives the dependency exponentially increasing breathing room. Total load from retries decays rapidly.
⚠️ Still has a serious flaw on its own.
Backoff alone causes synchronization. All clients that failed at the same instant retry at the same instant:
1,000 clients fail at t=0
All wait exactly 1s → all retry at t=1 → 1,000 simultaneous requests → all fail
All wait exactly 2s → all retry at t=3 → 1,000 simultaneous requests → all fail
A thundering herd, repeating forever, in perfect lockstep. The backoff spread out the average load and did nothing about the peaks — and peaks are what kill you.
Jitter randomizes the delay:
# Full jitter — AWS's recommendation, and the best performer in their testing
delay = random.uniform(0, min(cap, base * 2 ** attempt))
# Equal jitter — half fixed, half random. Slightly more predictable.
temp = min(cap, base * 2 ** attempt)
delay = temp / 2 + random.uniform(0, temp / 2)
# Decorrelated jitter — good for long-running retry loops
delay = min(cap, random.uniform(base, previous_delay * 3))
📐 AWS published simulations comparing these. Full jitter performed best — it minimized both total work and completion time, because it spreads retries evenly across the window instead of clustering them.
🚨 “Exponential backoff with jitter” is the phrase to use. Saying just “exponential backoff” misses the more important half — and interviewers who’ve operated systems will notice.
def retry(fn, max_attempts=5, base=0.1, cap=30):
for attempt in range(max_attempts):
try:
return fn()
except RetryableError:
if attempt == max_attempts - 1:
raise
time.sleep(random.uniform(0, min(cap, base * (2 ** attempt))))
🚨 Retrying the wrong thing wastes resources and can corrupt data.
| Response | Retry? | Why |
|---|---|---|
| Connection refused / reset | ✅ | Transient; the request likely never arrived |
| Timeout | ⚠️ Only if idempotent | Ambiguous — it may have succeeded |
429 Too Many Requests |
✅ | Honour Retry-After, don’t guess |
500 Internal Server Error |
✅ | Possibly transient |
502 / 503 / 504 |
✅ | Upstream problem, likely transient |
400 Bad Request |
❌ | Malformed. It will never succeed. |
401 / 403 |
❌ | Refresh the credential, don’t blindly retry |
404 |
❌ | It doesn’t exist |
409 Conflict |
⚠️ | Only after resolving the conflict |
422 Unprocessable |
❌ | Semantically invalid |
🚨 Retrying 4xx is a specific and common bug. The request will never succeed, so you burn capacity on guaranteed failures — and if the 4xx was caused by overload-induced misbehaviour, you’ve made it worse.
The timeout row is the interesting one. A timeout is ambiguous — the operation may have completed. Retry only if the operation is idempotent, which for anything with side effects means you need an idempotency key. → Idempotency ⭐
Retries at every layer of a call chain multiply:
Client retries 3× → Gateway retries 3× → Service A retries 3× → Service B
One user request can become 3 × 3 × 3 = 27 requests to Service B.
📐 With a five-layer chain and 3 retries each, one request becomes 243. During a partial outage, this is what converts a degradation into a total collapse.
The fix: retry at one layer only. Usually the outermost (closest to the user) or the innermost (closest to the failure), never both. Decide deliberately and document it.
A global cap on retries as a fraction of total traffic, rather than a per-request count.
Retry budget: retries ≤ 10% of successful requests over a rolling window.
Normal operation: few failures → retries well under budget → all retried ✅
Dependency degraded: many failures → budget exhausted → additional retries REJECTED
→ load stays bounded → the dependency can recover
🚨 This is the mechanism that actually prevents retry storms, and it’s what per-request retry counts cannot do. Per-request limits bound one client’s behaviour; a budget bounds the aggregate, which is what the failing dependency experiences.
It’s the correct answer to “how do you stop retries from causing a cascade?” and most candidates only know backoff and jitter.
Implemented in gRPC (retry throttling), Envoy, Finagle, and most service meshes.
A tripped circuit breaker stops retries entirely — which is the strongest form of retry limiting. Use both: budgets bound the retry rate during partial degradation; the breaker cuts it to zero during total failure.
Client-side (in the caller): the caller knows whether the operation is idempotent and what the business impact of failure is. It’s the most common and usually correct place.
Service mesh / sidecar: configured centrally, applied uniformly, no application code. → Service Mesh 🚨 Watch for double-retrying — if both the application and the mesh retry, you get multiplication without anyone intending it. This is a real and confusing production issue.
Load balancer: can retry an idempotent request on a different backend. Only safe for idempotent
methods, which is why L7 load balancers default to retrying GET but not POST.
Message queue: the broker redelivers unacknowledged messages, with a dead-letter queue as the terminal state. → Message Queues
Cap the total time, not just the attempt count. A user waiting 30 seconds through five retries has already left. Bound the overall deadline.
Log retries as a metric. A rising retry rate is a leading indicator of trouble — often visible before error rates or latency move. It’s an excellent alert.
Make the retryable/non-retryable distinction explicit in your error types, rather than re-deriving it from status codes at each call site.
Don’t retry inside a database transaction. You’ll hold locks across the backoff sleep. → Transactions
Consider hedged requests for latency rather than failures: if no response by the p95, send a duplicate to another replica and take the first answer. Google reported large p99.9 improvements for a few percent extra load. It’s a nice answer to “how would you reduce tail latency?” → Performance Metrics
| Decision | Gain | Cost |
|---|---|---|
| Retries | Recover from transient failures invisibly | Amplify load exactly when the system is struggling |
| Exponential backoff | Gives the dependency room to recover | Later attempts are slow for the user |
| Jitter | Prevents synchronized retry waves | None worth mentioning — always add it |
| Retry budget | Bounds aggregate retry load | Some recoverable requests fail |
| Retry at one layer only | No multiplication | That layer must handle it well |
| Aggressive timeout | Bounded resource use; fail fast | Kills slow-but-successful requests |
| Hedged requests | Much better tail latency | ~5% extra load; requires idempotency |
1. Cause a retry storm. Service A calls service B with 3 retries, no backoff. Make B fail 50% of requests. Graph B’s incoming request rate. Watch it multiply and watch B get worse. Then add exponential backoff with jitter and watch the curve flatten.
2. Prove jitter matters. Simulate 1,000 clients all failing at t=0, retrying with pure exponential backoff. Plot requests-per-second over time — you’ll see sharp spikes at 1 s, 3 s, 7 s. Add full jitter and re-plot. The spikes become a smooth curve. This visual makes the point better than any explanation.
import random
# Without jitter: every client retries at exactly the same moments
# With jitter: delay = random.uniform(0, min(cap, base * 2**attempt))
3. Measure amplification. Instrument a 3-layer call chain with 3 retries each. Count requests arriving at the bottom service for one client request during a failure. Confirm it’s 27.
4. Implement a retry budget. Add a rolling counter capping retries at 10% of successes. Trigger a dependency failure and watch retries get rejected once the budget is exhausted — and watch the dependency’s load stay flat instead of climbing.