system-design

Chaos Engineering

Deliberately breaking your system in production to find out how it fails — before it fails on its own at 3am. Counter-intuitive, and one of the most effective reliability practices there is.

Prerequisites: Failure Modes, Resilience Patterns Time to read: ~16 minutes

The core idea

You built resilience — retries, circuit breakers, failover, redundancy. But you don’t actually know it works until something fails. And the first time it fails for real is the worst possible time to discover your failover doesn’t work or your retry logic causes a cascade.

🚨 Chaos engineering flips this: deliberately inject failures, on purpose, during business hours, when everyone is available and watching — so you discover the problems on a calm Tuesday afternoon instead of at 3am during a real outage.

The insight: untested failure handling is broken failure handling. Code paths that only run during failures never run in testing, so they’re the least exercised and most likely to be broken — exactly the ones you’re relying on when things go wrong.

🎙️ “You don’t know your resilience works until something fails. Chaos engineering makes failure happen on your terms — during business hours with everyone watching — so you find the broken failover on a Tuesday afternoon, not at 3am in a real incident.”

The method

Chaos engineering is a scientific experiment, not random destruction:

HYPOTHESIS   — "If we kill an instance, the load balancer reroutes and users see no impact."
DEFINE STEADY STATE — what "healthy" looks like (error rate, latency, throughput).
INJECT FAILURE — kill the instance.
OBSERVE       — did the system behave as hypothesized? Did users notice?
LEARN & FIX   — if the hypothesis was wrong, you found a real weakness. Fix it.

🚨 The hypothesis is what separates chaos engineering from vandalism. You predict the outcome based on your belief that the system is resilient, then test that belief. When reality contradicts the hypothesis, you’ve found a real gap — the whole point.

Start small, expand carefully:

Start in staging, not production. Build confidence before touching real traffic.
Limit the blast radius — affect a small percentage of traffic or one instance first, not everything.
🚨 Have an abort button — be able to stop the experiment instantly if it goes worse than expected.
Then graduate to production — because 🚨 staging is never a faithful copy of production (scale, real traffic patterns, real data, real dependencies), so some failures only manifest in production. This is the uncomfortable but essential part.

What you inject

Failures map to real-world failure modes (Failure Modes):

Injection	Simulates	Tests
Kill an instance	Server crash	Load balancer failover, redundancy
Add latency	Slow dependency / degraded network	🚨 Timeouts, circuit breakers — the slow failure that’s worse than crashing
Inject errors	Downstream service failing	Retry logic, graceful degradation, fallbacks
Network partition	Split cluster	Split-brain handling, consensus, quorum behaviour
Resource exhaustion	CPU/memory/disk pressure	Autoscaling, load shedding, backpressure
Region/AZ failure	Datacenter outage	Multi-region failover, DR
Clock skew	NTP failure	Time-dependent logic, token expiry
Dependency down	Third-party outage	Circuit breakers, degraded mode

🚨 Latency injection is the most valuable and underused — because a slow dependency is more dangerous than a dead one (it ties up threads while looking alive), and it’s the failure teams most often haven’t handled. Killing an instance is the famous one, but injecting latency finds more real bugs.

What you typically discover

Chaos experiments reliably surface problems that testing misses:

Missing or wrong timeouts — a call with no timeout that hangs forever when the dependency slows.
Retry storms — retries with no backoff/budget that amplify load and cause a cascade.
Failover that doesn’t work — the standby was never actually tested, or fails over too slowly.
Cascading failures — one component’s failure taking down others through shared resources.
Hidden dependencies — “this service is non-critical” turns out to be in the critical path.
Cold-cache / warm-up problems — a restarted service hammering the database.
Alerts that don’t fire — the monitoring gap you’d otherwise find during a real incident.
Runbooks that are wrong — the documented recovery procedure doesn’t actually work.

🚨 Every one of these is far cheaper to find in a controlled experiment than in a real outage. And finding that your alerting didn’t fire is itself hugely valuable — it means a real incident would go undetected.

Game days

A game day is a scheduled, planned chaos exercise — often a broader simulated incident rather than a single injection. The team practices the full incident response: detect, declare, assign an IC, mitigate, resolve, and postmortem — on a simulated incident.

🚨 Game days test the humans and the process, not just the system. They reveal whether people know the runbooks, whether escalation paths work, whether communication flows, and whether the team stays calm — all things you can’t learn from automated injection. Practicing the incident when it’s not real is how you’re competent when it is.

The prerequisites

🚨 You should not do chaos engineering until you have:

Good observability — you must be able to see the impact of an experiment (the three pillars). If you can’t observe, you can’t tell whether the system handled the failure. → Three Pillars
Monitoring and alerting — to detect if an experiment goes wrong.
A way to abort — instantly stop the experiment.
Basic resilience already — chaos engineering validates resilience; it doesn’t create it. Injecting failures into a system with no redundancy just breaks it.
Organizational buy-in — deliberately causing failures needs cultural acceptance and a blameless attitude.

🎙️ “Chaos engineering validates resilience, it doesn’t create it — so you need observability to see the impact, an abort button, and basic redundancy already in place before you start. Injecting failures into a fragile system just causes an outage.”

The trust problem

⚖️ Deliberately breaking production is scary, and getting buy-in is a real challenge. Approaches:

Start in staging to build confidence and a track record.
Small blast radius initially — one instance, 1% of traffic.
Business hours only, with the team watching and ready.
Frame it as insurance — the cost of a controlled experiment is tiny versus a real outage.
Share the wins — “chaos testing found that our payment failover was broken; we fixed it before it cost us a real outage.” Concrete saves build support.

In the real world

Netflix’s Chaos Monkey originated the practice — it randomly terminates production instances during business hours, forcing every service to be resilient to instance loss by default, because if it isn’t, Chaos Monkey finds out on a Tuesday. It grew into the “Simian Army” (Latency Monkey, Chaos Gorilla for AZ failures, etc.) and then Gremlin and open-source tools (Chaos Mesh, LitmusChaos). Netflix’s willingness to run this in production is the canonical example.
AWS’s GameDay exercises and internal chaos practices are standard at large infrastructure companies — deliberately failing regions and services to validate that the redundancy actually works.
The “we thought we had failover but didn’t” discovery is near-universal among teams that adopt chaos engineering — the standby that was configured years ago and never tested, the retry logic that causes a cascade, the “non-critical” dependency in the critical path. These are found cheaply in experiments and expensively in real outages.

🚨 Interview traps

Not knowing chaos engineering for a high-reliability system — it’s how you validate resilience.
Random destruction without a hypothesis — that’s vandalism, not chaos engineering.
Doing it without observability — you can’t see the impact, so you learn nothing (or cause an outage).
Injecting failures into a system with no resilience — chaos validates, it doesn’t create.
Only killing instances — latency injection finds more real bugs.
No blast-radius limit or abort button.
Forgetting game days test the humans, not just the system.

🎙️ Soundbites

“Untested failure handling is broken failure handling — the code paths that only run during failures are the least exercised and most likely broken. Chaos engineering runs them deliberately, during business hours, so we find the problems on a calm Tuesday instead of at 3am.”
“It’s a scientific experiment, not random destruction: hypothesize how the system will handle a failure, inject it, and observe. When reality contradicts the hypothesis, you’ve found a real gap — that’s the point.”
“Latency injection is the most valuable and most underused — a slow dependency is more dangerous than a dead one, and it’s the failure teams most often haven’t handled.”
“Chaos engineering validates resilience; it doesn’t create it. You need observability to see the impact, an abort button, and basic redundancy already in place — otherwise you’re just causing an outage.”
“Game days test the humans and the process, not just the system — do people know the runbooks, does escalation work, does the team stay calm. You can’t learn that from automated injection.”

🛠️ Try it

1. Kill an instance under load. Run three service instances behind a load balancer, put them under load, and kill one. Form the hypothesis first (“the LB reroutes, users see no errors”), then test it. Measure how many requests actually failed and how long recovery took. Was your hypothesis right? (Often the answer is “more requests failed than expected.”)

2. Inject latency — the valuable one. Add 5 seconds of latency to a dependency (with tc netem or Toxiproxy). Hypothesis: “our timeout and circuit breaker contain it.” Test it. You’ll very likely find a missing timeout or a retry storm — the exact bugs latency injection is best at surfacing.

3. Test your alerts. Break something and check whether your monitoring actually alerts. A surprising number of teams discover their alerts don’t fire — which means a real incident would go undetected. Finding that gap in an experiment is a big win.

4. Run a mini game day. Break something in staging without telling the team the cause, and have them respond as if it’s real. Time the detection, the mitigation, and whether the runbook worked. The gaps you find are what a real 3am incident would have hit.

Check yourself

1. Why deliberately break a working system?

Because you don't actually know your resilience works until failure occurs, and the first real failure is the worst time to discover that your failover is broken, your retry logic causes a cascade, or your "non-critical" dependency is actually in the critical path. The code paths that handle failures only run *during* failures, so they're the least exercised by normal testing and the most likely to be broken — precisely the paths you're depending on when things go wrong. Chaos engineering inverts the timing: it injects failures deliberately, during business hours, with the whole team available and watching, so you discover and fix these gaps in a controlled experiment on a calm Tuesday afternoon rather than in a real, high-stress outage at 3am with users affected and information scarce. Untested failure handling is, in effect, broken failure handling.

2. What distinguishes chaos engineering from just breaking things randomly?

The hypothesis and the scientific method around it. Chaos engineering is an experiment: you first define your system's healthy "steady state" (error rate, latency, throughput), then form a specific, falsifiable hypothesis about how the system will handle a particular failure ("if we kill an instance, the load balancer reroutes and users see no impact"), *then* inject that failure and observe whether reality matches the prediction. When the system behaves as hypothesized, you've confirmed resilience; when it doesn't, you've found a real, actionable weakness — which is the entire value. Random destruction with no hypothesis and no observation is just vandalism: you break things, cause harm, and learn nothing systematic. The prediction-then-test structure, plus limited blast radius, an abort button, and careful observation, is what makes it a disciplined engineering practice rather than recklessness.

3. Why is latency injection considered more valuable than killing instances?

Because a slow dependency is more dangerous than a dead one, and it's the failure teams most often haven't handled. A killed instance produces a clean, obvious failure that health checks detect and load balancers route around — the common, well-tested case. A *slow* dependency keeps responding, so it passes health checks and keeps receiving traffic, while each call ties up a thread, connection, and timeout budget on the caller for the full slow duration — silently exhausting the caller's resources until it stops serving everything, including requests that don't touch the slow dependency. This is exactly the failure mode that missing timeouts and absent circuit breakers fail to contain, and it's subtle enough that teams frequently overlook it. So injecting latency reliably surfaces missing timeouts, retry storms, and cascade risks — more real bugs than the famous instance-killing, which tests the case people usually already handle.

4. What must you have in place before starting chaos engineering?

Chaos engineering *validates* resilience — it doesn't create it — so you need the resilience and the tooling to observe it already present. Specifically: **good observability** (metrics, logs, traces) so you can actually see the impact of an experiment and tell whether the system handled the failure; **monitoring and alerting** to detect if an experiment goes worse than expected; **an abort button** to stop the experiment instantly; **basic resilience already** (redundancy, failover, timeouts) — injecting failures into a system with no redundancy just produces an outage and teaches nothing; and **organizational buy-in** with a blameless culture, since deliberately causing failures requires cultural acceptance. Without observability especially, you can't distinguish "the system handled it" from "the system broke and we didn't notice," which defeats the purpose. You build the resilience and the visibility first, then use chaos engineering to prove they work.

5. What do game days test that automated failure injection doesn't?

The humans and the process. Automated injection (like Chaos Monkey) tests whether the *system* technically survives a failure — whether the load balancer reroutes, whether the circuit breaker trips. A game day simulates a full incident and tests whether the *team and its procedures* work: whether people notice the problem (detection), whether they declare an incident and someone takes the Incident Commander role, whether they know and correctly follow the runbooks, whether escalation paths function, whether communication flows to stakeholders, whether the documented mitigation actually works, and whether the team stays coordinated and calm under pressure. These are things you can only learn by practicing the human response, and they're often where real incidents go wrong — the system might have recovered fine, but the people fumbled the response, communicated poorly, or followed a runbook that turned out to be incorrect. Practicing the incident when it isn't real is how the team performs competently when it is.