system-design

Monitoring and Alerting That Doesn’t Suck

An alert should mean “a human needs to act, now.” Most alerting systems fail this test, and the result — alert fatigue — is more dangerous than having no alerts at all.

Prerequisites: The Three Pillars, Availability Time to read: ~18 minutes

The problem

You set up monitoring. Soon: 200 alerts a day, most of them noise. People start ignoring the pager. Then a real incident fires an alert that looks exactly like the 199 false ones, and nobody acts for 40 minutes.

🚨 Alert fatigue is the central failure of monitoring, and it’s worse than under-alerting — because a team drowning in noise is trained to ignore alerts, so the real one is missed. The goal isn’t more alerts; it’s alerts that mean something.

The rule that fixes almost everything: “Every alert must be actionable and urgent. If a human doesn’t need to do something right now, it’s not an alert — it’s a dashboard or a ticket.”

Alert on symptoms, not causes

🚨 The single most important principle, and the one that separates good alerting from bad.

Cause-based alerting — alert on internal conditions: high CPU, a full disk, a service down, high memory.

Symptom-based alerting — alert on what users experience: elevated error rate, high latency, failed checkouts.

Why symptoms win:

❌ Cause-based:  "CPU is at 95%"
   → But users are fine (it's a batch job). False alarm, 3am, nobody needed to wake up.
   → OR you didn't alert on the CPU pattern that mattered, and users suffered silently.

✅ Symptom-based: "Checkout error rate is 8%, up from 0.1%"
   → Users are definitely affected. Someone needs to act. Always meaningful.

🚨 The deep reason: you can’t enumerate every cause of a user-facing problem, but you can measure the user-facing problem directly. A checkout might fail from high CPU, a full disk, a bad deploy, a dependency outage, a network issue, or a cause you’ve never seen — you can’t alert on all of them, but you can alert on “checkouts are failing.” Symptom alerts catch problems you didn’t anticipate, which are exactly the ones that hurt.

Causes belong on dashboards — you look at CPU when investigating a symptom alert, not as an alert itself.

🎙️ “I’d alert on symptoms — user-facing error rate and latency — not causes like CPU. You can’t enumerate every cause of a failure, but you can measure the failure directly, and it means every alert corresponds to real user impact.”

What to alert on: SLO burn rate

🚨 The modern approach, and a strong thing to bring up.

Instead of static thresholds (“alert if latency > 500ms”), alert on burning through your error budget too fast.

If your SLO is 99.9% availability, you have an error budget of 0.1% — about 43 minutes/month. → SLIs, SLOs, SLAs

Burn-rate alerting fires when you’re consuming that budget faster than sustainable:

Fast burn:  consuming a month's budget in an hour → page immediately (major incident)
Slow burn:  consuming it steadily over days → a ticket, not a page (a real but non-urgent problem)

Why this is better than static thresholds:

Tied to actual user impact (the SLO), not an arbitrary number someone guessed.
Multi-window — a brief blip that recovers doesn’t page; a sustained problem does. This dramatically cuts false alarms.
Severity follows burn rate — fast burn wakes someone up, slow burn is a ticket.

This is Google’s SRE approach and it directly attacks alert fatigue by making alerts proportional to real impact.

The alert quality checklist

Every alert should be:

Property	Meaning
Actionable	There’s a specific action to take. If not, it’s a dashboard.
Urgent	It needs action now. If it can wait until morning, it’s a ticket.
User-impacting	It reflects a real problem for users (symptom-based).
Documented	🚨 A runbook — what to check, what to do. An alert with no runbook wastes the responder’s night.
Rare	If it fires constantly, either fix the underlying issue or the alert is wrong.
Owned	Routed to whoever can actually fix it.

🚨 Every alert must link to a runbook. Being paged at 3am with “error rate high” and no guidance is how incidents drag on. The runbook says: here’s what this means, here’s what to check first, here’s how to mitigate.

Tiering: page vs ticket vs dashboard

🚨 Not everything is a page. Route by urgency:

Page (wake someone up): user-impacting, urgent, needs immediate human action. Checkout down, data loss risk, security incident. These should be rare — a well-tuned system pages a few times a month, not daily.
Ticket (handle during business hours): a real problem, but not urgent. Slow error-budget burn, a degraded non-critical feature, an approaching capacity limit.
Dashboard (look at when investigating): causes, resource metrics, trends. Not an alert at all.

Getting this tiering right is most of what makes alerting sustainable — a page at 3am should be a genuine “the building is on fire” event, not “CPU touched 80%.”

Avoiding alert fatigue

Concrete techniques:

Symptom-based alerting (above) — the biggest lever.
Multi-window burn-rate alerts — don’t fire on brief blips.
Alert grouping / deduplication — one incident that trips 50 checks should page once, not 50 times. 🚨 A cascading failure that fires every alert simultaneously is useless if it’s 50 separate pages.
Dependency-aware alerting — if the database is down, don’t also page for the 30 services that depend on it. Suppress downstream alerts when the root cause is already alerting.
Auto-resolution — alerts that clear themselves when the condition passes.
Regular alert review — 🚨 audit alerts that fired: which were actionable? which were noise? Delete or tune the noisy ones. Alert hygiene is ongoing work, not set-and-forget.

Dashboards done right

Dashboards are for investigating (once an alert fires) and for situational awareness, not alerting.

Good dashboards:

Lead with the golden signals (latency, traffic, errors, saturation) at the top. → Three Pillars
Tell a story top-to-bottom — overview first, then drill-down detail.
Show what’s normal — a number without context (“340ms”) is useless; a trend against baseline is actionable.
🚨 Split by success/failure — a service failing fast looks great on average latency. → Performance

A common anti-pattern: the “wall of graphs” — 60 panels nobody reads, that answer no specific question. A dashboard should answer “is the system healthy?” at a glance and support drilling into “why not?”

On-call and human factors

Monitoring exists to be acted on by humans, so the human system matters:

A sustainable on-call rotation — nobody should be paged constantly (that’s the fatigue signal that your alerting is broken).
Escalation policies — if the primary doesn’t ack within N minutes, escalate to secondary, then to a manager. Alerts must never go unanswered.
Blameless culture — → Incident Response.
🚨 Track alert-to-action ratio as a metric. If most pages don’t result in action, your alerting is noise, and that’s a fixable problem you should be measuring.

⚖️ Trade-offs

Choice	Gain	Cost
Symptom-based alerts	Every alert is real user impact; catches unknown causes	Doesn’t tell you the cause (that’s the dashboard’s job)
Burn-rate alerts	Proportional to real impact; fewer false alarms	More complex to set up than static thresholds
Aggressive alerting	Catch everything early	Alert fatigue; ignored pages; burnout
Conservative alerting	Sustainable, meaningful pages	Might miss an early signal
Multi-window	Ignores brief blips	Slightly slower to fire on real issues

🚨 The core tension: sensitivity vs specificity. Alert on everything and you get fatigue; alert on too little and you miss incidents. Symptom-based, burn-rate, multi-window alerting is how you get high specificity (few false alarms) without sacrificing sensitivity to real problems.

In the real world

Google’s SRE book (“Monitoring Distributed Systems” and “Alerting on SLOs”) is the canonical reference for symptom-based, burn-rate alerting. The principles here come directly from it.
Alert fatigue is a documented, serious problem — studied in healthcare (where alarm fatigue has killed patients) and equally real in software, where teams that page constantly miss real incidents. The 2017 studies on hospital alarm fatigue are a sobering parallel.
The “we alerted on CPU and everyone ignored it” experience is near-universal — cause-based alerts on resource metrics generate constant noise (CPU spikes are normal), so teams tune them out, and then a symptom nobody was watching for causes an outage. Migrating to symptom + SLO alerting is a common maturity step.

🚨 Interview traps

Alerting on causes (CPU, memory) instead of symptoms (user-facing errors, latency).
Not addressing alert fatigue. It’s the central failure mode.
Alerts with no runbook.
Everything is a page — no tiering into page/ticket/dashboard.
Static thresholds without mentioning SLO burn-rate as the better approach.
No alert deduplication — a cascade pages 50 times.
Wall-of-graphs dashboards that answer no question.

🎙️ Soundbites

“Alert on symptoms, not causes — user-facing error rate and latency, not CPU. You can’t enumerate every cause of a failure, but you can measure the failure directly, so every alert means real user impact and it catches problems you didn’t anticipate.”
“I’d use SLO burn-rate alerting: page when we’re consuming the error budget fast, ticket when it’s slow. It ties alerts to real impact and, being multi-window, ignores brief blips that recover.”
“Every alert needs a runbook. Being paged at 3am with ‘error rate high’ and no guidance is how incidents drag on.”
“Not everything is a page — page for urgent user impact, ticket for real-but-not-urgent, dashboard for causes. A 3am page should be a genuine emergency, and those should be rare.”
“Alert fatigue is the real risk — a team drowning in noise ignores the pager, so the real incident gets missed. I’d track alert-to-action ratio and delete alerts that don’t lead to action.”

🛠️ Try it

1. Build a symptom-based alert. Instrument a service’s error rate, then alert on “error rate > 2% for 5 minutes” rather than on CPU. Trigger it by making requests fail, and confirm the alert corresponds to real (simulated) user impact. Then spike the CPU with a batch job and confirm it does not alert — because users are fine.

2. Implement burn-rate alerting. Define an SLO (99.9%), compute the error budget, and alert on fast vs slow burn with different severities. Simulate a brief blip (shouldn’t page) and a sustained problem (should page). Watching the multi-window logic suppress the blip is the “aha.”

3. Audit real alerts. Take any monitoring setup and list every alert. For each, ask: is it actionable? urgent? does it have a runbook? has it ever led to action? Most setups have several alerts that fail this test — that’s your noise to delete.

4. Cause an alert storm. Make a shared dependency (database) fail and watch how many alerts fire across dependent services. Then implement dependency-aware suppression so the root cause pages once, not fifty times.

Check yourself

1. Why alert on symptoms rather than causes?

Because symptoms (user-facing error rate, latency, failed transactions) directly measure real impact, while causes (CPU, memory, disk) often don't correspond to user impact — high CPU might be a harmless batch job, so alerting on it produces false alarms that train people to ignore the pager. More fundamentally, you cannot enumerate every possible cause of a user-facing problem: a checkout can fail from a bad deploy, a dependency outage, a network issue, a full disk, or a cause you've never seen. You can't write an alert for each, but you *can* alert on "checkouts are failing," which catches the problem regardless of cause — including the causes you didn't anticipate, which are exactly the ones that hurt. Causes belong on dashboards, consulted when investigating a symptom alert, not as alerts themselves.

2. What is alert fatigue and why is it worse than under-alerting?

Alert fatigue is what happens when a system generates so many alerts — especially noisy, non-actionable ones — that responders become desensitized and start ignoring them. It's worse than under-alerting because it actively *trains* the team to disregard the pager: when a real, urgent incident finally fires an alert, it looks identical to the hundreds of false ones people have learned to dismiss, so nobody acts, and the real problem festers. Under-alerting at least leaves people attentive to the alerts that do fire. The dangerous state isn't "no alerts," it's "alerts that have been rendered meaningless by noise." This is why the goal of alerting is *fewer, better* alerts — every one actionable and urgent — not more coverage. It's a documented, serious failure mode (with fatal parallels in hospital alarm fatigue).

3. What is SLO burn-rate alerting and why is it better than static thresholds?

Instead of alerting when a metric crosses a fixed number ("latency > 500ms"), burn-rate alerting fires when you're consuming your error budget — the allowable failure implied by your SLO — faster than sustainable. If your SLO is 99.9%, you have ~43 minutes of failure budget per month; a "fast burn" (consuming a month's budget in an hour) pages immediately, while a "slow burn" (consuming it over days) becomes a ticket. It's better because: the threshold is tied to *actual user impact* (the SLO) rather than an arbitrary guessed number; it's multi-window, so a brief blip that recovers doesn't fire while a sustained problem does — dramatically reducing false alarms; and severity follows burn rate, so urgency is proportional to real impact. It directly attacks alert fatigue by making every alert meaningful.

4. What's the difference between a page, a ticket, and a dashboard, and why does it matter?

A **page** wakes someone up: it's for urgent, user-impacting problems needing immediate human action (checkout down, data-loss risk), and it should be *rare* — a well-tuned system pages a few times a month. A **ticket** is for real but non-urgent problems handled during business hours (slow error-budget burn, a degraded non-critical feature, approaching a capacity limit). A **dashboard** is for causes, resource metrics, and trends — consulted when investigating, not an alert at all. The distinction matters because it's most of what makes alerting sustainable: if everything pages, you get fatigue and burnout; if urgent things become tickets, you miss incidents. Routing each signal to the right tier — reserving pages for genuine emergencies — keeps the pager meaningful and the team functional. A 3am page should feel like "the building is on fire," not "CPU touched 80%."

5. How do you prevent a cascading failure from producing an alert storm?

Two main techniques. **Alert grouping/deduplication**: one incident that trips many checks should generate a single notification, not one per check — 50 correlated alerts for one root cause are useless (and drown any other signal). **Dependency-aware suppression**: when a root cause is already alerting (the database is down), suppress the downstream alerts from the services that depend on it, since they're all symptoms of the same problem — you want one page saying "database down," not 30 pages saying "service X can't reach database." Additionally, multi-window burn-rate alerting avoids firing on the transient spikes a cascade produces, and auto-resolution clears alerts once conditions recover. The goal is that the responder gets one clear, actionable page pointing at the actual cause, rather than a wall of correlated noise they have to sift through under pressure.