system-design

Design a Metrics & Monitoring System (Prometheus / Datadog)

Difficulty: Tier 2 Asked at: Datadog, Amazon, Google, infra-heavy companies Time budget: 45 min

A monitoring system ingests a firehose of numbers — every server’s CPU, every service’s latency, every counter — and lets you query and alert on them. It’s a time-series problem: high-volume append-only writes, aggregation over time windows, and a storage engine tuned for exactly that. The signature ideas are the time-series data model, downsampling/retention, and the write-heavy ingestion pipeline.

Prerequisites: Time-Series Databases, Batch vs Stream, Observability

1. Requirements

Functional:

Ingest metrics (name + tags + timestamp + value) from many sources.
Query/aggregate them over time ranges (avg CPU last hour, p99 latency per service).
Dashboards and alerting on thresholds/anomalies.
Retention: keep recent data at high resolution, older data downsampled.

Non-functional:

Write-heavy — millions of data points/sec from thousands of hosts. 🚨 The defining property.
Fast range/aggregation queries over time windows.
Scalable storage with retention/downsampling (you can’t keep everything forever at full resolution).
High availability of ingestion (don’t lose metrics during incidents — that’s when you need them most).

Out of scope: log aggregation (separate), distributed tracing (mention the three pillars).

2. Estimation

10,000 hosts × 1,000 metrics each, reported every 10s → 10M metrics / 10s = ~1M data points/sec.
Each point is tiny (~timestamp + float + tags), but the volume and cardinality (unique metric+tag combinations) are the challenge.
Storage: 1M points/sec × ~10 bytes × retention → huge, but compresses extremely well (time-series compression) and downsamples over time.

3. The data model: time series

🚨 A metric is a time series: (metric_name, {tags}) → [(timestamp, value), ...].

cpu_usage{host="web1", dc="karachi"}  → [(t0, 0.42), (t1, 0.45), ...]
http_latency_p99{service="checkout"}  → [(t0, 120), (t1, 135), ...]

Tags/labels give dimensionality (query “avg cpu across all hosts in dc=karachi”).
Cardinality (number of unique series) is the key scaling concern — high-cardinality tags (e.g. user_id as a tag) explode the series count. 🚨 Name this: avoid unbounded-cardinality labels.

4. High-level design

flowchart LR
    Agents[Agents / exporters<br/>on each host] -->|push/pull metrics| Ingest[Ingestion tier]
    Ingest --> TSDB[(Time-Series DB<br/>append-only, compressed)]
    Ingest --> Stream[Stream processing<br/>rollups, alerting]
    TSDB --> Query[Query engine<br/>range + aggregation]
    Query --> Dash[Dashboards]
    Stream --> Alert[Alerting engine]
    TSDB --> Downsample[Downsampling / retention jobs]

Ingest (push from agents, or pull/scrape à la Prometheus) → write to a time-series database optimized for append-heavy writes and range reads → query engine for dashboards → stream processing for real-time rollups and alerting → downsampling/retention to age out old data.

5. Deep dives

5a. Why a time-series DB (not a general one)?

Metrics have a special shape: append-only, timestamp-ordered, queried by time range + aggregation. A purpose-built TSDB exploits this:

Columnar/append storage ordered by time → fast range scans.
Aggressive compression — consecutive timestamps and similar values compress ~10× (delta-of-delta, XOR encoding). 🚨 This is why metrics at this volume are affordable.
LSM-style writes for high write throughput. (Time-Series Databases) A general SQL DB would choke on the write volume and store it inefficiently.

5b. Downsampling & retention

You can’t keep 1M points/sec at 10s resolution forever. Downsample: keep raw data for days, then aggregate to 1-min resolution for weeks, 1-hour for months/years. Old data loses resolution but stays queryable and cheap. Retention policies delete or roll up automatically. 🚨 Resolution decreases with age — nobody needs per-second data from a year ago.

5c. Ingestion at scale (write-heavy)

1M writes/sec needs a horizontally-scaled ingestion tier, sharded by series (hash of metric+tags) so each series goes to a consistent shard. Buffer/batch writes; the TSDB’s write path is append-optimized. Handle bursts (an incident spikes metric volume) with buffering. Push vs pull: pull (scrape) gives the system control over rate and knows what’s up; push suits ephemeral/serverless sources.

5d. Querying & aggregation

Queries are range + aggregation: “p99 latency for service=checkout over the last 6h, by 1-min buckets.” The engine reads the relevant series’ time ranges and aggregates. Precompute common rollups (recording rules) so dashboards don’t recompute heavy aggregations every refresh. Cache dashboard queries.

5e. Alerting

Continuously evaluate rules (threshold, rate-of-change, anomaly) against incoming/recent data via stream processing, firing alerts to notification channels. Must be reliable and low-latency — you need alerts during incidents. Deduplicate/group alerts to avoid storms. (Observability)

5f. High cardinality — the classic pitfall

🚨 Each unique combination of metric name + tag values is a separate series. Putting a high-cardinality value (user ID, request ID, full URL) in a tag creates millions of series → memory/index blowup. Guard against it: bound label cardinality, reject/limit runaway series. This is the operational failure mode of metrics systems.

6. Bottlenecks & scaling further

Write volume → TSDB with append/LSM writes, sharded ingestion by series.
Storage cost → time-series compression + downsampling + retention.
Cardinality explosion → bound label cardinality; the top real-world failure.
Query cost → precomputed rollups (recording rules), caching, time-partitioned reads.
Ingestion availability → buffering, redundancy — never lose metrics during incidents.

7. Trade-off summary

Decision	Chosen	Alternative	Why
Storage	Purpose-built TSDB	General SQL DB	Append/compression/range-scan fit metrics
Retention	Downsample by age	Keep all at full resolution	Old data rarely needs high resolution; huge savings
Ingestion	Sharded by series	Single writer	1M writes/sec needs horizontal scale
Aggregation	Precomputed rollups	Compute on every query	Dashboards recompute constantly
Labels	Bounded cardinality	Arbitrary tags	Unbounded cardinality kills the system

8. Follow-up questions

Why use a specialized time-series database instead of a regular database?

Because metrics have a specific shape that a purpose-built store exploits and a general database handles poorly. Metric data is append-only, arrives in timestamp order, and is almost always queried as an aggregation over a time range for a set of series — never random single-row updates. A time-series database is built for exactly this: it stores data ordered by time (often columnar) so range scans are fast, uses LSM-style append-optimized writes to absorb the enormous write volume, and applies time-series-specific compression (delta-of-delta on timestamps, XOR on values) that shrinks the data roughly tenfold because consecutive points are so similar — which is what makes storing a million points a second affordable. A general SQL database, tuned for transactional row updates and arbitrary queries, would struggle with the write throughput, store the data far less efficiently, and lack the built-in downsampling and retention that time-series workloads require. Matching the storage engine to the access pattern is the whole point.

You can't keep a million points a second forever. How do you manage storage over time?

Through downsampling and retention policies that trade resolution for age. Recent data is kept at full resolution (say every 10 seconds) because that's what you need when debugging something happening now, but as data ages you aggregate it into coarser buckets — raw for a few days, one-minute averages/percentiles for a few weeks, one-hour rollups for months or years — and eventually delete it entirely. This works because nobody needs per-second granularity from a year ago; for old data you only want trends, which coarse rollups capture at a tiny fraction of the storage. The system runs background jobs that roll up and expire data automatically according to configured retention tiers. Combined with the heavy compression time-series data already enjoys, downsampling keeps total storage bounded and cost-effective while preserving high-resolution recent data and long-term low-resolution history — resolution decreasing with age is the key idea.

What is a cardinality explosion and why is it the classic failure mode?

Cardinality is the number of distinct time series, and every unique combination of metric name plus tag/label values is a separate series that the system must index and hold state for. A cardinality explosion happens when a label takes on a huge or unbounded number of values — putting something like user ID, request ID, session ID, or a full URL into a tag — because each distinct value multiplies the series count, quickly producing millions or billions of series. That blows up the in-memory index and per-series overhead, causing memory exhaustion, slow queries, and ingestion failures, often taking the monitoring system down precisely when it's needed. It's the classic failure mode because it's easy to do accidentally (a developer adds a convenient high-cardinality label) and its cost is nonlinear and often invisible until it's severe. The defense is to keep labels low-cardinality and bounded (use tags for things with few stable values like service, host, datacenter — not per-user or per-request identifiers), and to enforce limits that reject or cap runaway series before they overwhelm the system.

How do you keep dashboard and alert queries fast over huge volumes?

By precomputing common aggregations and reading only the data a query needs. Dashboards and alerts repeatedly ask for the same heavy aggregations (p99 latency per service per minute, average CPU per datacenter), so instead of recomputing those from raw points on every refresh, you evaluate recording rules that continuously roll raw data into precomputed aggregate series; queries then read the cheap rollup rather than scanning millions of raw points. Time-partitioned, time-ordered storage means a range query touches only the relevant time blocks, not the whole dataset, and reading downsampled tiers for older ranges cuts data volume further. Caching dashboard query results for their refresh interval avoids redundant work, and alerting evaluates rules against the incoming stream and recent data rather than re-querying history. So the combination of precomputed rollups, time-partitioned reads, downsampled tiers, and caching keeps interactive queries fast despite the underlying volume.

9. What junior / mid / senior answers look like

Junior: stores metrics as rows in a SQL DB and queries with GROUP BY. Works tiny; can’t handle the write volume, storage, or cardinality.
Mid: uses a time-series DB, tags/labels, downsampling and retention, sharded ingestion, a query engine and alerting.
Senior: all that plus time-series compression as the affordability lever, cardinality management as the top failure mode, precomputed recording rules for query speed, push-vs-pull ingestion trade-offs, and reliable ingestion/alerting that survives the incidents it’s meant to observe.