system-design

The Three Pillars: Logs, Metrics, Traces

You can’t operate, debug, or secure what you can’t see. These three signals are how you see inside a running distributed system — and each answers a different question.

Prerequisites: Performance Metrics, The 8 Fallacies Time to read: ~20 minutes

The problem

Your system is a hundred services across a thousand machines. A user reports “it’s slow.” You have no direct way to look inside — you can’t SSH into a thousand boxes, and the problem is already over by the time you’d get there.

🚨 Observability is the property of being able to understand a system’s internal state from its external outputs. The distinction from monitoring is worth knowing:

Monitoring — watching known failure modes (“is CPU above 90%?”). You decided in advance what to watch.
Observability — being able to answer questions you didn’t anticipate (“why is this specific user’s checkout slow, only on mobile, only in Karachi?”). You can explore.

You need observability because at scale, the failures you didn’t predict dominate — and you can’t predict every question you’ll need to ask.

The three pillars, and the question each answers

🚨 The key framing: each pillar answers a different question, and you need all three.

Pillar	Question	Shape
Metrics	Is something wrong, and how bad?	Numbers over time (aggregated)
Logs	What exactly happened?	Discrete events (detailed)
Traces	Where, across services, did it happen?	One request’s path through the system

The workflow ties them together: a metric alert tells you something’s wrong (error rate spiked); a trace shows you which service in the request path is failing; a log in that service tells you exactly why (the specific error, the stack trace, the parameters).

🎙️ “Metrics tell me something is wrong, traces tell me where, logs tell me why. That’s the debugging workflow, and it’s why you need all three, not one.”

Metrics

Numbers measured over time, aggregated. Cheap to store (a metric is a few numbers per interval), efficient to query, and ideal for dashboards and alerts.

http_requests_total{status="500", endpoint="/checkout"}  = 4,213
request_duration_seconds{quantile="0.99"}                = 0.340
db_connections_active                                    = 87

What to measure — the frameworks:

The Four Golden Signals (Google SRE) — if you instrument nothing else:

Latency — how long requests take (split successful vs failed! → Performance)
Traffic — demand (requests/second)
Errors — rate of failures
Saturation — how full the constrained resource is (the leading indicator)

RED (for services): Rate, Errors, Duration. USE (for resources): Utilization, Saturation, Errors.

🚨 Metrics are aggregates, which is their strength and their limit. They tell you the error rate is 5% but not which requests failed or why — you can’t drill into an individual event. And 🚨 high-cardinality labels kill metrics systems (user_id as a label creates millions of series, as in time-series databases). Metrics are for bounded dimensions; individual events go in logs and traces.

Store percentiles correctly. 🚨 You can’t average percentiles across hosts — metrics systems store histograms and compute percentiles from the merged distribution. → Performance Metrics

Tools: Prometheus (pull-based, the standard), Grafana (dashboards), Datadog, CloudWatch.

Logs

Discrete, timestamped records of events. Detailed — the full context of what happened.

🚨 Structured logging is non-negotiable at scale. Unstructured text is unqueryable:

# ❌ Unstructured — you can't query this
[2026-07-22 10:00:03] Error processing order 4821 for user 42

# ✅ Structured (JSON) — queryable, filterable, aggregatable
{"ts": "2026-07-22T10:00:03Z", "level": "error", "msg": "order processing failed",
 "order_id": 4821, "user_id": 42, "trace_id": "7f3a...", "error": "payment_declined"}

With structured logs you can query “all errors for user 42” or “all payment_declined events in the last hour.” With text logs you’re grepping and hoping.

🚨 Every log line must carry the trace/correlation ID (trace_id above). This is what lets you find every log across every service for one request — the single most valuable thing about structured logging in a distributed system. → Distributed Tracing

The costs, which are real:

Volume and cost. Logs are the most expensive pillar — high-volume systems generate terabytes, and storage/ingestion (Datadog, Splunk) gets very expensive. Sample or filter aggressively; you don’t need every debug log in production.
🚨 Never log secrets or PII. Logs are a breach target with weaker access controls than your database, and over-logging PII creates a compliance exposure. → Privacy
Log levels matter — ERROR/WARN/INFO/DEBUG, with production usually at INFO and the ability to raise verbosity temporarily.

Tools: ELK/Elasticsearch (the inverted index again → Search), Loki, Splunk, CloudWatch Logs.

Traces

🚨 The pillar that makes distributed systems debuggable, and the one people underuse.

A trace follows a single request across every service it touches, showing where the time went.

Trace: checkout request (total: 340ms)
├─ api-gateway          [██] 12ms
├─ auth-service         [█] 8ms
├─ order-service        [████████████████████] 280ms   ← the problem is here
│  ├─ inventory-check   [███] 40ms
│  └─ payment-service   [████████████████] 230ms        ← specifically here
│     └─ stripe-api     [███████████████] 220ms          ← waiting on Stripe
└─ notification-service [█] 5ms (async)

A trace is made of spans — each span is one unit of work (one service call), with a start time, duration, and metadata. Spans link into a tree via a shared trace ID propagated through the whole request, and each span records its parent span.

🚨 Why this is transformative: without tracing, “checkout is slow” means reading logs from six services and manually correlating by time — hours of work. With tracing, you look at one waterfall and see that 220ms is spent waiting on Stripe. It turns distributed debugging from archaeology into reading a picture.

How trace context propagates: the trace ID travels in request headers (the W3C Trace Context standard, traceparent), so every service adds its span to the same trace. Every service must propagate it, or the trace breaks. → Distributed Tracing

Tools: Jaeger, Zipkin, Tempo, Datadog APM, and OpenTelemetry as the vendor-neutral standard for instrumentation.

How the pillars work together

The unified debugging workflow, which is the point of the chapter:

flowchart LR
    M[Metric alert:<br/>error rate 5%] --> T[Trace:<br/>which service?]
    T --> L[Log:<br/>exact error + context]
    L --> F[Fix]

Metric fires an alert: checkout error rate jumped to 5%.
Trace the failing requests: they’re all failing in payment-service, waiting on Stripe.
Log in payment-service: stripe_timeout, Stripe is returning 503s.
Root cause: Stripe outage. Fix: circuit-break Stripe, degrade gracefully.

🚨 The trace/correlation ID is the thread through all three. A metric points at a symptom, you find example traces, each trace carries an ID, and that ID pulls up every relevant log. Without the shared ID, the three pillars are three disconnected islands.

OpenTelemetry: the unifying standard

🚨 Worth knowing as the current direction. OpenTelemetry (OTel) is a vendor-neutral standard for generating and collecting all three signals with one instrumentation library and one wire format.

Historically you instrumented separately for metrics (Prometheus), logs (some logger), and traces (Jaeger) — and were locked into each vendor. OTel unifies instrumentation: instrument once, export to any backend (Datadog, Grafana, Honeycomb, whatever), and switch backends without re-instrumenting.

🎙️ “I’d instrument with OpenTelemetry so we’re not locked into a vendor — one instrumentation layer for metrics, logs, and traces, exportable to any backend.” Mentioning OTel is a good currency signal.

The costs of observability

⚖️ Observability isn’t free, and a mature answer acknowledges it:

Storage and ingestion cost — often a significant fraction of infrastructure spend; logs especially. Sampling (keep 1% of traces, more of the errors) and retention limits control it.
Performance overhead — instrumentation adds a little latency and CPU (usually <1%, but real).
Cardinality explosions — the recurring failure mode; watch label/tag cardinality.
Alert fatigue — too many alerts and people ignore all of them. → Monitoring & Alerting

🚨 The trade-off in one line: observability data can cost as much as the system it observes if unmanaged. Sample intelligently — full fidelity for errors and a sample of the rest.

In the real world

Google’s SRE practice codified the Four Golden Signals and the metrics-first approach, and the free SRE book is the reference. “Instrument the golden signals” is the expected baseline answer.
OpenTelemetry has become the industry standard, backed by every major vendor, precisely because the previous lock-in (re-instrument to switch observability vendors) was painful. It’s the CNCF’s second-most-active project after Kubernetes.
The “we have logs but can’t debug” problem is common — teams log everything but without structure, without trace IDs, and without traces, so a distributed issue still takes hours to correlate manually. Structured logs + trace IDs + traces is what actually makes systems debuggable.

🚨 Interview traps

Only mentioning logs. You need metrics (is it wrong?), traces (where?), and logs (why?).
Unstructured logs at scale — unqueryable.
No trace/correlation ID tying the pillars together.
High-cardinality labels in metrics (user_id) — kills the metrics system.
Logging secrets or PII.
Not knowing distributed tracing — the pillar that makes microservices debuggable.
Ignoring the cost of observability data.

🎙️ Soundbites

“Metrics tell me something’s wrong, traces tell me where in the request path, logs tell me exactly why. I’d instrument all three — they answer different questions and together give the debugging workflow.”
“Structured logging with a trace ID on every line — that’s what lets me pull every log for one request across every service. Grepping unstructured text doesn’t scale.”
“The Four Golden Signals as the baseline: latency, traffic, errors, saturation — with latency split by success and failure, since a fast-failing service looks healthy on average latency.”
“I’d instrument with OpenTelemetry so we’re not locked into one vendor, and sample intelligently — full fidelity for errors, a sample of the rest — because observability data can cost as much as the system it observes.”
“High-cardinality goes in traces and logs, not metric labels — user_id as a metric label creates millions of series and takes down the metrics backend.”

🛠️ Try it

1. Instrument a service with all three. Add Prometheus metrics (request count, latency histogram), structured JSON logs with a trace ID, and OpenTelemetry tracing across two services. Then trigger a slow request and debug it using the three pillars in sequence — metric shows the latency, trace shows which service, log shows why. Doing the workflow once makes the “three pillars” framing concrete.

2. Generate and read a trace. Two services calling each other, instrumented with OTel + Jaeger. Make one slow, then look at the trace waterfall. Seeing the time-spent-per-service picture is the “aha” for why tracing matters.

3. Break your logs, then fix them. Log an error as plain text and try to answer “how many errors for user 42 today?” You can’t easily. Switch to structured JSON and answer it with a query.

4. Cause a cardinality explosion. Add user_id as a Prometheus label and generate traffic from many users. Watch the series count and memory climb. That’s why high cardinality belongs in logs, not metrics.

Check yourself

1. What question does each of the three pillars answer, and why do you need all three?

**Metrics** answer "*is* something wrong, and how bad?" — aggregated numbers over time (error rate, latency, saturation) that are cheap to store and ideal for dashboards and alerts, but can't tell you about an individual event. **Logs** answer "*what exactly* happened?" — detailed, discrete event records with full context (the specific error, parameters, stack trace), but they're expensive and you can't easily see cross-service flow. **Traces** answer "*where*, across services, did it happen?" — following one request's path through the whole system to show where the time or failure occurred. You need all three because they operate at different resolutions: a metric alerts you that something's wrong, a trace localizes it to a service, and a log explains why. Any one alone leaves a gap — metrics without traces can't localize, traces without logs can't explain, logs without metrics can't alert.

2. Why is structured logging essential at scale?

Because unstructured text logs are effectively unqueryable at scale. When you have terabytes of logs across hundreds of services and need to answer "show me all payment-declined errors for user 42 in the last hour," grepping free-text lines is slow, fragile (a format change breaks your patterns), and can't aggregate. Structured logs (JSON with typed fields) let you query, filter, and aggregate like a database: filter by `user_id`, count by `error` type, group by `service`. Critically, structured logs carry the trace/correlation ID as a field, which is what lets you retrieve every log across every service for a single request — impossible to do reliably by correlating timestamps in text. Structure turns logs from a haystack into a queryable dataset.

3. Why does distributed tracing make microservices debuggable in a way logs alone don't?

Because it reconstructs the causal, cross-service path of a single request and shows where time was spent — automatically. Without tracing, debugging "checkout is slow" means pulling logs from every service the request touched and manually correlating them by timestamp and IDs to reconstruct the sequence and find the slow step, which takes hours and is error-prone. A trace does this for you: each service adds a span (its unit of work, with duration) to a shared trace identified by a propagated trace ID, and the spans form a waterfall you can read at a glance — you *see* that 220ms of a 340ms request was spent waiting on Stripe inside payment-service. It turns distributed debugging from manual log archaeology into reading a picture, which is why it's the pillar that specifically addresses the microservices "where did it go wrong?" problem.

4. Why is high cardinality a problem for metrics but fine for logs and traces?

Because of how each is stored. Metrics systems store a separate time series for every unique combination of label values, and query performance and memory depend on the total series count. A high-cardinality label like `user_id` (millions of values) multiplies the series count into the millions or billions, exhausting memory and grinding queries to a halt — it's the most common way to take down a metrics backend. Logs and traces, by contrast, store individual events, so a unique `user_id` per event is just a field value, not a new series — exactly what they're designed for. So the rule is: metrics carry *bounded, low-cardinality* dimensions (status code, endpoint, region) for aggregation; high-cardinality identifiers (user ID, request ID, trace ID) go in logs and traces where you drill into individual events.

5. What ties the three pillars together into a debugging workflow?

The trace/correlation ID, propagated through the entire request and recorded in all three signals. The workflow: a **metric** alert fires (error rate spiked) — telling you something is wrong but not what. You find example failing requests and pull their **traces**, which show which service in the request path is failing and where the time went. Each trace carries a trace ID, and because every **log** line in every service also records that same trace ID, you use it to retrieve exactly the logs for that request across all services — which contain the specific error and context that explain the root cause. Without the shared ID, the three pillars are disconnected: you'd have an alert, some traces, and a sea of logs with no reliable way to connect a specific metric anomaly to the specific traces and logs that explain it. The ID is the thread that makes them a system rather than three separate tools.