system-design

Distributed Tracing

Following one request across fifty services to answer “where did the time go?” The pillar that turns distributed debugging from hours of log archaeology into reading a picture.

Prerequisites: The Three Pillars, Microservices Time to read: ~18 minutes

The problem

A user’s request is slow — 2 seconds. It touched the API gateway, auth, the order service, inventory, payments, and notifications. Which one was slow? Why?

Without tracing, you:

Grep logs in six services.
Try to correlate them by timestamp (which don’t quite line up — clocks disagree).
Manually reconstruct the sequence.
Guess.

🚨 This is hours of work per incident, and it’s the reason “we moved to microservices and can’t debug anything” is a common complaint. Distributed tracing solves it directly: one view showing the request’s entire path and where every millisecond went.

Traces and spans

A trace represents one request’s complete journey. It’s made of spans — each span is one unit of work (typically one service handling the request, or one operation within it).

Trace: checkout (trace_id: 7f3a...) — total 2,000ms
│
├─ Span: api-gateway          [█]                     20ms
│   └─ Span: auth-service     [█]                     15ms
│       └─ Span: order-service [███████████████████]  1,900ms   ← here
│           ├─ Span: inventory [██]                    100ms
│           └─ Span: payment   [█████████████████]     1,750ms   ← specifically here
│               └─ Span: stripe-api [████████████████] 1,700ms   ← waiting on Stripe
└─ Span: notification (async) [·]                      5ms

Each span records:

A span ID and its parent span ID (forming the tree).
The shared trace ID (linking all spans of one request).
Start time and duration.
Tags/attributes (service, endpoint, status, user tier, etc.).
Events/logs within the span.

🚨 Reading the waterfall, you see that 1,700ms of a 2,000ms request was spent waiting on Stripe. No grepping, no timestamp correlation — the picture tells you. This is the transformative value.

How trace context propagates

🚨 The core mechanism, and the thing that breaks if you get it wrong: the trace context must travel with the request through every service.

First service (gateway) creates a trace: generates trace_id, creates the root span.
When it calls the next service, it injects the context into the request HEADERS:
     traceparent: 00-7f3a9c2e...-a1b2c3d4-01
The next service EXTRACTS the context from the headers, creates a CHILD span
   (same trace_id, its own span_id, parent = the caller's span_id).
And so on through every service — each adds its span to the same trace.
Spans are exported to a collector, which assembles them into the full trace tree.

🚨 Every service must propagate the context, or the trace breaks — you get two disconnected partial traces instead of one. This is why a single service that doesn’t forward the headers creates a gap in every trace that passes through it. And it must propagate across all boundaries: HTTP calls, gRPC (metadata), message queues (in the message), and async work.

The W3C Trace Context standard (traceparent header) is the vendor-neutral format everyone now uses, so services instrumented with different tools still link into one trace. Knowing this standard is a good signal.

Sampling: you can’t trace everything

🚨 Tracing every request is prohibitively expensive at scale — the storage and processing cost of a full trace for billions of requests is enormous. So you sample.

Head-based sampling — decide at the start of the request whether to trace it (e.g. trace 1%). ✅ Simple, low overhead — you don’t instrument the un-sampled requests. ❌ 🚨 You decide before you know if the request is interesting. You’ll sample away most of the errors and slow requests — exactly the ones you want — because they’re rare and random sampling mostly catches boring successful requests.

Tail-based sampling — buffer the whole trace, then decide after it completes whether to keep it, based on what happened (keep it if it errored, was slow, or hit a rare path). ✅ 🚨 Keeps the interesting traces — errors and slow requests — which is what you actually want. ❌ More complex and expensive (must buffer all traces until they complete before deciding).

🎙️ The strong answer: “I’d use tail-based sampling so we keep the errors and slow requests, which are the traces worth having. Head-based sampling is cheaper but samples away exactly the interesting ones. Common approach: keep 100% of errors and slow traces, and a small percentage of the rest.”

A hybrid — always keep errors/slow, sample the rest — is the pragmatic default, and stating it is a strong signal.

What tracing gives you beyond debugging

Latency breakdown — which service, which operation, is the bottleneck. The core use.
Service dependency mapping — 🚨 traces reveal what actually calls what, derived from real traffic, versus the architecture diagram you think is accurate. Often eye-opening.
Error propagation — see where a failure originated and how it cascaded.
Detecting N+1 patterns — a trace showing 50 identical sequential database spans is a distributed N+1 staring at you.
Understanding real user journeys — how requests actually flow, including surprising paths.

Correlating traces, logs, and metrics

🚨 The trace ID is the thread through all three pillars (Three Pillars):

Every log line includes the trace_id → jump from a trace directly to its logs.
Metrics can be exemplar-linked to traces → click a latency spike on a graph, land on an example slow trace.
The debugging flow: metric alert → find slow/failing traces → jump to their logs for the exact cause.

This is why structured logging with a trace ID on every line matters so much — it’s what makes the three pillars a connected system rather than three islands.

Instrumentation and OpenTelemetry

Manual instrumentation — you add spans in code. Precise, but tedious and easy to miss.

Auto-instrumentation — libraries automatically create spans for common operations (HTTP handlers, database queries, gRPC calls) with no code changes. 🚨 The practical default — you get useful traces immediately and add manual spans only for custom business logic.

OpenTelemetry (OTel) is the standard: vendor-neutral instrumentation for traces (and metrics and logs), so you instrument once and export to any backend (Jaeger, Tempo, Datadog, Honeycomb). It’s the industry direction and mentioning it is a good currency signal. → Three Pillars

Tools: Jaeger and Zipkin (open source), Grafana Tempo, Datadog APM, Honeycomb (which pioneered the high-cardinality, tail-sampling approach).

The costs

⚖️ Tracing isn’t free:

Performance overhead — creating and exporting spans adds a little latency and CPU (usually <1%, but real). Async export minimizes it.
Storage cost — even sampled, traces are voluminous; retention is limited (days, not months).
Instrumentation effort — auto-instrumentation helps, but full coverage takes work, and one un-instrumented service breaks traces passing through it.
Complexity — a collection pipeline and backend to run.

🚨 The cost is worth it for any non-trivial distributed system — the alternative (debugging by correlating logs manually) costs far more in engineer-hours per incident. For a monolith, tracing matters much less.

In the real world

Google’s Dapper paper (2010) introduced the trace/span model and low-overhead sampling that every modern tracing system descends from. It’s the origin of this entire field.
Honeycomb popularized high-cardinality observability and tail-based sampling, arguing that aggregate metrics hide the individual weird requests that matter — and that you need to keep the interesting traces, not a random sample. Their approach shaped the modern direction.
The “microservices we can’t debug” problem is exactly what tracing solves, and teams that adopt microservices without tracing consistently hit a wall where cross-service issues take days to diagnose — then adopt tracing and the diagnosis drops to minutes. It’s not optional at scale.

🚨 Interview traps

Not mentioning distributed tracing for a microservices design. It’s the tool for debugging them.
Not knowing context propagation — the trace breaks if any service doesn’t forward the headers.
Proposing head-based sampling without noting it samples away the interesting (error/slow) traces.
Not connecting the trace ID to logs and metrics — that correlation is the point.
Ignoring the propagation across queues — async boundaries break naive tracing.
Forgetting the overhead/cost of full tracing.

🎙️ Soundbites

“Distributed tracing follows one request across every service and shows where the time went — I look at one waterfall and *see that 1.7 seconds of a 2-second request was waiting on Stripe, instead of grepping logs in six services and correlating by timestamp.”*
“The trace context propagates in the request headers — W3C traceparent — and every service must forward it, or the trace breaks. It has to cross HTTP, gRPC, and message queues, or async work disconnects from the trace.”
“I’d use tail-based sampling so we keep the errors and slow traces — the ones worth having. Head-based sampling decides before it knows the request is interesting and samples exactly those away. Keep 100% of errors, sample the rest.”
“The trace ID is on every log line, so I can jump from a trace straight to its logs. That correlation across traces, logs, and metrics is what makes the three pillars one system.”
“Auto-instrumentation with OpenTelemetry gets us useful traces immediately without vendor lock-in, and I’d add manual spans for the custom business logic.”

🛠️ Try it

1. Trace across two services. Instrument two services (calling each other) with OpenTelemetry, export to Jaeger. Make a request and view the trace. Seeing the waterfall — the two spans, the parent-child link, the durations — is the “aha” for why tracing matters.

2. Break the trace. Remove context propagation from one service (don’t forward the headers). Watch the trace split into two disconnected partial traces. This makes the “every service must propagate” rule concrete — and it’s a real bug you’ll recognize.

3. Propagate across a queue. Have service A publish a message that service B consumes. Naively, they’re separate traces. Then inject the trace context into the message and extract it in the consumer, linking them. Now the async work appears in the same trace.

4. Find an N+1 in a trace. Build an endpoint with a distributed N+1 (a loop calling a service per item). Look at the trace — you’ll see N identical sequential spans, the problem visible at a glance. Then batch the calls and see the trace collapse to one span.

Check yourself

1. What is a span, and how do spans form a trace?

A span represents a single unit of work within a request — typically one service handling the request, or one operation like a database query — recording a start time, a duration, tags/attributes (service, endpoint, status), and any events. Each span has its own span ID and records its *parent* span ID (the operation that caused it). All spans belonging to one request share the same *trace ID*. This parent- child linkage via shared trace ID assembles the spans into a tree: the root span (the first service) has children for the services it called, which have their own children, and so on. When you export all these spans to a collector, it reconstructs the tree and renders it as a waterfall showing the request's full path through the system and how long each step took.

2. How does trace context propagate across services, and what breaks it?

The first service to handle a request generates a trace ID and creates a root span, then *injects* the trace context (trace ID and its current span ID) into the outgoing request's headers — the W3C standard uses a `traceparent` header. Each downstream service *extracts* that context from the incoming headers, creates a child span with the same trace ID, its own new span ID, and the caller's span ID as parent, then propagates the (updated) context onward. What breaks it: any service that fails to forward the context — it creates a new root span with a fresh trace ID, splitting one logical request into two disconnected partial traces and leaving a gap in every trace passing through it. It also breaks at boundaries where the headers don't naturally travel: message queues (you must inject context into the message and extract it in the consumer) and async work, which otherwise become orphaned traces.

3. Why is tail-based sampling usually preferable to head-based?

Because it keeps the traces you actually want. Tracing every request is too expensive at scale, so you sample. **Head-based** sampling decides at the *start* of a request whether to trace it (e.g. 1% randomly) — cheap, but it makes the decision before knowing whether the request will error or be slow, so it samples away most of the errors and slow requests, which are rare and randomly missed, keeping mostly boring successful traces. **Tail-based** sampling buffers the complete trace and decides *after* it finishes, based on what happened — keep it if it errored, was slow, or hit a rare path — so you retain exactly the interesting traces that are worth investigating. The cost is complexity and memory (you buffer all traces until they complete before deciding). The pragmatic default is a hybrid: always keep 100% of error and slow traces, and a small sample of the normal ones.

4. How does distributed tracing connect to the other two observability pillars?

Through the trace ID, which is the common thread. Every log line includes the request's trace ID as a field, so from any span in a trace you can jump directly to all the logs that request produced across every service — no timestamp guessing. Metrics can be linked to traces via exemplars, so clicking a latency spike on a dashboard takes you to an example trace that was slow. This enables the unified debugging workflow: a *metric* alert tells you error rate spiked, you find example failing *traces* which localize the problem to a specific service and show where time went, and you jump from those traces to their *logs* for the exact error and context. Without the shared trace ID, the three pillars are three separate tools you'd have to manually correlate; with it, they're one connected system — which is why structured logging with a trace ID on every line is so important.

5. Besides debugging latency, what does distributed tracing reveal?

Several things. **Service dependency maps** derived from real traffic — traces show what actually calls what, which frequently differs from the architecture diagram you believe is accurate, revealing unexpected dependencies and call patterns. **Error propagation** — where a failure originated and how it cascaded through the request path. **N+1 patterns** — a trace showing dozens of identical sequential spans (a loop calling a service or database per item) makes a distributed N+1 problem visible at a glance, where it would be nearly invisible in logs. **Real user journeys** — how requests genuinely flow through the system, including surprising paths and unnecessary hops. And **fan-out analysis** — seeing how many downstream services one request touches, which relates directly to tail-latency amplification (each additional synchronous dependency multiplies the chance of a slow response).