system-design

Design a Log Aggregation System (ELK / Splunk / Loki)

Difficulty: Tier 2 Asked at: Amazon, Datadog, Splunk, infra teams Time budget: 45 min

When you have thousands of servers, you can’t ssh in to read logs. A log aggregation system collects the log lines from everywhere, ships them to central storage, indexes them for search, and lets you query “show me all errors for request X across all services.” It’s the metrics system’s cousin — write-heavy ingestion — but the data is unstructured text you need to search, so the inverted index reappears.

Prerequisites: Design Metrics System, Search Systems, Message Queues

1. Requirements

Functional:

Collect logs from thousands of hosts/services/containers.
Ship to central storage; search and filter (by text, service, time, level, trace/request ID).
Structured (JSON) and unstructured logs.
Tail/live view; retention.

Non-functional:

Write-heavy, bursty — millions of log lines/sec; incidents spike volume.
Fast search over recent logs (debugging is time-sensitive).
Scalable, cheap storage — logs are voluminous and mostly never read.
Don’t lose logs during incidents (when you need them most), but some loss is tolerable (best-effort-ish).

Out of scope: metrics (separate), tracing (mention the three pillars), alerting internals.

2. Estimation

10,000 hosts × 100 log lines/sec × ~500 bytes → ~1M lines/sec, ~500 MB/sec ≈ ~40 TB/day. 🚨 Volume and cost dominate; most logs are never read.
Search must be fast on recent data (last hours/days); old data is cold and rarely queried.

3. The pipeline

flowchart LR
    Agents[Log agents<br/>on each host] -->|ship| Buffer[[Buffer / Queue<br/>Kafka]]
    Buffer --> Proc[Processing<br/>parse, enrich, structure]
    Proc --> Index[(Search index<br/>inverted, time-partitioned)]
    Proc --> Cold[(Cheap object storage<br/>raw logs, long retention)]
    Index --> Query[Search / Dashboard]
    Cold -.rehydrate.-> Query

Agents on each host tail log files and ship lines → a buffer/queue (Kafka) absorbs bursts and decouples → processing parses/enriches (extract fields, add service/host/trace tags) → writes to a search index (recent, hot) and cheap object storage (all logs, cold, long retention).

🚨 The queue is essential: log volume is bursty (an incident 10×’s it), and a buffer prevents the ingestion spike from overwhelming or losing data. (Message Queues)

4. Deep dives

4a. Searching logs — the inverted index returns

To answer “all errors mentioning ‘timeout’ in service=checkout in the last hour,” you need full-text search → an inverted index (term → log lines), time-partitioned so queries hit only relevant time ranges. Elasticsearch is the canonical choice. 🚨 Indexing is expensive (CPU + storage); you often index only recent/hot data and keep raw logs cheaply for the rest.

4b. Hot/warm/cold tiering — managing cost

40 TB/day, mostly unread → tier storage:

Hot: recent logs (hours–days) fully indexed on fast storage for instant search.
Warm: older logs, less-indexed, slower storage.
Cold: raw logs in cheap object storage (S3), searchable only by rehydration or coarse scan.
Retention: delete after N days (or archive). 🚨 Index the recent, archive the rest — you can’t afford to index everything forever.

4c. Ingestion at scale & backpressure

Agents ship to a queue; the queue absorbs bursts and provides backpressure so processing/indexing isn’t overwhelmed. If indexing lags during an incident spike, the queue buffers rather than dropping (up to retention). Sample or rate-limit noisy sources. Structure logs (JSON) at the source so parsing is cheap.

4d. Correlation — trace/request IDs

The killer feature: follow one request across services. Propagate a trace/request ID through all services (context propagation); logs carry it; search by it reconstructs the full journey. 🚨 This is why structured logging + a correlation ID matters — it turns scattered lines into a coherent story. Ties into distributed tracing (the third observability pillar).

4e. Reliability during incidents

Logs are most needed during incidents — exactly when volume spikes and systems are stressed. Design for graceful degradation: buffer aggressively, drop low-priority logs before high-priority (ERROR), and never let log shipping take down the application (bounded local buffers, async shipping). Some loss is acceptable; taking down the app to guarantee logs is not.

5. Bottlenecks & scaling further

Ingestion bursts → queue buffer + backpressure; async agents.
Search speed → time-partitioned inverted index on hot data.
Storage cost → hot/warm/cold tiering, index only recent, archive raw cheaply, retention limits.
Correlation → propagate trace IDs; structured logs.
Indexing cost → sample/limit; structure at source.

6. Trade-off summary

Decision	Chosen	Alternative	Why
Ingestion	Buffer via queue	Direct to index	Absorb bursts; backpressure; avoid loss
Search	Inverted index on hot data	Index everything	Indexing all 40 TB/day is unaffordable
Storage	Hot/warm/cold tiering	All on fast storage	Most logs never read; tier by access
Correlation	Trace ID + structured logs	Grep unstructured text	Follow a request across services
Loss policy	Best-effort, drop low-priority first	Guarantee every log	Never take down the app for logs

7. Follow-up questions

How is this like and unlike the metrics system?

Both are write-heavy ingestion pipelines fed by agents on thousands of hosts, both buffer bursty input, and both use hot/warm/cold tiering with retention to manage enormous volume affordably — so the ingestion and storage-tiering halves are very similar. The key difference is the data and the query. Metrics are small, structured numeric time series queried by aggregation over time ranges, so they use a compact time-series database with heavy numeric compression. Logs are large, semi-structured or unstructured *text* that you need to *search* — "find lines containing this term for this service in this window" — which requires a full-text inverted index, the same structure as a search engine, and that indexing is far more expensive per byte than storing metrics. So while the front of the pipeline (collect, buffer, tier) rhymes with metrics, the back (index for text search rather than store for numeric aggregation) is where logs diverge and inherit the search-engine machinery.

You generate 40 TB of logs a day, mostly never read. How do you keep it affordable?

By not treating all logs equally — tiering storage and indexing by how likely data is to be queried. Recent logs (the last hours to days), which are what you search during active debugging, are kept hot: fully indexed in a fast search store for instant queries. Older logs move to warmer, cheaper, less-indexed storage, and the bulk moves to cold storage — raw logs in cheap object storage with long retention, searchable only by rehydration or coarse scan when you occasionally need history. Beyond a retention window you delete or archive entirely. The crucial cost lever is indexing selectively: full-text indexing is expensive in CPU and storage, so you index only the recent/hot data everyone actually searches and keep the rest as cheap raw blobs. You can also sample or rate-limit noisy log sources at ingestion. This "index the recent, archive the rest, delete the old" tiering matches spend to access and keeps a 40-TB-a-day firehose economical.

Why put a queue between the agents and the index?

To decouple ingestion from processing and absorb bursts without losing data or overwhelming downstream. Log volume is highly bursty — an incident or a deploy gone wrong can multiply output tenfold in seconds — and the indexing tier has finite throughput, so writing agents directly to the index would either drop logs or topple the index exactly when logs matter most. A durable queue (like Kafka) sits in between: agents ship to it cheaply and quickly, it buffers the surge, and the processing/indexing tier consumes at a sustainable rate, catching up after the spike. This provides backpressure (the buffer grows instead of overwhelming consumers), decouples the many producers from the indexing consumers so each scales independently, and adds durability so a temporary indexing outage doesn't lose logs still sitting in the queue. It's the same shock- absorber role queues play throughout system design, and it's especially critical here because the input is so spiky and the data is most valuable during the very incidents that cause the spikes.

How do you follow a single request across many services in the logs?

By propagating a correlation ID — a trace/request ID generated at the entry point and passed through every service the request touches via context propagation, with each service including that ID in every log line it writes. Then searching the aggregated logs for that one ID returns every log line from every service for that request, in time order, reconstructing its full journey — which is what lets you debug "what happened to this specific request" across a distributed system instead of staring at disconnected lines. This depends on structured logging (so the ID is a searchable field, not buried in free text) and on disciplined propagation so the ID survives across service boundaries and async hops. It's the logging side of distributed tracing — the correlation ID is the thread that ties scattered, independent log streams into one coherent story, and it's the single most valuable practice for making aggregated logs actually useful during an incident.

8. What junior / mid / senior answers look like

Junior: ships logs to a central store and greps them. Works small; can’t search at scale, no burst handling, no cost control.
Mid: agents → queue → processing → inverted-index search + storage, with retention and time- partitioned indexing.
Senior: all that plus hot/warm/cold tiering with selective indexing for cost, queue-based burst absorption and backpressure, trace-ID correlation with structured logging, graceful degradation that drops low-priority logs before taking down the app, and a clear contrast with the metrics system (text-search index vs numeric time-series store).