system-design

Batch vs Stream Processing

Process everything at 2 a.m., or process each event as it arrives. The choice looks like a latency decision and is really a complexity decision.

Prerequisites: Message Queues, Analytics Storage Time to read: ~24 minutes

The problem

You need daily revenue by region.

Batch: at 2 a.m., query yesterday’s orders, aggregate, write the result. Simple, correct, restartable, and the number is up to 26 hours stale.

Stream: every order updates a running total as it happens. The number is always current, and now you have to answer: what if events arrive out of order? What if one arrives three hours late? What if a worker crashes mid-aggregation? What if you need to recompute after a bug?

🚨 The decision is rarely about whether real-time is “better.” It’s about whether the value of fresh data exceeds the very real cost of stream processing’s complexity. Most of the time, for most questions, it doesn’t.

Batch processing

Run periodically over a bounded dataset.

Every night at 02:00:
  read all of yesterday's orders     (bounded — the set is known and complete)
  aggregate
  write results

✅ Simple. A finite input, a deterministic output. It’s a function. ✅ Restartable. Failed at 60%? Run it again. The input hasn’t changed. ✅ Easy to test. Feed known input, assert known output. ✅ Efficient. Bulk reads, large sequential I/O, high throughput per unit of compute. ✅ Easy to reason about correctness. You either processed the day’s data or you didn’t.

❌ Latency equals the interval. A daily job means up to 24 hours of staleness. ❌ Bursty resource usage. Idle for 23 hours, then saturating the cluster. ❌ Late-arriving data needs reprocessing of an already-completed window.

Tools: Spark, plain SQL in a warehouse, dbt, Airflow-orchestrated scripts, Hadoop MapReduce (historically).

Stream processing

Run continuously over an unbounded dataset.

Forever:
  read the next event
  update state
  emit results

✅ Seconds of latency. ✅ Smooth resource usage — steady load rather than a nightly spike. ✅ Enables genuinely real-time products — fraud blocking, live dashboards, alerting, dynamic pricing.

❌ State management is hard. The running aggregate lives somewhere and must survive crashes. ❌ Out-of-order and late events are a permanent design concern, not an edge case. ❌ Reprocessing is harder. Found a bug? You need to replay history and reconcile. ❌ Debugging is harder. There’s no “the input file” to inspect. ❌ Always-on means always-on-call.

Tools: Flink, Kafka Streams, Spark Structured Streaming, ksqlDB, Materialize, Beam.

The hard parts of streaming

These four are what make streaming genuinely difficult, and knowing them separates people who’ve read about it from people who’ve operated it.

1. Event time vs processing time

🚨 The single most important concept in stream processing.

Event time: when the thing actually happened (a click at 14:59:58).
Processing time: when your system saw it (15:00:12, after the mobile device reconnected).

If you aggregate by processing time, that click lands in the 15:00 bucket — wrong. Your hourly report is incorrect, and worse, it’s non-deterministic: reprocess the same data and you get different answers because the timings differ.

Always aggregate by event time. Which immediately creates the next problem.

2. Windowing

You must group unbounded data into finite chunks to aggregate it.

Window type	Shape	Use for
Tumbling	Fixed, non-overlapping: `[0-5m)[5-10m)`	“Orders per 5 minutes”
Hopping / sliding	Fixed size, overlapping: 5-min window every 1 min	“Rolling 5-minute average”
Session	Gap-based: closes after N minutes of inactivity	“User session length”
Global	One window, forever	Running totals

Session windows are the interesting one — the window boundary is defined by the data, not the clock. A user active at 10:00, 10:02, 10:04, then nothing until 11:30 has two sessions, and you only know the first ended when the gap elapsed.

3. Watermarks and late data

A window can’t stay open forever, but events arrive late. When do you emit the result?

A watermark is the system’s assertion: “I believe I’ve now seen all events with event time ≤ T.”

Watermark at 15:05 means: the 14:55–15:00 window can be closed and emitted.
An event for 14:58 arriving at 15:07 is LATE.

⚖️ The fundamental trade-off, and it has no correct answer:

Aggressive watermark (allow 10 s lateness)  → fast results, more data missed
Conservative watermark (allow 1 hour)       → complete results, an hour of latency

Options for handling late events:

Drop them. Simple. Your numbers are slightly wrong, permanently.
Update the emitted result. Correct, but downstream consumers must handle retractions — which means they can’t have already sent an email based on the old value.
Route to a side output for separate handling or manual reconciliation.
Allow a grace period, then drop.

🚨 In mobile-heavy products, events can arrive hours or days late — a phone that was offline in a tunnel, on a plane, or simply had the app suspended. Any streaming design for mobile must state its lateness policy explicitly.

4. Exactly-once processing and state

The consumer crashes mid-window. What happens to the partial aggregate?

Checkpointing is the answer: periodically snapshot the operator state and the input offsets together, atomically. On recovery, restore the state and rewind the input to the matching offset. Flink’s distributed snapshot algorithm (based on Chandy-Lamport) does exactly this.

🚨 This gives “exactly-once state semantics”, not exactly-once side effects. If your job sends an email or charges a card, replay after recovery will do it again. External effects still need idempotency. This is the same distinction as Kafka’s transactional guarantees, and overclaiming it is a common interview error.

The comparison

	Batch	Streaming
Data	Bounded	Unbounded
Latency	Minutes to hours	Milliseconds to seconds
State	In the job, discarded after	Persistent, checkpointed
Correctness	Straightforward	Watermarks, late data, retractions
Reprocessing	Just re-run it	Replay + reconcile
Testing	Easy (known input → known output)	Hard (timing matters)
Resource profile	Bursty	Steady
Failure	Restart the job	Restore from checkpoint
Operational burden	Low	High

Lambda and Kappa architectures

Two named patterns you should be able to discuss.

Lambda architecture

Run both. A speed layer (streaming) gives approximate real-time results; a batch layer gives accurate results later; a serving layer merges them.

flowchart LR
    D[Data] --> B[Batch layer<br/>accurate, hours late]
    D --> S[Speed layer<br/>approximate, seconds]
    B --> V[Serving layer]
    S --> V
    V --> Q[Queries]

✅ Accurate and fast. ❌ 🚨 You maintain the same business logic twice, in two systems, in two languages. They drift. The bugs are the divergences, and they’re miserable to find. This is the well-known criticism.

Kappa architecture

Only streaming. To reprocess, replay the log from the beginning with new code.

flowchart LR
    D[Data] --> K[[Kafka<br/>durable, replayable log]]
    K --> S[Stream processor]
    S --> V[Serving layer]
    K -.replay from offset 0.-> S2[New version<br/>backfill]

✅ One codebase, one system. Reprocessing is “run a second instance from offset 0.” ❌ Requires retaining all history in the log (or in object storage), and replaying a year of events takes real time and compute.

🎙️ Kappa is generally the modern preference, and knowing why is a good signal: “I’d avoid Lambda — maintaining the same aggregation logic in both Spark and Flink means they inevitably diverge, and the discrepancies are extremely hard to debug. Kappa keeps one implementation and uses Kafka’s replay for backfills.”

Micro-batching: the middle ground

Process small batches very frequently — every second, every 10 seconds. Spark Structured Streaming originally worked this way.

✅ Batch’s simplicity (bounded chunks, easy restart) with near-streaming latency. ✅ Better throughput than per-event processing, because of batching efficiency. ❌ Latency is bounded below by the batch interval.

🚨 This covers far more real requirements than people admit. When a stakeholder says “real-time,” they usually mean “not tomorrow.” A 30-second micro-batch satisfies most “real-time dashboard” requests at a fraction of the complexity.

Choosing

Use batch when:

Latency of minutes to hours is acceptable (most reporting, billing, ML training).
You need complex joins across large historical datasets.
Correctness matters more than freshness — financial reconciliation, invoicing.
The team is small. Operational simplicity has real value.

Use streaming when:

The value of the data decays in seconds: fraud blocking, alerting, live pricing, personalization, operational dashboards.
Continuous processing is genuinely cheaper than a huge periodic spike.
You’re feeding a real-time product feature, not a report.

🎙️ The answer that scores well: “I’d start with a batch job every 15 minutes. That satisfies ‘near real-time’ for the dashboard, and it’s dramatically simpler to build, test, and reprocess when something is wrong. If the business genuinely needs sub-minute latency — for fraud blocking, say — I’d add streaming for that specific path rather than converting everything.”

🚨 Interrogate the requirement. “We need real-time” is one of the most commonly overstated requirements in engineering. Ask: what decision is made from this data, and how quickly must it be made? If the answer is “an analyst looks at it each morning,” a nightly job is correct.

⚖️ Trade-offs

Decision	Gain	Cost
Batch	Simple, testable, restartable, efficient	Stale by up to the interval; bursty load
Streaming	Seconds of latency; enables real-time products	State management, watermarks, late data, always-on ops
Micro-batch	Most of streaming’s latency, most of batch’s simplicity	Floor on latency
Lambda	Accurate and fast	Two implementations that drift
Kappa	One codebase; replay for backfill	Must retain history; replays are slow
Aggressive watermark	Fast results	More late data dropped
Conservative watermark	Complete results	Higher latency

In the real world

Uber runs both: streaming for surge pricing and driver matching (where seconds matter), batch for financial reconciliation and ML training (where correctness matters). The split is by use case, not by ideology — which is the practical answer.
Netflix uses Flink for real-time personalization signals and Spark batch jobs for the heavy offline model training that feeds them. Again, both, chosen per problem.
The Lambda architecture’s decline is instructive: it was widely adopted around 2014, and by 2020 most teams had abandoned it because dual-maintenance costs exceeded the benefit. Jay Kreps’s Kappa article arguing this is worth reading.

🚨 Interview traps

Proposing streaming without justifying the latency requirement. Ask what decision depends on it.
Not knowing event time vs processing time. This is the streaming concept, and it’s a very likely follow-up.
Ignoring late-arriving data. For mobile, events can be hours late. State the policy.
Claiming exactly-once for external side effects. Checkpointing gives exactly-once state, not exactly-once emails.
Proposing Lambda without noting the dual-maintenance problem.
No reprocessing story. “We found a bug in the aggregation from last month” — what now?
Forgetting micro-batching exists as a cheaper middle option.

🎙️ Soundbites

“What decision gets made from this number, and how fast? If an analyst reads it each morning, a nightly batch job is correct and I’d rather not operate a streaming system for it.”
“I’d aggregate by event time, not processing time — otherwise mobile events that arrive late land in the wrong bucket and the numbers become non-deterministic on reprocessing.”
“Watermarks are the trade: allow 10 seconds of lateness and results are fast but incomplete; allow an hour and they’re complete but an hour behind. For mobile I’d need a grace period plus a side-output for very late events.”
“I’d use Kappa rather than Lambda — maintaining the same logic in a batch and a streaming system guarantees they drift, and debugging the divergence is far worse than a slow replay.”
“Checkpointing gives us exactly-once state recovery, but the emails our job sends still need idempotency keys — the external side effect isn’t inside the checkpoint.”

🛠️ Try it

1. Build a tumbling window by hand. Kafka + a small consumer that aggregates events into 1-minute buckets by event time. Then inject an event with a timestamp 5 minutes in the past and watch what your naive implementation does with it. That’s the late-data problem, discovered rather than read about.

2. Compare event time and processing time. Generate events with random delays between event time and arrival. Aggregate both ways. Compare the outputs. Then reprocess the same data with different delays and observe that the processing-time result changes while the event-time result doesn’t.

3. Try Kafka Streams or Flink with windowing:

stream.groupByKey()
      .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(1)))
      .count()

Change the grace period and observe how completeness and latency trade off.

4. Kill it mid-window. Start an aggregation, kill the process halfway, restart it. Without checkpointing, the partial state is lost. With checkpointing, it resumes. Doing this once makes the whole state-management topic concrete.

Check yourself

1. What's the difference between event time and processing time, and why does it matter?

Event time is when the thing actually happened; processing time is when your system observed it. They differ because of network delays, mobile devices being offline, queue backlogs, and retries. Aggregating by processing time puts events in the wrong buckets — a click at 14:59 that arrives at 15:00 lands in the wrong hour — and, critically, makes results **non-deterministic**: reprocess the same data and you get different answers because the timings differ. Event-time aggregation is correct and reproducible, but it forces you to decide how long to wait for stragglers, which is what watermarks are for.

2. What is a watermark and what does it trade off?

A watermark is the stream processor's assertion that it believes it has seen all events with event time up to T, so windows ending at or before T can be closed and emitted. It trades **latency against completeness**: an aggressive watermark (allow 10 seconds of lateness) emits results quickly but drops or must retract more late-arriving data; a conservative one (allow an hour) captures nearly everything but delays every result by an hour. There's no correct setting — it depends on how late your data realistically arrives and whether downstream consumers can handle corrections. For mobile-heavy products, events can arrive hours late, which forces either a long grace period or an explicit late-data path.

3. Why has Kappa largely won over Lambda?

Lambda requires implementing the same business logic twice — once in the batch layer and once in the speed layer — usually in different frameworks and sometimes different languages. They inevitably drift, and the resulting discrepancies between the "fast" and "accurate" numbers are extremely hard to debug because you have to reason about two systems simultaneously. Kappa keeps a single streaming implementation and handles reprocessing by replaying the durable log from an earlier offset with the new code, then swapping the output. One codebase, one set of semantics. The cost is retaining history and the time to replay it, which is a far more tractable problem than logic divergence.

4. Your streaming job has exactly-once checkpointing. Does that mean it never sends a duplicate email?

No. Checkpointing gives exactly-once semantics for the job's **internal state and its input offsets** — on recovery, the operator state and the consumed position are restored consistently, so aggregates aren't double-counted. But an email sent to an external SMTP service before the crash is not part of that checkpoint, so replaying from the last checkpoint will send it again. Any external side effect — emails, payments, writes to a non-transactional store, API calls — needs its own idempotency mechanism, typically a deduplication key derived from the event.

5. When is micro-batching the right answer?

When the stated requirement is "real-time" but the actual requirement is "not tomorrow" — which is most of the time. A 10–30 second micro-batch satisfies dashboards, near-real-time reporting, search index updates, and most alerting, while keeping batch's operational simplicity: bounded inputs, easy restart-on-failure, straightforward testing, and no watermark or late-data machinery. You'd move to true streaming when the latency requirement is genuinely sub-second (fraud blocking during a transaction, live bidding) or when per-event processing with complex session state makes batching awkward.