Not a queue — a distributed, durable, replayable log. Understanding that one distinction explains everything else about it.
Prerequisites: Message Queues, Replication Time to read: ~28 minutes
LinkedIn, around 2010, had a familiar mess: dozens of systems producing data (user activity, page views, metrics, database changes) and dozens consuming it (search, analytics, monitoring, recommendations, the data warehouse). Each producer–consumer pair had its own bespoke pipeline.
N producers × M consumers = N×M integrations
Every new consumer meant touching every producer. Every format change broke things nobody knew about. Adding a data warehouse meant a six-month project.
Kafka’s answer: put one durable log in the middle.
N producers → [ Kafka ] → M consumers = N + M integrations
Producers write once. Consumers read independently, at their own pace, from wherever they choose in the history. That’s it — and every design decision in Kafka follows from making that log fast, durable, and scalable.
🧠 Mental model. A traditional queue is a to-do list — you take an item off and it’s gone. Kafka is a journal — entries are appended, never removed on read, and each reader keeps a bookmark.
Partition 0: [0][1][2][3][4][5][6][7][8][9] ← new messages appended here
↑ ↑
analytics group email group
(offset 3) (offset 8)
Consequences that make Kafka different from RabbitMQ or SQS:
🚨 If a design needs replay, or multiple independent consumers of the same stream, that’s the argument for Kafka. If it just needs work distributed to workers, SQS or RabbitMQ is simpler and you should say so.
A topic is a named stream (user.events, orders). Each topic is split into partitions —
the unit of parallelism and ordering.
Topic: user.events
├── Partition 0: [msg][msg][msg]... → Broker 1
├── Partition 2: [msg][msg]... → Broker 2
└── Partition 3: [msg][msg][msg]... → Broker 3
Which partition does a message go to?
partition = hash(message_key) % num_partitions # if a key is provided
partition = round-robin / sticky # if no key
🚨 The key decision: messages with the same key always land in the same partition, so they’re
ordered relative to each other. Key by user_id and all events for that user are strictly
ordered, while different users process in parallel.
Ordering is per-partition only. There is no global ordering across a topic, and asking for it means one partition, which means one consumer, which means no parallelism. Design around per-key ordering instead. → Message Queues
How many partitions? This is a real design question:
max(target_throughput / per_partition_throughput,
target_consumers), then round up generously.% N changes). Over-provision modestly at the start; 12–50 is common
for a busy topic.The mechanism that gives you both queue and pub/sub semantics from one system.
flowchart LR
T["Topic: orders<br/>P0 P1 P2 P3"]
T --> G1
T --> G2
subgraph G1["Group: email-service"]
C1[consumer 1 → P0,P1]
C2[consumer 2 → P2,P3]
end
subgraph G2["Group: analytics"]
C3[consumer 1 → P0,P1,P2,P3]
end
🚨 A consumer group can’t have more active consumers than partitions. 4 partitions, 10 consumers → 6 sit idle. This surprises people trying to scale out, and it’s the reason partition count is a capacity decision, not a detail.
Rebalancing happens when a consumer joins, leaves, or dies: partitions are reassigned. During a
rebalance, consumption pauses. Frequent rebalances (from slow processing exceeding
max.poll.interval.ms) cause a nasty failure mode where the group spends its time rebalancing
instead of consuming. Modern Kafka uses cooperative/incremental rebalancing to reduce the pain.
Kafka routinely handles millions of messages per second on ordinary hardware. Four reasons, and all of them come straight from Computer Fundamentals:
1. Sequential disk I/O only. Kafka only ever appends. Sequential writes on a spinning disk reach hundreds of MB/s; random writes manage a few MB/s. By never updating in place, Kafka gets memory-like throughput from cheap disks.
2. The OS page cache does the caching. Kafka doesn’t maintain its own cache in the JVM heap. It writes to the page cache and lets the OS handle it. Consumers reading recent data — which is most consumers — are served from RAM without Kafka doing anything.
3. Zero-copy transfer. sendfile() moves data from the page cache to the network socket without
passing through userspace. No copy into the JVM, no serialization, no GC pressure.
4. Batching and compression. Producers batch messages and compress the batch. Better compression ratios (similar messages compress well together) and fewer, larger network operations. The batch stays compressed on disk and is decompressed by the consumer.
🎙️ “Kafka’s throughput comes from working with the hardware rather than around it — append-only sequential writes, the OS page cache instead of an application cache, and zero-copy to the socket.” This is a great answer to “why is Kafka fast?” and it demonstrates you understand the fundamentals, not just the product.
Each partition has a leader and followers on other brokers.
Partition 0: Leader = Broker 1, Followers = Broker 2, Broker 3
All reads and writes go to the leader; followers replicate. If the leader dies, a follower is promoted.
ISR (In-Sync Replicas) — the set of replicas caught up with the leader. A replica that falls behind is removed from the ISR and isn’t eligible for promotion.
The producer’s acks setting is the durability knob, and it’s a good interview topic:
acks |
Waits for | Durability | Speed |
|---|---|---|---|
0 |
Nothing — fire and forget | ❌ Messages lost freely | Fastest |
1 |
Leader only | ⚠️ Lost if the leader dies before replicating | Fast |
all (-1) |
All in-sync replicas | ✅ Survives broker failure | Slower |
🚨 acks=all alone isn’t enough. If the ISR has shrunk to just the leader, “all in-sync
replicas” means one replica, and you’re back to acks=1 durability without noticing. You must also
set min.insync.replicas=2 — then a write fails if fewer than 2 replicas are in sync, which is
correct behaviour: refuse the write rather than silently lose durability.
📐 The standard durable configuration:
replication.factor = 3
min.insync.replicas = 2
acks = all
This survives one broker failure with no data loss, and still accepts writes with one broker down. It’s the configuration to state in an interview.
unclean.leader.election.enable=false — don’t promote an out-of-sync replica even if it means
the partition is unavailable. Choosing consistency over availability
(CP). The default is false for good reason; setting it
true trades silent data loss for uptime.
The consumer decides when to commit its offset, and that decision determines your semantics:
# At-most-once: commit first, then process
consumer.commit()
process(msg) # crash here → message lost
# At-least-once: process first, then commit ← the standard choice
process(msg)
consumer.commit() # crash here → message reprocessed (duplicate)
Kafka’s “exactly-once semantics” (EOS) uses idempotent producers (sequence numbers deduplicate retries) and transactions (atomically write to several partitions and commit offsets together).
🚨 It’s exactly-once within Kafka. A read-process-write pipeline that stays inside Kafka can be exactly-once. The moment your consumer charges a credit card or writes to Postgres, you’re outside the transaction boundary and you need idempotency again. Say this precisely — it’s a common place where candidates overclaim.
An alternative to time-based retention: instead of deleting old messages, keep the latest value for each key forever.
Before: (user1, "A") (user2, "X") (user1, "B") (user3, "Z") (user1, "C")
After: (user2, "X") (user3, "Z") (user1, "C")
Why it’s powerful: the topic becomes a durable, replayable snapshot of current state. A new service can read the compacted topic from the beginning and rebuild a complete picture without querying anyone. This is how Kafka Connect stores connector state, how Kafka itself stores consumer offsets, and how CDC topics stay bounded.
Use for: database change streams, configuration, entity state, materialized views.
KRaft — modern Kafka has removed the ZooKeeper dependency, using an internal Raft implementation for metadata. If you’re describing a current deployment, “Kafka with KRaft, no ZooKeeper” is the up-to-date answer.
⚖️ Kafka is genuinely operationally heavy: brokers to size and monitor, partition and retention planning, consumer lag management, rebalance tuning, and a real learning curve.
Don’t use it for:
🎙️ “We need a queue, not an event log — no replay, one consumer group, modest volume. SQS gives us that with zero operational burden. I’d revisit Kafka if we needed multiple independent consumers or replay.”
| Gain | Cost | |
|---|---|---|
| Kafka over a queue | Replay, multiple consumer groups, enormous throughput, long retention | Real operational complexity |
| More partitions | More parallelism | More file handles/memory; slower failover; can’t reduce later |
acks=all + min.insync=2 |
No data loss on broker failure | Higher write latency; writes fail if replicas are down |
| Keyed messages | Per-key ordering | Hot keys concentrate on one partition |
| Log compaction | A replayable snapshot of current state | Loses history; only the latest value per key |
| Long retention | Replay far back; new consumers get full history | Storage cost |
min.insync.replicas when claiming durability with acks=all.1. Run it. docker compose with a single-broker Kafka (KRaft mode — no ZooKeeper needed
anymore). Create a topic with 3 partitions:
kafka-topics --create --topic orders --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-console-producer --topic orders --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
> user1:order-A
> user2:order-B
> user1:order-C
2. See partitioning and ordering. Run two consumers in the same group, then check which
partition each message landed in. Confirm that user1’s messages are always in the same partition
and always in order.
3. Prove replay works. Consume everything. Then:
kafka-consumer-groups --reset-offsets --to-earliest --group my-group --topic orders --execute --bootstrap-server localhost:9092
Watch your consumer reprocess history from the start. This is the thing you cannot do with SQS, and doing it once makes the distinction permanent.
4. Find the partition limit. Create a topic with 2 partitions and start 5 consumers in one group. Watch 3 of them receive nothing. Then increase partitions and watch them wake up.
acks=all not sufficient for durability on its own?