Putting a buffer between two services. It decouples them in time, absorbs traffic spikes, makes retries possible — and quietly changes your system from synchronous to eventually consistent.
Prerequisites: Client–Server Model, Consistency Models Time to read: ~26 minutes
A user signs up. Your handler does:
def signup(email, password):
user = db.create_user(email, password) # 20 ms
send_welcome_email(user) # 800 ms (external SMTP)
create_stripe_customer(user) # 400 ms (external API)
index_in_search(user) # 100 ms
notify_analytics(user) # 50 ms
return user # ─────────
# 1,370 ms
Four problems, all serious:
The fix: write the user, put a message on a queue, return immediately. Workers do the rest.
def signup(email, password):
user = db.create_user(email, password) # 20 ms
queue.publish("user.created", {"id": user.id})
return user # 25 ms total
A waiter takes your order. They could stand at the kitchen window until your food is ready (synchronous) — but then they serve four tables a night.
Instead they clip the ticket to a rail and go serve other tables (asynchronous). The kitchen works through tickets at its own pace. If the kitchen is slow, tickets accumulate — visibly, so the manager can see the backlog and act. If a cook drops a ticket, it’s still on the rail.
The rail is a queue. And note what it gives you beyond speed: visibility into the backlog, and work that survives the worker walking away.
Two patterns, frequently confused, and interviewers check.
flowchart LR
P[Producer] --> Q[[Queue]]
Q --> C1[Worker 1]
Q -.-> C2[Worker 2]
Q -.-> C3[Worker 3]
Each message is delivered to exactly one worker. Adding workers increases throughput — this is how you distribute work.
Use for: task distribution. Image resizing, sending emails, processing payments, generating reports.
Systems: RabbitMQ, AWS SQS, Redis lists/streams, Beanstalkd.
flowchart LR
P[Producer] --> T[[Topic: user.created]]
T --> S1[Email service]
T --> S2[Analytics service]
T --> S3[Search indexer]
T --> S4[Billing service]
Each message goes to all subscribers. The producer doesn’t know or care who’s listening.
Use for: event broadcasting. One event, many independent reactions.
Systems: Kafka, Google Pub/Sub, AWS SNS, Redis pub/sub, NATS.
🚨 The architectural difference matters more than the mechanism. Pub/sub means you can add a new consumer — a fraud detector, a recommendation trainer — without changing the producer at all. That’s the property that makes event-driven architectures extensible, and it’s the real reason to choose it.
Consumer groups combine both: Kafka delivers each message to one consumer within each group, and to every group. So the “email service” group load-balances across its instances, while the “analytics” group independently receives everything.
| Benefit | Detail |
|---|---|
| Decoupling in time | The producer doesn’t wait. The consumer can be down for an hour. |
| Decoupling in knowledge | The producer doesn’t know who consumes. Add consumers freely. |
| Load levelling | A 10× spike becomes a longer queue, not a crashed service. This is the big one. |
| Retries | Failed work goes back on the queue. No user-facing failure. |
| Backpressure | Queue depth is a visible, measurable signal that you’re behind. |
| Durability | Work survives a worker crash — the message is still there. |
| Independent scaling | Scale workers separately from your API tier. |
| Ordering (sometimes) | Per-partition ordering guarantees, where supported. |
📐 Load levelling, concretely. Without a queue: a spike to 10,000 signups/second means 10,000 concurrent calls to your email provider, who rate limits you and starts returning errors, which you then retry, making it worse. With a queue: the queue absorbs the burst and 50 workers drain it at a steady 500/second. The spike takes 20 seconds to clear. Nobody notices, nothing fails.
This is the single most valuable property of a queue and the one to lead with in an interview.
⚖️ You are trading synchronous simplicity for asynchronous complexity. Be honest about the bill.
1. Eventual consistency. The user signs up and the welcome email arrives 2 seconds later. Usually fine. But: the user signs up and their profile isn’t in search for 5 seconds — so they search for themselves and get nothing. Every async boundary is a place where the UI must handle “not yet.”
2. Debugging is harder. A request no longer has one stack trace. It has a producer, a broker, and a consumer, possibly minutes apart, possibly on different machines. Distributed tracing stops being optional. → Distributed Tracing
3. Duplicate delivery is guaranteed to happen. See below — this is the big one.
4. Ordering is not free. Most queues only guarantee order within a partition, and many guarantee nothing at all.
5. A new component to operate. The broker needs monitoring, capacity planning, and its own failure handling. And it becomes a critical dependency.
6. Failure moves, it doesn’t disappear. The email still fails — it just fails in a worker at 3 a.m. instead of in the user’s request. You need dead-letter queues and alerting, or failures become silent.
At-most-once: deliver, don’t retry. Messages can be lost. Fine for metrics; unacceptable for orders.
At-least-once: retry until acknowledged. Duplicates will occur. This is what nearly every real queue provides.
Exactly-once: the message is processed precisely once.
🚨 Exactly-once delivery is impossible in a distributed system, and claiming otherwise is a red flag in interviews. Here’s why:
Worker receives the message
Worker processes it successfully
Worker sends ACK ──────✗ network drops the ACK
Broker never got the ACK → redelivers
The broker cannot distinguish “the worker died before processing” from “the worker processed it and the ACK was lost.” Both look identical. So it must redeliver, and you get a duplicate.
What you actually build: at-least-once delivery + idempotent processing.
def handle_payment(msg):
# Deduplicate by a stable business key
if db.exists("processed_events", msg.event_id):
return # already done — safe to ignore
with db.transaction():
charge_card(msg.amount, idempotency_key=msg.event_id)
db.insert("processed_events", msg.event_id)
This is effectively-once, which is what Kafka’s “exactly-once semantics” actually means — it’s exactly-once within Kafka’s own read-process-write transaction boundary, not across your external side effects.
→ Idempotency ⭐ — genuinely one of the most valuable things to bring up unprompted in a design interview.
Global ordering across a whole topic is expensive — it requires a single partition, which means a single consumer, which means no parallelism. You almost never want it.
Per-key ordering is what you actually need, and it’s cheap. Partition by a key so all messages for that key go to the same partition and are processed in order:
partition = hash(user_id) % num_partitions
Now all events for user 42 are ordered relative to each other, while different users process in parallel. Kafka, Kinesis, and SQS FIFO (via message group ID) all work this way.
🚨 Design so ordering doesn’t matter, where you can. Instead of “increment balance by 10,” send “balance is now 150 as of timestamp T.” Order-independent messages are far more robust, and this reframing is a good thing to suggest.
After N failed attempts, move the message to a separate queue rather than retrying forever.
Message fails → retry (backoff) → fails → retry → fails → DLQ
Without a DLQ, one poison message — malformed, or referencing deleted data — is retried forever, consuming worker capacity and filling your logs. This is a real and common outage.
🚨 A DLQ with no alerting is a place where work goes to die silently. Alert on DLQ depth > 0. That’s the whole point of it.
attempt 1: immediate
attempt 2: 1 s + random(0, 1s)
attempt 3: 2 s + random(0, 2s)
attempt 4: 4 s + random(0, 4s)
attempt 5: 8 s + random(0, 8s) → DLQ
Exponential gives the downstream service room to recover. Jitter prevents all failed messages from retrying in the same instant and re-killing it. → Retries & Timeouts
The worker takes a message; it becomes invisible to other workers for N seconds. If the worker finishes, it ACKs and the message is deleted. If the worker crashes, the timeout expires and another worker picks it up.
🚨 Setting this correctly matters. Too short: a slow job gets processed twice concurrently. Too long: a crashed worker’s message is stuck for ages. Rule of thumb: several times your p99 processing time, and extend it (heartbeat) for genuinely long jobs.
| RabbitMQ | Kafka | SQS | Redis Streams | NATS | |
|---|---|---|---|---|---|
| Model | Queue + exchanges | Distributed log | Managed queue | Log | Messaging |
| Throughput | ~50k/s | Millions/s | High (managed) | ~1M/s | Very high |
| Retention | Until consumed | Configurable — days/forever | 14 days max | Configurable | Optional |
| Replay | ❌ | ✅ Rewind to any offset | ❌ | ✅ | Limited |
| Ordering | Per queue | Per partition | FIFO queues only | Per stream | Per subject |
| Routing | Rich (topic, fanout, headers) | By partition | Basic | Basic | Subject wildcards |
| Ops burden | Medium | High | None | Low (if you run Redis) | Low |
| Best for | Complex routing, task queues | Event streaming, high volume, replay | “Just give me a queue” | Already using Redis | Low-latency messaging |
How to choose, in practice:
🚨 Kafka is not a queue, and treating it as one causes problems. It’s a distributed, durable, replayable log. Consumers track their own offset. That’s why you can add a new consumer that reads all of history — and it’s also why Kafka doesn’t do per-message acknowledgment or selective redelivery the way RabbitMQ does.
🎙️ “I’d use SQS here. We need a queue, not an event log — no replay requirement, no multiple consumer groups — and Kafka would be a significant operational burden for what a managed queue does in an afternoon.”
Choosing the simpler option with reasoning scores better than reaching for Kafka reflexively.
🚨 A subtle bug that appears in almost every queue-based design, and spotting it is a strong signal.
db.create_order(order) # succeeds
queue.publish("order.created", order) # ✗ fails — broker unreachable
# Order exists. No downstream system knows. Silent inconsistency forever.
Or the reverse: the publish succeeds and the transaction rolls back, so consumers act on an order that doesn’t exist.
You cannot atomically write to a database and a message broker — they’re separate systems with no shared transaction.
Solutions:
1. Transactional outbox. Write the event to an outbox table in the same database transaction
as the business data. A separate process reads the outbox and publishes.
BEGIN;
INSERT INTO orders (...) VALUES (...);
INSERT INTO outbox (event_type, payload) VALUES ('order.created', '...');
COMMIT; -- atomic: both or neither
Then a relay publishes from outbox and marks rows as sent. Duplicates are possible (publish
succeeds, mark fails) — which is fine, because consumers are idempotent.
2. Change data capture. Read the database’s replication log and publish changes automatically. No outbox table, no application changes. → CDC
3. Event sourcing. The event log is the source of truth; state is derived from it. No dual write exists. → Event Sourcing
🎙️ “Writing to the database and publishing to the queue isn’t atomic, so I’d use a transactional outbox — the event goes into the same transaction as the order, and a relay publishes from there.”
| Decision | Gain | Cost |
|---|---|---|
| Add a queue | Fast responses, spike absorption, retries, decoupling | Eventual consistency, harder debugging, duplicates, a broker to run |
| At-least-once + idempotency | Never lose work | Consumers must be idempotent; dedup storage |
| Per-key ordering | Correct ordering where it matters, with parallelism | Hot keys concentrate on one partition |
| Global ordering | Total order | One partition, one consumer, no parallelism |
| DLQ | Poison messages don’t block the queue | Must be monitored, or failures are silent |
| Managed (SQS) | No operations | Less control, per-message cost, feature limits |
| Kafka | Replay, huge throughput, many consumer groups | Real operational complexity |
1. Move work off the request path. Take any endpoint doing something slow. Add RabbitMQ or Redis in Docker, publish a message, and process it in a worker. Measure the endpoint’s p99 before and after. The number will be dramatic.
2. Cause and fix a duplicate. Have your worker ACK after processing, then kill it mid-process
with kill -9. Watch the message get redelivered and processed twice. Then add idempotency keyed on
the message ID and watch the second attempt no-op.
3. Watch load levelling work. Producer publishing at 1,000 msg/s; consumer processing 100/s. Graph queue depth over time. Then start 10 consumers and watch it drain. Seeing the backlog build and drain makes queue-depth monitoring feel obvious rather than abstract.
4. Build a poison message. Publish a message your consumer can’t parse. Watch it retry forever and consume a worker. Add a DLQ after 3 attempts. Now watch it get quarantined instead.