system-design

Message Queues & Pub/Sub

Putting a buffer between two services. It decouples them in time, absorbs traffic spikes, makes retries possible — and quietly changes your system from synchronous to eventually consistent.

Prerequisites: Client–Server Model, Consistency Models Time to read: ~26 minutes


The problem

A user signs up. Your handler does:

def signup(email, password):
    user = db.create_user(email, password)      #   20 ms
    send_welcome_email(user)                    #  800 ms  (external SMTP)
    create_stripe_customer(user)                #  400 ms  (external API)
    index_in_search(user)                       #  100 ms
    notify_analytics(user)                      #   50 ms
    return user                                 # ─────────
                                                # 1,370 ms

Four problems, all serious:

  1. The user waits 1.4 seconds for work they don’t care about.
  2. Any failure fails the signup. Stripe is having a bad day → nobody can register. Your availability is now the product of five services’ availability (the arithmetic).
  3. You can’t retry safely. If the email fails after the user is created, do you fail the whole request? Roll back? Leave it inconsistent?
  4. A traffic spike hits everything at once. 10,000 signups/second means 10,000 concurrent calls to your email provider, who will rate limit you.

The fix: write the user, put a message on a queue, return immediately. Workers do the rest.

def signup(email, password):
    user = db.create_user(email, password)      # 20 ms
    queue.publish("user.created", {"id": user.id})
    return user                                 # 25 ms total

🧠 Mental model: the restaurant order rail

A waiter takes your order. They could stand at the kitchen window until your food is ready (synchronous) — but then they serve four tables a night.

Instead they clip the ticket to a rail and go serve other tables (asynchronous). The kitchen works through tickets at its own pace. If the kitchen is slow, tickets accumulate — visibly, so the manager can see the backlog and act. If a cook drops a ticket, it’s still on the rail.

The rail is a queue. And note what it gives you beyond speed: visibility into the backlog, and work that survives the worker walking away.


Queues vs pub/sub

Two patterns, frequently confused, and interviewers check.

Point-to-point queue: one message, one consumer

flowchart LR
    P[Producer] --> Q[[Queue]]
    Q --> C1[Worker 1]
    Q -.-> C2[Worker 2]
    Q -.-> C3[Worker 3]

Each message is delivered to exactly one worker. Adding workers increases throughput — this is how you distribute work.

Use for: task distribution. Image resizing, sending emails, processing payments, generating reports.

Systems: RabbitMQ, AWS SQS, Redis lists/streams, Beanstalkd.

Publish/subscribe: one message, every subscriber

flowchart LR
    P[Producer] --> T[[Topic: user.created]]
    T --> S1[Email service]
    T --> S2[Analytics service]
    T --> S3[Search indexer]
    T --> S4[Billing service]

Each message goes to all subscribers. The producer doesn’t know or care who’s listening.

Use for: event broadcasting. One event, many independent reactions.

Systems: Kafka, Google Pub/Sub, AWS SNS, Redis pub/sub, NATS.

🚨 The architectural difference matters more than the mechanism. Pub/sub means you can add a new consumer — a fraud detector, a recommendation trainer — without changing the producer at all. That’s the property that makes event-driven architectures extensible, and it’s the real reason to choose it.

Consumer groups combine both: Kafka delivers each message to one consumer within each group, and to every group. So the “email service” group load-balances across its instances, while the “analytics” group independently receives everything.


What a queue actually buys you

Benefit Detail
Decoupling in time The producer doesn’t wait. The consumer can be down for an hour.
Decoupling in knowledge The producer doesn’t know who consumes. Add consumers freely.
Load levelling A 10× spike becomes a longer queue, not a crashed service. This is the big one.
Retries Failed work goes back on the queue. No user-facing failure.
Backpressure Queue depth is a visible, measurable signal that you’re behind.
Durability Work survives a worker crash — the message is still there.
Independent scaling Scale workers separately from your API tier.
Ordering (sometimes) Per-partition ordering guarantees, where supported.

📐 Load levelling, concretely. Without a queue: a spike to 10,000 signups/second means 10,000 concurrent calls to your email provider, who rate limits you and starts returning errors, which you then retry, making it worse. With a queue: the queue absorbs the burst and 50 workers drain it at a steady 500/second. The spike takes 20 seconds to clear. Nobody notices, nothing fails.

This is the single most valuable property of a queue and the one to lead with in an interview.


What a queue costs you

⚖️ You are trading synchronous simplicity for asynchronous complexity. Be honest about the bill.

1. Eventual consistency. The user signs up and the welcome email arrives 2 seconds later. Usually fine. But: the user signs up and their profile isn’t in search for 5 seconds — so they search for themselves and get nothing. Every async boundary is a place where the UI must handle “not yet.”

2. Debugging is harder. A request no longer has one stack trace. It has a producer, a broker, and a consumer, possibly minutes apart, possibly on different machines. Distributed tracing stops being optional.Distributed Tracing

3. Duplicate delivery is guaranteed to happen. See below — this is the big one.

4. Ordering is not free. Most queues only guarantee order within a partition, and many guarantee nothing at all.

5. A new component to operate. The broker needs monitoring, capacity planning, and its own failure handling. And it becomes a critical dependency.

6. Failure moves, it doesn’t disappear. The email still fails — it just fails in a worker at 3 a.m. instead of in the user’s request. You need dead-letter queues and alerting, or failures become silent.


Delivery semantics — the most-asked question here

At-most-once: deliver, don’t retry. Messages can be lost. Fine for metrics; unacceptable for orders.

At-least-once: retry until acknowledged. Duplicates will occur. This is what nearly every real queue provides.

Exactly-once: the message is processed precisely once.

🚨 Exactly-once delivery is impossible in a distributed system, and claiming otherwise is a red flag in interviews. Here’s why:

Worker receives the message
Worker processes it successfully
Worker sends ACK ──────✗ network drops the ACK
Broker never got the ACK → redelivers

The broker cannot distinguish “the worker died before processing” from “the worker processed it and the ACK was lost.” Both look identical. So it must redeliver, and you get a duplicate.

What you actually build: at-least-once delivery + idempotent processing.

def handle_payment(msg):
    # Deduplicate by a stable business key
    if db.exists("processed_events", msg.event_id):
        return                                  # already done — safe to ignore
    with db.transaction():
        charge_card(msg.amount, idempotency_key=msg.event_id)
        db.insert("processed_events", msg.event_id)

This is effectively-once, which is what Kafka’s “exactly-once semantics” actually means — it’s exactly-once within Kafka’s own read-process-write transaction boundary, not across your external side effects.

Idempotency ⭐ — genuinely one of the most valuable things to bring up unprompted in a design interview.


Ordering

Global ordering across a whole topic is expensive — it requires a single partition, which means a single consumer, which means no parallelism. You almost never want it.

Per-key ordering is what you actually need, and it’s cheap. Partition by a key so all messages for that key go to the same partition and are processed in order:

partition = hash(user_id) % num_partitions

Now all events for user 42 are ordered relative to each other, while different users process in parallel. Kafka, Kinesis, and SQS FIFO (via message group ID) all work this way.

🚨 Design so ordering doesn’t matter, where you can. Instead of “increment balance by 10,” send “balance is now 150 as of timestamp T.” Order-independent messages are far more robust, and this reframing is a good thing to suggest.


Failure handling

Dead letter queues (DLQ)

After N failed attempts, move the message to a separate queue rather than retrying forever.

Message fails → retry (backoff) → fails → retry → fails → DLQ

Without a DLQ, one poison message — malformed, or referencing deleted data — is retried forever, consuming worker capacity and filling your logs. This is a real and common outage.

🚨 A DLQ with no alerting is a place where work goes to die silently. Alert on DLQ depth > 0. That’s the whole point of it.

Retries with backoff and jitter

attempt 1: immediate
attempt 2: 1 s  + random(0, 1s)
attempt 3: 2 s  + random(0, 2s)
attempt 4: 4 s  + random(0, 4s)
attempt 5: 8 s  + random(0, 8s) → DLQ

Exponential gives the downstream service room to recover. Jitter prevents all failed messages from retrying in the same instant and re-killing it. → Retries & Timeouts

Visibility timeout / acknowledgment

The worker takes a message; it becomes invisible to other workers for N seconds. If the worker finishes, it ACKs and the message is deleted. If the worker crashes, the timeout expires and another worker picks it up.

🚨 Setting this correctly matters. Too short: a slow job gets processed twice concurrently. Too long: a crashed worker’s message is stuck for ages. Rule of thumb: several times your p99 processing time, and extend it (heartbeat) for genuinely long jobs.


Choosing a broker

  RabbitMQ Kafka SQS Redis Streams NATS
Model Queue + exchanges Distributed log Managed queue Log Messaging
Throughput ~50k/s Millions/s High (managed) ~1M/s Very high
Retention Until consumed Configurable — days/forever 14 days max Configurable Optional
Replay Rewind to any offset Limited
Ordering Per queue Per partition FIFO queues only Per stream Per subject
Routing Rich (topic, fanout, headers) By partition Basic Basic Subject wildcards
Ops burden Medium High None Low (if you run Redis) Low
Best for Complex routing, task queues Event streaming, high volume, replay “Just give me a queue” Already using Redis Low-latency messaging

How to choose, in practice:

🚨 Kafka is not a queue, and treating it as one causes problems. It’s a distributed, durable, replayable log. Consumers track their own offset. That’s why you can add a new consumer that reads all of history — and it’s also why Kafka doesn’t do per-message acknowledgment or selective redelivery the way RabbitMQ does.

🎙️ “I’d use SQS here. We need a queue, not an event log — no replay requirement, no multiple consumer groups — and Kafka would be a significant operational burden for what a managed queue does in an afternoon.”

Choosing the simpler option with reasoning scores better than reaching for Kafka reflexively.


The dual-write problem

🚨 A subtle bug that appears in almost every queue-based design, and spotting it is a strong signal.

db.create_order(order)                   # succeeds
queue.publish("order.created", order)    # ✗ fails — broker unreachable
# Order exists. No downstream system knows. Silent inconsistency forever.

Or the reverse: the publish succeeds and the transaction rolls back, so consumers act on an order that doesn’t exist.

You cannot atomically write to a database and a message broker — they’re separate systems with no shared transaction.

Solutions:

1. Transactional outbox. Write the event to an outbox table in the same database transaction as the business data. A separate process reads the outbox and publishes.

BEGIN;
  INSERT INTO orders (...) VALUES (...);
  INSERT INTO outbox (event_type, payload) VALUES ('order.created', '...');
COMMIT;                                   -- atomic: both or neither

Then a relay publishes from outbox and marks rows as sent. Duplicates are possible (publish succeeds, mark fails) — which is fine, because consumers are idempotent.

2. Change data capture. Read the database’s replication log and publish changes automatically. No outbox table, no application changes. → CDC

3. Event sourcing. The event log is the source of truth; state is derived from it. No dual write exists. → Event Sourcing

🎙️ “Writing to the database and publishing to the queue isn’t atomic, so I’d use a transactional outbox — the event goes into the same transaction as the order, and a relay publishes from there.”


⚖️ Trade-offs

Decision Gain Cost
Add a queue Fast responses, spike absorption, retries, decoupling Eventual consistency, harder debugging, duplicates, a broker to run
At-least-once + idempotency Never lose work Consumers must be idempotent; dedup storage
Per-key ordering Correct ordering where it matters, with parallelism Hot keys concentrate on one partition
Global ordering Total order One partition, one consumer, no parallelism
DLQ Poison messages don’t block the queue Must be monitored, or failures are silent
Managed (SQS) No operations Less control, per-message cost, feature limits
Kafka Replay, huge throughput, many consumer groups Real operational complexity

In the real world


🚨 Interview traps


🎙️ Soundbites


🛠️ Try it

1. Move work off the request path. Take any endpoint doing something slow. Add RabbitMQ or Redis in Docker, publish a message, and process it in a worker. Measure the endpoint’s p99 before and after. The number will be dramatic.

2. Cause and fix a duplicate. Have your worker ACK after processing, then kill it mid-process with kill -9. Watch the message get redelivered and processed twice. Then add idempotency keyed on the message ID and watch the second attempt no-op.

3. Watch load levelling work. Producer publishing at 1,000 msg/s; consumer processing 100/s. Graph queue depth over time. Then start 10 consumers and watch it drain. Seeing the backlog build and drain makes queue-depth monitoring feel obvious rather than abstract.

4. Build a poison message. Publish a message your consumer can’t parse. Watch it retry forever and consume a worker. Add a DLQ after 3 attempts. Now watch it get quarantined instead.


Check yourself

1. Why is exactly-once delivery impossible, and what do you build instead? Because a lost acknowledgment is indistinguishable from a crashed consumer. The broker sends a message, the consumer processes it and ACKs, the ACK is lost — the broker must assume failure and redeliver, producing a duplicate. There's no way to close this gap without a distributed transaction across the broker and the consumer's side effects, which is impractical. What you build is **at-least-once delivery plus idempotent processing**: dedupe on a stable business key, ideally in the same transaction as the side effect. That gives you effectively-once behaviour.
2. What's the dual-write problem and how does the outbox pattern solve it? You need to update the database *and* publish an event, but they're separate systems with no shared transaction. Either can fail after the other succeeded, leaving your database and your consumers permanently inconsistent — and it fails silently. The transactional outbox writes the event into an `outbox` table inside the same database transaction as the business data, so both commit or neither does. A separate relay process then reads the outbox and publishes, marking rows sent. If the relay crashes between publishing and marking, you get a duplicate — which is harmless because consumers are idempotent.
3. When do you need a message queue, and when is a direct synchronous call better? Use a queue when: the work isn't needed for the response, the work is slow or calls an unreliable external service, traffic is spiky and you need load levelling, multiple independent consumers care about the event, or the work must survive failures and be retried. Use a direct call when: the caller needs the result to respond, the operation must be strongly consistent, latency budget is tight and the callee is fast and reliable, or the added complexity (eventual consistency, tracing, duplicate handling, another system to operate) isn't justified. Not everything needs to be async.
4. Your queue depth is growing steadily. What's happening and what are your options? Consumers can't keep up — arrival rate exceeds processing rate. Diagnose first: is it a traffic increase, a slowdown in consumers (a slow downstream dependency, a bad deploy, a missing index), or a reduction in worker count? Options: scale out consumers (if the work parallelizes and downstream systems can take it); make processing faster (batch, remove a slow call); shed or defer low-priority messages; apply backpressure to producers; or, if it's a temporary spike, do nothing — absorbing spikes is what the queue is *for*. The key distinction is whether depth is growing because of a bounded burst (fine) or because throughput is structurally insufficient (needs a fix).
5. Why is global message ordering usually a bad requirement, and what's the alternative? Global ordering requires a single partition processed by a single consumer, which eliminates all parallelism and caps your throughput at one worker — and creates a single point of failure with no ability to scale. The alternative is **per-key ordering**: partition by an entity key (user ID, account ID, order ID) so all messages for that entity are ordered relative to each other while different entities process concurrently. Better still, design messages to be order-independent (send absolute state, "balance is now 150," rather than relative deltas, "add 10"), so ordering stops being a correctness requirement at all.

Further reading