system-design

Design a Chat System (WhatsApp / Messenger / Slack)

Difficulty: Tier 2 Asked at: Meta, Careem, Talabat, Slack, Amazon Time budget: 45–60 min

Chat flips the feed problem on its head. Feeds are read-heavy pull systems; chat is a real-time, push, stateful-connection system — messages must arrive in milliseconds, in order, exactly once, even when the recipient is offline. The heart of this question is persistent connections (WebSockets), presence, and delivery guarantees. It’s the canonical push-messaging design.

Prerequisites: WebSockets & SSE, Message Queues, Idempotency

1. Requirements

Functional:

Non-functional:

Out of scope: end-to-end encryption internals (mention it), voice/video calls.

2. Estimation

3. API / protocol

Not REST for the message path — a persistent connection:

WebSocket connect  → authenticated, long-lived
  send:    { type: "message", to, chatId, clientMsgId, text }
  receive: { type: "message", from, chatId, msgId, text, ts }
           { type: "receipt", msgId, status: "delivered"|"read" }
           { type: "presence", userId, status: "online"|"offline" }

REST for history/media: GET /chats/{id}/messages?before=..., POST /media.

🚨 WebSockets, not polling. Polling for new messages at this scale is wasteful and slow; a persistent connection lets the server push instantly. (WebSockets & SSE)

4. High-level design

flowchart TB
    A[User A] <-->|WebSocket| CS1[Connection Server 1]
    B[User B] <-->|WebSocket| CS2[Connection Server 2]
    CS1 --> Router[Message Router / Service]
    CS2 --> Router
    Router --> Store[(Message store<br/>wide-column)]
    Router --> Presence[(Presence store<br/>Redis)]
    Router --> Registry[(Connection registry<br/>user → server)]
    Router --> Push[Push service<br/>APNs/FCM for offline]

Send flow: A sends over its WebSocket to Connection Server 1 → the message router persists the message, looks up where B is connected (connection registry: user → server) → forwards to Connection Server 2 → pushes down B’s WebSocket. If B is offline, store it and send a mobile push notification.

🚨 The connection registry (which server holds which user’s socket) is the key piece that lets one server route a message to a user connected to a different server.

5. Deep dives

5a. Managing millions of connections

Connections are stateful — a user is pinned to one connection server for the life of the socket. Connection servers are horizontally scaled; a service discovery/registry (Redis) maps user_id → connection_server. When A messages B, the router uses the registry to find B’s server and forwards there. Heartbeats/pings detect dead connections; on disconnect, update presence and the registry.

5b. Delivery guarantees & ordering

5c. Offline delivery

If the recipient is offline (no active socket), persist the message in their inbox / mailbox and send a push notification (APNs/FCM). When they reconnect, they sync undelivered messages from the store (everything since their last-received sequence number). 🚨 This “store-and-sync on reconnect” is what makes chat reliable across flaky mobile networks.

5d. Group chats & fan-out

A group message must reach all members. For small/medium groups, fan out to each member’s connection (or inbox if offline) — like a mini feed fan-out. For very large groups (Slack channels, broadcast lists), fan-out on read or a hybrid, to avoid amplification. Store the message once; deliver references.

5e. Presence

Presence (online/offline/last-seen) is high-churn. Store in Redis, updated on connect/disconnect and heartbeats. Broadcasting every presence change to everyone is expensive — only push presence for a user’s contacts / open chats, and use a short TTL so a crashed connection expires to offline automatically.

6. Bottlenecks & scaling further

  1. Concurrent connections → many connection servers; registry maps users to servers.
  2. Routing across servers → connection registry + a message router / pub-sub between servers.
  3. Message storage → wide-column store sharded by chat/user; time-sortable IDs.
  4. Offline sync → per-user mailbox + sync-since-sequence on reconnect.
  5. Presence churn → Redis + TTL, scoped broadcasts.
  6. Group amplification → hybrid fan-out for large groups.

7. Trade-off summary

Decision Chosen Alternative Why
Transport WebSocket (push) HTTP polling Instant delivery; no wasteful polling at scale
Connection state Stateful, pinned to a server Stateless Long-lived sockets must live somewhere; registry routes
Delivery At-least-once + idempotency Exactly-once (hard) Reliable without impossible guarantees
Ordering Per-chat sequence numbers Global order Only per-chat order matters; cheaper
Offline Persist + push + sync Drop or block Reliable over flaky mobile networks

8. Follow-up questions

How does a message reach a user connected to a different server? Through the connection registry. Each connection server holds the live WebSockets for the users pinned to it, and a shared registry (e.g. Redis) maps every online user to the server currently holding their socket. When user A (on server 1) sends to user B, the message router persists the message, looks up B in the registry, sees B is on server 2, and forwards the message to server 2 (via an internal RPC or a pub-sub channel between servers), which pushes it down B's socket. So no single server needs everyone's connection — the registry lets any server deliver to any user by locating their server first.
How do you guarantee a message is never lost but also never shown twice? Persist before acknowledging, and dedupe on delivery. The message is written durably to the store before the sender gets its "sent" ack, so even if the recipient's delivery fails, the message survives and can be re-delivered on reconnect — nothing is lost. Delivery to the recipient is at-least-once (retried until acked), which can cause duplicates, so each message carries a client-generated ID and the recipient (and the store) dedupe on it — a redelivered message with a seen ID is dropped, not displayed again. This combination gives the practical "exactly-once *effect*" without needing true exactly-once delivery, which is effectively impossible across unreliable networks and devices.
A user was offline for a day. How do they get their messages on reconnect? Messages sent while they were offline were persisted to their per-user mailbox/inbox (and triggered a push notification at the time). Each user tracks the sequence number / timestamp of the last message they've received. On reconnect, the client sends that watermark and the server streams every message since it, in order, so the client syncs the full backlog. Delivery is idempotent, so any message that was mid-flight when they dropped won't be duplicated. This store-and-sync-on-reconnect model is what makes chat robust to the constant disconnects of mobile networks.
How do you keep presence (online status) cheap at a billion users? Store presence in an in-memory store (Redis) updated on connect/disconnect and periodic heartbeats, and use a short TTL so that if a connection dies without a clean disconnect, the key simply expires and the user flips to offline automatically — no explicit cleanup needed. Crucially, don't broadcast every presence change to everyone: only notify the people who care — a user's contacts or the participants of chats they have open — which bounds the fan-out. Presence is high-churn, so scoping who gets updates and relying on TTL expiry instead of exact bookkeeping keeps it affordable.
How would you handle a 10,000-member group? Don't fan out one message to 10,000 sockets synchronously. Store the message once, and for large groups use a fan-out-on-read or hybrid model: online members can be pushed a lightweight notification/reference and pull the message, while offline members get it from their mailbox on reconnect. For truly broadcast-scale groups (huge Slack channels), this looks like the celebrity problem from the feed designs — you avoid amplification by not eagerly delivering to every member and instead let clients pull, with the message stored a single time. Small and medium groups can simply fan out to each member's connection/inbox.

9. What junior / mid / senior answers look like

Further reading

This site is open source. Improve this page.