Getting data to a client the moment it changes. The server can’t call you, so every solution is a variation on “the client asks and waits.”
Prerequisites: Client–Server Model, HTTP & TLS Time to read: ~22 minutes
HTTP is request–response. The client asks; the server answers. The server has no way to start a conversation.
But you constantly need server-initiated updates: a chat message arrives, a ride is matched, a price moves, a notification fires, a collaborator edits the document.
And the client is unreachable — behind carrier NAT, on a phone whose IP changes, with no listening port. The client must establish and hold the connection. Every technique below is a different way of doing that.
Ask repeatedly.
setInterval(async () => {
const messages = await fetch('/api/messages?since=' + lastId).then(r => r.json());
render(messages);
}, 3000);
✅ Trivially simple. Works everywhere. Stateless servers — any instance serves any poll. ❌ Latency up to the poll interval. ❌ Massively wasteful. Most responses are empty.
📐 The arithmetic that kills it: 100,000 users polling every 3 seconds = 33,000 requests/second — and if updates are rare, 99% return nothing. You’re paying full request cost (TCP, TLS on new connections, auth, database query) to say “nothing new.”
Use when: updates are genuinely infrequent and multi-second latency is fine. Checking for app updates, refreshing a dashboard every minute. Don’t dismiss it — for many features it’s the right, boring answer.
Ask, and the server holds the request open until there’s something to say (or a timeout).
async function poll() {
try {
const res = await fetch('/api/messages/wait?since=' + lastId); // may hang 30s
if (res.ok) render(await res.json());
} catch (e) { await sleep(1000); }
poll(); // immediately re-establish
}
✅ Near-instant delivery — the server responds the moment data exists.
✅ Works through every proxy and firewall; it’s just HTTP.
✅ No empty responses.
❌ One held connection per client, so servers become connection-bound.
❌ A reconnect after every message — wasteful for high-frequency streams.
❌ 🚨 There’s a gap between responses. A message arriving in the microsecond between the server
responding and the client reconnecting can be missed — unless you use a cursor (since=lastId), which
you must.
Use when: you need real-time delivery but can’t use WebSockets, or as a fallback. It’s still widely deployed for exactly that reason.
One long-lived HTTP response that the server streams into. Server → client only.
const events = new EventSource('/api/stream');
events.onmessage = (e) => render(JSON.parse(e.data));
events.addEventListener('price', (e) => updatePrice(e.data));
// Automatic reconnection is built in
HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
data: {"msg": "hello"}\n\n
id: 42\n
event: price\n
data: {"symbol": "AAPL", "price": 195.3}\n\n
✅ Plain HTTP — works with existing infrastructure, auth, and compression.
✅ Automatic reconnection built into the browser, including Last-Event-ID replay so you don’t
lose messages across a reconnect. This is genuinely excellent and underappreciated.
✅ Simpler than WebSockets on both ends.
❌ One-way only. Client→server needs a separate normal request.
❌ Text only (base64 your binary).
❌ 🚨 Over HTTP/1.1, browsers limit ~6 connections per domain — so 6 tabs and the seventh hangs.
Over HTTP/2 this disappears (multiplexing). Worth knowing.
🚨 SSE is under-used. For a notification feed, a live dashboard, an activity stream, or LLM token streaming — anything one-directional — SSE is simpler than WebSockets and gives you reconnection for free. Proposing SSE where it fits, instead of reflexively reaching for WebSockets, is a good signal.
A persistent, full-duplex connection. Starts as HTTP, then upgrades:
GET /chat HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
After that, it’s a raw bidirectional message channel with ~2–14 bytes of framing overhead per message.
✅ True bidirectional, low latency, minimal per-message overhead. ✅ Binary or text. ❌ Stateful servers — with everything that implies (below). ❌ Not plain HTTP, so proxies and load balancers need explicit configuration. ❌ You implement reconnection, heartbeats, and message replay yourself.
Use when: you genuinely need client→server messages at high frequency — chat, collaborative editing, multiplayer games, trading interfaces.
| Short poll | Long poll | SSE | WebSocket | |
|---|---|---|---|---|
| Direction | Client pulls | Client pulls | Server → client | Bidirectional |
| Latency | Up to interval | Near-instant | Near-instant | Lowest |
| Connections held | None | 1 per client | 1 per client | 1 per client |
| Server state | Stateless ✅ | Stateful | Stateful | Stateful |
| Auto-reconnect | N/A | Manual | Built in ✅ | Manual |
| Proxy-friendly | ✅ | ✅ | ✅ | Needs config |
| Overhead/message | Full HTTP | Full HTTP | ~10 bytes | ~2–14 bytes |
| Complexity | ⭐ | ⭐⭐ | ⭐⭐ | ⭐⭐⭐⭐ |
🎙️ The decision rule: “Do I need client→server messages at high frequency? If no, SSE. If yes, WebSockets. If the volume is low or latency tolerance is seconds, just poll.”
🚨 This is the part that matters in a design interview, and it’s what candidates who’ve only read about WebSockets miss.
📐 Capacity. Each connection costs kernel buffers plus application state — call it ~10–50 KB.
1 million concurrent connections × 30 KB = 30 GB of connection memory alone
Plus a file descriptor each (raise ulimit -n), plus per-connection heap in your runtime. Realistic
per-node capacity is 50,000–500,000 connections depending on language and how much state you keep,
which means 10–20 nodes for a million users just to hold connections.
And here’s the structural problem: user A’s connection lives on gateway node 3. A message for user A arrives at node 7. How does node 7 reach user A?
flowchart LR
A[User A] -.WebSocket.-> G3[Gateway 3]
B[User B] -.WebSocket.-> G7[Gateway 7]
G7 -->|1 . send to user A| PS[[Redis Pub/Sub<br/>or Kafka]]
PS -->|2 . delivered| G3
G3 -.->|3 . pushed| A
R[(Connection registry<br/>user → node)] --- G3
R --- G7
Two mechanisms are needed:
user_id → gateway_node, in Redis with a TTL, updated on connect and
disconnect. So any node can find out where a user is connected.🚨 Simpler variant worth mentioning: skip the registry and have every gateway subscribe to a
Redis channel per connected user (user:42). Publishing to user:42 reaches whichever node has
them, with no lookup. Costs more subscriptions; removes a lookup and a consistency problem. Both
answers are good — knowing the trade-off is better.
This routing problem is the heart of the chat system design question.
Load balancers need configuration.
location /ws {
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 3600s; # or the LB kills idle connections
}
Forgetting the Upgrade headers or leaving a 60-second idle timeout is the single most common
“WebSockets don’t work in production but work locally” bug.
Deploys disconnect everyone. Rolling a fleet of gateway nodes drops every connection on each node. At scale, all clients reconnecting simultaneously is a thundering herd that can take down the new instances.
Mitigations: drain slowly (disconnect a small percentage at a time), and require clients to reconnect with exponential backoff and jitter. The jitter is not optional — without it, a million clients reconnect in the same second.
Heartbeats are mandatory. TCP connections die silently — NAT tables expire (often ~5 minutes idle), mobile networks drop, laptops sleep. Without a ping/pong every ~30 seconds, both sides hold dead connections believing they’re fine. Servers leak memory; clients think they’re connected and receive nothing.
Message loss across reconnects. When a client reconnects, what did it miss? You need either:
sequence numbers so the client requests everything after its last seen ID (SSE gives you
Last-Event-ID for free), or server-side buffering of recent messages per user, or a “fetch missed
messages” REST call on reconnect. Design this explicitly — “reconnect and hope” loses messages.
Backpressure. A slow client (bad mobile connection) can’t consume as fast as you produce. The server’s send buffer grows until it OOMs. You need: bounded per-connection buffers, and a policy — drop old messages, drop the connection, or slow the producer.
Auth. The browser WebSocket API can’t set custom headers, so you can’t send
Authorization: Bearer. Options: a token in the query string (ends up in logs — use a short-lived
one-time ticket), a cookie (works, but consider CSRF — check the Origin header), or authenticate as
the first message after connecting.
Separate the connection tier from the business logic tier.
Clients → [Gateway nodes: hold connections only] → [Stateless services: business logic]
Gateways are dumb, memory-heavy, and scaled by connection count. Services are stateless, CPU-bound, and scaled by request rate. Deploying a business-logic change then doesn’t disconnect anyone — this alone justifies the split.
Use sticky routing at the load balancer so reconnects prefer the same node (warm state), but never depend on it.
Consider a managed service. Pusher, Ably, AWS API Gateway WebSockets, PubNub, or Supabase Realtime handle connection management, scaling, and reconnection for you. For many products this is the right call — connection management at scale is a real engineering investment.
🎙️ “For a million concurrent connections I’d separate the gateway tier from business logic, keep a user→node registry in Redis, and route messages through pub/sub. If this weren’t core to our product, I’d seriously consider a managed realtime service instead.”
| Decision | Gain | Cost |
|---|---|---|
| Polling | Stateless, trivially simple, scales like normal HTTP | Latency; wasted requests |
| Long polling | Instant delivery over plain HTTP | Held connections; reconnect per message |
| SSE | Instant, simple, free reconnection + replay | One-way; HTTP/1.1 connection limit |
| WebSockets | Bidirectional, lowest overhead | Stateful servers, routing layer, manual reconnect/heartbeat |
| Managed service | No connection infrastructure to build | Cost, vendor dependency, less control |
| Separate gateway tier | Deploys don’t disconnect users | An extra tier and an internal hop |
Last-Event-ID replay for free.”1. Build all four. The same feature — a live counter — as short polling, long polling, SSE, and WebSockets. Open 100 tabs against each and compare: requests/second in the server log, server memory, and observed latency. The numbers make the trade-offs concrete in a way no table does.
2. Find your connection ceiling. Write a script opening WebSocket connections until something
breaks. You’ll hit ulimit -n first (raise it), then memory. Divide 1,000,000 by your result — that’s
how many nodes you’d need.
3. Break it through a proxy. Run your WebSocket server behind Nginx without the Upgrade
headers. Watch it fail. Add them. Then set proxy_read_timeout 60s and watch idle connections drop
after a minute. This is the most common production WebSocket bug, and causing it once means you’ll
never be confused by it.
4. Simulate the reconnect storm. Open 1,000 connections, kill the server, restart it. Watch what happens without backoff (all reconnect instantly), then with jittered exponential backoff.