system-design

Hot Keys and Celebrity Problems

Perfectly even sharding, and one node is on fire. The reason: real traffic isn’t uniform — a few keys get most of the load, and no amount of clever hashing distributes load when it’s concentrated on one key.

Prerequisites: Consistent Hashing, Sharding Time to read: ~16 minutes

The problem

You shard perfectly. Consistent hashing distributes keys evenly across nodes. And yet one node is at 100% while the rest idle.

🚨 The cause: consistent hashing distributes keys evenly, but not load. Real traffic follows a Zipf/power-law distribution — a small number of keys get the overwhelming majority of requests. A celebrity with 100M followers, a viral post, a trending product, one enterprise tenant 1000× the size of the others. That one hot key maps to one node, and no hashing scheme can spread a single key’s load across multiple nodes, because they’d all need the same key.

📐 This is the “celebrity problem” and it’s fundamental — you can’t hash your way out of it, because the problem is that a single logical entity is too popular for one machine.

Where hot keys hurt

Hot keys appear at every layer:

Cache hot key — one Redis key so popular it saturates the CPU/network of the one node holding it.
Database hot row / hot partition — a celebrity’s row, or all of “today’s” writes hitting one time-partitioned shard.
Hot counter — millions of writes to one value (viral post likes → scaling writes).
Hot shard — one shard (a whale tenant, a popular product) getting disproportionate traffic. → Sharding
Celebrity fan-out — a post from someone with 100M followers must fan out to 100M feeds → News Feed.

The fixes

🚨 The core strategies — spread the load off the single key:

1. Replicate the hot key

🚨 Make N copies of the hot key (hotkey:0 .. hotkey:9), each on a different node; reads pick a random copy, spreading read load across N nodes.

✅ Spreads read load N-fold. ❌ Writes must update all N copies (fine for read-heavy hot keys — most are). Adds complexity.

2. Cache it in-process on every server

🚨 For an extremely hot read key, cache it locally on every application server (in-process, L1). Then it never even reaches the shared cache/database — every server serves it from its own memory.

✅ Ultimate read scaling — the hot key’s read load is distributed across all app servers automatically. ❌ Per-server memory; brief inconsistency (each server’s copy updates independently, accept a short TTL). This is the standard fix for the hottest read keys.

3. Serve it from the CDN

If the hot key is public and cacheable, push it to the CDN — served from the edge, never touching your origin. Perfect for a viral public post.

4. Sharded counters (for hot writes)

Split a hot counter into N sub-counters, increment a random one, sum on read. → Scaling Writes

5. Dedicated resources for the whale

🚨 For a hot shard (a huge tenant), give it its own dedicated node/shard — isolate the whale so it doesn’t affect everyone else. Needs directory-based sharding to place specific keys on specific nodes. This is the multi-tenancy tiering answer.

6. Request coalescing

🚨 When many requests hit the same hot key simultaneously and it’s a cache miss, deduplicate them — one request fetches, the rest wait for the result. Prevents a hot key’s cache miss from stampeding the database.

The celebrity fan-out problem

🚨 A specific, famous hot-key case worth knowing (news feed):

Fan-out-on-write precomputes each user’s feed at write time — great, until a celebrity with 100M followers posts, and that one write must fan out to 100M feeds. The write becomes catastrophically expensive; a hot key at the write path.

The fix: hybrid fan-out. Fan out on write for normal users (cheap), but for celebrities, don’t fan out — instead, followers pull the celebrity’s posts at read time and merge them into their feed. 🚨 So most of the feed is precomputed (fast), and celebrity posts are pulled (avoiding the 100M-write explosion). This hybrid is the canonical answer to “design Twitter’s feed.”

Detecting hot keys

🚨 You can’t fix what you can’t see — detect hot keys before they cause an incident:

Track per-key request counts — approximate with a Count-Min Sketch (bounded memory, finds heavy hitters) rather than an exact counter per key.
Monitor per-node load imbalance — one node hotter than the rest signals a hot key/shard.
Redis has hot-key detection (--hotkeys) built in.

Detection lets you respond (add replicas, cache locally, dedicate a shard) before the hot key takes down a node.

⚖️ Trade-offs

Fix	Gain	Cost
Replicate hot key	Spreads read load N×	Writes update N copies; complexity
In-process cache	Ultimate read scaling	Per-server memory; brief inconsistency
CDN	Off origin entirely	Only for public cacheable data
Sharded counters	Scales hot writes	Sum on read; approximation
Dedicated shard	Isolates the whale	Needs directory sharding; more nodes
Hybrid fan-out	Handles celebrities	Two code paths; read-time merge
Request coalescing	Prevents stampede	Slight added latency for waiters

In the real world

The celebrity problem is why Twitter/Instagram use hybrid fan-out — a pure precompute approach breaks on accounts with tens of millions of followers, so they precompute for normal users and pull for celebrities. It’s the textbook hot-key solution and the crux of the “design a news feed” question.
In-process caching of hot keys is a standard technique at large scale — Facebook and others cache the hottest data locally on every server (multi-layer caching), so a viral item’s read load spreads across the entire fleet automatically rather than hammering one cache node.
Hot partitions in DynamoDB/Cassandra are a common operational pain — a poorly-chosen partition key or a naturally-skewed access pattern concentrates load on one partition, and the fix (better key, write sharding, caching) is a recurring lesson.

🚨 Interview traps

Assuming even sharding solves load imbalance — it distributes keys, not load. This is the core point.
Not knowing the celebrity/hot-key problem — it’s a common follow-up (“what about a user with 100M followers?”).
Not knowing hybrid fan-out for the news-feed celebrity case.
Not detecting hot keys — you need to see them.
Only one fix — different hot keys need different fixes (read vs write, public vs private, key vs shard).

🎙️ Soundbites

“Consistent hashing distributes keys evenly, not *load — real traffic is Zipf, so a celebrity or viral item is one key getting most of the requests, and it maps to one node. You can’t hash your way out; the entity is too popular for one machine.”*
“For a hot read key I’d cache it in-process on every app server, so it never reaches the shared cache — the read load spreads across the whole fleet automatically. Or replicate the key across N nodes and read a random one.”
“The celebrity fan-out problem: a 100M-follower post can’t fan out to 100M feeds. So hybrid fan-out — precompute feeds for normal users, but pull celebrity posts at read time and merge. That’s the crux of designing Twitter’s feed.”
“For a whale tenant I’d give them a dedicated shard so they don’t affect everyone else — the multi-tenancy tiering answer, which needs directory-based sharding.”
“I’d detect hot keys with a Count-Min Sketch — bounded memory, finds the heavy hitters — so we respond before one takes down a node.”

🛠️ Try it

1. Create a hot key. Distribute requests with a Zipf distribution (real traffic) instead of uniform, across a sharded cache. Watch one node take most of the load despite even key distribution — the hot key problem, made visible.

2. Fix it with replication. Replicate the hot key across N nodes, read a random copy. Watch the load spread N-fold. Then try in-process caching and watch the hot key stop hitting the shared cache entirely.

3. Build hybrid fan-out. Implement fan-out-on-write, then add a celebrity with a huge follower count and watch the write explode. Add the hybrid path (pull for celebrities) and watch the write cost collapse while reads merge the pulled posts. This is the news-feed answer, built.

4. Detect heavy hitters. Implement a Count-Min Sketch over a request stream with a Zipf distribution and use it to find the hot keys with bounded memory. See how you’d detect a hot key in production before it causes an incident.

Check yourself

1. Why doesn't even sharding (consistent hashing) solve hot keys?

Because consistent hashing distributes *keys* evenly across nodes, but real request *load* isn't evenly distributed across keys — it follows a Zipf/power-law distribution where a small number of keys receive the overwhelming majority of requests. A celebrity account, a viral post, a trending product, or a whale tenant is a single key (or small set of keys) getting perhaps thousands of times the traffic of an average key. Consistent hashing maps that one hot key to exactly one node, and no hashing scheme can spread a *single* key's load across multiple nodes, because to serve requests for key X you need the data for key X, and putting it on multiple nodes means replicating it. So even with a perfectly balanced distribution of the millions of keys, the one node holding the hot key is overwhelmed while the rest idle — you have even *key* distribution and wildly uneven *load* distribution. It's a fundamental limitation: the problem is that a single logical entity is too popular for one machine, and hashing addresses "which node holds which key," not "how do we serve a key that's too hot for one node." The fixes all involve spreading the hot key's load off the single node (replication, in-process caching, CDN, dedicated resources), not better hashing.

2. How do you scale a hot read key?

Several approaches, chosen by how hot it is and whether it's public. **In-process caching on every app server**: for the very hottest read keys, cache the value locally in each application server's memory (an L1 cache), so requests never reach the shared cache or database — the hot key's read load is automatically distributed across the entire fleet of app servers, each serving from its own memory. The cost is per-server memory and brief inconsistency (each copy updates independently, so accept a short TTL), and it's the standard fix for the hottest keys. **Replicate the hot key across N nodes** (`hotkey:0`..`hotkey:9`), each on a different cache node, and have reads pick a random copy — spreading read load N-fold, at the cost of writes needing to update all N copies (fine since hot keys are usually read-heavy). **Serve from a CDN**: if the hot key is public and cacheable (a viral public post), push it to the CDN edge so it never touches your origin at all. These aren't mutually exclusive — a viral public item might be served from the CDN *and* cached in-process. The common thread is distributing the single key's read load across many serving points rather than concentrating it on the one node the hash assigns it to.

3. What is the celebrity fan-out problem and how does hybrid fan-out solve it?

Fan-out-on-write precomputes each user's feed at write time — when someone posts, the post is pushed (fanned out) into each of their followers' precomputed feeds, so reads are cheap lookups. This works beautifully until a celebrity with, say, 100 million followers posts: that single write must now fan out to 100 million feeds, an enormous, slow, resource-crushing operation, and it happens for every celebrity post — the post has become a hot key at the *write* path. **Hybrid fan-out** solves it by treating celebrities differently: for normal users (few followers), fan out on write as usual (cheap); for celebrities (many followers), *don't* fan out — instead, when a follower reads their feed, the system *pulls* the celebrity's recent posts at read time and merges them into the precomputed feed. So a follower's feed is mostly precomputed (fast) with celebrity posts pulled in and merged (avoiding the 100-million-write explosion). The write cost for a celebrity post collapses from 100M writes to essentially zero (it's just stored once, pulled on demand), at the cost of a small read-time merge and maintaining two code paths. This hybrid is the canonical answer to "design Twitter/Instagram's feed," and recognizing that the celebrity case breaks pure fan-out-on-write — and knowing the pull-based fix — is exactly what the interview is probing.

4. When would you give a hot shard its own dedicated resources?

When the hot key is actually a hot *shard* — a large tenant, a hugely popular product, or an entity whose entire data set and traffic dwarf the others — and the concern is that this one heavy entity degrades service for everyone else sharing its shard (the noisy-neighbor problem). Rather than replicating a single key or caching a single value, you isolate the whole heavy entity onto its own dedicated node or shard, so its load is contained and can't affect other tenants, and it can be scaled, tuned, and monitored independently. This is the multi-tenancy tiering answer: detect the outlier tenants (a customer 1000× the median size) and give them dedicated infrastructure while the long tail of small tenants shares pooled resources. It requires *directory-based sharding* — an explicit mapping of keys to shards — rather than pure hash-based sharding, because you need to deliberately place specific keys on specific nodes (the whale on its own shard) rather than letting the hash decide. It's usually justified both technically (isolating the load) and commercially (such large tenants are typically paying enterprise contracts that fund the dedicated resources). You'd use this for a *hot shard* (a large entity with lots of data and traffic), whereas for a *hot key* (a single small value getting enormous read traffic, like a viral post) you'd use replication or in-process/CDN caching instead.

5. Why is detecting hot keys important, and how do you do it efficiently?

Because hot keys cause node-level saturation and outages that even distribution can't prevent, and you can only apply the fixes (replicate the key, cache it in-process, dedicate a shard, add request coalescing) if you know *which* keys are hot — a hot key that takes down a node during a viral event, undetected, becomes an incident, whereas a detected one can be handled proactively. Detecting hot keys efficiently is itself a challenge because you can't afford an exact request counter per key when there are millions of keys (that's huge memory and its own hot-write problem). The efficient approach is a **Count-Min Sketch** — a probabilistic data structure that estimates per-key frequencies in bounded (fixed) memory regardless of the number of distinct keys, and is specifically good at finding "heavy hitters" (the keys with disproportionately high counts), which is exactly what a hot key is. It never underestimates, so it won't miss a genuinely hot key, and it trades a small amount of accuracy on rare keys for constant memory. You'd also **monitor per-node load imbalance** — if one node is consistently hotter than its peers despite even key distribution, that signals a hot key or shard living there — and use built-in tooling where available (Redis's `--hotkeys` mode samples and reports the hottest keys). Together these let you spot a hot key emerging (say, as a post goes viral) and respond — spinning up replicas or in-process caching — before it saturates the node.