The most powerful optimization available, and the source of the most subtle bugs you will ever debug. Everything about caching is a trade of correctness for speed.
Prerequisites: Latency Numbers, Consistency Models Time to read: ~35 minutes. This chapter earns the time.
A query takes 20 ms. It runs 50,000 times a second. It returns the same answer nearly every time.
You are recomputing an identical result 50,000 times per second — burning database CPU, disk IOPS, and network bandwidth on work you already did. Meanwhile RAM is 1,000× faster than SSD and you’re not using it.
Caching is: store the result, serve it again. Conceptually trivial. Every difficulty comes from one question — when does the stored copy stop being true?
“There are only two hard things in Computer Science: cache invalidation and naming things.” — Phil Karlton
He wasn’t joking.
Caching happens at every layer. A well-designed system uses several, and knowing the full stack is itself an interview signal.
flowchart TB
B["1 · Browser cache<br/>free, closest, zero network"]
C["2 · CDN / edge<br/>~10 ms from user"]
L["3 · Reverse proxy cache<br/>Nginx, Varnish"]
A["4 · Application in-process<br/>~100 ns, per-instance"]
D["5 · Distributed cache<br/>Redis, Memcached — ~0.5 ms"]
E["6 · Database buffer pool<br/>the page cache"]
F["7 · Materialized views<br/>precomputed results"]
B --> C --> L --> A --> D --> E --> F
| Layer | Latency | Shared? | Best for |
|---|---|---|---|
| Browser | 0 | No | Static assets, user’s own data |
| CDN | ~10 ms | Globally | Static assets, public API responses → CDN |
| Reverse proxy | ~1 ms | Per-datacenter | Full HTTP responses |
| In-process (local) | ~100 ns | No — per instance | Tiny, hot, rarely-changing data (config, feature flags) |
| Distributed (Redis) | ~0.5 ms | Yes | Sessions, query results, computed data |
| Database buffer pool | ~100 ns | Within the DB | Automatic; size your RAM for it |
| Materialized view | query cost | Yes | Expensive aggregations |
🚨 The cheapest cache is the one closest to the user. Candidates jump to Redis immediately and
never mention HTTP caching — but a Cache-Control header costs nothing to add, requires no
infrastructure, and eliminates the request entirely. Mention the browser and CDN layers first.
In-process vs distributed is a real design decision:
| In-process | Distributed (Redis) | |
|---|---|---|
| Latency | ~100 ns | ~500 µs (network-bound) |
| Shared across instances | ❌ Each has its own copy | ✅ |
| Invalidation | Hard — must notify every instance | Easy — one place |
| Survives restart | ❌ | ✅ |
| Memory cost | × number of instances | Once |
| Capacity | Limited by app instance RAM | Scales with the cluster |
Use in-process for small, hot, slowly-changing data (feature flags, config, reference tables) and accept a few seconds of inconsistency. Use distributed for everything else. Many systems layer both — an in-process L1 in front of a Redis L2, which cuts Redis load dramatically for the hottest keys.
The application manages the cache explicitly.
def get_user(user_id):
key = f"user:{user_id}"
cached = cache.get(key)
if cached is not None:
return cached # HIT
user = db.query("SELECT * FROM users WHERE id = ?", user_id) # MISS
cache.set(key, user, ttl=300)
return user
def update_user(user_id, data):
db.update("UPDATE users SET ... WHERE id = ?", user_id)
cache.delete(f"user:{user_id}") # invalidate, don't update
Why it dominates: only requested data is cached (no waste), cache failure is survivable (you just hit the database), and it works with any datastore.
Costs: every miss pays the full latency plus a cache round trip; there’s a window after a write where the cache is stale; and there’s a subtle race condition described below.
🚨 Delete, don’t update, on write. Updating the cache from the write path introduces a race:
T1: writer A updates DB to X
T2: writer B updates DB to Y
T3: writer B writes Y to cache
T4: writer A writes X to cache ← cache now has X, database has Y. Permanently wrong.
Deleting is safe — the next read repopulates from the current truth. This is a great detail to mention; it demonstrates you’ve actually operated a cache.
The cache itself fetches on a miss; your code only ever talks to the cache. Cleaner application code, but requires a cache that supports it (or a wrapper library). Same characteristics as cache-aside otherwise.
Write to the cache and the database synchronously, together.
def update_user(user_id, data):
db.update(...)
cache.set(f"user:{user_id}", data, ttl=300)
✅ Cache is never stale. ❌ Every write pays both latencies, and you cache data that may never be read. Good when reads follow writes closely; wasteful otherwise.
Write to the cache immediately, return, and flush to the database asynchronously.
✅ Very fast writes; absorbs write bursts; batches database writes. ❌ Data loss if the cache dies before flushing. Only acceptable when losing recent writes is tolerable — metrics, counters, analytics, view counts.
📐 This is how high-volume counters are actually built: increment in Redis, flush aggregates to the database every 10 seconds. A view counter losing 10 seconds of increments in a crash is fine. A payment ledger is not.
Proactively refresh entries that are about to expire and are being accessed frequently. Users never
experience a miss on hot keys. Adds complexity and wasted refreshes for keys that go cold.
stale-while-revalidate at the HTTP layer is the same idea.
| Pattern | Read latency | Write latency | Staleness | Data-loss risk |
|---|---|---|---|---|
| Cache-aside | Fast on hit, slow on miss | DB only | Brief after write | None |
| Read-through | Same as aside | DB only | Brief | None |
| Write-through | Fast | DB + cache | None | None |
| Write-behind | Fast | Cache only ⚡ | None | Yes |
| Refresh-ahead | Always fast | DB only | Minimal | None |
The cache is full. Something must go.
| Policy | Evicts | Good for |
|---|---|---|
| LRU (Least Recently Used) | Longest since last access | The default. Right ~90% of the time. |
| LFU (Least Frequently Used) | Fewest accesses | Stable popularity distributions; resists one-off scans |
| FIFO | Oldest inserted | Rarely what you want |
| TTL-only | Whatever expires | Time-sensitive data |
| Random | Anything | Surprisingly decent, very cheap (Redis approximates LRU this way) |
🚨 LRU’s weakness — cache pollution by scans. A batch job that reads a million rows once will
evict your entire hot working set, and every real user request misses afterward. Symptoms: a nightly
job runs and morning latency is terrible. Fixes: LFU, a separate cache for batch workloads, or
segmented LRU (Redis 4+ offers allkeys-lfu for exactly this).
Sizing: apply the 80/20 rule. Cache the ~20% of data serving ~80% of traffic. Measure hit rate and grow until the marginal improvement flattens.
📐 Hit rate is the number that determines whether a cache is worth anything:
DB query: 20 ms. Cache hit: 0.6 ms.
50% hit rate: 0.5×0.6 + 0.5×20.6 = 10.6 ms ← barely worth it
90%: 0.9×0.6 + 0.1×20.6 = 2.6 ms ← 7.7× improvement
99%: 0.99×0.6 + 0.01×20.6 = 0.8 ms ← 25× improvement
Note the shape: below ~80%, a cache adds complexity and a failure mode for modest gain. “What hit rate do we expect?” is the right first question before adding one.
Four strategies, in increasing order of precision and effort.
cache.set(key, value, ttl=300) # stale for at most 5 minutes
Simple, self-healing, requires no coordination. The right default. The whole design question is: how stale can this data be? Ask it per data type — a product price might tolerate 60 seconds; a product description, an hour; inventory, zero.
Delete the key when the underlying data changes. Precise, but:
🚨 You must find every key affected by a write. Updating a user’s name invalidates user:42 —
but also user:42:profile, the team:7:members list that embeds their name, the search index entry,
and the cached HTML fragment. Missing one means permanent staleness.
Tag-based invalidation solves this: tag entries with the entities they depend on, and invalidate by tag.
cache.set("team:7:members", data, tags=["user:42", "user:99", "team:7"])
cache.invalidate_tag("user:42") # kills every entry involving user 42
Redis doesn’t do this natively (you build it with sets), but CDNs do — surrogate keys. → CDN
Instead of deleting, change the key.
version = cache.get("user:42:version") or 1
data = cache.get(f"user:42:v{version}")
# On write: cache.incr("user:42:version") ← old keys become unreachable, expire naturally
No deletion race, works across many cached derivatives at once, and rollback is trivial. Costs memory until old versions expire. This is the same trick as content-hashed asset URLs.
Publish change events (via CDC or your message bus); cache invalidators subscribe. Decoupled and reliable — but eventually consistent, and now you’re operating a pipeline.
These four are the most commonly asked caching questions in interviews.
A hot key expires. 10,000 concurrent requests all miss and hit the database simultaneously. The database dies. When it recovers, it happens again.
Fixes:
if time.time() - value.computed_at > ttl - beta * abs(math.log(random.random())):
refresh()
ttl = 300 + random(0, 60) so keys created together don’t expire together.Requests for keys that don’t exist anywhere. Every one misses the cache and misses the database. An attacker requesting random user IDs bypasses your cache entirely.
Fixes: cache the negative result (cache.set(key, NULL_SENTINEL, ttl=60)), or use a
Bloom filter to reject keys that definitely don’t exist
before touching anything.
Many keys expire simultaneously — usually because they were all created at once (a deploy, a cache warm, a restart). The database receives the full uncached load in one instant.
Fixes: jittered TTLs (the same one-line fix as above, and it’s the most valuable), staged cache warming, and a circuit breaker in front of the database.
One key is so popular that the single node holding it saturates its CPU or network. Consistent hashing doesn’t help — the key still maps to one node.
Fixes: replicate the hot key across several nodes with a random suffix (hot:key:1..N, read a
random one); cache it in-process on every app server; or put it in a CDN.
→ Hot Keys
| Redis | Memcached | |
|---|---|---|
| Data types | Strings, lists, sets, sorted sets, hashes, streams, HyperLogLog, geo | Strings only |
| Persistence | RDB snapshots + AOF | None |
| Replication | Yes | No |
| Clustering | Built in | Client-side sharding |
| Threading | Single-threaded commands (threaded I/O in 6+) | Multi-threaded |
| Extras | Pub/sub, Lua scripts, transactions, TTL per key | Just cache |
| Memory efficiency | Good | Slightly better for simple values |
Practical answer: use Redis unless you have a specific reason not to. The data structures alone justify it — sorted sets give you leaderboards and rate limiters, sets give you tag invalidation, and pub/sub gives you cross-instance invalidation. Memcached is simpler and marginally more memory-efficient for pure string caching at very large scale.
🚨 Redis is not just a cache. Sorted sets for leaderboards (case study), sliding-window rate limiting, distributed locks (carefully), session storage, and pub/sub fan-out. Knowing which structure solves which problem is very useful in interviews.
🎙️ “I wouldn’t cache this — it’s a primary-key lookup on an indexed column that already returns in 2 ms, and adding a cache buys us 1.5 ms while adding an invalidation problem and a new failure mode.” Saying no to a cache, with reasoning, scores as well as saying yes.
| Decision | Gain | Cost |
|---|---|---|
| Add a cache | 10–100× faster reads; big database load reduction | Staleness; invalidation complexity; a new failure mode |
| Longer TTL | Higher hit rate, less origin load | More staleness |
| Cache-aside | Simple, resilient to cache failure | Miss penalty; brief post-write staleness |
| Write-through | Never stale | Slower writes; caches unread data |
| Write-behind | Fastest writes | Data loss on cache failure |
| In-process cache | ~100 ns; no network | Per-instance memory; hard to invalidate |
| Distributed cache | Shared, invalidatable, survives restarts | Network hop; a system to operate |
1. Cause a stampede. Redis + a database + an endpoint doing cache-aside with a 10-second TTL. Load-test at 500 concurrent requests for one key. Watch the database QPS spike every 10 seconds like a heartbeat. Then implement single-flight locking and watch it flatten. This is the single most instructive caching exercise there is.
2. Measure hit rate honestly. Instrument hits and misses. Run a realistic access pattern (Zipf distribution, not uniform — real traffic is heavily skewed). Vary the cache size and plot hit rate. You’ll find the curve is steep at first and then flat, which is exactly why “cache 20% of the data” is a real rule of thumb.
3. Break invalidation. Cache a user profile. Cache a team page that embeds usernames. Update the
username and invalidate only user:42. Watch the team page stay wrong forever. That’s the
find-every-affected-key problem, and once you’ve seen it you’ll always ask about it.