system-design

Caching — The Complete Guide ⭐

The most powerful optimization available, and the source of the most subtle bugs you will ever debug. Everything about caching is a trade of correctness for speed.

Prerequisites: Latency Numbers, Consistency Models Time to read: ~35 minutes. This chapter earns the time.


The problem

A query takes 20 ms. It runs 50,000 times a second. It returns the same answer nearly every time.

You are recomputing an identical result 50,000 times per second — burning database CPU, disk IOPS, and network bandwidth on work you already did. Meanwhile RAM is 1,000× faster than SSD and you’re not using it.

Caching is: store the result, serve it again. Conceptually trivial. Every difficulty comes from one question — when does the stored copy stop being true?

“There are only two hard things in Computer Science: cache invalidation and naming things.” — Phil Karlton

He wasn’t joking.


Where caches live

Caching happens at every layer. A well-designed system uses several, and knowing the full stack is itself an interview signal.

flowchart TB
    B["1 · Browser cache<br/>free, closest, zero network"]
    C["2 · CDN / edge<br/>~10 ms from user"]
    L["3 · Reverse proxy cache<br/>Nginx, Varnish"]
    A["4 · Application in-process<br/>~100 ns, per-instance"]
    D["5 · Distributed cache<br/>Redis, Memcached — ~0.5 ms"]
    E["6 · Database buffer pool<br/>the page cache"]
    F["7 · Materialized views<br/>precomputed results"]
    B --> C --> L --> A --> D --> E --> F
Layer Latency Shared? Best for
Browser 0 No Static assets, user’s own data
CDN ~10 ms Globally Static assets, public API responses → CDN
Reverse proxy ~1 ms Per-datacenter Full HTTP responses
In-process (local) ~100 ns No — per instance Tiny, hot, rarely-changing data (config, feature flags)
Distributed (Redis) ~0.5 ms Yes Sessions, query results, computed data
Database buffer pool ~100 ns Within the DB Automatic; size your RAM for it
Materialized view query cost Yes Expensive aggregations

🚨 The cheapest cache is the one closest to the user. Candidates jump to Redis immediately and never mention HTTP caching — but a Cache-Control header costs nothing to add, requires no infrastructure, and eliminates the request entirely. Mention the browser and CDN layers first.

In-process vs distributed is a real design decision:

  In-process Distributed (Redis)
Latency ~100 ns ~500 µs (network-bound)
Shared across instances ❌ Each has its own copy
Invalidation Hard — must notify every instance Easy — one place
Survives restart
Memory cost × number of instances Once
Capacity Limited by app instance RAM Scales with the cluster

Use in-process for small, hot, slowly-changing data (feature flags, config, reference tables) and accept a few seconds of inconsistency. Use distributed for everything else. Many systems layer both — an in-process L1 in front of a Redis L2, which cuts Redis load dramatically for the hottest keys.


Caching patterns

Cache-aside (lazy loading) — the default

The application manages the cache explicitly.

def get_user(user_id):
    key = f"user:{user_id}"
    cached = cache.get(key)
    if cached is not None:
        return cached                       # HIT
    user = db.query("SELECT * FROM users WHERE id = ?", user_id)   # MISS
    cache.set(key, user, ttl=300)
    return user

def update_user(user_id, data):
    db.update("UPDATE users SET ... WHERE id = ?", user_id)
    cache.delete(f"user:{user_id}")         # invalidate, don't update

Why it dominates: only requested data is cached (no waste), cache failure is survivable (you just hit the database), and it works with any datastore.

Costs: every miss pays the full latency plus a cache round trip; there’s a window after a write where the cache is stale; and there’s a subtle race condition described below.

🚨 Delete, don’t update, on write. Updating the cache from the write path introduces a race:

T1: writer A updates DB to X
T2: writer B updates DB to Y
T3: writer B writes Y to cache
T4: writer A writes X to cache      ← cache now has X, database has Y. Permanently wrong.

Deleting is safe — the next read repopulates from the current truth. This is a great detail to mention; it demonstrates you’ve actually operated a cache.

Read-through

The cache itself fetches on a miss; your code only ever talks to the cache. Cleaner application code, but requires a cache that supports it (or a wrapper library). Same characteristics as cache-aside otherwise.

Write-through

Write to the cache and the database synchronously, together.

def update_user(user_id, data):
    db.update(...)
    cache.set(f"user:{user_id}", data, ttl=300)

✅ Cache is never stale. ❌ Every write pays both latencies, and you cache data that may never be read. Good when reads follow writes closely; wasteful otherwise.

Write-behind (write-back)

Write to the cache immediately, return, and flush to the database asynchronously.

✅ Very fast writes; absorbs write bursts; batches database writes. ❌ Data loss if the cache dies before flushing. Only acceptable when losing recent writes is tolerable — metrics, counters, analytics, view counts.

📐 This is how high-volume counters are actually built: increment in Redis, flush aggregates to the database every 10 seconds. A view counter losing 10 seconds of increments in a crash is fine. A payment ledger is not.

Refresh-ahead

Proactively refresh entries that are about to expire and are being accessed frequently. Users never experience a miss on hot keys. Adds complexity and wasted refreshes for keys that go cold. stale-while-revalidate at the HTTP layer is the same idea.

The comparison

Pattern Read latency Write latency Staleness Data-loss risk
Cache-aside Fast on hit, slow on miss DB only Brief after write None
Read-through Same as aside DB only Brief None
Write-through Fast DB + cache None None
Write-behind Fast Cache only ⚡ None Yes
Refresh-ahead Always fast DB only Minimal None

Eviction

The cache is full. Something must go.

Policy Evicts Good for
LRU (Least Recently Used) Longest since last access The default. Right ~90% of the time.
LFU (Least Frequently Used) Fewest accesses Stable popularity distributions; resists one-off scans
FIFO Oldest inserted Rarely what you want
TTL-only Whatever expires Time-sensitive data
Random Anything Surprisingly decent, very cheap (Redis approximates LRU this way)

🚨 LRU’s weakness — cache pollution by scans. A batch job that reads a million rows once will evict your entire hot working set, and every real user request misses afterward. Symptoms: a nightly job runs and morning latency is terrible. Fixes: LFU, a separate cache for batch workloads, or segmented LRU (Redis 4+ offers allkeys-lfu for exactly this).

Sizing: apply the 80/20 rule. Cache the ~20% of data serving ~80% of traffic. Measure hit rate and grow until the marginal improvement flattens.

📐 Hit rate is the number that determines whether a cache is worth anything:

DB query: 20 ms.  Cache hit: 0.6 ms.

50% hit rate:  0.5×0.6 + 0.5×20.6 = 10.6 ms   ← barely worth it
90%:           0.9×0.6 + 0.1×20.6 =  2.6 ms   ← 7.7× improvement
99%:          0.99×0.6 + 0.01×20.6 = 0.8 ms   ← 25× improvement

Note the shape: below ~80%, a cache adds complexity and a failure mode for modest gain. “What hit rate do we expect?” is the right first question before adding one.


Invalidation: the actual hard part

Four strategies, in increasing order of precision and effort.

1. TTL — just let it expire

cache.set(key, value, ttl=300)   # stale for at most 5 minutes

Simple, self-healing, requires no coordination. The right default. The whole design question is: how stale can this data be? Ask it per data type — a product price might tolerate 60 seconds; a product description, an hour; inventory, zero.

2. Explicit invalidation on write

Delete the key when the underlying data changes. Precise, but:

🚨 You must find every key affected by a write. Updating a user’s name invalidates user:42 — but also user:42:profile, the team:7:members list that embeds their name, the search index entry, and the cached HTML fragment. Missing one means permanent staleness.

Tag-based invalidation solves this: tag entries with the entities they depend on, and invalidate by tag.

cache.set("team:7:members", data, tags=["user:42", "user:99", "team:7"])
cache.invalidate_tag("user:42")     # kills every entry involving user 42

Redis doesn’t do this natively (you build it with sets), but CDNs do — surrogate keys. → CDN

3. Versioned keys

Instead of deleting, change the key.

version = cache.get("user:42:version") or 1
data = cache.get(f"user:42:v{version}")
# On write: cache.incr("user:42:version")   ← old keys become unreachable, expire naturally

No deletion race, works across many cached derivatives at once, and rollback is trivial. Costs memory until old versions expire. This is the same trick as content-hashed asset URLs.

4. Event-driven invalidation

Publish change events (via CDC or your message bus); cache invalidators subscribe. Decoupled and reliable — but eventually consistent, and now you’re operating a pipeline.


The failure modes you must know

These four are the most commonly asked caching questions in interviews.

Cache stampede / thundering herd

A hot key expires. 10,000 concurrent requests all miss and hit the database simultaneously. The database dies. When it recovers, it happens again.

Fixes:

Thundering Herd

Cache penetration

Requests for keys that don’t exist anywhere. Every one misses the cache and misses the database. An attacker requesting random user IDs bypasses your cache entirely.

Fixes: cache the negative result (cache.set(key, NULL_SENTINEL, ttl=60)), or use a Bloom filter to reject keys that definitely don’t exist before touching anything.

Cache avalanche

Many keys expire simultaneously — usually because they were all created at once (a deploy, a cache warm, a restart). The database receives the full uncached load in one instant.

Fixes: jittered TTLs (the same one-line fix as above, and it’s the most valuable), staged cache warming, and a circuit breaker in front of the database.

Hot key

One key is so popular that the single node holding it saturates its CPU or network. Consistent hashing doesn’t help — the key still maps to one node.

Fixes: replicate the hot key across several nodes with a random suffix (hot:key:1..N, read a random one); cache it in-process on every app server; or put it in a CDN. → Hot Keys


Redis vs Memcached

  Redis Memcached
Data types Strings, lists, sets, sorted sets, hashes, streams, HyperLogLog, geo Strings only
Persistence RDB snapshots + AOF None
Replication Yes No
Clustering Built in Client-side sharding
Threading Single-threaded commands (threaded I/O in 6+) Multi-threaded
Extras Pub/sub, Lua scripts, transactions, TTL per key Just cache
Memory efficiency Good Slightly better for simple values

Practical answer: use Redis unless you have a specific reason not to. The data structures alone justify it — sorted sets give you leaderboards and rate limiters, sets give you tag invalidation, and pub/sub gives you cross-instance invalidation. Memcached is simpler and marginally more memory-efficient for pure string caching at very large scale.

🚨 Redis is not just a cache. Sorted sets for leaderboards (case study), sliding-window rate limiting, distributed locks (carefully), session storage, and pub/sub fan-out. Knowing which structure solves which problem is very useful in interviews.


What not to cache

🎙️ “I wouldn’t cache this — it’s a primary-key lookup on an indexed column that already returns in 2 ms, and adding a cache buys us 1.5 ms while adding an invalidation problem and a new failure mode.” Saying no to a cache, with reasoning, scores as well as saying yes.


⚖️ Trade-offs

Decision Gain Cost
Add a cache 10–100× faster reads; big database load reduction Staleness; invalidation complexity; a new failure mode
Longer TTL Higher hit rate, less origin load More staleness
Cache-aside Simple, resilient to cache failure Miss penalty; brief post-write staleness
Write-through Never stale Slower writes; caches unread data
Write-behind Fastest writes Data loss on cache failure
In-process cache ~100 ns; no network Per-instance memory; hard to invalidate
Distributed cache Shared, invalidatable, survives restarts Network hop; a system to operate

In the real world


🚨 Interview traps


🎙️ Soundbites


🛠️ Try it

1. Cause a stampede. Redis + a database + an endpoint doing cache-aside with a 10-second TTL. Load-test at 500 concurrent requests for one key. Watch the database QPS spike every 10 seconds like a heartbeat. Then implement single-flight locking and watch it flatten. This is the single most instructive caching exercise there is.

2. Measure hit rate honestly. Instrument hits and misses. Run a realistic access pattern (Zipf distribution, not uniform — real traffic is heavily skewed). Vary the cache size and plot hit rate. You’ll find the curve is steep at first and then flat, which is exactly why “cache 20% of the data” is a real rule of thumb.

3. Break invalidation. Cache a user profile. Cache a team page that embeds usernames. Update the username and invalidate only user:42. Watch the team page stay wrong forever. That’s the find-every-affected-key problem, and once you’ve seen it you’ll always ask about it.


Check yourself

1. Why delete the cache key on write instead of updating it? Because concurrent writers can interleave: writer A reads/updates the database to X, writer B updates it to Y, B writes Y to the cache, then A's delayed write puts X in the cache. The database says Y and the cache says X — permanently, until the TTL expires. Deleting has no such race: whoever reads next repopulates from the current database state. (If you must update, you need versioning or a compare-and-set.)
2. Your database gets a traffic spike every 5 minutes, matching your TTL. What's happening and what are the fixes? A cache stampede combined with an avalanche: many keys were populated at the same moment (deploy, restart, or warm-up) with identical TTLs, so they all expire together and every concurrent request misses at once. Fixes: **jitter the TTLs** (`300 + random(0,60)`) so expiry spreads out; single-flight locking so only one request per key recomputes; probabilistic early expiration; and `stale-while-revalidate` so users get the old value while one background task refreshes.
3. An attacker requests random non-existent user IDs at high rate. Why is your cache useless, and what do you do? Cache penetration — those keys aren't in the cache (nothing to cache) and aren't in the database either, so every request is a full miss that reaches the database. The cache provides zero protection. Fixes: cache the *negative* result with a short TTL so repeated requests for the same missing key are absorbed; put a Bloom filter in front so keys that definitely don't exist are rejected without touching anything; and rate limit by client so the attack is bounded regardless.
4. When would you choose an in-process cache over Redis? When the data is small, extremely hot, read constantly, and tolerant of a few seconds of inconsistency across instances: feature flags, configuration, reference tables, routing rules, compiled templates. The gain is ~100 ns instead of ~500 µs and no network dependency. The costs are per-instance memory duplication and hard invalidation (you must broadcast to every instance, usually via pub/sub, and accept propagation delay). Many systems use both: in-process L1 in front of a Redis L2, which slashes Redis load for the hottest keys.
5. Your Redis cluster goes down completely. What happens, and what should you have built? Every read becomes a database read. If the cache was absorbing 95% of a 100,000 QPS read load, the database suddenly receives 20× its normal traffic and almost certainly collapses — turning a cache outage into a total outage. You should have: a circuit breaker that detects the cache is down and sheds load or serves degraded responses rather than stampeding the database; database-side rate limiting or admission control; an in-process L1 cache that keeps serving the hottest data; graceful degradation paths (show stale data, or a reduced feature set); and enough database headroom to survive partial cache loss. Also: test this. Turn the cache off in staging under load and see what actually happens.

Further reading