system-design

Design a Recommendation System (Netflix / YouTube / Amazon)

Difficulty: Tier 3 Asked at: Netflix, YouTube, Amazon, Noon, TikTok-style companies Time budget: 45–60 min

“You might also like…” powers Netflix, YouTube, Amazon, and every feed. The interview isn’t about the ML math — it’s about the system: the two-stage candidate generation → ranking pipeline, a feature store, offline training vs online serving, and serving personalized results in milliseconds at massive scale. It generalizes the news feed ranking and reuses patterns from across the repo.

Prerequisites: Design News Feed, Batch vs Stream, Caching

1. Requirements

Functional:

Recommend items (videos, products, movies) personalized to each user.
Use signals: user history, item features, behavior of similar users, context (time, device).
Update recommendations as the user interacts (near-real-time-ish).
Support “because you watched X,” “trending,” “similar items.”

Non-functional:

Low-latency serving — recommendations render in a page load (< 100–200 ms).
Massive scale — millions of users × millions of items.
Personalized & fresh — reflect recent behavior and new items.
Quality (relevance) is the product; measurable via engagement.

Out of scope: the ML model internals/math (reference), the feed rendering, cold data pipelines’ details.

2. Estimation

Millions of users, millions of items → the full user×item matrix is astronomically large. 🚨 You cannot score every item for every user at request time — that’s the central constraint that forces a two-stage design.
Serving: hundreds of thousands of requests/sec, each needing personalized results in ~100 ms.

3. The core: candidate generation → ranking

🚨 The universal two-stage recommender pattern (identical framing to feed ranking):

flowchart LR
    Req[User requests recs] --> CG[1. Candidate Generation<br/>millions -> hundreds]
    CG --> RK[2. Ranking<br/>score the hundreds precisely]
    RK --> Filt[3. Filtering / diversity / business rules]
    Filt --> Recs[Final recommendations]
    FS[(Feature Store)] --> RK
    Sources[(Collaborative filtering<br/>content-based<br/>trending / embeddings)] --> CG

Candidate generation: cheaply narrow millions of items to a few hundred plausible candidates, using fast methods — collaborative filtering, content-based similarity, embedding nearest-neighbor search (ANN), trending. Multiple sources, each recall-oriented.
Ranking: apply an expensive, precise ML model to score only those hundreds of candidates, using rich features. Sort by predicted engagement.
Filtering: remove already-seen items, apply diversity, business rules (in-stock, region), and blend.

🚨 Why two stages? You can’t run the expensive model over millions of items per request; cheap candidate generation bounds the expensive ranking to a few hundred. This is the whole architectural insight.

4. Deep dives

4a. Candidate generation approaches

Collaborative filtering: “users like you also liked X” — from the user-item interaction matrix (matrix factorization / embeddings). Powerful but needs interaction data (cold-start problem).
Content-based: recommend items similar to ones the user liked (item features/tags/embeddings). Handles new items better.
Embedding + ANN: represent users and items as vectors; find items whose vectors are nearest the user’s vector via approximate nearest-neighbor search (fast at scale). 🚨 A very common modern approach.
Trending/popular: a cheap, always-available source (great for cold-start). Combine several sources → a richer candidate pool.

4b. Offline training vs online serving

🚨 Split the system: heavy offline work (training models, computing embeddings, precomputing candidate lists / user & item features) runs in batch pipelines; online serving is a fast lookup + lightweight ranking. Embeddings and features land in a feature store / vector index for millisecond retrieval. The model is trained offline on logged interactions, deployed to the serving tier. This is the same batch-build / online-serve split as typeahead and search.

4c. The feature store

Ranking needs many features (user profile, item stats, cross features) fast. Precompute and store them in a feature store keyed by user/item so ranking is inference-over-fetched-features, not compute-from-raw. Same component as fraud detection and feed ranking — 🚨 point out the reuse.

4d. Freshness — reacting to recent behavior

Users expect recs to reflect what they just did. Blend the offline-precomputed recommendations with real-time signals (this session’s clicks) via streaming features and light online candidate sources, so the model sees fresh behavior without a full retrain. Two-speed again: slow offline model + fast online signals.

4e. Cold start

New users (no history) → fall back to popularity/trending, onboarding preferences, and content-based recs. New items (no interactions) → content-based and exploration (show them to some users to gather data). 🚨 Name cold-start explicitly; it’s a classic follow-up.

4f. Evaluation & exploration

Measure with engagement metrics; run A/B tests for model changes. Balance exploitation (recommend what you know works) with exploration (try new items to learn) — pure exploitation creates filter bubbles and never discovers new hits. Mention it as a real concern.

5. Bottlenecks & scaling further

Can’t score all items → two-stage candidate-gen → ranking.
Candidate gen at scale → embeddings + ANN index; precomputed lists.
Ranking latency → bounded candidate set + feature store + fast inference + result caching.
Freshness → offline model + real-time streaming signals.
Cold start → trending/content-based fallbacks + exploration.
Serving load → cache recs per user for a short window; stateless serving tier.

6. Trade-off summary

Decision	Chosen	Alternative	Why
Architecture	Two-stage (candidates → rank)	Score all items	Can’t run the model over millions/request
Candidate gen	Multiple sources (CF + content + ANN + trending)	One method	Coverage + cold-start handling
Compute split	Offline train + online serve	All online	Heavy work offline; serving is fast lookup
Features	Feature store (precomputed)	Compute at rank time	Millisecond ranking
Freshness	Offline model + streaming signals	Retrain constantly	React to recent behavior cheaply
Strategy	Exploit + explore	Pure exploit	Avoid filter bubbles; discover new items

7. Follow-up questions

Why can't you just score every item for a user and pick the best?

Because the numbers make it impossible within the latency budget. With millions of items and millions of users, running a rich ranking model over every item for every request would be billions of scoring operations per request at hundreds of thousands of requests per second — utterly infeasible in the ~100 ms you have to render a page. So the system uses two stages: cheap candidate generation first narrows the millions of items down to a few hundred plausible candidates using fast methods (collaborative-filtering lookups, embedding nearest-neighbor search, precomputed lists, trending), and only then does the expensive, accurate ranking model score those few hundred candidates precisely. The candidate stage optimizes for recall (don't miss good items) and speed; the ranking stage optimizes for precision over a small set. This funnel is the defining architectural decision of every large-scale recommender precisely because exhaustive scoring doesn't scale — you make the expensive computation proportional to a bounded candidate set, not the whole catalog.

What is the candidate-generation vs ranking split, and what methods feed candidates?

Candidate generation is the cheap first stage that reduces the entire item catalog to a small set of plausible recommendations, and ranking is the expensive second stage that precisely orders that small set. Candidates come from several complementary sources: collaborative filtering (users similar to you liked these, derived from the user-item interaction matrix or learned embeddings), content-based similarity (items resembling ones you engaged with, using item features/tags), embedding nearest-neighbor search (represent users and items as vectors and retrieve items whose vectors are closest to the user's via approximate nearest-neighbor search, which is fast even over millions of items), and simple popularity/trending (a cheap, always-available source useful for cold-start). Using multiple sources gives broad coverage and handles different situations (new users, new items). The ranking stage then takes the merged few-hundred candidates and applies a heavy ML model with rich features to predict engagement and sort them, after which filtering removes seen items and applies diversity and business rules. Cheap-and-broad then expensive-and-precise is the pattern.

How do you serve personalized recommendations in ~100 ms?

By doing the heavy work offline and making the online path a fast lookup plus bounded inference. Offline batch pipelines train the models, compute user and item embeddings, precompute candidate lists, and populate a feature store and a vector (ANN) index. At request time, the serving tier retrieves the user's candidates (precomputed lists and/or a fast nearest-neighbor query against the vector index), fetches the needed features for those few hundred candidates from the feature store (a fast key-value lookup, not a live computation), runs the ranking model inference over just those candidates, applies filtering, and returns — all bounded and fast. Caching per-user recommendations for a short window absorbs repeat requests, and the serving tier is stateless so it scales horizontally. The key is that nothing expensive happens on the request path: training, embedding computation, and feature aggregation are all precomputed offline, leaving online serving as retrieval plus lightweight ranking, which fits the millisecond budget.

How do you handle a brand-new user or a brand-new item (cold start)?

With fallbacks and exploration, because collaborative filtering needs interaction history that new users and items don't have. For a new user, you can't personalize from their (empty) history, so you fall back to popularity/trending recommendations, use any onboarding-declared preferences, and lean on content-based recs as they take a few actions; as interaction data accumulates, personalization kicks in. For a new item, it has no interactions for collaborative filtering to latch onto, so you recommend it based on its content/features (similarity to items users liked) and deliberately *explore* — show it to some users to gather interaction data — rather than letting it languish unseen forever. This exploration ties into the broader exploit-versus- explore balance: a system that only recommends proven items never learns about new ones and creates filter bubbles, so you allocate some traffic to trying uncertain items to keep discovering hits. Naming cold-start and addressing both the new-user and new-item sides with these fallbacks is the expected depth.

8. What junior / mid / senior answers look like

Junior: proposes scoring items with a model or simple “users who bought X bought Y” without addressing scale — can’t serve millions of items per request in time.
Mid: the two-stage candidate-generation → ranking pipeline, collaborative + content-based candidates, a feature store, offline training vs online serving.
Senior: all that plus embeddings + ANN for candidate generation, the offline/online split with a two-speed freshness model (offline model + streaming signals), explicit cold-start handling, exploit/explore and A/B evaluation, and pointed reuse of the feature-store and ranking patterns from feed/fraud — framing “you can’t score everything” as the driving constraint.