system-design

Design Instagram

Difficulty: Tier 2 Asked at: Meta, Careem, Noon, Amazon Time budget: 45–60 min

Instagram is Twitter plus images and video. The feed mechanics (fan-out, the celebrity problem) are the same — so this case study spends its time on what’s new: uploading, storing, transcoding, and serving media at scale, and generating thumbnails. If you’ve done Twitter, focus here on the media pipeline and CDN.

Prerequisites: Design Twitter, Object Storage, CDN

1. Requirements

Functional:

Upload photos/videos with captions.
Follow users; a home feed of recent posts from people you follow.
View a user’s profile grid.
Like, comment.
Generate multiple image sizes (thumbnail, feed, full).

Non-functional:

Read-heavy feed (like Twitter).
Media-heavy — the payload is images/videos, not text. Storage and bandwidth dominate.
Low-latency image loading globally → CDN is mandatory.
High availability, eventual consistency for the feed.

Out of scope: Stories/Reels internals, DMs, explore/recommendations (mention ranking exists).

2. Estimation

500M DAU, ~100M photos uploaded/day → ~1,200 uploads/sec, peak ~5×.
Each photo stored in ~4 sizes; average ~1.5 MB total across sizes → 100M × 1.5 MB/day ≈ 150 TB/day ingested. Over years → petabytes. 🚨 Media storage is the dominant cost.
Feed reads: hundreds of thousands/sec, each pulling many image URLs → CDN bandwidth is enormous.
Metadata (posts, likes, follows) is comparatively tiny.

3. API

POST /media/upload-url   → { uploadUrl (pre-signed), mediaId }   # client uploads directly to storage
POST /posts              { mediaId, caption }                    → 201 { postId }
GET  /feed?cursor=...                                            → [posts with image URLs]
GET  /users/{id}/posts?cursor=...
POST /posts/{id}/like  /  /comments

🚨 Uploads go directly to object storage via a pre-signed URL, not through your app servers — keeping multi-MB payloads off your compute tier (same trick as Pastebin / file storage).

4. Data model

Posts (metadata store): post_id, user_id, media_refs (per size), caption, created_at. Small.
Media blobs in object storage (S3), served via CDN.
Social graph and feed timelines: same as Twitter (fan-out on write + hybrid for celebrities).
Likes/comments: counters + lists, wide-column store.

5. The media pipeline (the new part)

flowchart LR
    Client -->|1. request upload URL| API[API]
    API -->|pre-signed URL| Client
    Client -->|2. PUT original| S3[(Object Storage)]
    S3 -->|3. event| Q[[Processing queue]]
    Q --> Worker[Transcode/Resize workers]
    Worker -->|thumbnails, sizes,<br/>video transcodes| S3
    Worker --> Meta[(Post metadata:<br/>media_refs ready)]
    S3 --> CDN[CDN] --> Viewer[Feed viewers]

Client requests a pre-signed upload URL; uploads the original directly to object storage.
The upload triggers an event onto a processing queue.
Async workers generate resized images (thumbnail, feed, full) and transcode video into multiple bitrates/formats; store all variants back in object storage.
Once processed, the post’s media becomes available; the feed serves the appropriate size via CDN.

🚨 Processing is asynchronous — the user’s upload completes immediately; resizing/transcoding happens in the background. The post can appear with a placeholder/blur until variants are ready.

6. Deep dives

6a. Serving images fast & cheap — CDN

Images are immutable once processed → perfectly cacheable at the CDN edge. The feed embeds CDN URLs; viewers fetch images from the nearest edge, not your origin. This offloads ~all image bandwidth from your servers and gives global low latency. 🚨 Without a CDN this design is impossible at scale (petabytes served worldwide). (CDN)

6b. Multiple resolutions

Storing/serving one giant original to a phone thumbnail wastes bandwidth and battery. Pre-generate sizes (thumbnail for grids, medium for feed, full for zoom) and serve the right one per context/device. Modern formats (WebP/AVIF) and responsive srcset reduce bytes further. This is why the pipeline transcodes on upload, not on read.

6c. The feed — reuse Twitter’s design

Fan-out on write into precomputed timelines (tweet IDs → here, post IDs), hybrid fan-out-on-read for celebrities, hydration on read. 🚨 Don’t re-derive it — say “the feed is the Twitter design; let me focus on media,” which shows you recognize the reusable pattern. (Twitter)

6d. Video

Video is heavier: transcode into an adaptive bitrate ladder (HLS/DASH) so playback adapts to network speed, store segments in object storage, serve via CDN. This is a mini video-streaming problem — reference it rather than fully designing it.

6e. Likes/comments at scale

A viral post gets millions of likes — a hot counter. Don’t do one-row-per-increment contention: use sharded/approximate counters or buffer increments and aggregate. Comments are a paginated list keyed by post. (Hot Keys)

7. Bottlenecks & scaling further

Image/video bandwidth → CDN (the single most important scaling lever here).
Upload load on app servers → pre-signed direct-to-storage uploads.
Transcoding compute → autoscaling worker pool draining the processing queue.
Feed reads → precomputed timelines + hybrid (Twitter).
Storage growth (petabytes) → object storage tiering; expire/cold-store old media.
Hot likes → sharded counters.

8. Trade-off summary

Decision	Chosen	Alternative	Why
Media storage	Object storage + CDN	In database / app servers	Cheap, durable, globally fast
Upload path	Pre-signed direct-to-storage	Through app servers	Keeps big payloads off compute
Processing	Async (queue + workers)	Synchronous on upload	Upload returns instantly; resize in background
Sizes	Pre-generate on upload	Resize on read	Read is hot; do the work once on write
Feed	Reuse Twitter hybrid fan-out	New design	Same problem; don’t reinvent

9. Follow-up questions

Why upload directly to object storage instead of through your servers?

Because routing multi-megabyte (and for video, hundreds-of-MB) uploads through your application servers wastes their bandwidth and compute, ties up connections, and makes them a scaling bottleneck for something they add no value to. With a pre-signed URL, the app server only issues a short-lived, scoped permission to PUT one object, and the client streams the bytes straight to object storage, which is built for exactly this. Your servers stay lean and handle only metadata and coordination. The upload completing then triggers async processing via an event, so the compute tier never touches the raw bytes.

Why generate image sizes on upload rather than on demand?

Because reads vastly outnumber writes, so doing the resize work once at upload time and serving cached variants is far cheaper than resizing on every read. On-demand resizing would put CPU-heavy image processing in the hot read path, hurting latency and cost, and would defeat CDN caching (each size/URL combination would be computed repeatedly). Pre-generating a fixed set of sizes at upload means every read is a cheap static fetch from the CDN. The trade-off is a little extra storage for the variants, which is cheap relative to the compute and latency saved.

How is the feed different from Twitter's?

Mechanically it isn't — it's the same fan-out-on-write-with-hybrid-for-celebrities design, storing post IDs in precomputed timelines and hydrating on read, with eventual consistency. What differs is what you hydrate *into*: the feed items carry CDN image/video URLs and the payload is media rather than text, so the emphasis shifts to serving that media cheaply (CDN, multiple sizes) rather than to the feed-building algorithm. The right move in an interview is to state that the feed reuses the Twitter design and spend your time on the media pipeline, which is what's actually different.

A post goes viral and gets millions of likes in minutes. What breaks?

A single like-counter row becomes a hot key with enormous write contention — every like is a read-modify-write on the same row, serializing behind locks and overwhelming that shard. Fix it by not maintaining a single exact counter under contention: shard the counter across many keys and sum them on read, or buffer increments in memory/Redis and flush aggregates periodically (approximate counts are fine for a like total). The image serving itself is unaffected because it's immutable and CDN-cached — only the mutable counter needs the hot-key treatment.

10. What junior / mid / senior answers look like

Junior: stores images, builds a feed, but streams uploads through app servers and may serve images from the origin — misses the CDN and async processing.
Mid: pre-signed uploads, object storage + CDN, async resize pipeline, reuses a precomputed feed.
Senior: recognizes the feed as the Twitter problem and moves on, designs the async media pipeline with a processing queue and multiple sizes, makes the CDN the centerpiece of read scaling, handles hot like- counters, and references adaptive-bitrate video as a bounded sub-problem.