system-design

Design Pastebin

Difficulty: Tier 1 (warm-up) Asked at: Arbisoft, Educative, Careem, Amazon phone screens Time budget: 45 min

Pastebin is the URL shortener’s slightly bigger sibling: instead of mapping a code to a URL, you map a code to a blob of text (sometimes megabytes). That one change — the payload is large — pulls in object storage, and that’s the whole lesson. If you’ve done the URL shortener, 80% of this transfers; focus on what’s different.

Prerequisites: URL Shortener, Object Storage, CDN

1. Requirements

Functional:

Create a paste (arbitrary text, up to some size limit) → get a short URL.
View a paste by its URL.
Optional: expiration (paste self-destructs after time/one view), custom URL, syntax highlighting, visibility (public/unlisted/private).

Non-functional:

Read-heavy — pastes are created once, read many times (a code snippet shared in a chat is read by everyone in it).
Large payloads — a paste can be a few KB to several MB, unlike a URL’s few hundred bytes. 🚨 This is the key difference.
High availability, low read latency, durability (a paste must not vanish).

Out of scope: rich collaboration/editing (that’s collaborative editing), accounts, comments.

Size limit: state one — e.g. 10 MB per paste. It bounds storage and informs the design.

2. Back-of-the-envelope estimation

Assume 10 million new pastes / month, average size 10 KB (most are small snippets; a few are huge).

Writes: 10M / 2.6M sec ≈ ~4 writes/sec. Trivial.
Reads: at 10:1 → ~40 reads/sec average, peak ~200/sec. Modest.
Storage: 10M/month × 10 KB × 12 × 5 years ≈ 6 TB over 5 years. (Note: storage dominates here, not request rate — the opposite emphasis from the URL shortener.)
Bandwidth: 40 reads/sec × 10 KB ≈ 400 KB/sec read. Small on average, but a viral 5 MB paste spikes it → CDN territory.

🚨 The numbers say: request rates are low, but payloads are large and storage grows fast. So the design centers on cheap durable blob storage + a CDN, not on request throughput.

3. API design

POST /pastes
  body: { content, expiresAt?, visibility?, customPath? }
  → 201 { url: "https://pst.bin/aB3xK9p" }

GET /{pasteId}
  → 200 { content, createdAt, ... }   (or the raw text for a raw endpoint)

GET /{pasteId}/raw   → 200 text/plain   (raw content, CDN-friendly)
DELETE /pastes/{pasteId}

4. Data model

Split metadata from content — the crucial design choice:

Metadata store (small, queried, key-value or relational):

Field	Type
`paste_id`	string (PK), base-62 code
`content_ref`	string — pointer to the blob in object storage
`size`	int
`created_at` / `expires_at`	timestamp
`visibility`	enum

Content store: the actual text lives in object storage (S3/GCS/Blob), keyed by content_ref. 🚨 Don’t put multi-MB blobs in your primary database — it bloats the DB, slows queries, and is far more expensive per byte than object storage.

5. High-level design

flowchart LR
    Client -->|POST| LB[Load Balancer]
    Client -->|GET| CDN[CDN]
    CDN -.miss.-> LB
    LB --> App[App Servers]
    App --> Meta[(Metadata Store<br/>key-value)]
    App --> Blob[(Object Storage<br/>S3 - the paste text)]
    CDN -.-> Blob

Write: generate paste_id (same range-based counter trick as the URL shortener), upload the content to object storage, store metadata (with the content pointer) in the metadata store, return the URL.

Read: the CDN serves popular pastes’ raw content directly from the edge (cache). On miss, the app reads metadata, fetches the blob (or hands back a URL to it), returns it, and the CDN caches it.

6. Deep dives

6a. Why split metadata from content?

Because they have opposite characteristics. Metadata is tiny, structured, and queried (list a user’s pastes, check expiration) — a database’s job. Content is large, opaque, and served whole — object storage’s job (cheap per GB, infinitely scalable, durable, CDN-integrated). Mixing them makes the database huge and slow and wastes money. 🚨 “Metadata in the DB, blobs in object storage” is a pattern you’ll reuse in every design with large payloads (file storage, video, images).

6b. Serving reads cheaply with a CDN

A viral paste (a leaked config, a popular snippet) can get hammered. Because content is immutable once created, it’s perfectly cacheable. Put a CDN in front: the edge serves the raw text, your origin barely gets touched, and global users get low latency. Immutability is what makes this clean — no invalidation needed. (CDN)

6c. Expiration & one-time pastes

Time-based: store expires_at; lazily 404 on read if expired; a background job (or object-storage lifecycle policy) deletes expired blobs to reclaim space.
Burn-after-read: mark viewed and delete on first read. 🚨 Watch the race: two simultaneous reads — use an atomic compare-and-set on a viewed flag so only one succeeds.

6d. Handling large uploads

For multi-MB pastes, let the client upload directly to object storage via a pre-signed URL rather than streaming the bytes through your app servers. The app issues a pre-signed PUT URL, the client uploads to S3 directly, then confirms — this keeps large payloads off your servers entirely. (Same pattern as file storage.)

7. Bottlenecks & scaling further

Storage growth (6 TB+) → object storage scales infinitely; use lifecycle policies to expire old pastes to cheaper tiers or delete them.
Read hotspots → CDN (immutable content caches perfectly).
Large-upload load on app servers → pre-signed direct-to-S3 uploads.
Metadata scale → shard the metadata store on paste_id (it’s small, so this is late to bite).

8. Trade-off summary

Decision	Chosen	Alternative	Why
Content storage	Object storage (S3)	In the database	Cheap per byte, durable, CDN-ready, keeps DB lean
Metadata	Separate key-value store	Combined with content	Opposite access patterns
Read scaling	CDN (immutable content)	App-server caching	Immutable blobs cache perfectly at the edge
Large uploads	Pre-signed direct-to-S3	Stream through app	Keeps big payloads off compute tier

9. Follow-up questions

Why is the design so different from a URL shortener if they look identical?

Because the payload size flips the emphasis. A URL is a few hundred bytes, so a URL shortener is about *request throughput* (20K reads/sec → cache) and storage is an afterthought. A paste is up to megabytes, so Pastebin is about *storage and bandwidth* (6 TB, big blobs → object storage + CDN) and request throughput is an afterthought (a few dozen/sec). Same skeleton (code → payload), but the size difference pulls in object storage, a CDN for large immutable blobs, and direct-to-storage uploads — none of which the URL shortener needs.

How do you prevent abuse (huge pastes, spam)?

Enforce the size limit at upload (reject > 10 MB), rate-limit creation per IP/user, scan for malware/spam patterns, and support reporting + takedown. Pre-signed URLs should be scoped to the exact size and expire quickly so they can't be reused for arbitrary uploads.

How do you support syntax highlighting?

Do it client-side (a JS library highlights on render) so your servers just store/serve plain text and stay CDN-friendly. Store an optional `language` hint in metadata. Server-side highlighting would defeat CDN caching of the raw content and add compute.

What consistency guarantees does a read need?

Read-after-create for the author is nice (they expect their paste to work immediately), so write metadata synchronously before returning the URL. For everyone else, eventual consistency of CDN caches is fine — content is immutable, so there's no stale-data problem, only a brief propagation delay for a brand-new paste, which is acceptable.

10. What junior / mid / senior answers look like

Junior: treats it exactly like a URL shortener, stores the text in the database. Works, but misses the storage/bandwidth implications of large payloads.
Mid: splits metadata from content, puts blobs in object storage, adds a CDN, estimates storage correctly and sees it’s the dominant concern.
Senior: all that plus pre-signed direct uploads, exploits content immutability for clean CDN caching, handles the burn-after-read race, and explicitly contrasts the emphasis with the URL shortener (throughput vs storage) to show they understand why the design differs.