system-design

Design a File Storage & Sync Service (Dropbox / Google Drive)

Difficulty: Tier 2 Asked at: Google, Dropbox, Amazon, Systems Ltd Time budget: 45–60 min

Dropbox looks like “store files in the cloud,” but the hard, interesting part is sync: keeping the same folder consistent across a laptop, a phone, and the cloud, efficiently, without re-uploading a whole file when one byte changes. The two signature ideas are chunking + deduplication and sync via metadata + notifications. This is where the metadata/blob split from Pastebin grows up.

Prerequisites: Object Storage, Design Pastebin, Notification System

1. Requirements

Functional:

Upload/download files; organize in folders.
Sync across a user’s devices — a change on one appears on all.
Sharing with other users; permissions.
File versioning / history.
Work efficiently on large files and slow/intermittent networks.

Non-functional:

Durability — never lose a file (the #1 promise).
Efficient sync — don’t re-transfer unchanged data.
Consistency — devices converge on the same state; conflicts resolved sanely.
Scale to billions of files, petabytes.

Out of scope: real-time collaborative editing (separate), full-text search of contents.

2. Estimation

500M users, average 100 files, ~1 GB each stored → petabytes, growing.
Reads/writes are modest per user but the sync notification traffic (telling N devices about changes) and dedup savings dominate the design.
Chunk size ~4 MB → a 1 GB file = 256 chunks; a small edit re-uploads 1 chunk, not 256. 🚨 This ratio is the whole efficiency argument.

3. Core design: chunking + metadata

🚨 Split every file into fixed-size chunks (e.g. 4 MB). Store:

Chunks in object storage, addressed by a content hash of the chunk.
File metadata in a database: the file = an ordered list of chunk hashes + name, size, version, permissions.

This unlocks the two big wins:

Deduplication: identical chunks (same hash) are stored once, even across files/users. Re-upload the same file → zero new storage. (Content-addressed storage)
Delta sync: editing a file changes only some chunks; only those chunks’ hashes change, so the client uploads only the changed chunks, not the whole file.

4. High-level design

flowchart TB
    Client[Client / Sync agent] -->|which chunks changed?| MetaSvc[Metadata Service]
    MetaSvc --> MetaDB[(Metadata DB<br/>files, chunk lists, versions)]
    Client -->|upload new chunks| Block[Block/Chunk Service]
    Block --> S3[(Object Storage<br/>chunks by hash)]
    MetaSvc --> Notify[Notification Service]
    Notify -.->|"changed!"| OtherDevices[Other devices]
    OtherDevices -->|pull new metadata + chunks| MetaSvc

Upload/edit: the client computes chunk hashes locally, asks the metadata service which chunks are new (not already stored), uploads only those to the block service → object storage, then commits new file metadata (new version = new chunk list).

Sync: when metadata changes, the notification service tells the user’s other devices; they pull the updated metadata, see which chunk hashes are new to them, and download only those. 🚨 Sync moves metadata first, chunks second, and only the diff.

5. Deep dives

5a. Efficient sync (the heart)

The sync agent keeps a local view of file → chunk-hash lists. On any change:

Recompute chunk hashes locally.
Diff against the last-known server metadata → the set of changed chunks.
Upload only changed chunks + commit new metadata. For download, the reverse: fetch new metadata, download only chunks you don’t already have (dedup applies locally too). 🚨 A one-byte edit to a 1 GB file transfers ~4 MB, not 1 GB.

5b. Notifying devices of changes

Devices need to know quickly when something changed elsewhere. Options: a long-poll/WebSocket “notification” channel per device, or push. The server maintains, per user, a change log / cursor; devices poll or get pushed “you have changes since cursor X,” then pull. (Notification System) Offline devices catch up on reconnect via the cursor (like chat sync).

5c. Conflict resolution

Two devices edit the same file offline, then both sync. You can’t silently lose one. Common approach: detect the divergence (both changed from the same base version) and keep both as conflicting copies (“file (conflicted copy from Bob’s laptop)”), letting the user reconcile. Simpler and safer than trying to auto-merge arbitrary binary files. Use version vectors to detect concurrency. (Conflict Resolution)

5d. Durability

The core promise. Object storage replicates chunks across zones (11 nines of durability). Metadata is replicated too. Versioning means even a bad edit is recoverable. Never acknowledge a write until it’s durably stored.

A shared folder = the same chunk lists referenced by multiple users’ namespaces, with an access-control list. Dedup means the shared data isn’t duplicated per user. Permission checks gate metadata access.

6. Bottlenecks & scaling further

Storage → object storage + chunk-level dedup (huge savings from identical chunks).
Transfer efficiency → delta sync (only changed chunks).
Sync notification fan-out → per-user change log + push/long-poll; catch up via cursor.
Metadata scale → shard the metadata DB by user; it’s small relative to chunks.
Hot files (widely shared) → CDN/cache popular chunks.

7. Trade-off summary

Decision	Chosen	Alternative	Why
File storage	Chunks by content hash	Whole-file blobs	Enables dedup + delta sync
Sync	Metadata diff + changed chunks	Re-upload whole file	Transfers only what changed
Dedup	Content-addressed, global	Per-user copies	Massive storage savings
Conflicts	Keep both (conflicted copy)	Auto-merge / last-write-wins	Never lose data; safe for binaries
Notify	Change log + push/long-poll	Periodic full scan	Fast, cheap change propagation

8. Follow-up questions

Why chunk files instead of storing them whole?

Chunking unlocks the two features that define the service. First, deduplication: chunks are addressed by a hash of their content, so if two files (or two users) contain the same chunk, it's stored once — re- uploading an identical file costs zero new storage, and common data (shared documents, OS files) is stored a single time across everyone. Second, delta sync: when a file is edited, only some of its chunks change, so the client re-uploads just those chunks rather than the entire file — a one-byte change to a gigabyte file transfers one ~4 MB chunk, not a gigabyte. Whole-file storage gives you neither: every copy costs full storage and every edit costs a full re-transfer. The cost of chunking is more metadata (a file is now an ordered list of chunk hashes), which is small and well worth it.

How does a change on my laptop reach my phone efficiently?

Metadata first, then only the missing chunks. When the laptop commits a change, the server records it in the user's change log and notifies the user's other devices (push or long-poll). The phone pulls the updated file metadata — the new ordered list of chunk hashes — and compares it to what it already has locally. It then downloads only the chunks whose hashes it doesn't already possess (dedup applies on the device too), and reassembles the file. So the phone transfers just the delta, and if it was offline, it catches up on reconnect using a cursor into the change log. The heavy data (chunks) moves only when genuinely new; the lightweight metadata drives the whole sync.

Two devices edit the same file offline. What happens when both sync?

The server detects that both edits descend from the same base version (using version vectors), meaning they diverged concurrently rather than one building on the other. Rather than silently overwriting one edit (which would lose a user's work) or trying to auto-merge arbitrary binary content (which is unsafe and often impossible), the safe standard approach is to keep both: one becomes the file, the other is saved as a clearly-labeled conflicted copy ("file (conflicted copy from Bob's laptop)") so the user can reconcile them manually. The guiding principle is that a storage service's cardinal rule is never to lose data, so when in doubt it preserves both versions.

How do you guarantee a file is never lost?

Durability is layered. Chunks live in object storage that replicates each chunk across multiple availability zones, giving extremely high durability (on the order of eleven nines), and metadata is replicated too. A write is never acknowledged to the client until the data is durably persisted, so a crash mid-upload can't report false success. Versioning keeps prior versions, so even a destructive edit or accidental deletion is recoverable. Together — replicated durable chunk storage, durable acked writes, and version history — these make loss effectively impossible short of catastrophic correlated failure, which cross-zone replication is designed to survive.

9. What junior / mid / senior answers look like

Junior: stores whole files in object storage with metadata in a DB; sync = re-download changed files. Works but re-transfers everything and can’t dedup.
Mid: chunks files, dedups by content hash, syncs by uploading only changed chunks, notifies devices of changes, handles versioning.
Senior: all that plus the local-diff sync agent, change-log + cursor for offline catch-up, version- vector conflict detection with keep-both resolution, durability as the cardinal guarantee, and sharing via shared chunk references + ACLs — framing sync as “move metadata + the diff, never the whole file.”