system-design

Scaling Writes

The hard direction. You can’t duplicate a write the way you duplicate a read — there’s one source of truth — so scaling writes eventually forces the most consequential decision in system design: sharding.

Prerequisites: Sharding, Storage Engines Time to read: ~16 minutes

Why writes are the hard direction

🚨 Writes are fundamentally harder to scale than reads because they can’t be freely duplicated. A read can be served from any of a hundred copies; a write must go to the source of truth and be coordinated there. Read replicas don’t help — they all replicate from the primary, so writes still bottleneck on one machine.

📐 So while reads scale by adding copies, writes eventually force you to either do less writing, do it more efficiently, or split the data itself (sharding) — and sharding is the biggest, most irreversible decision in system design. → Sharding

The write-scaling toolkit (in order of preference)

🚨 Exhaust the cheaper options before sharding — sharding is close to irreversible:

1. Vertical scaling first

🚨 A single primary handles far more writes than people assume — tens of thousands per second on good hardware. Before anything clever, a bigger machine (more IOPS, RAM, CPU) is the simplest write scaling and buys a lot of runway. → Scalability

2. Batch writes

🚨 Combine many small writes into fewer larger ones. Because much of a write’s cost is per-operation overhead (round trips, transaction setup, index updates), batching amortizes it.

1000 individual inserts (1000 round trips + 1000 index updates)
vs 1 batch insert of 1000 rows (1 round trip, batched index update) → far cheaper

Applies to database inserts (bulk insert), and to a write-behind cache (accumulate in memory, flush aggregates periodically — how high-volume counters work).

3. Async writes / write buffering

🚨 Take writes off the critical path. Instead of writing synchronously, accept the write into a queue and process it asynchronously — which also levels write spikes (a 10× burst becomes a longer queue, not an overwhelmed database).

✅ Fast response, absorbs spikes, batches naturally. ❌ Eventual consistency; must handle idempotency and the dual-write problem. Data-loss risk if the buffer isn’t durable.

4. Use a write-optimized storage engine

🚨 LSM-tree engines (Cassandra, RocksDB) sustain far higher write throughput than B-trees because they only append (sequential writes) instead of updating in place (random writes). If the workload is genuinely write-heavy (time series, events, logs), the engine choice is a major lever before sharding.

📐 Sequential writes are ~10-100× faster than random. This is why write-heavy systems use LSM stores.

5. Fewer indexes

🚨 Every index is updated on every write — write amplification. A write-heavy table wants the minimum indexes the queries require. Dropping unused indexes directly increases write throughput.

6. Shard — the last resort, and the biggest step

🚨 When one primary genuinely can’t handle the write volume, you split the data across multiple databases so each handles a subset of writes. This is the only way to scale writes past one machine horizontally — and it’s expensive and irreversible:

Choose a shard key matching the access pattern (the highest-stakes decision — get it wrong and every query fans out).
Lose cross-shard joins and transactions (→ sagas).
Handle hot shards, rebalancing (virtual shards), distributed IDs.

🎙️ “For write scaling I’d exhaust the cheaper options first — vertical scaling (one primary handles tens of thousands of writes/second), batching, async writes through a queue, an LSM engine if it’s genuinely write-heavy, and dropping unused indexes. Only when one primary truly can’t keep up would I shard, because that’s close to irreversible and costs us joins and transactions.”

This “exhaust the alternatives before sharding” ordering is the strong answer.

Reduce writes at the source

🚨 Often the best write-scaling is writing less, and it’s underused:

Don’t write what you don’t need — do you need to persist every event, or can you sample/aggregate?
Deduplicate — idempotency so retries don’t double-write.
Coalesce updates — if a value changes 100 times a second, write the latest periodically, not every change (write-behind).
Approximate — for counters/analytics, CRDTs or probabilistic structures reduce write coordination.

The special case: high-volume counters and hot writes

🚨 A common write-scaling problem worth knowing: a single counter (likes on a viral post, a global metric) written millions of times a second is a hot write — every write contends on one row/key. Fixes:

Sharded counters — split the counter into N sub-counters (counter:0..9), increment a random one, sum on read. Spreads the write contention across N keys.
Write-behind / batch — accumulate in memory, flush aggregates.
CRDT counters — increment locally, merge — no coordination.

→ Hot Keys

⚖️ Trade-offs

Technique	Gain	Cost
Vertical scaling	Simple, big runway	Ceiling; SPOF; expensive at top
Batching	Amortizes per-write overhead	Higher per-item latency
Async writes	Fast response, spike absorption	Eventual consistency; durability/idempotency
LSM engine	Much higher write throughput	Slower/variable reads; compaction
Fewer indexes	Faster writes	Slower reads
Sharding	Unlimited write scale	No joins/transactions; irreversible; hot shards
Write less / coalesce	Fewer writes entirely	Approximation; lost granularity

In the real world

Discord’s message-storage journey (MongoDB → Cassandra → ScyllaDB) is the canonical write-scaling story — each migration was driven by outgrowing the previous store’s write capacity, ending on an LSM engine (ScyllaDB) tuned for their write volume. It shows write scaling as the thing that forces database migrations.
Sharded counters are a standard pattern for hot-write problems (viral post likes, ad impressions) — splitting one contended counter into many sub-counters is how systems handle millions of increments per second to “one” value.
The “one primary handles more than you think” reality — many teams shard prematurely, when vertical scaling, batching, and dropping unused indexes would have bought years. Sharding’s irreversibility makes premature sharding especially costly.

🚨 Interview traps

Sharding first — exhaust vertical scaling, batching, async, LSM, fewer indexes first.
Thinking read replicas help write scaling — they replicate from the primary.
Not knowing LSM vs B-tree for write-heavy workloads.
Ignoring write amplification from indexes.
Not handling the hot-write / counter problem (sharded counters).
Sharding without a justified shard key or a plan for lost joins/transactions.
Not considering writing less at the source.

🎙️ Soundbites

“Writes are the hard direction — you can’t duplicate a write like a read, and replicas don’t help because they replicate *from the primary. So it’s do less writing, do it more efficiently, or split the data.”*
“I’d exhaust the cheaper options before sharding: vertical scaling handles tens of thousands of writes/second, then batching, async writes through a queue that also levels spikes, an LSM engine if it’s genuinely write-heavy, and dropping unused indexes. Sharding is last because it’s irreversible and costs us joins and transactions.”
“For a viral like-counter — millions of writes to one row — I’d use sharded counters: split into N sub-counters, increment a random one, sum on read. That spreads the write contention.”
“An LSM engine sustains far more writes than a B-tree because it only appends — sequential writes are 10 to 100× faster than the random writes of in-place updates.”
“Often the best write scaling is writing less — coalescing rapid updates, sampling events, or deduplicating with idempotency so retries don’t double-write.”

🛠️ Try it

1. Find the single-primary write ceiling. Load a database with concurrent writers, ramping up until write throughput plateaus. That number is your real ‘when would we shard?’ threshold — usually much higher than expected, which is the argument against premature sharding.

2. Measure batching’s impact. Insert 100,000 rows one at a time, then in batches of 1,000. Compare the time — batching is often 10-50× faster because it amortizes per-write overhead.

3. Compare LSM vs B-tree writes. Benchmark write throughput on Postgres (B-tree) vs Cassandra/RocksDB (LSM) for a write-heavy workload. See the sequential-append advantage — the reason write-heavy systems use LSM.

4. Build a sharded counter. Implement a counter as one row vs N sharded sub-counters under heavy concurrent increments. Watch the single row become a contention bottleneck while the sharded version scales — the hot-write fix, demonstrated.

Check yourself

1. Why can't read replicas help scale writes?

Because read replicas are copies that replicate *from* the primary — they exist to serve reads, and every write still has to go to the primary first (which then propagates it to the replicas). The primary is the single source of truth for writes, and replicas can't accept writes independently (that would create conflicting sources of truth). So no matter how many read replicas you add, all writes funnel through the one primary, and its write capacity is unchanged — if anything, more replicas add *more* replication load on the primary. This is the fundamental asymmetry: reads can be served from any of many copies (so you scale reads by adding copies), but writes must be coordinated at one authoritative source (so adding read copies does nothing for write throughput). Scaling writes past one machine requires actually splitting the data so different machines own different writes — which is sharding, with all its costs — or reducing/batching/async-buffering the writes so the single primary can keep up. Confusing "add replicas" (a read-scaling move) with write scaling is a common mistake.

2. What should you try before sharding to scale writes, and in what order?

Sharding is close to irreversible and costs you cross-shard joins and transactions, so you exhaust the cheaper options first, roughly in this order. **Vertical scaling**: a single primary on good hardware handles tens of thousands of writes per second — more than most people assume — so a bigger machine (more IOPS, RAM, CPU) is the simplest first step and buys significant runway. **Batching**: combine many small writes into fewer larger ones (bulk inserts), amortizing the per-write overhead of round trips, transaction setup, and index updates — often a 10-50× improvement. **Async writes / buffering**: accept writes into a queue and process them asynchronously, taking them off the critical path and *leveling* spikes (a burst becomes a longer queue, not an overwhelmed database), at the cost of eventual consistency and needing idempotency. **A write-optimized (LSM) storage engine**: if the workload is genuinely write-heavy, Cassandra/RocksDB sustain far more writes than a B-tree because they only append (sequential) rather than update in place (random). **Dropping unused indexes**: every index is updated on every write, so a write-heavy table wants minimal indexes. Only when one primary genuinely can't keep up after all this do you **shard** — and then with a carefully chosen shard key and a plan for the lost joins and transactions.

3. Why does an LSM-tree engine sustain higher write throughput than a B-tree?

Because LSM-trees only ever *append*, turning all writes into sequential disk I/O, while B-trees update data *in place*, requiring random I/O. In a B-tree, a write finds the right page (possibly reading it first), modifies it, and writes it back to its specific location on disk — a random write — and page splits can cascade into more random writes, plus every index update is another in-place modification. In an LSM-tree, writes go to an in-memory buffer (memtable) and a sequential commit log, and when the buffer fills it's flushed to disk as one large sequential write to an immutable sorted file (SSTable); compaction later merges these files, also sequentially. Since sequential disk writes are roughly 10-100× faster than random writes (no seeking, batchable into large contiguous operations — dramatically so on spinning disks but significant even on SSDs), the LSM design gets far more write throughput from the same hardware. The trade-off is on the read side (a read may check multiple SSTables, needing Bloom filters to stay fast) and compaction overhead (background CPU and I/O), so LSM engines suit write-heavy workloads (time series, event logs, messaging) while B-trees suit read-heavy transactional ones — which is why the engine choice is itself a major write-scaling lever for genuinely write-dominated systems.

4. How do you scale a single high-volume counter (like likes on a viral post)?

A single counter written millions of times per second is a hot-write problem — every increment contends on the same row or key, serializing writes at one point that can't keep up regardless of overall database capacity. The standard fix is **sharded counters**: split the logical counter into N physical sub-counters (`counter:0`, `counter:1`, ... `counter:9`), have each increment go to a *randomly chosen* sub-counter, and compute the total by summing all N on read. This spreads the write contention across N independent keys, so N sub-counters can absorb roughly N× the write rate, at the cost that reads must sum N values (cheap for small N). Alternatives: **write-behind / batching** — accumulate increments in memory (or a fast store like Redis) and flush the aggregate to the durable store periodically (say every few seconds), trading a small window of potential loss for a massive reduction in database writes (a counter losing a few seconds of increments in a crash is usually fine); and **CRDT counters** — each node increments its own local count and counts merge deterministically without coordination, ideal for distributed or eventually-consistent settings. All three convert "millions of writes to one place" into "writes spread across many places, combined on read," which is the general principle for hot writes.

5. Why is "write less" often the best write-scaling strategy?

Because the cheapest write to scale is the one you never make — reducing the write volume at the source attacks the problem directly, without the cost and complexity of batching infrastructure, async pipelines, or (especially) sharding's irreversible loss of joins and transactions. Several forms: **don't write what you don't need** — question whether every event must be persisted, or whether sampling or pre-aggregation suffices (do you need every individual analytics event, or counts per minute?). **Coalesce rapid updates** — if a value changes 100 times a second, writing the latest value periodically instead of every change turns 100 writes into 1, with no meaningful loss for most use cases (the write-behind pattern). **Deduplicate** — idempotency ensures retries and duplicate deliveries don't cause redundant writes, which matters because at-least-once systems naturally produce duplicates. **Approximate** — for counters and analytics, CRDTs and probabilistic structures (HyperLogLog for unique counts, Count-Min Sketch for frequencies) achieve the goal with far less write coordination than exact counting. Each of these reduces the fundamental amount of writing rather than making the writes faster, which is often a larger and simpler win — and it's underused because the instinct is to scale the infrastructure to handle the writes rather than to question whether all the writes are necessary. The same "do less work" principle that makes it the best *read* optimization (an index eliminating a scan) applies to writes: eliminating unnecessary work beats doing unnecessary work efficiently.