Turning your database’s replication log into an event stream. It solves the dual-write problem completely, and it’s how derived systems stay in sync without anyone remembering to update them.
Prerequisites: Message Queues, Storage Engines, Replication Time to read: ~22 minutes
An order is created. Five systems need to know:
Postgres (source of truth) — the order row
Elasticsearch — so it's searchable
Redis — cache invalidation
The data warehouse — analytics
The notification service — email the customer
Attempt 1 — dual writes:
db.insert(order)
search.index(order)
cache.delete(f"orders:{order.user_id}")
warehouse.insert(order)
notifications.send(order)
🚨 This is broken, and it’s broken in a way that fails silently. If search.index throws, the
order exists in Postgres and is invisible in search — forever, with no error anyone will notice. If
the process dies after the database insert, four systems never hear about it. There is no
transaction spanning these systems, so partial failure is guaranteed at scale.
Attempt 2 — publish an event:
db.insert(order)
kafka.publish("order.created", order) # ← what if this fails?
Same problem, one system smaller. The database write and the publish aren’t atomic. → The dual-write problem
Attempt 3 — poll the database:
SELECT * FROM orders WHERE updated_at > :last_check;
Better, but: it loads production on every poll, it misses deletes entirely, it misses rows
created and deleted between polls, it requires an updated_at column that everything updates
correctly, and clock skew or transactions committing out of order cause missed rows.
Your database already maintains a perfect, ordered, durable log of every committed change. It has to — that’s how crash recovery and replication work.
Postgres: the WAL (write-ahead log)
MySQL: the binlog
MongoDB: the oplog
Change Data Capture reads that log and publishes the changes as events.
flowchart LR
A[Application] -->|writes| DB[(PostgreSQL)]
DB -->|WAL| C[Debezium connector]
C --> K[[Kafka]]
K --> ES[Elasticsearch]
K --> CA[Cache invalidator]
K --> DW[Data warehouse]
K --> N[Notification service]
🚨 Why this is qualitatively better than every alternative:
🎙️ This is one of the most valuable things to propose unprompted in a design interview, because most candidates reach for dual writes and don’t notice the problem.
Debezium is the standard implementation. It registers as a logical replication client — the database thinks it’s just another replica.
{
"op": "u",
"ts_ms": 1735689600000,
"source": {"db": "shop", "table": "orders", "lsn": 24023119, "txId": 5821},
"before": {"id": 42, "status": "pending", "total_cents": 4200},
"after": {"id": 42, "status": "paid", "total_cents": 4200}
}
Every event carries the operation (create, update, delete, read-for-snapshot), the row
before and after, and source metadata including the log position and transaction ID.
🚨 The before image is genuinely useful and easy to overlook: it lets consumers compute deltas
(“the status changed from pending to paid” rather than just “it’s paid now”), which is what makes
event-driven reactions precise.
Setup requirements:
wal_level = logical, plus a replication slot and a publication.
🚨 Replication slots retain WAL until consumed. If your CDC consumer stops and nobody notices,
the WAL grows until the disk fills and the database stops accepting writes. This is a real,
recurring production incident. Monitor replication slot lag as a first-class alert.binlog_format = ROW (not STATEMENT), plus binlog_row_image = FULL for the before
image.The initial snapshot. When CDC starts, the log only has recent changes. So the connector first
takes a consistent snapshot of existing rows (emitted as r events), then switches to streaming from
the exact log position where the snapshot was taken. Debezium handles this transition, and doing it
without downtime for very large tables is the operationally tricky part (incremental snapshotting
exists for this).
| Use case | Why CDC fits |
|---|---|
| Keeping a search index current | Never miss an update; no application changes → Search |
| Cache invalidation | Precisely invalidate when the underlying row actually changes |
| Feeding the data warehouse | Near-real-time, no heavy periodic exports → Analytics |
| Microservice data sync | Service B needs a read-only copy of Service A’s data |
| Audit logs | An immutable record of every change, including who and when |
| Zero-downtime migrations | Dual-write to the new schema without touching application code → Migrations |
| Materialized views | Maintain denormalized read models incrementally |
| Strangler fig migrations | Keep the legacy and new systems in sync during a gradual cutover |
🚨 The microservice sync case deserves emphasis because it’s an increasingly standard pattern. Rather than Service B calling Service A’s API on every request (coupling, latency, availability multiplication), Service B subscribes to A’s change stream and maintains its own local read-only copy. Reads become local and fast, and A being down doesn’t take B down.
⚖️ The cost: B’s copy is eventually consistent, and you’ve created a dependency on A’s schema rather than its API — which is a real coupling concern (see below).
🚨 The most important nuance in this chapter.
Raw CDC publishes table rows. That means consumers are coupled to your internal schema — rename a
column and you break every downstream system. And a row change often doesn’t map cleanly to a
business event: an order going from pending to paid is meaningful; the same row’s updated_at
changing is not.
The transactional outbox pattern fixes this:
BEGIN;
INSERT INTO orders (id, user_id, total_cents, status)
VALUES (42, 7, 4200, 'paid');
INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload)
VALUES ('order', '42', 'OrderPaid',
'{"order_id":42,"user_id":7,"amount":4200,"currency":"PKR"}');
COMMIT; -- atomic: both rows or neither
Then CDC streams the outbox table, not the business tables.
✅ You publish deliberate business events with a stable, versioned contract — not incidental row
changes.
✅ Still atomic — the event is in the same transaction as the data.
✅ Consumers aren’t coupled to your internal schema. You can refactor orders freely.
✅ You choose exactly which changes are events.
❌ Application code must write to the outbox (unlike raw CDC). ❌ The outbox table needs periodic cleanup (delete rows after they’re captured, or use a short-retention partitioned table).
🎙️ The distinction to draw in an interview: “Raw CDC on the business tables couples every consumer to our internal schema and turns every column change into an event. I’d use the transactional outbox — write a deliberate business event in the same transaction, and stream that. We get atomicity without exposing our schema as a public contract.”
Schema changes. 🚨 What happens downstream when you ALTER TABLE? Adding a nullable column is
usually fine; renaming or dropping one breaks consumers. Mitigations: a
schema registry with compatibility enforcement, the outbox
pattern (so consumers never see your table schema), and treating any CDC-exposed table as a public
API with a change process.
Deletes. CDC emits a delete event with the before image and (in Debezium) a tombstone — a
message with a null value — which enables Kafka
log compaction to actually remove the key. Soft
deletes appear as ordinary updates, which consumers must interpret correctly.
Ordering. Preserved per-table and per-key if you partition Kafka by primary key. Order across
tables is not guaranteed unless you’re careful — so a consumer might see an order_items row before
the orders row it references. Handle this by tolerating out-of-order arrival, or by using the
outbox so one event carries the whole aggregate.
Transaction boundaries. By default, CDC emits per-row events, so a transaction that updated 3 tables becomes 3+ independent events and consumers see intermediate states. Debezium can emit transaction metadata markers to let consumers reassemble them; the outbox pattern sidesteps it entirely.
Backfill. The initial snapshot of a billion-row table is heavy and slow. Incremental snapshotting (chunked, interleaved with streaming) exists for this and is worth knowing about.
Replication slot management. As above — an unconsumed slot fills your disk and stops the database. Alert on it.
Sensitive data. 🚨 CDC streams everything in the table, including password hashes, PII, and payment details. Column filtering and masking must be configured deliberately, and this is a common compliance oversight. → Privacy & Compliance
| Decision | Gain | Cost |
|---|---|---|
| CDC over dual writes | No lost events; atomic by construction | A pipeline to operate; eventual consistency |
| CDC over polling | Captures deletes; no production query load; nothing missed | Requires log access and configuration |
| Raw table CDC | Zero application changes | Consumers coupled to internal schema; noisy events |
| Outbox pattern | Deliberate, versioned business events | Application must write the outbox; cleanup needed |
| Log compaction on CDC topics | Bounded storage; replayable current state | Loses intermediate history |
1. Set up Debezium end to end. Docker Compose with Postgres, Kafka, and Debezium — about 20 minutes with the official images:
ALTER SYSTEM SET wal_level = logical; -- then restart
CREATE TABLE orders (id SERIAL PRIMARY KEY, status TEXT, total INT);
Register the connector, then INSERT, UPDATE, and DELETE a row while watching the Kafka topic:
kafka-console-consumer --topic dbserver1.public.orders --from-beginning \
--bootstrap-server localhost:9092
Watching an UPDATE produce an event with both before and after is the moment this clicks.
2. Implement the outbox pattern. Add an outbox table, write to it in the same transaction as a
business change, and configure Debezium’s outbox event router. Compare the events you get to raw
table CDC — the difference in cleanliness is the argument.
3. Break it deliberately. Stop your CDC consumer and keep writing to the database. Then check:
SELECT slot_name, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots;
Watch the retained WAL grow. That number is what fills your disk if you don’t alert on it.
4. Prove polling misses deletes. Write a poller using updated_at, then delete a row. The poller
never learns about it. CDC does.
WHERE updated_at > last_check?