Services announce what happened instead of telling each other what to do. It decouples systems beautifully and makes the overall behaviour much harder to see.
Prerequisites: Message Queues, Service Decomposition Time to read: ~26 minutes
An order is placed. Five things must happen:
def place_order(order):
db.save(order)
inventory_service.reserve(order.items) # synchronous call
payment_service.charge(order.total) # synchronous call
email_service.send_confirmation(order) # synchronous call
analytics_service.track(order) # synchronous call
loyalty_service.add_points(order) # synchronous call
Everything wrong with this:
The event-driven inversion:
def place_order(order):
db.save(order)
publish("OrderPlaced", order) # that's it
Ordering announces a fact. Whoever cares, reacts. Adding a fraud service requires zero changes to ordering.
🚨 A distinction that matters more than it appears, and getting the vocabulary right signals real understanding.
| Command | Event | |
|---|---|---|
| Intent | “Do this” | “This happened” |
| Tense | Imperative — ChargePayment |
Past tense — PaymentCharged |
| Recipients | One, named | Any number, unknown to the sender |
| Can be rejected | ✅ Yes | ❌ No — it already happened |
| Coupling | Sender knows the receiver | Publisher knows nothing about consumers |
Naming events in the past tense is not a style preference. It forces the right mental model: an
event is an immutable historical fact. OrderPlaced cannot be refused. If you find yourself wanting
to name an event ProcessOrder, you’ve written a command and disguised it as an event — and the
coupling comes back with it.
🚨 The most common mistake in event-driven systems is publishing commands dressed as events:
SendEmailRequested is a command with an event’s grammar. The publisher still knows exactly what
should happen next, so nothing was decoupled.
1. Event notification — minimal payload, just “something happened, go look.”
{"type": "OrderPlaced", "orderId": "ord_4821", "occurredAt": "2026-07-22T10:00:00Z"}
✅ Tiny messages, no stale data. ❌ 🚨 Every consumer must call back to the producer, which recreates the coupling and the load you were avoiding — and now with an availability dependency in the reverse direction.
2. Event-carried state transfer — the event carries the data consumers need.
{"type": "OrderPlaced", "orderId": "ord_4821", "customerId": "cus_7",
"items": [{"sku": "SHOE-42", "qty": 2, "unitPriceCents": 4500}],
"totalCents": 9000, "shippingAddress": {...}, "occurredAt": "..."}
✅ Consumers need no callback. They can build a local read model and serve queries without ever calling the producer — so the producer being down doesn’t affect them. ❌ Larger messages; the payload is a contract you must version; some duplication.
🚨 This is usually the right default, and it’s what makes event-driven architecture genuinely decoupling rather than just asynchronous.
3. Event sourcing — the event log is the source of truth; state is derived by replaying it. A much bigger commitment. → Event Sourcing
Same distinction as in sagas, and it’s the central design choice here.
Choreography — each service reacts to events and emits its own. No coordinator.
flowchart LR
O[Ordering] -->|OrderPlaced| I[Inventory]
O -->|OrderPlaced| P[Payments]
I -->|StockReserved| S[Shipping]
P -->|PaymentCaptured| S
P -->|PaymentCaptured| L[Loyalty]
✅ Maximum decoupling. Adding a consumer changes nothing. ❌ 🚨 The business process exists nowhere. To understand what happens when an order is placed you must read every service and infer the flow from subscriptions. There is no artifact that describes it, and no one can answer “where is order 4821 in the process?”
Orchestration — a coordinator explicitly drives the sequence and tracks state.
✅ The flow is explicit, testable, and observable. ❌ More coupling; another component.
🎙️ The balanced answer: “Choreography for genuine fan-out — many independent consumers reacting to a fact. Orchestration when there’s a business process with a defined sequence and a state I need to be able to query. Most systems need both, and using choreography for a multi-step workflow is how you end up unable to explain your own system.”
1. Extensibility without modification. The strongest argument. A new consumer subscribes; nothing
upstream changes. Over time this is enormously valuable — the analytics team, the ML team, the fraud
team, and a partner integration all consume OrderPlaced and the ordering team never knew.
2. Temporal decoupling. The consumer can be down for an hour. Messages queue. Nothing is lost.
3. Load levelling. A traffic spike becomes a longer queue rather than a cascade of failures. → Message Queues
4. Independent scaling and failure. Each consumer scales to its own workload; a slow consumer doesn’t slow the producer.
5. Auditability. The event stream is a record of everything that happened.
6. Replay. With a durable log (Kafka), fix a bug and reprocess history. This is genuinely transformative for derived data.
⚖️ Event-driven systems trade local clarity for global opacity. That’s the essential bargain.
1. Eventual consistency, everywhere. The order exists; the inventory isn’t reserved yet. Your UI must handle “processing” states, and users will refresh and see inconsistent things.
2. The flow is invisible. 🚨 No single place describes what happens when an order is placed. New engineers cannot read the code and understand the system. This is the biggest practical cost and it compounds over years.
3. Debugging is genuinely hard. “Why didn’t this email send?” requires tracing across services and time. Distributed tracing with correlation IDs is mandatory, not optional. → Distributed Tracing
4. Duplicate delivery. At-least-once means every consumer must be idempotent. Every one. → Idempotency ⭐
5. Ordering is only per-partition. Events for different entities can be processed out of order.
6. Schema evolution across many unknown consumers. You can’t coordinate a breaking change with consumers you don’t know about. → Serialization
7. The dual-write problem. Saving to the database and publishing aren’t atomic. → Transactional outbox
8. Accidental cycles. Service A’s event triggers B, which emits an event that triggers A. Infinite loops that only appear under specific conditions.
Past tense, business language. OrderPlaced, not OrderTableRowInserted. The event should be
meaningful to a domain expert, not describe your database.
Include what consumers need, not your entire internal model. 🚨 An event is a public contract. Dumping your internal entity into it means you can never refactor it.
Include metadata:
{
"eventId": "evt_9f3a...", // for idempotent consumers
"eventType": "OrderPlaced",
"eventVersion": "2", // for schema evolution
"occurredAt": "2026-07-22T10:00:00Z",
"correlationId": "req_7c2b...", // ties it back to the originating request
"causationId": "evt_prior...", // which event caused this one
"data": { ... }
}
🚨 correlationId and causationId are what make debugging possible. Correlation groups
everything from one user action; causation forms the chain. Without them, an event-driven system is
genuinely opaque during an incident. Add them from day one — retrofitting is painful.
Version from the start. Add optional fields only; never repurpose one. Use a schema registry that rejects incompatible changes in CI.
Right-size the granularity. OrderPlaced is useful. OrderFieldChanged is a database changelog
pretending to be a domain event, and consumers can’t do anything meaningful with it.
🚨 Two things every event-driven design must address:
At-least-once means duplicates. Every consumer must be idempotent — dedupe on eventId in the
same transaction as the side effect. There is no way around this.
→ Idempotency
Publishing is not atomic with your database write.
db.save(order) # ✅ committed
publish("OrderPlaced", order) # ❌ broker unreachable
# The order exists. Nothing downstream knows. Silently, forever.
Use the transactional outbox: write the event to an outbox table in the same transaction, and
have a relay (or CDC) publish it.
🎙️ “Saving the order and publishing the event aren’t atomic, so I’d use a transactional outbox — the event goes in the same transaction as the order, and a relay publishes from there.”
🚨 The single biggest practical problem is that nobody can see the system. Concrete mitigations worth naming:
Distributed tracing with correlation IDs across every event hop.
An event catalogue — a registry documenting every event type, its schema, its producer, and its known consumers. Tools like AsyncAPI and EventCatalog exist for this. Without one, nobody knows who consumes what, and every schema change is a gamble.
Process documentation — an actual diagram of the flow, maintained. Yes, it can drift from reality; having a stale diagram is still far better than having none.
Orchestration for the flows that matter. If you need to answer “where is this order?”, make it explicit rather than inferring it.
Consumer-driven contract tests — consumers publish what they expect; producers verify it in CI. This is how you learn who depends on you before you break them.
⚖️ Event-driven is not a default. Reasons to prefer a synchronous call:
🎙️ “Queries stay synchronous — the caller needs an answer. State changes that others react to become events. Making everything asynchronous would give us eventual consistency in places where we don’t need it and can’t easily explain it to users.”
| Decision | Gain | Cost |
|---|---|---|
| Events over synchronous calls | Decoupling, extensibility, load levelling, availability | Eventual consistency; invisible flow; harder debugging |
| Event-carried state | Consumers are fully independent | Larger payloads; the schema is a contract; duplication |
| Event notification | Small messages; no stale data | Callbacks recreate coupling and load |
| Choreography | Maximum decoupling | The process exists nowhere |
| Orchestration | Visible, queryable flow | Coupling; another component |
| Durable log (Kafka) | Replay, many consumers, audit trail | Operational complexity |
| Transactional outbox | No lost events | An extra table and a relay |
SendEmail as an “event” — the coupling is still there.OrderPlaced and doesn’t know who consumes it. Adding a fraud check later
means subscribing to an existing event, with zero changes to ordering — that extensibility is the
main reason to do this.”1. Convert a synchronous flow. Take an endpoint making three synchronous downstream calls. Replace them with a published event and three consumers. Measure the endpoint’s p99 before and after. Then kill one consumer and confirm the endpoint still works — that’s the availability benefit, demonstrated.
2. Cause the dual-write bug. Save to the database, then publish — with the broker stopped. The record exists and no consumer knows. Then implement an outbox table and a relay, and confirm the event is eventually published even if the broker was down at commit time.
3. Build the debugging problem, then fix it. Chain four services via events, with no correlation IDs. Now try to trace one request through the logs. It’s genuinely hard. Add a correlation ID propagated through every event and try again.
4. Create an accidental cycle. Service A emits an event that B consumes; B emits one that A
consumes. Watch it loop. Then add cycle detection via causationId chains and see how you’d catch it.