system-design

The Saga Pattern

How you get “transactions” across services when real transactions aren’t available: a sequence of local commits, each with an undo. You trade isolation for availability, and the undo logic is business logic.

Prerequisites: Distributed Transactions, Message Queues Time to read: ~24 minutes

The problem

Placing an order touches four services:

Order Service      → create the order
Payment Service    → charge the card
Inventory Service  → reserve the items
Shipping Service   → schedule delivery

Each has its own database. There is no transaction spanning them, and 2PC blocks if the coordinator dies.

🚨 What happens if step 3 fails after step 2 succeeded? The customer has been charged and will receive nothing. You need a way to undo.

The core idea

A saga is a sequence of local transactions. Each one commits immediately in its own service. If a later step fails, you run compensating transactions to undo the earlier ones, in reverse.

Forward:      T1 → T2 → T3 → T4
Compensating: C1 ← C2 ← C3        (if T4 fails)

Step	Forward	Compensation
1	Create order (pending)	Mark order cancelled
2	Charge card	Refund
3	Reserve inventory	Release reservation
4	Schedule shipment	Cancel shipment

🚨 Compensation is not rollback. A database rollback erases the transaction as though it never happened. A compensation is a new business fact: the charge happened and then a refund happened. Both appear on the customer’s statement. Both are in your ledger. The intermediate state was real and was visible to everyone.

That distinction is the heart of the pattern, and it’s where the design work is.

Two ways to coordinate

Choreography — services react to events

No central coordinator. Each service listens for events and emits its own.

flowchart LR
    O[Order Service] -->|OrderCreated| P[Payment Service]
    P -->|PaymentCompleted| I[Inventory Service]
    I -->|InventoryReserved| S[Shipping Service]
    I -->|InventoryFailed| P2[Payment: refund]

✅ Loosely coupled — no service knows about the whole flow. Adding a step means adding a subscriber. ✅ No single point of failure or bottleneck.

❌ 🚨 The flow exists nowhere. To understand what happens when an order is placed, you must read four codebases and infer the sequence from event subscriptions. There’s no diagram that is authoritative. ❌ Debugging is hard — “why didn’t step 3 run?” requires tracing across services. ❌ Cyclic dependencies creep in as the flow grows.

Use for: short sagas (2–4 steps) with simple flows.

Orchestration — a coordinator drives it

A saga orchestrator explicitly tells each service what to do and tracks the state.

flowchart TB
    SO[Order Saga Orchestrator]
    SO -->|1. reserve| I[Inventory]
    SO -->|2. charge| P[Payment]
    SO -->|3. ship| S[Shipping]
    I -.->|ok / fail| SO
    P -.->|ok / fail| SO
    S -.->|ok / fail| SO

✅ The flow is explicit and in one place. You can read it, draw it, and test it. ✅ State is tracked — you always know which step an order is on. ✅ Easier to debug, monitor, and modify. ✅ Complex conditional logic is manageable.

❌ More coupling — the orchestrator knows about every service. ❌ Another component to build and run (though workflow engines provide it). ❌ Risk of becoming a god object with all the business logic.

Use for: longer sagas, anything with conditional branching, anything where you need to answer “where is order 4821 right now?”

🎙️ The recommendation to state: “I’d use orchestration here. With five steps and conditional compensation, choreography means the flow exists only implicitly across five codebases — nobody can tell you what happens without reading all of them. An orchestrator makes the process explicit and gives us a place to track state.”

Choreography is often the default because it feels more “microservices,” and it’s frequently the wrong choice for anything non-trivial. Saying so is a good signal.

Designing compensations

🚨 This is where sagas actually get hard, and it’s what interviewers probe.

Not everything can be undone

Action	Compensation	Reality
Reserve inventory	Release it	✅ Clean
Charge a card	Refund	⚠️ Money moved twice; fees may not be refunded; it’s visible to the customer
Send an email	—	❌ Impossible. You cannot unsend it
Ship a package	Recall it	❌ Not really — it’s on a van
Publish to a partner API	Call their delete endpoint	⚠️ If they support it, and if it succeeds

The design response: order your steps so irreversible actions come last.

❌ send_email → charge_card → reserve_inventory
   (if inventory fails, you've emailed a confirmation for an order that won't happen)

✅ reserve_inventory → charge_card → send_email
   (the irreversible step only runs once everything reversible has succeeded)

🚨 “Put the irreversible steps last” is one of the most useful concrete rules in this chapter.

Pivot transactions are the formal version: the step after which the saga can no longer be compensated and must run to completion. Everything before it is compensatable; everything after must be retried until it succeeds. Identify yours explicitly.

Compensations can fail

The refund API is down. Now what?

🚨 Compensations must be retried until they succeed, which means they must be:

Idempotent — retrying a refund must not refund twice → Idempotency
Durable — the intent to compensate must survive a crash
Monitored — a compensation stuck after N retries needs a human. There must be an alert and a manual resolution path.

There is no “give up” option — a failed compensation means a customer has been charged for nothing. It escalates to a person.

Semantic locks

Since there’s no isolation, mark records to signal an in-flight saga:

UPDATE orders SET status = 'payment_pending' WHERE id = 42;

Other processes see the state and behave accordingly — don’t ship it, don’t let it be edited, show “processing” in the UI. It’s a convention, not a lock, and every reader must honour it.

The isolation problem

🚨 Sagas provide A, C, and D — but not I. Intermediate states are visible to everything else, and that causes real anomalies:

Dirty reads. Another transaction sees the inventory as reserved during a saga that later compensates. It made a decision on state that was rolled back.

Lost updates. Someone modifies the order while the saga is mid-flight; the compensation overwrites their change.

The classic: inventory is reserved, another customer is told “out of stock,” then the saga compensates and the item is available again — but that customer has gone.

Countermeasures (from Garcia-Molina’s original saga paper, and still the standard vocabulary):

Technique	How
Semantic lock	A status flag marking the record as in-flight
Commutative updates	Design operations so order doesn’t matter (increment, not set)
Pessimistic view	Reorder steps so the risky ones happen when least data is exposed
Re-read value	Verify data hasn’t changed before acting on it
By-value	Route high-value transactions through a real distributed transaction, low-value ones through a saga

⚖️ That last one is a genuinely practical idea: use a saga for a $20 order and something stricter for a $2 million one, because the cost of an anomaly scales with value.

Implementation

State must be durable. The saga’s current step must survive a crash — persist it in a database or a workflow engine, never in memory.

CREATE TABLE saga_instances (
    id            UUID PRIMARY KEY,
    saga_type     TEXT NOT NULL,
    current_step  INT NOT NULL,
    state         JSONB NOT NULL,
    status        TEXT NOT NULL,     -- running | compensating | completed | failed
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

Every step must be idempotent. Steps will be retried — timeouts are ambiguous (the fallacies) and the orchestrator may crash after a step succeeded but before recording it.

Use the transactional outbox to publish saga events atomically with the local state change. → CDC

Timeouts on every step. A saga waiting forever on an unresponsive service is stuck. Define what happens on timeout — usually compensate, but sometimes retry.

Workflow engines do all of this for you. Temporal, AWS Step Functions, Camunda, and Cadence provide durable execution, retries, compensation, timeouts, and visibility.

🚨 Temporal’s model deserves a mention because it’s the modern answer: you write ordinary sequential code, and the engine records every step’s result. If the process dies, execution resumes by replaying the recorded history. Waiting three days for a webhook is one line.

@workflow.defn
class OrderSaga:
    @workflow.run
    async def run(self, order):
        try:
            await workflow.execute_activity(reserve_inventory, order)
            await workflow.execute_activity(charge_payment, order)
            await workflow.execute_activity(schedule_shipping, order)
        except ActivityError:
            await workflow.execute_activity(compensate, order)
            raise

🎙️ “I’d use a workflow engine rather than hand-rolling the orchestrator. Durable state, retries, timeouts, and visibility are all things we’d otherwise build badly.”

When not to use a saga

⚖️ Sagas are not free. Before reaching for one:

Can the data live in one database? Then use a real transaction. This is the best answer. → Distributed Transactions
Do the steps genuinely need to be atomic? Often “create the order, then asynchronously do the rest with retries” is sufficient — no compensation needed, because the steps aren’t conditional on each other.
Is the operation short and low-value? Idempotent retries plus a reconciliation job may be simpler than a saga.
Are the service boundaries wrong? 🚨 If most business operations require a saga, your services are split incorrectly. → Service Decomposition

⚖️ Trade-offs

	Gain	Cost
Saga vs 2PC	No locks, no blocking, scales, high availability	No isolation; compensation logic to write and test
Orchestration	Explicit flow, visible state, easy debugging	Coupling; another component
Choreography	Loose coupling, no coordinator	The flow exists nowhere; hard to debug
Workflow engine	Durability, retries, visibility for free	A platform to learn and operate
Semantic locks	Reduces anomalies from missing isolation	Every reader must honour the convention
Sagas at all	Cross-service business transactions	Intermediate states are user-visible

In the real world

Temporal (from Uber’s Cadence) exists precisely because hand-rolled sagas over message queues produce systems where nobody can answer “what state is order 4821 in?” Uber had hundreds of multi-step workflows — trip lifecycle, driver onboarding, payment reconciliation — and needed durable, observable execution.
Booking and travel systems are the canonical saga domain: reserve a flight, then a hotel, then a car, with compensations at every step and genuinely irreversible actions (a confirmation email, a non-refundable fare) that force careful step ordering.
The visible-compensation problem is real: customers who see a charge and a refund on their statement contact support, even when nothing went wrong from the business’s perspective. Some systems therefore authorize rather than capture payment until the saga completes — turning a visible compensation into an invisible one. That’s a nice example of choosing operations that compensate cleanly.

🚨 Interview traps

Not mentioning the lack of isolation. It’s the defining cost. Say it and name a countermeasure.
Assuming compensation is rollback. It’s a new business fact, visible to users.
Not ordering irreversible steps last. An emailed confirmation for a cancelled order.
Compensations that aren’t idempotent. A retried refund that refunds twice.
No plan for a failed compensation. It must escalate to a human.
Choosing choreography for a complex flow without acknowledging that the process becomes invisible.
Using a saga when one database would do.

🎙️ Soundbites

“A saga is local transactions plus compensating actions. We give up isolation — intermediate states are visible — in exchange for not holding locks across services.”
“Compensation isn’t rollback. The charge and the refund both happened and both appear on the customer’s statement. So I’d authorize rather than capture, and only capture once the saga completes — that way the compensation is invisible.”
“I’d order the steps so irreversible actions come last. Sending the confirmation email before reserving inventory means we’ve confirmed an order we can’t fulfil.”
“Orchestration rather than choreography. With five steps and conditional compensation, choreography means the flow only exists implicitly across five codebases and nobody can tell you what happens.”
“Every step and every compensation needs to be idempotent, because the orchestrator can crash after a step succeeded but before recording it — and a compensation that fails has to retry until it succeeds, then page someone.”

🛠️ Try it

1. Build a saga by hand. Three services (order, payment, inventory) with their own databases and an orchestrator. Make step 3 fail deliberately. Watch the compensations run. Then:

Kill the orchestrator between step 2 succeeding and it being recorded. Restart it. Watch it charge the card twice — that’s why steps must be idempotent.
Make a compensation fail. Notice you have no good options. That’s the part nobody plans for.

2. Feel the isolation problem. While a saga has inventory reserved mid-flight, have another process read the inventory count and make a decision. Then let the saga compensate. The second process acted on state that was rolled back — and there was no way for it to know.

3. Try Temporal. Implement the same order saga as a Temporal workflow. Kill the worker process mid-execution and watch it resume exactly where it left off. The durability is genuinely impressive, and it makes clear how much you’d otherwise be building.

4. Compare coordination styles. Implement the same flow as choreography (services reacting to events) and orchestration. Then ask a colleague to explain the flow from the code. The difference in how long that takes is the argument.

Check yourself

1. How is a compensating transaction different from a rollback?

A rollback erases a transaction as though it never happened — no other transaction ever saw it (given sufficient isolation), and no trace remains. A compensation is a **new transaction that semantically undoes** a previous one, and both are permanent, visible facts: the card was charged and then refunded, and both entries appear on the statement and in your ledger. The intermediate state was real and was observable by other processes during the window. This is why compensations are business logic rather than a database feature, why they can fail (and must therefore be retried and monitored), and why some actions have no compensation at all.

2. Why should irreversible steps come last in a saga?

Because once an irreversible step has executed, the saga cannot be fully compensated — you can't unsend an email, un-ship a package, or fully undo a call to a third party that doesn't support deletion. If such a step runs early and a later step fails, you're left with a partially-completed operation you cannot clean up: a customer holding a confirmation email for an order that was never fulfilled. Ordering reversible steps first means that by the time you reach the irreversible one, everything that could still fail has already succeeded. The formal term is the **pivot transaction** — the point of no return — and identifying yours explicitly is part of the design.

3. What's the isolation problem with sagas, and how do you mitigate it?

Sagas provide atomicity, consistency, and durability, but **not isolation**: each local transaction commits immediately, so intermediate states are visible to every other process. This causes dirty reads (another transaction acts on inventory that's later released), lost updates (someone edits a record mid-saga and the compensation overwrites them), and user-visible oddities (a customer is told "out of stock" for an item that becomes available seconds later). Mitigations: **semantic locks** (status flags marking a record as in-flight, which all readers must honour), **commutative operations** (increment rather than set, so order doesn't matter), **re-reading and verifying** before acting, and routing high-value operations through a stricter mechanism.

4. When should you choose orchestration over choreography?

When the flow has more than a few steps, involves conditional logic or branching, or when you need to answer "what state is this order in right now?" With choreography, the business process exists only implicitly in the union of several services' event subscriptions — no single artifact describes it, so understanding or changing it requires reading every service, and debugging "why didn't step 4 run?" means tracing across all of them. Orchestration puts the flow in one readable, testable place with explicit state tracking. Choreography's advantage — loose coupling — is real and worth it for short, simple sagas, but it's often chosen by default because it feels more microservice-y, which is the wrong reason.

5. What happens if a compensating transaction fails, and how do you design for it?

You're in the worst state the saga can produce: a customer charged with nothing delivered, or inventory reserved indefinitely. There's no "give up" option, because giving up means leaving inconsistent state permanently. Design for it by making compensations **idempotent** (so unlimited retries are safe), **durable** (the intent to compensate is persisted, so a crash doesn't lose it), and **retried with backoff indefinitely** rather than a fixed count. Then add an **escalation path**: after N failures, move the saga to a dead-letter state and alert a human, with tooling to inspect and manually resolve it. Also design compensations to depend on as few external systems as possible — marking a record for later refund is more reliable than calling a payment API synchronously.