system-design

Design a Payment System (Stripe / Tabby / a Wallet)

Difficulty: Tier 3 Asked at: Stripe, Tabby, Careem Pay, PayPal, Amazon, any fintech Time budget: 45–60 min

Payments is where “eventual consistency is fine” goes to die. Money must be exactly right: never double-charged, never lost, always auditable, reconcilable to the cent. This question is about strong consistency, idempotency, the double-entry ledger, and integrating unreliable external processors — a correctness-first design in a repo mostly about scale-first designs. It’s a MENA fintech favourite (Tabby, Careem Pay).

Prerequisites: Transactions / ACID, Idempotency, Saga Pattern, Two-Phase Commit / Distributed Transactions

1. Requirements

Functional:

Process a payment: charge a customer, credit a merchant (or move money between wallets).
Support external processors (card networks, banks) that are slow and can fail.
Refunds, reversals; transaction history/statements.
Idempotent payment requests (retries must not double-charge).

Non-functional:

Correctness above all — no money created or destroyed; every cent accounted for.
Strong consistency & durability — a committed payment is never lost.
Auditability — a complete, immutable record of every money movement.
Reconcilable — internal records must match the external processors’ records.
Reasonable availability, but 🚨 correctness beats availability here (unlike feeds).

Out of scope: fraud detection (separate), PCI/card-vault details (mention), the card-network internals.

2. The foundation: the double-entry ledger

🚨 The single most important concept. Money movements are recorded as double-entry bookkeeping: every transaction is two entries — a debit from one account and a credit to another — that must sum to zero. Money is never created or destroyed, only moved.

Payment of $100 from Customer to Merchant:
  Customer account:  -100   (debit)
  Merchant account:  +100   (credit)
  Sum: 0  ✅

The ledger is append-only / immutable — you never edit an entry; a correction is a new compensating entry. This gives a complete audit trail.
Account balances are derived from the sum of entries (or maintained as a running balance updated atomically with each entry). 🚨 This structure makes correctness checkable: the books must always balance.

3. Idempotency: never double-charge

🚨 Every payment request carries a client-generated idempotency key. The payment service records it; if the same key arrives again (a retry after a timeout, a double-click), it returns the original result instead of charging again.

Store (idempotency_key → result) durably, checked-and-set atomically at the start of processing.
This is essential because networks are unreliable: a client that doesn’t get a response must retry, and the idempotency key is what makes that retry safe. (Idempotency)

4. High-level design

flowchart TB
    Client -->|charge + idempotency key| PaySvc[Payment Service]
    PaySvc --> Idem[(Idempotency store)]
    PaySvc --> Ledger[(Ledger DB<br/>ACID, double-entry)]
    PaySvc --> PSP[Payment Service Provider<br/>card network / bank]
    PaySvc --> Recon[Reconciliation Service]
    Recon --> Ledger
    Recon --> PSP
    PaySvc --> Events[[Payment events]]

Flow: request (with idempotency key) → check idempotency store → create a pending ledger entry → call the external processor → on success, commit the entry (mark completed, update balances); on failure, mark failed/reverse. All money-state changes are ACID transactions in the ledger.

5. Deep dives

5a. State machine of a payment

A payment isn’t instant — the external processor is slow/async. Model it as a state machine:

INITIATED → PENDING (awaiting processor) → SUCCEEDED
                                         → FAILED
SUCCEEDED → REFUNDED (a new reverse entry)

Each transition is durable. 🚨 Never assume the processor call succeeded — you might not get a response; you must reconcile (below) to learn the true outcome. Handle the “we don’t know” state explicitly.

5b. Integrating unreliable external processors

Card networks/banks time out, return late, or send async webhooks. Defend:

Idempotency toward the processor too — send an idempotency key so a retried charge isn’t double-run externally.
Async confirmation — treat the payment as PENDING until the processor confirms (webhook/callback or polling), then finalize.
Timeouts + reconciliation — if you don’t get a response, don’t assume failure or success; reconcile.

5c. Reconciliation (the safety net)

🚨 Periodically compare your ledger against the processor’s records (settlement files) and fix discrepancies. This catches lost webhooks, ambiguous timeouts, and bugs. Reconciliation is what ultimately guarantees “every cent accounted for” — it’s non-negotiable in real payment systems, and a strong thing to raise unprompted.

5d. Consistency across services (transfers/wallets)

Moving money between two accounts/services that could be in different databases needs atomicity — you can’t debit one and fail to credit the other. Within one ledger DB, use an ACID transaction. Across services, use a saga with compensations, or a coordinator, keeping each step idempotent. Prefer keeping the ledger in one strongly-consistent store so the core money-move is a single transaction. (Saga, Distributed Transactions)

5e. Correctness over availability

Unlike a feed, if the payment system is unsure, it should refuse or hold rather than risk a wrong charge. A brief outage is better than a double-charge or lost money. State this CAP stance explicitly: CP, not AP. (CAP)

6. Bottlenecks & scaling further

Correctness under concurrency → ACID ledger, atomic balance updates, idempotency.
Slow external processors → async state machine, webhooks, timeouts.
Ambiguous outcomes → reconciliation against processor records.
Throughput → shard the ledger by account; most transactions touch few accounts; keep per-transaction atomicity within a shard where possible.
Audit/history → append-only ledger is naturally the audit log.

7. Trade-off summary

Decision	Chosen	Alternative	Why
Records	Double-entry, append-only ledger	Mutable balance field	Auditable, checkable, corrections are new entries
Retries	Idempotency keys (client + processor)	Best-effort	Networks force retries; must not double-charge
Outcome handling	Explicit PENDING + reconciliation	Assume success/fail	Processor responses are unreliable
CAP stance	CP (correctness)	AP (availability)	A wrong charge is worse than downtime
Cross-account move	ACID txn (same store) / saga	Uncoordinated writes	Money must move atomically

8. Follow-up questions

What is a double-entry ledger and why use it for money?

A double-entry ledger records every money movement as two matching entries — a debit from one account and an equal credit to another — so that every transaction sums to zero, encoding the invariant that money is never created or destroyed, only moved. It's used for money because it makes correctness structurally checkable and auditable: at all times the books must balance (all entries sum to zero, and each account's balance is the sum of its entries), so a bug or discrepancy is detectable rather than silent. The ledger is append-only and immutable — you never edit or delete an entry; a mistake is fixed by adding a compensating entry — which yields a complete, tamper-evident history of every cent, exactly what audits and reconciliation require. Contrast a naive design that just stores a mutable balance number: it has no history, no way to verify how a balance was reached, and a lost update silently corrupts money. Double-entry is the centuries-old accounting foundation precisely because it turns "did we handle the money correctly?" into a verifiable property.

How do you make sure a customer is never charged twice?

With idempotency keys, at both boundaries. The client generates a unique idempotency key for each distinct payment intent and sends it with the request; the payment service atomically checks a durable idempotency store before processing, and if that key has been seen, it returns the original result rather than charging again. This is essential because the network is unreliable — a client that times out waiting for a response cannot know whether the charge went through, so it must retry, and the idempotency key is what makes retrying safe. You also pass an idempotency key to the external processor so that if *your* call to the card network is retried, the network doesn't run the charge twice either. Together these ensure that no matter how many times a request is retried at any layer, the customer's card is charged exactly once for a given intent.

You call the card network and get no response. What do you do?

You treat the outcome as unknown and never guess — assuming success could credit a merchant for money you didn't collect, and assuming failure could tell the customer it failed when they were actually charged. So the payment stays in an explicit PENDING state, and you determine the true outcome out-of-band: wait for the processor's asynchronous confirmation (a webhook/callback), and/or actively query the processor for that transaction's status using your idempotency key, and back it all with reconciliation against the processor's settlement records. Because your call carried an idempotency key, safely retrying the query (or the charge) won't double-run it. Only once you've positively confirmed the real outcome do you transition PENDING to SUCCEEDED or FAILED and finalize the ledger entries. Explicitly modeling and resolving the "we don't know" state — rather than optimistically assuming — is what keeps the books correct across unreliable processors.

Why is reconciliation necessary if you already have idempotency and an ACID ledger?

Because idempotency and ACID transactions guarantee correctness *within your system*, but a payment system's truth is shared with external parties (card networks, banks) whose records you must match, and the boundary between systems is where things go wrong: webhooks get lost, calls time out ambiguously, the processor's view and yours can diverge due to bugs or partial failures. Reconciliation is the periodic process of comparing your ledger against the processor's authoritative settlement files and resolving any discrepancies — a charge they recorded that you marked pending, a refund that didn't propagate, a timing mismatch. It's the safety net that ultimately delivers the "every cent accounted for" guarantee, catching the cases that in-system mechanisms can't see because they involve the external world. Real payment systems treat reconciliation as non-negotiable precisely because no amount of in-process rigor can substitute for verifying against the counterparty's books.

Why choose consistency over availability here when the rest of the repo favors availability?

Because the cost of incorrectness is categorically different. In a feed or a cache, serving slightly stale data during a partition is harmless, so you favor availability (AP) and keep working. In payments, acting on uncertain or inconsistent state can create or destroy money — a double charge, a lost payment, an unbalanced ledger — which is unacceptable and often legally serious, while a brief period of unavailability is merely inconvenient. So the correct CAP stance is CP: when the system can't be sure it can act correctly (it can't reach the ledger, can't confirm a processor outcome), it refuses or holds rather than guessing. A payment that's delayed a minute is fine; a payment that's wrong is a disaster. This is the defining inversion of this problem versus most others in system design — correctness is the top requirement, and availability yields to it.

9. What junior / mid / senior answers look like

Junior: stores a balance field and updates it on payment; calls the processor and assumes success. Vulnerable to double-charges, lost updates, and no audit trail.
Mid: uses an ACID store, idempotency keys to prevent double-charges, a payment state machine, and understands the processor is async.
Senior: builds on a double-entry append-only ledger as the correctness foundation, idempotency at both boundaries, explicit PENDING/unknown handling with async confirmation, reconciliation against the processor as the ultimate safety net, and a clear CP-over-AP stance — treating “every cent accounted for and never double-charged” as the non-negotiable requirement.