Networks force retries. Retries cause duplicates. Idempotency is what makes duplicates harmless — and it’s the single most valuable thing you can bring up unprompted in a design interview.
Prerequisites: The 8 Fallacies, Message Queues Time to read: ~26 minutes
A customer taps “Pay $500.” The request times out.
Client → POST /payments {amount: 500}
⏱️ 30 seconds pass
❌ timeout
Did the payment succeed?
The client cannot tell. Three completely different situations produce an identical observation:
🚨 This ambiguity is the fundamental problem of distributed systems, and there is no protocol that removes it. The client must decide: retry (and risk charging twice) or don’t (and risk not charging at all).
Idempotency makes the choice easy: retry, and design the operation so the second attempt is a no-op.
An operation is idempotent if performing it multiple times has the same effect as performing it once.
# ✅ Idempotent — the end state is the same however many times you run it
set_status(order, "shipped")
delete_user(42)
UPDATE accounts SET balance = 500 WHERE id = 1
# ❌ Not idempotent — each execution changes the result
increment_counter()
UPDATE accounts SET balance = balance - 100 WHERE id = 1
append_to_list(item)
send_email()
🚨 Note the second one carefully. balance = balance - 100 is not idempotent, even though it’s
a single atomic SQL statement. Atomicity and idempotency are different properties, and conflating
them is a common error.
HTTP methods and their guarantees (from HTTP):
| Method | Idempotent? |
|---|---|
GET, HEAD, OPTIONS |
✅ Also safe — no side effects at all |
PUT |
✅ Setting a value repeatedly gives the same result |
DELETE |
✅ Deleting twice leaves it deleted (return 204 or 404, not an error) |
POST |
❌ By default. This is where the work is. |
PATCH |
❌ Usually not ({"increment": 5} is not idempotent) |
The client generates a unique key and sends it with the request. The server records it.
POST /payments HTTP/1.1
Idempotency-Key: 7f3a9c2e-4b1d-4e8a-9f2c-1a5b7d3e9f01
Content-Type: application/json
{"amount": 500, "currency": "PKR", "card": "..."}
def create_payment(idempotency_key, request):
with db.transaction():
existing = db.query(
"SELECT * FROM idempotency_keys WHERE key = %s FOR UPDATE",
idempotency_key
)
if existing:
if existing.status == "completed":
return existing.response # ← return the ORIGINAL response
if existing.status == "in_progress":
raise Conflict409("Request already in progress")
db.insert("idempotency_keys",
key=idempotency_key, status="in_progress",
request_hash=hash(request))
result = charge_card(request) # the actual work
with db.transaction():
db.update("idempotency_keys", key=idempotency_key,
status="completed", response=result)
return result
🚨 Four details that matter, and each is a thing interviewers probe:
1. Return the original response, not just “already done.” The client needs the payment ID. A retry should be indistinguishable from the original success.
2. Handle the in-progress case. Two concurrent requests with the same key: one proceeds, the
other gets a 409 Conflict telling it to retry shortly. Without the FOR UPDATE lock, both proceed
and you’ve charged twice.
3. Store a hash of the request body. If the same key arrives with different parameters, that’s
a client bug — return 422, don’t silently return the old response for a different request.
4. Expire keys. Stripe retains them for 24 hours. Keeping them forever means unbounded storage. Choose a window longer than any realistic retry sequence.
Where the key comes from: the client generates it (a UUID), so the same logical operation keeps the same key across every retry. 🚨 If the server generated it, retries would get different keys and the whole mechanism would fail — this is a surprisingly common misunderstanding.
Often you don’t need a separate key table — the data already has a unique business identifier.
-- Idempotent by construction
INSERT INTO invoices (customer_id, period, amount_cents)
VALUES (42, '2026-07', 150000)
ON CONFLICT (customer_id, period) DO NOTHING;
-- Or with an explicit event ID
INSERT INTO processed_events (event_id) VALUES (%s)
ON CONFLICT (event_id) DO NOTHING;
-- 0 rows affected → we've already handled this event → return
⚖️ Prefer this when a natural key exists. It’s simpler, has no extra table to maintain, no expiry policy, and the uniqueness constraint is enforced by the database for every writer — including a migration script or someone running SQL manually.
🎙️ “I’d make it idempotent on (customer_id, billing_period) with an upsert rather than adding a
separate idempotency key table. The constraint does the work, and it protects us regardless of which
code path writes.”
🚨 Claiming otherwise is a red flag in interviews. Here’s the proof, and it’s short:
Broker sends message → Consumer processes it → Consumer sends ACK ──✗ ACK is lost
Broker never received the ACK. It must assume failure. It redelivers.
→ duplicate
The broker cannot distinguish “the consumer died before processing” from “the consumer processed it and the ACK was lost.” Both look identical. Closing the gap would require an atomic commit spanning the broker and the consumer’s side effects — which is a distributed transaction, with all its problems.
What you actually build:
at-least-once delivery + idempotent processing = effectively-once
🚨 “Effectively-once” is the correct term, and using it precisely is a strong signal.
What about Kafka’s “exactly-once semantics”? It’s real, and it’s narrower than the name suggests: Kafka’s transactions make a read-process-write cycle within Kafka atomic — consume from a topic, produce to another topic, and commit the consumer offset, all or nothing. The moment your consumer charges a card or writes to Postgres, you’re outside that boundary and you need idempotency again.
🎙️ “Kafka’s exactly-once is exactly-once within Kafka’s transaction boundary. Since our consumer calls a payment API, that’s an external side effect outside the transaction — so we still need an idempotency key.”
| Strategy | How | Best for |
|---|---|---|
| Idempotency key table | Store keys, check before processing | API endpoints with side effects |
| Natural unique constraint | ON CONFLICT DO NOTHING on a business key |
When one exists — prefer this |
| Conditional update | UPDATE ... WHERE status = 'pending' |
State machines |
| Optimistic concurrency | WHERE version = 7 |
Concurrent edits |
| Bloom filter + storage | Fast negative check before a lookup | Very high volume, some false positives OK |
| Sequence numbers | Track the highest processed per producer | Ordered streams |
🚨 The transactional guarantee is what makes it work. Recording that you processed the event must be atomic with the side effect:
# ✅ Atomic — either both happen or neither
with db.transaction():
if db.exists("processed_events", event_id):
return
apply_business_change(event)
db.insert("processed_events", event_id)
# ❌ Broken — crash between them and you reprocess
apply_business_change(event)
db.insert("processed_events", event_id) # crash here → duplicate on retry
⚖️ And when the side effect is external (a payment API, an email), you can’t include it in your
transaction. Then you need the external system to be idempotent too — which is exactly why Stripe
accepts an Idempotency-Key header. Idempotency has to compose across service boundaries.
Counters. balance = balance - 100 isn’t idempotent. Options:
INSERT INTO ledger (id, account, amount) VALUES (%s, 42, -100)
ON CONFLICT (id) DO NOTHING;
-- balance = SELECT sum(amount) FROM ledger WHERE account = 42
🚨 This is how real financial systems work — an immutable append-only ledger with derived balances, never a mutable balance field. It’s idempotent, auditable, and reconstructible.
UPDATE accounts SET balance = 400, version = 8 WHERE id = 1 AND version = 7Emails and notifications. You cannot unsend. So deduplicate before sending:
if db.insert_if_not_exists("sent_notifications", (user_id, event_id)):
send_email(...) # only runs the first time
Note the ordering: record first, then send. 🚨 The reverse means a crash between them sends twice. This deliberately biases toward possibly not sending over definitely sending twice — which is usually the right trade for a notification, and the wrong one for a critical alert. Decide consciously.
Appending to a list. Use a set keyed by item ID, or an upsert, rather than a blind append.
Calling a third-party API. Use their idempotency mechanism if they have one. If not, record the attempt with its result before and after, and reconcile.
Almost everywhere, once you look:
| Context | Why |
|---|---|
| Any API with side effects | Clients retry on timeout |
| Message consumers | At-least-once delivery guarantees duplicates |
| Webhook receivers | Providers retry aggressively; you’ll get duplicates |
| Background jobs | Locks expire, workers crash, humans re-run them |
| Saga steps and compensations | Orchestrators crash between “did it” and “recorded it” |
| Payment flows | The stakes are money |
| Data pipelines | Reprocessing after a bug fix must not double-count |
| CDC consumers | Events replay on restart |
🚨 Webhooks deserve special mention. Every serious provider (Stripe, GitHub, Twilio) retries webhooks on non-2xx responses and on timeouts, and they all document that you will receive duplicates. A webhook handler without deduplication is a bug waiting to be triggered.
| Approach | Gain | Cost |
|---|---|---|
| Idempotency keys | Safe retries for any operation | A key store to maintain and expire; an extra lookup per request |
| Natural unique constraints | Simple, enforced for all writers | Only when a business key exists |
| Append-only ledger | Idempotent, auditable, reconstructible | More storage; balance requires aggregation (or a materialized view) |
| Dedupe before an external call | Never double-send | May not send if you crash after recording |
| Dedupe after | Never miss | May double-send |
| Longer key retention | Safe for slow retries | More storage |
Idempotency-Key header is the reference implementation: client-generated, results
cached for 24 hours, request-body hashing to detect key reuse with different parameters, and a
documented 409 for concurrent requests with the same key. Their public documentation on it is
worth reading as a spec.409.1. Cause a double charge. An endpoint that inserts a payment row. A client that retries on timeout. Make the server sleep past the client’s timeout, then complete successfully. Watch two payments appear for one user action. Then add an idempotency key and watch the second attempt return the first result.
This is the single most valuable exercise in Part 4 — it takes fifteen minutes and it makes the concept permanent.
2. Break it with concurrency. Send two requests with the same idempotency key simultaneously.
Without SELECT ... FOR UPDATE, both pass the existence check and both proceed. Add the lock and
watch one get a 409.
3. Break the atomicity. Split the side effect and the dedup record into two transactions. Kill the process between them. Retry. Observe the duplicate. Then combine them into one transaction.
4. Test a real webhook. Point a Stripe (test mode) or GitHub webhook at your endpoint and return a 500. Watch the retries arrive. Count how many duplicates you receive — then implement deduplication on the provider’s event ID.