system-design

Webhooks and Callbacks

The server calling the client, over HTTP. It inverts the usual direction — and inherits every unreliable-network problem, now from the receiving end.

Prerequisites: Idempotency, Client–Server Model Time to read: ~18 minutes

The problem

A customer’s payment completes on Stripe. Your system needs to know so it can fulfil the order. How does Stripe tell you?

Polling: your server asks “any updates?” every few seconds. ❌ Wasteful (most responses are empty), high latency (up to the poll interval), and it doesn’t scale — a million integrations each polling is a lot of pointless traffic. → Realtime Communication

Webhooks: Stripe calls your URL when the event happens. ✅ Real-time, efficient — the server pushes only when there’s news.

🚨 A webhook is just an HTTP POST from a provider to a URL you registered. It’s the server-to-server version of the server-push problem: you register https://myapp.com/webhooks/stripe, and Stripe POSTs event data there when something happens.

The flow

sequenceDiagram
    participant U as User
    participant S as Provider (Stripe)
    participant M as Your server
    U->>S: pays
    Note over S: payment.succeeded event
    S->>M: POST /webhooks/stripe {event data}
    M->>M: verify signature, dedupe, enqueue
    M-->>S: 200 OK (fast!)
    Note over M: process async: fulfil order

🚨 The critical design rule: acknowledge fast, process async.

def handle_webhook(request):
    verify_signature(request)                    # reject forgeries
    event = parse(request.body)
    if already_processed(event.id):              # dedupe
        return 200
    queue.publish(event)                         # enqueue, don't process inline
    return 200                                   # ACK immediately
    # The actual work happens in a worker, off the request path.

Why acknowledge before processing: providers have short timeouts (often 5–30 seconds) and retry on any non-2xx or timeout. If you fulfil the order inline — a slow, multi-step operation — and it takes 40 seconds, the provider times out, marks it failed, and retries — even though you succeeded. Now you’ve processed it twice. Verify, enqueue, ACK in milliseconds; do the real work in a worker.

The three hard problems

Webhooks inherit every property of the unreliable network, now as the receiver. Three problems dominate, and interviewers probe all three.

1. 🚨 Duplicates are guaranteed

Providers use at-least-once delivery and retry aggressively — on timeouts, on non-2xx, and sometimes just because. You will receive the same event multiple times.

This is the exactly-once-is-impossible problem from the receiving side: the provider can’t tell “your server crashed before processing” from “your server processed it but the ACK was lost,” so it redelivers.

The fix: idempotent processing. Every event has a unique ID; record processed IDs and skip repeats:

with db.transaction():
    if db.exists("processed_webhooks", event.id):
        return                                   # already handled
    fulfil_order(event)
    db.insert("processed_webhooks", event.id)    # atomic with the side effect

🚨 A webhook handler without deduplication is a bug waiting to fire — double-fulfilled orders, duplicate emails, double-counted revenue. Every serious provider’s docs say explicitly “you will receive duplicates; make your handler idempotent.” → Idempotency ⭐

2. 🚨 Authenticity — anyone can POST to your URL

Your webhook endpoint is a public URL. Anyone can send a fake “payment succeeded” event and trick you into fulfilling an unpaid order. This is a genuine, exploited attack.

The fix: signature verification. The provider signs each payload with a shared secret; you verify the signature before trusting anything:

expected = hmac_sha256(webhook_secret, request.body)
if not constant_time_compare(expected, request.headers["Signature"]):
    return 401                                   # forged — reject

🚨 Two details that matter:

Sign the raw body, not the parsed JSON — re-serializing can change bytes and break the signature.
Constant-time comparison — a naive == leaks timing information an attacker can exploit.
Many providers also include a timestamp in the signature to prevent replay attacks (rejecting events older than a few minutes).

Never trust a webhook payload’s contents over the provider’s API. 🚨 A stronger pattern for high-value events: the webhook tells you that something happened; you then call the provider’s API to fetch the authoritative state, rather than acting on the webhook’s data directly. This defends against both forgery and stale/out-of-order events.

3. 🚨 Ordering is not guaranteed

Events can arrive out of order. A subscription.updated may arrive before the subscription.created it depends on, because they took different network paths or were retried at different times.

Fixes:

Design handlers to be order-independent where possible — act on absolute state, not deltas.
Fetch current state from the API when order matters (see above), rather than trusting the event sequence.
Use timestamps or version numbers to detect and ignore stale events: if event.version <= last_processed_version: skip.

Delivering webhooks (the provider side)

If you’re building the webhook system, you face the mirror-image problems, and they’re a real design question:

Retries with backoff. The receiver might be down. Retry with exponential backoff and jitter (e.g. after 1 min, 5 min, 30 min, 2 hr, up to 24–72 hr), then give up and mark the endpoint failed. → Retries

A durable queue. Never deliver inline from your main flow (that’s the dual-write problem). Emit events to a queue; a delivery worker POSTs them. This decouples your core system from slow or dead receivers.

Timeouts and circuit breakers. A receiver taking 30 seconds shouldn’t tie up delivery workers. Short timeouts, and stop trying a persistently-failing endpoint. → Resilience Patterns

A dead-letter queue and a dashboard. After max retries, park the event and let the customer see failed deliveries and replay them. 🚨 This is essential — receivers will have outages, and “your webhooks were silently lost during your downtime” destroys trust.

Delivery logs and manual replay. Let customers see every attempt and re-trigger delivery. Stripe’s webhook dashboard is the reference.

SSRF protection. 🚨 The receiver URL is customer-controlled, so validate it — block internal IPs (10.x, 169.254.169.254 the cloud metadata endpoint, localhost), or an attacker registers a webhook pointing at your own internal services and uses your delivery worker to reach them. This is a real, serious vulnerability. → OWASP

Practical concerns

The receiver-behind-a-firewall problem. Webhooks require the receiver to have a public HTTPS endpoint. For local development, tunnels (ngrok, Stripe CLI’s listen) expose your local server. Corporate receivers behind firewalls can’t receive webhooks at all — which is a real limitation and why some providers also offer polling.

Event filtering. Let receivers subscribe to specific event types rather than firing everything.

Verification handshakes. Some providers require you to prove you own the URL (echo back a challenge) before they’ll send events.

Versioning. Webhook payloads are an API and evolve like one. → Versioning

Webhooks vs the alternatives

Approach	Latency	Efficiency	Receiver needs	Reliability
Polling	Up to interval	❌ Wasteful	Nothing special	Simple, no lost events
Webhooks	Real-time	✅	Public HTTPS endpoint	Duplicates, ordering, delivery to handle
WebSockets/SSE	Real-time	✅	Persistent connection	Stateful connections → Realtime
Message queue	Real-time	✅	Queue access	Best for internal systems → Queues

🎙️ “Webhooks for notifying third parties of events over HTTP — real-time and efficient. I’d acknowledge fast and process async, verify signatures, dedupe on the event ID because delivery is at-least-once, and not assume ordering. If a partner can’t receive webhooks — behind a firewall — I’d offer polling as a fallback.”

For internal event delivery, prefer a message queue — you get durability, ordering, and replay without the HTTP-delivery complexity. Webhooks are specifically for reaching external parties you can’t put a queue between.

⚖️ Trade-offs

Choice	Gain	Cost
Webhooks over polling	Real-time, efficient	Duplicates, ordering, delivery reliability to handle
ACK fast, process async	Avoids timeout-induced duplicates	Need a queue and workers
Signature verification	Rejects forgeries	Secret management; raw-body signing
Fetch-state-from-API pattern	Defends against forgery and stale data	An extra API call per event
Provider-side DLQ + replay	Receivers survive their own outages	More to build and operate

In the real world

Stripe’s webhooks are the reference: signed payloads with timestamps (replay protection), a dashboard showing every delivery attempt with manual replay, explicit at-least-once semantics with “make your handler idempotent” front and center, and the recommended pattern of fetching authoritative state from the API for critical events. Their docs are worth reading as a spec.
GitHub webhooks power CI/CD and integrations, with HMAC signatures and delivery logs, and are a good second reference.
The forged-webhook attack is real and exploited: endpoints without signature verification have been tricked into granting access or fulfilling unpaid orders. It’s why “verify the signature” is non-negotiable.
SSRF via webhook URLs is a recognized vulnerability class — a customer registers a webhook pointing at 169.254.169.254 (cloud metadata) and the provider’s delivery worker dutifully fetches cloud credentials. Providers must validate destination URLs.

🚨 Interview traps

Processing inline instead of ACK-then-async. Causes timeout-induced duplicates.
No deduplication. Delivery is at-least-once; duplicates are guaranteed.
No signature verification. Anyone can POST a fake event.
Assuming ordered delivery. Events arrive out of order.
Trusting webhook payload data for high-value actions instead of fetching authoritative state.
No SSRF protection on customer-controlled receiver URLs (provider side).
No dead-letter queue / replay (provider side) — receiver outages silently lose events.
Using webhooks internally where a message queue is better.

🎙️ Soundbites

“Acknowledge in milliseconds and process async — verify, dedupe, enqueue, return 200. If we process inline and it’s slow, the provider times out and retries something we already did.”
“Delivery is at-least-once, so the handler must be idempotent — dedupe on the event ID in the same transaction as the side effect. A webhook handler without dedup double-fulfils orders.”
“The endpoint is public, so I’d verify the HMAC signature on the raw body with a constant-time compare, and reject old timestamps to prevent replay. For high-value events I’d fetch authoritative state from the provider’s API rather than trusting the payload.”
“Events arrive out of order, so I’d make handlers act on absolute state, not deltas, and use version numbers to skip stale events.”
“If I were building the delivery side: a durable queue, retries with backoff, a dead-letter queue with manual replay so receivers survive their own outages, and SSRF validation on the customer’s URL.”

🛠️ Try it

1. Build a webhook receiver, then break it. Accept a POST, and process it inline with a 40-second delay. Point Stripe’s test webhooks (or a script) at it with a 10-second timeout. Watch it get retried and processed twice. Then switch to ACK-fast-process-async and watch the duplicates stop.

2. Forge an event. Send a fake “payment succeeded” POST to your own endpoint with no signature check. Watch it get accepted. Then add HMAC verification and watch the forgery get rejected. This is the security lesson, in five minutes.

3. Dedupe. Send the same event ID three times. Without dedup, watch the order get fulfilled three times. Add an idempotency check keyed on the event ID and watch attempts 2 and 3 no-op.

4. Use a tunnel. Run your webhook receiver locally and expose it with ngrok or stripe listen. Trigger a real test event and watch it arrive. This is how you actually develop webhooks locally, and it makes the “receiver needs a public endpoint” constraint concrete.

Check yourself

1. Why must you acknowledge a webhook before processing it?

Because providers have short delivery timeouts (typically 5–30 seconds) and retry on any non-2xx response or timeout. If you do the actual work — fulfilling an order, a multi-step process — *inline* and it takes longer than the timeout, the provider concludes delivery failed and retries, even though your processing actually succeeded. Now you've done the work twice. The fix is to verify the signature, deduplicate, enqueue the event to a queue, and return 200 — all in milliseconds — then do the real work in a background worker off the request path. Fast acknowledgment decouples the provider's timeout from your processing time.

2. Why is deduplication mandatory in a webhook handler?

Because webhook delivery is at-least-once and providers retry aggressively — on timeouts, on non-2xx, and after network failures — so you will receive the same event multiple times. It's the receiving side of the exactly-once-is-impossible problem: the provider can't distinguish "your server crashed before processing" from "your server processed it but the acknowledgment was lost," so it must redeliver. Without deduplication, a retried `payment.succeeded` fulfils the order twice, sends the email twice, and counts the revenue twice. The fix is idempotent processing: every event has a unique ID, and you record processed IDs (in the same transaction as the side effect) and skip repeats.

3. How do you verify a webhook is authentic, and why does the raw body matter?

The provider signs each payload with a shared secret — typically an HMAC-SHA256 over the request body, sent in a header — and you recompute the signature with your copy of the secret and compare, using a constant-time comparison to avoid timing attacks. You must sign and verify the **raw body bytes**, not the parsed-and-re-serialized JSON, because parsing and re-serializing can reorder keys or change whitespace, producing different bytes and a different signature that no longer matches. Many providers also include a timestamp in the signed data so you can reject events older than a few minutes, preventing replay attacks. Verification is essential because the endpoint is a public URL — without it, anyone can POST a forged "payment succeeded" event and trick you into acting on it.

4. Why shouldn't you fully trust a webhook payload for high-value actions?

Because webhook payloads can be forged (if signature verification is weak or missing), can be stale or out of order (events don't arrive in guaranteed sequence, so the state described may already have changed), and can be replayed. For a high-value action like granting access or releasing goods, the stronger pattern is to treat the webhook as a *notification that something happened* and then call the provider's API to fetch the authoritative current state before acting. This defends against forgery (you're reading from the trusted API, not the untrusted POST), against stale data (you get the real current state, not what was true when the event fired), and against out-of-order delivery. The webhook tells you *when* to check; the API tells you *what's true*.

5. If you were building the webhook delivery system, what would you include and why?

A **durable queue** between your core system and delivery (never deliver inline — that's the dual-write problem, and a slow receiver would block your main flow). **Retries with exponential backoff and jitter** over a long window (hours to days), because receivers have outages. **Short timeouts and circuit breakers** so one slow receiver doesn't tie up delivery workers. A **dead-letter queue with a customer-facing dashboard and manual replay**, so events aren't silently lost during a receiver's downtime — this is essential for trust. **Delivery logs** showing every attempt. And critically, **SSRF protection** on the customer-controlled destination URL — validate it and block internal IPs and the cloud metadata endpoint, or an attacker registers a webhook pointing at your own internal infrastructure and uses your delivery worker to reach it.