system-design

Design a Notification System

Difficulty: Tier 2 Asked at: Careem, Talabat, Noon, Uber, Amazon Time budget: 45 min

Every product sends notifications — push, SMS, email, in-app. Behind that simple feature is a fan-out-heavy, third-party-dependent, must-not-lose-messages pipeline. This question rewards clean decoupling (queues everywhere), handling flaky external providers, and not spamming users. It’s the canonical “event → many channels → many users” problem.

Prerequisites: Message Queues, Idempotency, Background Jobs

1. Requirements

Functional:

Send notifications over multiple channels: push (iOS/Android), SMS, email, in-app.
Triggered by events (order shipped, driver arrived, promo).
User preferences — users opt in/out per channel and category; respect quiet hours.
Templates — reusable, localized message templates.
Support transactional (must-deliver: OTP, receipts) and promotional (best-effort, batchable).

Non-functional:

High throughput — millions of notifications (a promo blast to all users).
Reliable — transactional notifications must not be lost (at-least-once).
No duplicates / no spam — don’t send the same thing twice; respect rate limits per user.
Low latency for transactional (an OTP must arrive in seconds); promotional can lag.
Resilient to third-party failures — providers (APNs, Twilio, SES) go down.

Out of scope: the events themselves; analytics dashboards.

2. Estimation

10M notifications/day average ≈ ~115/sec, but a promo blast is 10M in a few minutes → ~50K/sec burst. 🚨 The system must absorb bursts → queues.
Fan-out: one “flash sale” event → millions of individual sends. The event count is tiny; the send count is huge.
Storage: notification history/status for auditing → billions of rows over time → a wide-column store.

3. API

POST /notifications
  { userId | segment, templateId, channel?, data: {...}, priority: "transactional"|"promotional" }
  → 202 Accepted { notificationId }

GET /notifications/{id}/status   → { delivered, failed, channelStatuses }
PUT /users/{id}/preferences      → opt-in/out per channel/category

Return 202 Accepted — sending is asynchronous. The caller isn’t blocked waiting for APNs/Twilio.

4. High-level design

flowchart LR
    Event[Event / API] --> Ingest[Notification Service]
    Ingest --> Pref{Preference &<br/>rate-limit check}
    Pref -->|allowed| Q[[Message Queue<br/>per channel/priority]]
    Pref -->|opted out| Drop[Drop]
    Q --> WPush[Push Workers] --> APNs[APNs/FCM]
    Q --> WSMS[SMS Workers] --> Twilio[SMS Provider]
    Q --> WEmail[Email Workers] --> SES[Email Provider]
    WPush --> Status[(Status/History store)]
    Templates[(Template store)] --> Ingest

Flow: event arrives → apply user preferences and rate limits → render the template → enqueue per channel → channel workers pull and call the external provider → record delivery status. 🚨 Queues decouple ingestion from delivery, absorbing bursts and isolating slow/flaky providers.

5. Deep dives

5a. Fan-out for a broadcast

A “flash sale to all users” is one request that becomes millions of sends. Don’t do it inline. The notification service expands the segment into individual user notifications and enqueues them (in batches), letting workers drain the queue at a sustainable rate. This is fan-out on write applied to notifications — the queue is the buffer between “1 event” and “10M sends.”

5b. Reliability & idempotency

At-least-once delivery: workers retry failed sends (provider timeout, 5xx) with exponential backoff + jitter. (Retries)
At-least-once means possible duplicates → make sends idempotent: a dedup key per (user, event, channel) so a retried or double-processed message sends only once. 🚨 Nobody wants two identical OTP texts.
Dead-letter queue for messages that fail repeatedly → alert/inspect rather than lose or loop forever.

5c. Handling third-party provider failures

Providers fail or throttle. Defend with:

Circuit breakers — stop hammering a down provider; fail fast, retry later. (Resilience Patterns)
Fallback providers — a second SMS vendor if the primary is down.
Rate-limit to the provider — respect the provider’s own limits (Twilio caps you) so you don’t get throttled/banned.
Per-provider queues so a slow provider doesn’t block others.

5d. Preferences, quiet hours, and anti-spam

Check before enqueuing: is the user opted in to this (channel × category)? Is it their quiet hours? Have they already received too many notifications today (per-user rate limit)? 🚨 The best notification system sends fewer notifications — dedup similar ones, batch digests, respect preferences. Getting this wrong means uninstalls.

5e. Prioritization

Separate transactional (OTP, receipts — low-latency, must-deliver) from promotional (batchable, best-effort) using separate queues / priority lanes. A promo blast must never delay an OTP. Transactional workers get dedicated capacity.

5f. Delivery tracking

Record status per notification per channel (queued → sent → delivered → failed/read), updated via provider webhooks/receipts. Store in a wide-column store (Cassandra) keyed by user or notification for history and auditing.

6. Bottlenecks & scaling further

Burst load → queues absorb; workers autoscale on queue depth.
Slow provider → per-provider queues + circuit breaker isolate it.
Fan-out amplification → batch the expansion; rate-limit enqueue.
Status write volume → wide-column store, async status updates.
Duplicate suppression at scale → a dedup store (Redis with TTL keyed on the idempotency key).

7. Trade-off summary

Decision	Chosen	Alternative	Why
Sync vs async	Async (202 + queues)	Synchronous send	Absorb bursts; isolate flaky providers
Delivery guarantee	At-least-once + idempotency	Exactly-once (hard)	Simpler; dedup key prevents duplicates
Provider failure	Circuit breaker + fallback	Retry blindly	Don’t hammer a down provider; stay available
Queues	Per channel + priority	One shared queue	OTP never waits behind a promo blast
Preferences	Check before enqueue	Send then filter	Cheaper; respects users; less spam

8. Follow-up questions

How do you make sure an OTP is never sent twice?

Attach an idempotency key derived from the meaningful identity of the send — e.g. (userId, eventId, channel) or a client-supplied request ID — and record it in a fast dedup store (Redis with a TTL) the first time you process it. Before sending, check-and-set the key atomically; if it already exists, skip the send. Because delivery is at-least-once, the same message may be processed more than once (a retry, a redelivered queue message), and the dedup key collapses those to a single actual send. This is cheaper and more robust than trying to achieve exactly-once delivery end-to-end, which is effectively impossible across external providers.

A promo blast to 10M users arrives. How do you not fall over — or delay OTPs?

Two mechanisms. First, absorb the burst with queues: the blast is expanded into per-user messages and enqueued, and workers drain at a sustainable rate rather than trying to send 10M at once — the queue is the shock absorber. Second, isolate priorities: transactional messages (OTPs) use separate queues and dedicated worker capacity, so a flood of promotional messages sits in the promotional lane and can't delay an OTP in the transactional lane. You also rate-limit sends to each provider so you don't get throttled. The blast takes minutes to drain, which is fine because promotional is best-effort, while OTPs keep flowing in seconds.

A provider (Twilio) goes down. What happens?

A circuit breaker around that provider trips after repeated failures, so workers stop hammering it and fail fast instead of piling up timeouts. Messages for that channel either wait in the queue for the provider to recover or fail over to a secondary provider if one is configured. Because each provider has its own queue, the outage is contained to SMS and doesn't block push or email. Messages that ultimately can't be delivered after retries go to a dead-letter queue for inspection rather than being lost or retried forever.

How do you avoid annoying users into uninstalling?

Treat *not sending* as a feature. Enforce per-user preferences (opt-in/out per channel and category) and quiet hours before enqueuing; apply a per-user frequency cap so nobody gets bombarded; deduplicate and batch similar notifications into digests rather than sending each separately; and prioritize relevance. The system should send the fewest notifications that achieve the goal — over-notification is the top cause of push opt-outs and uninstalls, so restraint is a design requirement, not an afterthought.

9. What junior / mid / senior answers look like

Junior: service calls APNs/Twilio directly and returns. Works for low volume; falls over on bursts and provider outages, and may double-send.
Mid: async with queues, per-channel workers, retries, at-least-once with idempotency, preference checks, delivery status.
Senior: all that plus fan-out handling for broadcasts, circuit breakers + fallback providers, transactional-vs-promotional priority lanes, anti-spam/frequency capping as a first-class requirement, and dead-letter handling — framing the whole thing as decoupling ingestion from unreliable external delivery.