system-design

Design a Feature Flag Service (LaunchDarkly)

Difficulty: Tier 2 Asked at: Amazon, LaunchDarkly, most product companies Time budget: 45 min

Feature flags let you turn features on/off and roll them out to a percentage of users without deploying — the backbone of safe releases, A/B tests, and kill switches. The system is deceptively simple but has a sharp constraint: flag evaluation happens on the hot path of every request, so it must add near-zero latency, which drives a local-evaluation + config-push design. It ties together config distribution, caching, and deployment safety.

Prerequisites: Caching, Deployment & Infra, Real-Time Communication

1. Requirements

Functional:

Define flags (on/off, or multivariate) and evaluate them per user/context.
Targeting rules — enable for specific users, segments, or a percentage rollout (e.g. 5% of users).
Change flags instantly without a deploy (including a kill switch).
A/B experiment support (consistent bucketing per user).
Audit changes; SDKs for many languages.

Non-functional:

Ultra-low latency evaluation — flags are checked on nearly every request; must add ~0 ms. 🚨 The defining constraint.
Fast propagation — a flag change reaches all app servers in seconds (especially kill switches).
Highly available — the flag service being down must not break the app (fail safe to defaults).
Scale — many flags × many services × huge request volume.

Out of scope: the experimentation stats engine, the dashboard UI internals.

2. Estimation

Evaluation happens inline with every request → effectively millions of evaluations/sec across a fleet. 🚨 You cannot make a network call per evaluation — that would add latency to every request.
Flag definitions are small (KBs) and change infrequently (human-driven) → the config is tiny; the challenge is distributing it fast and evaluating it locally at zero cost.

3. The core insight: evaluate locally, push config

🚨 Don’t call the flag service on each evaluation. Instead, push flag definitions to each app server (via an SDK) and evaluate locally in-process.

flowchart TB
    Dash[Flag config dashboard] --> FlagSvc[Flag Service]
    FlagSvc --> Store[(Flag config store)]
    FlagSvc -->|push updates<br/>stream / poll| SDK1[SDK in App Server 1<br/>local flag cache]
    FlagSvc --> SDK2[SDK in App Server 2<br/>local flag cache]
    App1[Request] -->|evaluate locally, ~0ms| SDK1

The SDK embedded in each app server holds the current flag rules in memory and evaluates them locally — no network call per request → ~0 latency.
Flag changes propagate from the service to all SDKs via a streaming push (SSE/WebSocket) or fast polling → updates land in seconds. 🚨 Local evaluation for latency + config push for freshness is the whole design.

4. Deep dives

4a. Percentage rollouts & consistent bucketing

“Enable for 5% of users” must be deterministic and sticky — the same user always gets the same result (no flickering), and 5% means a stable 5%. Achieve it by hashing (user_id + flag_key) → a number in [0,100) and comparing to the rollout percentage. 🚨 Same hash → same bucket every time, consistent across all servers (they share the rule), and increasing the percentage only adds users. This deterministic hashing is the key trick for rollouts and A/B bucketing.

4b. Fast propagation & kill switches

A kill switch (turn off a broken feature now) demands seconds-level propagation. Use a push channel (streaming) so SDKs get updates immediately rather than waiting for a poll interval. Trade-off: push is more complex but far faster than polling; many systems do streaming with polling as a fallback. (Real-Time Communication)

4c. Availability — fail safe

🚨 The flag service must never take down the app. If an SDK can’t reach the service:

It keeps serving the last-known flag config from local memory/cache (works fine offline).
On cold start with no config, evaluate to safe defaults (usually “feature off”).
Bootstrap from a cached/local file so a fresh instance isn’t flagless. So a flag-service outage degrades to “flags don’t change,” not “app breaks.” Design for the flag path to be non-critical.

4d. Targeting rules evaluation

Rules can target by user attributes (country, plan, beta group), segments, and percentages, combined with AND/OR. The SDK evaluates these locally against the request context. Keep rules simple enough to evaluate in microseconds. Order rules (specific overrides before percentage rollout).

4e. A/B experiments

An experiment = a multivariate flag with consistent per-user bucketing (the hashing trick), plus logging which variant each user saw (for the stats engine to correlate with outcomes). Ensure a user stays in the same variant for the experiment’s duration (sticky bucketing).

4f. Scale & consistency

Config is tiny and read-mostly; the service fans out updates to many SDKs. Eventual consistency of propagation is fine (a few seconds’ skew between servers during a change is acceptable — briefly some requests see old, some new). For changes that must be atomic across the fleet, that brief skew is the trade-off; usually acceptable.

5. Bottlenecks & scaling further

Evaluation latency → local in-SDK evaluation, zero network calls per request.
Propagation speed → streaming push to SDKs (+ polling fallback).
Availability → SDKs cache last-known config; safe defaults; never block the app.
Fan-out to many SDKs → the service streams small config diffs to all connected SDKs.
Consistent rollouts → deterministic hashing of (user, flag).

6. Trade-off summary

Decision	Chosen	Alternative	Why
Evaluation	Local in-process (SDK)	Remote call per check	~0 latency on the request hot path
Propagation	Streaming push	Polling only	Kill switches need seconds, not minutes
Rollout bucketing	Deterministic hash (user+flag)	Random per request	Sticky, consistent, monotonic rollout
Availability	Cache last-known + safe defaults	Depend on the service	Flag outage must not break the app
Consistency	Eventual (seconds) across fleet	Atomic fleet-wide	Brief skew acceptable; simpler, available

7. Follow-up questions

Why not just call the flag service to evaluate each flag?

Because flag checks happen on the hot path of nearly every request — often many per request — so a network call per evaluation would add latency and a dependency to essentially all of your traffic, which is unacceptable both for performance (milliseconds added everywhere, multiplied by many checks) and for reliability (the flag service becomes a hard dependency that can slow or break every request). Instead, the flag *definitions* — which are tiny and change infrequently — are pushed to an SDK embedded in each app server, which holds them in memory and evaluates flags locally in microseconds with no network call. Freshness comes from the service pushing config updates to the SDKs (via streaming, with polling as a fallback), so changes still propagate in seconds. This inverts the naive design: rather than sending each request's context to the flags, you send the (small, slow-changing) flags to where the requests are, getting near-zero evaluation latency and removing the flag service from the per-request critical path. Local evaluation plus config push is the core architectural decision.

How do you roll a feature out to exactly 5% of users, consistently?

By hashing a stable identifier for the user together with the flag key into a number in a fixed range (say 0–100) and enabling the feature when that number falls below the rollout percentage. Because the hash is deterministic, a given user always maps to the same bucket for that flag, so they consistently get the same result on every request and across every server (all of which share the same rule and hashing) — no flickering between on and off. Hashing the user *with the flag key* means a user's bucket differs per flag, so you're not always exposing the same 5% of people to every rollout. And because you compare against a threshold, raising the rollout from 5% to 10% only *adds* users (those whose hash is between 5 and 10) without churning the original 5% — a monotonic, sticky rollout. The same mechanism gives A/B experiments consistent per-user variant assignment. Deterministic hashing of (user, flag) is the trick that makes percentage rollouts stable, distributed-consistent, and monotonic.

The flag service goes down. What happens to the app?

Nothing bad, by design — a flag-service outage must degrade to "flags stop changing," never "the app breaks." Each SDK holds the last-known flag configuration in local memory, so if it can't reach the service it simply keeps evaluating flags against that cached config, and the application continues serving traffic normally with whatever flag states were last known. New instances starting up during an outage bootstrap from a cached or bundled local config file so they aren't flagless, and if truly no config is available, flags evaluate to safe defaults (typically "feature off"), which is the conservative choice. The whole flag path is engineered to be non-critical: it's a read-mostly, cached, locally-evaluated system precisely so that the availability of the central service never gates request handling. This fail-safe posture is essential because feature flags sit in the path of nearly every request, so they must be more available than the service that manages them.

How fast must a kill switch propagate, and how do you achieve it?

A kill switch needs to propagate in seconds, because its whole purpose is to instantly disable a feature that's causing problems (errors, an incident) without waiting for a deploy — minutes of delay could mean minutes of outage. You achieve near-instant propagation by pushing config changes to the SDKs over a streaming channel (server-sent events or a websocket) so that the moment an operator flips the flag, the update is pushed to all connected app servers and takes effect on the next request, rather than waiting for the SDKs to poll on some interval. Polling can serve as a fallback (and a safety net if a push is missed), but push is what delivers the seconds-level latency a kill switch requires. The trade-off is that maintaining streaming connections to a large fleet is more complex than periodic polling, but the fast propagation is worth it for the safety-critical case, and it's why production flag systems favor streaming for updates.

8. What junior / mid / senior answers look like

Junior: stores flags in a DB and queries them per request. Correct behavior but adds latency and a hard dependency to every request; slow to propagate.
Mid: SDK with local in-memory evaluation, config pushed/polled from a service, percentage rollouts via hashing, safe defaults on failure.
Senior: frames local-evaluation-plus-config-push as the latency solution, uses deterministic (user+flag) hashing for sticky consistent rollouts and A/B bucketing, streaming push for seconds-level kill switches with polling fallback, a rigorous fail-safe availability model (cache last-known + safe defaults so the app never breaks), and accepts brief eventual-consistency skew across the fleet.