system-design

Feature Flags and Progressive Delivery

A switch in your code that changes behaviour without a deploy. It decouples shipping code from releasing features — and that separation is quietly one of the most powerful ideas in modern delivery.

Prerequisites: Deployment Strategies, CI/CD Time to read: ~16 minutes

The core idea

🚨 Feature flags decouple deploy from release. These are usually the same event — you deploy code and the feature is live. Flags separate them:

if feature_flags.enabled("new-checkout", user):
    return new_checkout(user)      # the new code, deployed but dormant
else:
    return old_checkout(user)      # the current behaviour

You deploy the new code dark (flag off), then turn it on — for a percentage of users, specific users, or everyone — as a runtime configuration change, independent of deployment. And turn it off instantly if it breaks.

🚨 Why this is transformative: it solves the rollback problem better than any deployment strategy. Rolling back a bad feature is a flag flip in seconds — no rebuild, no redeploy, no touching infrastructure. And you can deploy incomplete work safely (hidden behind a flag), so a feature that takes weeks doesn’t block the main branch. → Trunk-based development

What flags let you do

Use case How
Progressive rollout Turn on for 1% → 5% → 100%, watching metrics (canary at the feature level)
Instant kill switch 🚨 A broken feature → flip off in seconds, no rollback deploy
Deploy incomplete work Merge unfinished features behind a flag; ship the branch, hide the feature
A/B testing Route users to variants, measure business outcomes
Targeted release Beta users, internal staff, a specific customer, one region
Operational toggles Disable an expensive feature under load (load shedding)
Circuit breaking Turn off a dependency-heavy feature when the dependency is down

🚨 The kill switch is the killer feature. When something breaks in production, “turn off the flag” is faster and simpler than any rollback — it’s the fastest possible mitigation.

Types of flags (they have different lifecycles)

🚨 A useful distinction — flags aren’t all the same, and treating them the same causes problems:

The lifecycle matters because 🚨 release and experiment flags must be removed after they’ve served their purpose — otherwise they become permanent technical debt.

The dark side: flag debt

⚖️ Feature flags are powerful and genuinely dangerous if unmanaged, and a mature answer says so:

🚨 Every flag is an if — and N flags create 2^N possible code paths, most of which are never tested together. Flags that should have been removed linger for years, and:

🚨 The famous Knight Capital disaster (2012) — $440 million lost in 45 minutes — was caused partly by repurposing an old flag whose stale code got activated on servers where it shouldn’t have been. It’s the canonical cautionary tale of flag debt.

Managing it:

🎙️ “Feature flags are powerful for decoupling deploy from release and giving us an instant kill switch, but they’re technical debt if not managed — every flag is untested code paths, and a stale flag caused the Knight Capital disaster. So temporary flags get expiry deadlines and get removed once rolled out, and I’d never repurpose an old flag.”

Progressive delivery

🚨 The broader practice that flags enable, and a good term to know: progressive delivery is gradually rolling out changes with control and observation at every step — combining canary deployments, feature flags, and automated analysis.

Deploy dark → enable for internal users → 1% of users → watch metrics →
5% → 25% → 100% → remove the flag

At each step, observability tells you if it’s safe to proceed, and the flag gives instant rollback. It’s the synthesis of everything in this part — deploy safely, release gradually, observe continuously, roll back instantly. The idea: no change is all-or-nothing; everything rolls out under control.

Implementation considerations

⚖️ Trade-offs

  Gain Cost
Feature flags Decouple deploy/release, instant kill switch, deploy incomplete work, gradual rollout Flag debt, untested combinations, complexity, a system to manage
Progressive delivery Nothing is all-or-nothing; controlled, observed rollout Slower; requires flags + canary + observability
Flag management system Visibility, audit, lifecycle Another dependency and cost
Long-lived ops flags Kill switches always available Kept forever (acceptable — they’re meant to be)

In the real world

🚨 Interview traps

🎙️ Soundbites

🛠️ Try it

1. Deploy dark, then release. Add a feature behind a flag (off). Deploy it — confirm the code is in production but the feature is invisible. Then flip the flag on and watch it appear with no deploy. That separation of deploy and release is the whole idea, felt directly.

2. Use the kill switch. With a feature behind a flag, “discover” it’s broken and flip it off. Time it — seconds — and compare to a rollback deploy. This is why the kill switch is the fastest mitigation.

3. Percentage rollout with consistent bucketing. Enable a flag for 10% of users by hashing user ID. Confirm a given user gets a stable answer across requests (no flickering). Then use a random check instead and watch users flicker — showing why consistent bucketing matters.

4. Simulate flag debt. Add five flags and count the code paths (2^5 = 32 combinations). Realize most are never tested together. Then set a stale flag to an unexpected state and find the resulting bug — a miniature Knight Capital, showing why flags must be cleaned up.

Check yourself

1. What does it mean that feature flags "decouple deploy from release," and why is that valuable? Normally, deploying code and releasing a feature are the same event — the moment your new code reaches production, its behaviour is live. Feature flags separate these: you deploy the code with the feature *off* (dark), so it's present in production but dormant, and then *release* it — turn it on — as a separate runtime configuration change, whenever you choose, for whatever subset of users you choose. This is valuable for several reasons. It gives you an **instant kill switch**: a broken feature is disabled by flipping the flag in seconds, no rebuild or redeploy — the fastest possible incident mitigation. It lets you **deploy incomplete work** safely behind a flag, so a feature that takes weeks doesn't block the main branch (enabling trunk-based development). It enables **gradual, controlled release** (1% → 100%) independent of the deployment. And it lets **release timing** be a product decision (release at a launch event) separate from when engineering deploys the code. Separating "the code is running" from "users see the feature" removes the coupling that makes deployment risky.
2. What is flag debt and why is it dangerous? Flag debt is the accumulation of feature flags that should have been removed but linger in the codebase. It's dangerous because every flag is an `if` statement creating two code paths, so N flags create up to 2^N possible combinations of code paths — the vast majority of which are never tested together, creating untested states where bizarre bugs hide. Temporary flags meant to live for days persist for years, filling the code with dead branches and stale conditionals that nobody remembers the purpose of or dares to remove. A forgotten flag left in an unexpected state can trigger a production incident — most infamously, the Knight Capital disaster, where repurposing an old flag activated stale code and lost $440 million in 45 minutes. Flags are powerful precisely because they change behaviour without a deploy, which is the same reason a mis-set or stale flag is dangerous. Managing the debt requires treating flag removal as part of the work: set expiry deadlines on temporary flags, remove them aggressively once a feature is fully rolled out, never repurpose an old flag, and use a management system for visibility into what flags exist and their age.
3. Why should you never repurpose an old feature flag? Because reusing an old flag's name for a new meaning is exactly the trap that caused the Knight Capital disaster. When you repurpose a flag, any old code still gated by that flag name — perhaps on servers that weren't fully cleaned up, or a code path you forgot existed — gets activated when you turn the "new" flag on, because the flag system doesn't know the difference between old and new usage; it just knows the flag is on. So enabling what you think is a new feature can silently reactivate dead or unintended code with the same flag. Knight Capital's incident involved a deployment that left old flag-gated code on some servers, and a repurposed flag triggered it, sending millions of erroneous stock orders in minutes. The safe practice is always to create a *new* flag with a new name for new behaviour, and to fully remove old flags and their code, so there's no ambiguity about what a flag controls. A flag's meaning should be immutable for its lifetime.
4. If the feature flag service is unreachable, what should happen? The application should fail to a *safe default* and continue operating — not break. Flag evaluation is on the request path, so a hard dependency on a reachable flag service would mean the flag service's outage takes down your entire application, which is unacceptable. The correct design: cache the last-known-good flag configuration locally, so if the flag service becomes unreachable, the app keeps using the most recent config it retrieved. And for the case where no config is available at all, each flag has a defined default — almost always "off" or the old, established behaviour, since falling back to proven behaviour is safer than falling back to a new, less-tested path. This way, a flag-service outage degrades gracefully (features stay in their last-known state, or safe defaults) rather than causing an application outage. It's the same resilience principle as any critical dependency: cache last-known-good, fail safe, and don't let the auxiliary system's failure cascade into the primary system.
5. What is progressive delivery and how do feature flags enable it? Progressive delivery is the practice of rolling out any change gradually and under continuous observation, so that nothing is ever all-or-nothing — every change ramps up in controlled stages with the ability to halt or reverse at each step. A typical flow: deploy the code dark, enable it for internal users, then 1% of real users, watch error rate and latency, then 5%, 25%, 100%, and finally remove the flag. Feature flags enable it by providing the granular runtime control: they let you target specific user segments and percentages, change the rollout without deploying, and flip off instantly if metrics degrade — giving you the "control at every step" that progressive delivery requires. It's the synthesis of the whole deployment part: canary deployments (gradual traffic shift) plus feature flags (gradual feature enablement, instant rollback) plus observability (metrics gate each step) plus automated analysis (auto-promote or auto-rollback). The unifying principle is that you never expose a change to everyone at once and hope — you expose it incrementally, observe real impact, and only proceed when the data says it's safe, with instant reversal always available.

Further reading

This site is open source. Improve this page.