system-design

Deployment Strategies: Blue-Green, Canary, Rolling

How to replace running software without dropping requests — and how to limit the damage when the new version is broken. Because it will sometimes be broken.

Prerequisites: CI/CD, Load Balancers Time to read: ~16 minutes

The problem

You have version 1 running, serving live traffic. You need to deploy version 2. The naive approach:

❌ Stop v1. Deploy v2. Start v2.

🚨 This causes downtime (the gap while nothing’s running) and is all-or-nothing — if v2 is broken, everyone hits the broken version simultaneously, and you’ve dropped every in-flight request.

The deployment strategies below solve two problems: zero downtime (no gap in service) and limited blast radius (a bad deploy affects few users, not all). Every strategy is a different point on the speed/safety/cost trade-off.

Rolling deployment

Replace instances gradually, a few at a time, while the rest keep serving.

Start: [v1][v1][v1][v1]
Step:  [v2][v1][v1][v1]   ← replace one, wait for it to be healthy
Step:  [v2][v2][v1][v1]
Step:  [v2][v2][v2][v1]
End:   [v2][v2][v2][v2]

✅ Zero downtime, no extra infrastructure (you reuse the same instances), gradual. ❌ 🚨 Two versions run simultaneously during the rollout — so v1 and v2 must be compatible (especially with a shared database — this is exactly the zero-downtime migration constraint). Rollback is slow (you have to roll forward/back instance-by-instance). A bad version does reach some users before you notice.

The default for Kubernetes and most orchestrators.

🚨 Graceful shutdown matters: each instance being replaced must drain (finish in-flight requests, stop accepting new ones) before it’s killed, or the rollout drops requests. → Load Balancers

Blue-green deployment

Run two complete environments — blue (current) and green (new). Deploy v2 to green (idle), test it, then switch all traffic from blue to green at once.

Blue (v1) ← all traffic          Blue (v1) ← idle now
Green (v2) ← idle, being tested   Green (v2) ← all traffic
              ─── switch ───►

✅ 🚨 Instant switch and instant rollback — the old environment (blue) stays running, so if green is bad, flip back to blue in seconds. This fast rollback is the key advantage. ✅ Test the new version fully before any real traffic. ❌ 🚨 Double the infrastructure (two full environments) — expensive. ❌ All-or-nothing — the switch sends everyone to v2 at once; if v2 has a subtle bug that testing missed, everyone hits it (though you can flip back fast). ❌ Database migrations are still tricky (both environments share the database, or you need careful handling).

Best when instant rollback is worth the cost of doubled infrastructure.

Canary deployment

🚨 The safest, and usually the best for high-stakes services. Route a small percentage of traffic to v2, watch it closely, and gradually increase if it’s healthy.

Step 1:  1% → v2, 99% → v1    ← watch error rate, latency for this 1%
Step 2:  5% → v2              ← metrics still good? continue
Step 3:  25% → v2
Step 4:  100% → v2            ← full rollout
(at any step, if metrics degrade → roll back, only that % was affected)

✅ 🚨 Minimal blast radius — a bad version affects 1% of users, not 100%. You catch problems with real production traffic before they hit everyone. ✅ Real-world validation — testing catches what tests can’t (real traffic, real data, real load). ✅ Gradual, controlled, data-driven progression. ❌ Slower (progressive rollout takes time), more complex (traffic splitting, metric monitoring), requires good observability to judge canary health. ❌ Two versions run simultaneously (compatibility, like rolling).

🚨 Automated canary analysis is the mature version: the system automatically compares the canary’s metrics (error rate, latency) to the baseline and auto-promotes or auto-rolls-back based on thresholds. Tools like Argo Rollouts, Flagger, and Spinnaker do this. Mentioning it is a strong signal.

🎙️ “For a high-stakes service I’d use canary deployment — route 1% of traffic to the new version, watch error rate and latency on real traffic, and progress to 5%, 25%, 100% only if metrics stay healthy. A bad version hits 1% of users, and automated canary analysis can promote or roll back based on the metrics.”

The comparison

	Rolling	Blue-Green	Canary
Downtime	Zero	Zero	Zero
Extra infrastructure	None	2× (double)	Small (canary instances)
Rollback speed	Slow (instance by instance)	Instant (flip back)	Fast (only % affected)
Blast radius of a bad deploy	Some users	All users (but flip back fast)	Minimal (1%)
Real-traffic validation	Partial	No (test before switch)	Yes
Complexity	Low	Medium	High (traffic split + metrics)
Best for	Default, most services	Instant rollback needed	High-stakes, gradual validation

Shadow / dark deployment — send v2 a copy of real traffic but don’t use its responses (serve v1’s). You validate v2 against real production load with zero user risk. 🚨 Great for testing performance and correctness of a risky change (and the technique behind strangler-fig migrations). Cost: v2 processes real traffic (double compute), and you must handle side effects (v2 must not actually charge the card).

A/B testing — route different users to different versions to compare business outcomes (conversion, engagement), not just technical health. Uses the same traffic-splitting mechanism as canary but for a different purpose (product experimentation, not deploy safety). Often built on feature flags.

Feature flags — 🚨 the most flexible option, and often the best. Deploy the code with the feature off, then turn it on for a percentage of users independently of deployment — and turn it off instantly if it breaks, with no rollback deploy. This decouples deploy from release, which is transformative. → Feature Flags

The universal constraint: version compatibility

🚨 Every zero-downtime strategy has old and new versions running simultaneously (rolling and canary literally, blue-green briefly during the switch and around database changes). This means:

The two versions must be compatible — especially with a shared database. A schema change must work for both v1 and v2 during the overlap, which is exactly the expand-contract pattern. → Zero-Downtime Migrations
API changes must be backward compatible during the overlap. → Versioning

This is why deployment strategy and database migration strategy are deeply linked — you can’t deploy safely if the versions can’t coexist.

Choosing

Rolling — the sensible default for most stateless services; zero-downtime, no extra cost.
Blue-green — when instant rollback is worth double the infrastructure (a critical service where you want to flip back in seconds).
Canary — when you want to validate on real traffic with minimal blast radius (high-stakes, large-scale services). The safest.
Feature flags — layer on top of any of these to decouple deploy from release and get instant feature-level rollback.

🎙️ The mature answer often combines them: “Rolling deploy as the mechanism, canary for the rollout (1% then gradual with automated analysis), and feature flags so we can deploy dark and control release independently — turning off a broken feature instantly without a rollback deploy.”

⚖️ Trade-offs

Strategy	Gain	Cost
Rolling	Zero downtime, no extra cost	Slow rollback; bad version reaches some users
Blue-green	Instant rollback; test before switch	Double infrastructure; all-or-nothing switch
Canary	Minimal blast radius; real-traffic validation	Slower; complex; needs good observability
Shadow	Validate on real load, zero user risk	Double compute; side-effect handling
Feature flags	Decouple deploy/release; instant off	Flag management; technical debt if not cleaned up

In the real world

Netflix and Google pioneered automated canary analysis — Netflix’s Spinnaker and Kayenta automatically compare canary metrics to a baseline and gate the rollout, so a bad deploy is caught and rolled back by automation before it affects most users. This is the state of the art.
Canary is standard at scale because the blast-radius argument is compelling: when you serve millions of users, “the bug hits 1% for 5 minutes before auto-rollback” is dramatically better than “the bug hits everyone.” The cost of the complexity is repaid the first time it catches a bad deploy.
Feature flags decoupling deploy from release has become a dominant practice precisely because it solves the rollback problem so elegantly — “turn it off” beats “roll back the deploy” in speed and simplicity, and it lets you deploy incomplete work safely (behind a flag).

🚨 Interview traps

Naive stop-deploy-start — causes downtime and drops requests.
Not knowing the strategies or their trade-offs (especially blast radius vs rollback speed).
Ignoring version compatibility — old and new run simultaneously; the database/API must support both.
Forgetting graceful drain on rolling deploys — drops in-flight requests.
Blue-green without acknowledging double infrastructure cost.
Canary without observability — you can’t judge canary health without good metrics.
Not mentioning feature flags as a way to decouple deploy from release.

🎙️ Soundbites

“Canary for high-stakes services — route 1% to the new version, watch error rate and latency on real traffic, and progress only if healthy. A bad version hits 1% of users, and automated canary analysis can auto-roll-back on the metrics.”
“Blue-green gives instant rollback because the old environment stays running — flip back in seconds. The cost is double the infrastructure, and the switch is all-or-nothing.”
“Every zero-downtime strategy runs old and new versions simultaneously, so they must be compatible — a database change has to work for both during the overlap. That’s why deployment and migration strategy are linked.”
“Feature flags decouple deploy from release — deploy the code dark, turn it on gradually, turn it off instantly if it breaks. That’s faster and simpler than a rollback deploy.”
“I’d combine them: rolling as the mechanism, canary for the rollout with automated analysis, and feature flags for instant feature-level control.”

🛠️ Try it

1. Do a rolling deploy and watch compatibility. With 4 instances behind a load balancer, roll out a new version one at a time. Then deploy a version that changes the API response shape incompatibly, and watch requests fail during the rollout while both versions run. That failure is the version- compatibility constraint, felt directly.

2. Blue-green with instant rollback. Run two environments, switch traffic to green, then “discover” green is broken and flip back to blue. Time the rollback — it’s seconds, because blue never stopped. Compare to how long a rolling rollback would take.

3. Canary with metrics. Route 10% of traffic to a new version that has a higher error rate. Watch your monitoring show the canary’s errors while the baseline stays healthy. Seeing the bad canary caught by metrics — while 90% of users are unaffected — is the whole argument for canary.

4. Feature flag rollback. Deploy code behind a flag (off). Turn it on, “discover” it’s broken, turn it off. No deploy, no rollback — just a flag flip in seconds. Compare the speed and simplicity to a deployment rollback.

Check yourself

1. What two problems do zero-downtime deployment strategies solve?

**Downtime** and **blast radius**. The naive approach (stop the old version, deploy the new, start it) creates a gap where nothing is serving — downtime — and drops every in-flight request. Zero-downtime strategies (rolling, blue-green, canary) all eliminate that gap by keeping some version serving throughout the transition. Separately, they address blast radius — how many users a *broken* new version affects. The naive approach and blue-green's switch are all-or-nothing (everyone hits the new version at once), while canary and rolling limit exposure (a bad version reaches a small fraction first). The strategies differ in how they balance these along with rollback speed and infrastructure cost: canary minimizes blast radius but is slower and complex; blue-green gives instant rollback but doubles infrastructure and switches everyone at once; rolling is cheap and gradual but rolls back slowly. The right choice depends on how much a bad deploy costs you.

2. What's the key advantage and disadvantage of blue-green deployment?

The key advantage is **instant rollback**: because you run two complete environments and the old one (blue) keeps running untouched while the new one (green) takes traffic, reverting a bad deploy is just flipping traffic back to blue — a change that takes *seconds*, with the previous version already warm and running. You also get to fully test green before sending it any real traffic. The key disadvantage is **double the infrastructure** — you must run two complete production environments simultaneously, which is expensive at scale. A secondary disadvantage is that the switch is all-or-nothing: it sends *everyone* to the new version at once, so a subtle bug that testing missed hits 100% of users immediately (though you can flip back fast). Blue-green is the right choice when the value of second-scale rollback justifies the doubled cost — typically a critical service where any prolonged degradation is unacceptable.

3. Why is canary deployment considered the safest strategy?

Because it minimizes blast radius while validating on real production traffic. It routes only a small percentage of traffic (start at 1%) to the new version, monitors that slice's error rate and latency against the baseline, and increases the percentage gradually (5%, 25%, 100%) only if the metrics stay healthy — rolling back at any step if they degrade. So a broken version affects at most that small percentage of users for the short time before it's caught, never everyone. Crucially, this validation happens against *real* traffic, real data, and real load, which catches problems that pre-production testing cannot (subtle production-only bugs, performance under real load, edge cases in real data). The mature form — automated canary analysis — has the system compare canary metrics to the baseline and automatically promote or roll back, removing human latency from the decision. The trade-offs are that it's slower (progressive rollout takes time) and requires good observability to judge canary health, but for high-stakes, large-scale services, limiting a bad deploy to 1% of users is worth it.

4. Why must old and new versions be compatible during any zero-downtime deploy?

Because every zero-downtime strategy has both versions running simultaneously for some period — rolling and canary literally run them side by side throughout the rollout, and blue-green runs them together briefly during the switch and around any shared database. During that overlap, both versions serve real requests against the *same* backing systems, most importantly a shared database. So a change must work for both: if v2 renames a database column, v1 (still running) breaks when it queries the old name; if v2 requires a new required field, v1's writes fail validation. This is exactly why database schema changes must follow the expand-contract pattern (add the new thing, make both versions work, then remove the old thing across multiple deploys) and why API changes must stay backward compatible during the overlap. It's the reason deployment strategy and database migration strategy are inseparable — you cannot deploy safely if the two versions can't coexist against shared state, so the migration must be designed to support the overlap window.

5. How do feature flags improve on deployment-based rollout, and what's the cost?

They decouple *deploy* from *release*. With deployment strategies alone, deploying code and activating a feature are the same event, so rolling back a broken feature means rolling back a deploy (rebuilding, redeploying, all instances). Feature flags let you deploy the code with the feature *off* (dark), then turn it on — for a percentage of users, specific users, or everyone — as a separate, instant configuration change, and turn it *off* just as instantly if it misbehaves, with no deploy at all. This is faster and simpler than any deploy rollback, enables gradual feature rollout independent of code deployment, allows deploying incomplete work safely (hidden behind a flag), and supports A/B testing on the same mechanism. The costs: flag management overhead (flags accumulate and must be cleaned up, or they become permanent technical debt and a source of untested code-path combinations), the risk of a mis-configured flag, and added conditional complexity in the code. But the ability to release and un-release a feature in seconds without touching the deployment makes flags one of the most valuable deployment-safety tools, usually layered on top of a deployment strategy rather than replacing it.