system-design

CI/CD Pipelines

The assembly line from a commit to production. Done well, deploying is a non-event that happens fifty times a day. Done badly, it’s a terrifying quarterly ritual.

Prerequisites: Containers Time to read: ~16 minutes

The problem

Code that isn’t deployed provides no value. But deploying is risky — bugs, broken environments, manual mistakes. The historical response was to deploy rarely (once a quarter, a big scary event), which paradoxically made it more dangerous: huge batches of changes, so when something broke, you had no idea which of 500 changes caused it.

🚨 The counter-intuitive insight: deploying more often is safer, not riskier. Small, frequent deploys mean each change is tiny, easy to review, easy to test, and — if it breaks — easy to identify and roll back. CI/CD is the automation that makes frequent, safe deployment possible.

Deploy once a quarter:  500 changes at once → something breaks → which one? → hours to find
Deploy 50× a day:       1 change → it breaks → obviously that change → roll back in 2 min

CI vs CD

Three terms, often blurred:

Continuous Integration (CI) — every commit is automatically built and tested. The goal: catch problems at commit time, and keep the main branch always in a working state.
Continuous Delivery (CD) — every change that passes CI is automatically prepared and ready to deploy, but the actual production deploy is a manual button-press (human decides when).
Continuous Deployment (CD) — goes further: every change that passes all tests deploys to production automatically, no human gate.

🚨 The distinction that matters: continuous delivery keeps a human in the loop for the production push; continuous deployment removes it entirely. Which you want depends on your confidence in your tests and your risk tolerance — a payment system might keep the human gate; a content site might fully automate.

The pipeline stages

flowchart LR
    C[Commit] --> B[Build]
    B --> T[Test]
    T --> S[Security scan]
    S --> A[Build artifact/image]
    A --> ST[Deploy to staging]
    ST --> IT[Integration tests]
    IT --> P[Deploy to prod]
    P --> V[Verify/monitor]

Build — compile, resolve dependencies, produce an artifact. 🚨 Build the deployable artifact once (a container image) and promote that same artifact through every environment — never rebuild per environment, or you lose the guarantee that what you tested is what you ship.

Test — the critical stage, in a pyramid:

Unit tests — many, fast, isolated (the base of the pyramid).
Integration tests — fewer, test components together.
End-to-end tests — few, slow, test the whole system. 🚨 Fast feedback matters — if the pipeline takes 40 minutes, developers context-switch and stop trusting it. Parallelize, cache dependencies, and run the fast tests first (fail fast).

Security scan — dependency vulnerability scanning (npm audit, Snyk), secret scanning (catch committed secrets → Secrets), static analysis (SAST), container image scanning. → OWASP

Artifact — the immutable, versioned output, stored in a registry.

Deploy — to staging, then (with a gate or automatically) to production, using a safe deployment strategy (rolling, canary, blue-green).

Verify — 🚨 the stage people forget: after deploy, automatically check it’s healthy (smoke tests, monitoring) and auto-rollback if not. A deploy isn’t done when the code is out; it’s done when you’ve confirmed it works.

What makes a pipeline good

🚨 These principles come up, and they’re what separate a pipeline that helps from one that hurts:

1. Fast. A slow pipeline (>10-15 min) breaks the feedback loop — developers stop waiting for it and lose trust. Parallelize, cache, fail fast.

2. Reliable / deterministic. 🚨 Flaky tests are poison — a test that fails randomly trains people to re-run and ignore failures, so a real failure gets ignored too (the same dynamic as alert fatigue). Fix or quarantine flaky tests aggressively.

3. Automated end to end. Every manual step is a place for human error and a bottleneck. The goal is that a merge to main can reach production with no manual intervention (whether it does automatically is the delivery-vs-deployment choice).

4. Reproducible. The same commit produces the same result every time — pinned dependencies, hermetic builds, no “it worked yesterday.”

5. Fast rollback. 🚨 The most important safety property. When a deploy goes bad, you need to undo it in seconds — because mitigate-before-diagnose means rolling back is your first move. A pipeline that can deploy fast but not roll back fast is dangerous.

6. Observable. You can see the state of every deploy and its history.

The deployment safety net

CI/CD pairs with several practices to make frequent deploys safe:

Automated tests — the primary gate.
Safe deployment strategies — canary/blue-green so a bad deploy hits few users.
Feature flags — 🚨 decouple deploy from release. Deploy code dark (flag off), turn it on separately, turn it off instantly if it breaks — no rollback needed.
Automated rollback — verify-after-deploy triggers rollback on failure.
Monitoring — catch what tests missed.

🚨 Together, these make a bad deploy a non-event: it hits 1% of users (canary), the flag is off for most, monitoring catches it, and it rolls back automatically. That’s the point — not preventing all bad deploys (impossible), but making them cheap.

Trunk-based development

🚨 A practice that goes hand-in-hand with CI/CD, and a good thing to know:

Instead of long-lived feature branches that diverge for weeks (and produce painful “merge hell”), developers integrate small changes into the main branch (trunk) frequently — at least daily.

✅ Small changes, continuous integration (the “CI” is real), no giant merges, always-deployable trunk. ❌ Requires discipline: feature flags to hide incomplete work, and a fast, reliable pipeline to keep trunk healthy.

This contrasts with heavy branching models (GitFlow) that batch changes into big releases — the opposite of what CI/CD wants. Small, frequent, integrated changes are the whole philosophy.

Pipeline security

🚨 The CI/CD pipeline is a high-value attack target — it has access to production and can inject code into everything you ship (supply chain, SolarWinds):

Secure the pipeline’s credentials — it has production access; those secrets are crown jewels. Use short-lived credentials (OIDC federation to the cloud, no stored long-lived key). → Secrets
Be careful with untrusted PRs — a malicious pull request from a fork can try to exfiltrate CI secrets if the pipeline runs untrusted code with secret access. Restrict what fork PRs can do.
Sign artifacts — so you can verify what you deploy is what you built (integrity).
Least privilege — the pipeline gets only the access it needs.

⚖️ Trade-offs

Choice	Gain	Cost
Frequent small deploys	Safer, easier to diagnose, faster feedback	Requires automation and discipline
Continuous deployment (no gate)	Fastest delivery	Requires very high test confidence
Continuous delivery (human gate)	Human judgment on timing	Slower; a manual step
Comprehensive test pyramid	Catches bugs before prod	Test maintenance cost; slower pipeline if unbalanced
Trunk-based development	Continuous integration, no merge hell	Needs feature flags and discipline

In the real world

High-performing teams deploy dozens to hundreds of times per day (per the DORA/State of DevOps research), and — counter-intuitively — have lower change-failure rates and faster recovery than teams that deploy rarely. The research is the strongest evidence that frequent deployment is safer.
Amazon reportedly deploys every few seconds across its services — an existence proof that deployment can be a complete non-event at extreme scale, enabled entirely by automation.
The “flaky test” problem is a near-universal pipeline killer — teams that tolerate flaky tests end up with pipelines nobody trusts, where real failures get re-run and ignored. Aggressive flaky- test management (quarantine, fix, or delete) is what keeps a pipeline credible.

🚨 Interview traps

Not knowing CI vs continuous delivery vs continuous deployment.
Thinking frequent deploys are riskier — they’re safer (small changes).
Rebuilding the artifact per environment — build once, promote the same one.
No fast rollback — the key safety property.
No verify-after-deploy step — a deploy isn’t done until confirmed healthy.
Tolerating flaky tests — they destroy pipeline trust.
Ignoring pipeline security — it has production access.

🎙️ Soundbites

“Frequent small deploys are safer, not riskier — a tiny change that breaks is obviously the culprit and rolls back in minutes, versus a quarterly batch of 500 changes where you can’t tell which broke.”
“Build the artifact once — a container image — and promote that same image through staging to production. Rebuilding per environment breaks the guarantee that what you tested is what you ship.”
“The most important property is fast rollback. Mitigate-before-diagnose means rolling back is the first move in an incident, so a pipeline that deploys fast but can’t roll back fast is dangerous.”
“Feature flags decouple deploy from release — deploy the code dark, turn it on separately, turn it off instantly if it breaks. That makes a bad rollout a non-event.”
“Flaky tests are poison — they train people to re-run and ignore failures, so a real failure gets ignored too. I’d quarantine or fix them aggressively, same as alert fatigue.”

🛠️ Try it

1. Build a real pipeline. Set up GitHub Actions (or GitLab CI) for a small project: on push, build, run tests, build a container image, push it to a registry. Then push a commit that breaks a test and watch the pipeline stop it before it ships. That gate — catching a bug automatically at commit time — is the whole value.

2. Feel the fast-feedback problem. Add a deliberately slow test (a 5-minute sleep) to your pipeline. Notice how you stop waiting for it and context-switch. Then parallelize/cache to bring it back under 2 minutes. The difference in how much you use the pipeline is real.

3. Practice rollback. Deploy something, then deploy a broken version. Time how long it takes to roll back. If it’s slow or manual, that’s your safety gap. Automate it (redeploy the previous image) and re-time.

4. Introduce a flaky test. Add a test that fails randomly 20% of the time. Run the pipeline several times and feel the instinct to just “re-run it.” That instinct is exactly how a real failure gets ignored — which is why flaky tests must be eliminated.

Check yourself

1. Why are frequent, small deployments safer than infrequent, large ones?

Because they shrink the blast radius and the diagnostic problem of each deploy. A quarterly release bundles hundreds of changes, so when something breaks, you can't tell which of the 500 changes caused it — hours of investigation — and rolling back means reverting everyone's work. A deploy containing a single small change is trivially diagnosable (if it breaks, it's obviously that change), trivially reviewable, and trivially reversible (roll back one change in minutes). Small changes are also less likely to break in the first place — less code, fewer interactions, easier to test thoroughly. Counter-intuitively, the DORA research confirms that teams deploying many times per day have *lower* change-failure rates and faster recovery than teams deploying rarely. The historical instinct to deploy rarely "to be safe" actually made deployment more dangerous by batching risk; CI/CD inverts this by making frequent, small, automated deployment the safe path.

2. What's the difference between continuous delivery and continuous deployment?

Both automate everything up to production, but they differ on the final production push. **Continuous delivery** keeps a human in the loop: every change that passes all automated tests is built, verified, and made *ready* to deploy, but a person presses the button to actually release it to production, deciding *when*. **Continuous deployment** removes that human gate entirely: every change that passes all tests deploys to production automatically, with no manual intervention. The choice depends on confidence and risk tolerance — continuous deployment requires very high trust in your automated tests and safety mechanisms (since nothing else stands between a merge and production), so it suits systems where the cost of a bad deploy is low and easily reversible, while continuous delivery's human gate suits higher-stakes systems (payments) where you want deliberate timing and a final human judgment. Both are "CD"; the distinction is whether a human approves the production release.

3. Why should you build the deployable artifact once and promote it?

To guarantee that what you tested is exactly what you ship. If you build a fresh artifact for each environment — one for staging, another for production — then subtle differences can creep in between builds (a dependency version resolved differently, a changed base image, a build-time variable, a non-deterministic step), meaning the artifact you validated in staging is *not* the one running in production. That gap is where "it passed staging but broke in production" bugs live. Building the artifact once (typically an immutable, versioned container image) and promoting that same bit-for-bit image through staging, integration testing, and production means every environment runs identical code, so passing tests in staging genuinely predicts production behaviour. It also makes rollback trivial (the previous image still exists) and builds faster (no rebuild per stage). Build once, promote the same artifact everywhere.

4. Why is fast rollback the most important pipeline safety property?

Because rollback is your first response when a deploy goes wrong, and incident response depends on recovering fast. The principle of "mitigate before you diagnose" means that when a deploy breaks production, you don't spend time understanding the bug while users suffer — you undo the change to restore service immediately, *then* investigate at leisure. If your pipeline can deploy quickly but rolling back is slow, manual, or risky (doesn't handle a schema change, requires a full rebuild), then you've lost your fastest, safest mitigation exactly when you need it, and the outage drags on while you either debug live or fumble a slow rollback. Fast, reliable, one-command rollback (redeploy the previous immutable artifact) turns a bad deploy from a crisis into a two-minute non-event. This is also why feature flags are valuable — they let you "roll back" a feature by flipping a flag, faster than even a deploy rollback. A deployment system's rollback capability matters more than its deploy speed.

5. Why are flaky tests so damaging to a CI/CD pipeline?

Because they destroy the trust that makes the pipeline useful. A flaky test — one that passes and fails non-deterministically on the same code — trains developers to respond to a failure by re-running the pipeline rather than investigating, since "it's probably just flaky." Once that habit forms, a *real* failure — an actual bug the test correctly caught — gets the same treatment: re-run, dismiss, ship anyway. So the tests stop being a reliable gate; the pipeline reports failures that everyone ignores, which is worse than no tests because it provides false confidence. It's exactly the dynamic of alert fatigue: too many false signals and people tune out all signals, including the true ones. The damage compounds — flaky tests also slow the pipeline (re-runs), cause spurious deploy blocks, and erode belief in the whole automated-testing investment. The fix is aggressive: fix flaky tests immediately, or quarantine them (remove from the blocking gate) until fixed, so that a red pipeline always means a genuine problem worth stopping for.