system-design

Secrets Management

Passwords, API keys, and certificates that your systems need but must never leak. The rule is simple — never in code — and the reasons it’s hard are all about rotation and scale.

Prerequisites: Encryption, Containers Time to read: ~16 minutes

The problem

Your application needs credentials: a database password, a Stripe API key, a signing key, a third-party service token. These secrets must be available to the running application but invisible to everyone and everything else.

The tempting, wrong answers:

# ❌ Hardcoded — now in git history forever, visible to everyone with repo access
DB_PASSWORD = "hunter2"

# ❌ In a committed config file — same problem
# config.yaml: db_password: hunter2

# ❌ In a .env file committed to the repo — the classic mistake

🚨 Secrets in git are the single most common credential leak. And git never forgets — deleting the secret in a later commit doesn’t remove it from history. Anyone who ever cloned the repo, and every fork, has it. You must rotate any secret that has ever touched a repository, because you must assume it’s compromised. GitHub scans public repos for leaked keys and there’s a thriving criminal market in harvesting them — leaked AWS keys are exploited within minutes.

The rules

1. Never in code or committed config. Non-negotiable. Add .env, *.pem, and credential files to .gitignore from the first commit.

2. Inject at runtime, not build time. Secrets come from the environment when the app starts, not baked into the artifact.

3. Store in a dedicated secrets manager, encrypted, with access controls and audit logs.

4. Rotate regularly, and immediately on any suspected compromise.

5. Least privilege — each service gets only the secrets it needs. → Least privilege

6. Audit access — know who (and what) accessed which secret, when.

Where secrets should live

Environment variables — the baseline, with caveats

The common way to inject secrets into a process.

DB_PASSWORD=... APP_ENV=prod ./myapp

✅ Simple, universal, and keeps secrets out of code. ❌ 🚨 Not actually that secure. Env vars are readable via /proc/<pid>/environ, leak into child processes, often get logged (crash dumps, error reporters that dump the environment), and are visible to anyone who can inspect the process. They’re a step up from hardcoding, not a strong solution.

Better than committed config, worse than a secrets manager. Fine for non-critical config; not ideal for high-value secrets.

Secrets managers — the real answer

Dedicated systems: HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault.

App → authenticates to Vault (via its workload identity, not a stored secret) → fetches the secret

🚨 What they give you beyond storage:

Encryption at rest with a KMS. → Encryption
Fine-grained access control — this service can read this secret, nothing else.
Audit logging — every access recorded (essential for compliance and incident response).
Automatic rotation — some can rotate a database password and update the database, coordinated.
🚨 Dynamic secrets (Vault’s standout feature): instead of a long-lived shared password, the app requests a short-lived, unique credential generated on demand — a database user valid for one hour, then automatically revoked. No long-lived secret to leak, and every access is attributable.

🎙️ “I’d store secrets in Vault or the cloud’s secrets manager, not environment variables — env vars leak into logs and child processes. And where possible, dynamic secrets: the app requests a short-lived database credential per session rather than everyone sharing one long-lived password.”

Dynamic secrets are a strong thing to mention — they change the model from “protect this long-lived secret forever” to “there is no long-lived secret.”

The bootstrap problem

🚨 A subtle chicken-and-egg worth knowing: to fetch secrets from Vault, the app needs to authenticate to Vault — which needs some credential. How does the app get that without it being a hardcoded secret?

The answer is workload identity. The app proves who it is by where it runs, not by a stored secret:

Kubernetes: a service account token the cluster injects and Vault trusts.
Cloud IAM roles: an EC2 instance / Lambda / GKE pod has an IAM role; the cloud provides short-lived credentials automatically; no secret is stored anywhere.
SPIFFE/SPIRE: cryptographic workload identity tied to the runtime environment.

🚨 This is the modern best practice: the application never holds a long-lived secret at all — its identity comes from its runtime, and it exchanges that for short-lived credentials. That eliminates the root secret entirely, which is the whole game.

Secrets in containers and CI/CD

Two places secrets commonly leak:

Container images. 🚨 Never bake secrets into an image — image layers are cached, shared, and inspectable (docker history shows build args), so a secret in a Dockerfile or a build arg is exposed to anyone who pulls the image. Inject secrets at runtime (env from the orchestrator, mounted files, or a fetch from Vault on startup). → Containers

CI/CD pipelines. Build systems need secrets (to deploy, to push images, to sign). Use the CI’s encrypted secret store (GitHub Actions secrets, GitLab CI variables), mark them masked so they don’t appear in logs, and 🚨 be careful with pull requests from forks — a malicious PR can be crafted to exfiltrate CI secrets if the pipeline runs untrusted code with secret access. Scope CI secrets tightly and use short-lived deployment credentials (OIDC federation with the cloud provider, so no long-lived cloud key sits in CI at all).

Kubernetes Secrets are worth a specific warning: 🚨 they are base64-encoded, not encrypted, by default — anyone with API access reads them trivially. Enable encryption at rest for etcd, use RBAC to restrict access, and consider an external secrets operator (External Secrets, Vault) so the real secret lives in Vault and K8s only holds a reference.

Rotation

🚨 Secrets must be rotatable, and the design must support rotation without downtime. A secret you can’t rotate is a permanent liability — when it leaks (and eventually one will), you can’t respond.

The challenge: rotating a shared secret while services are using it. Rotate it and every service still holding the old one breaks.

The pattern (same shape as zero-downtime migrations):

Create the new secret alongside the old (both valid).
Deploy services to use the new one.
Verify nothing still uses the old one.
Revoke the old one.

Dynamic secrets sidestep this entirely — each credential is short-lived, so rotation is automatic and continuous; there’s never a long-lived secret to coordinate.

Rotate immediately (not on schedule) when: an employee with access leaves, a secret may have leaked, a dependency is breached, or a secret ever appeared in logs or a repo.

Detecting leaks

🚨 Assume secrets will occasionally leak, and detect it:

Pre-commit hooks / secret scanners (git-secrets, truffleHog, GitHub push protection) block secrets from being committed in the first place.
Repository scanning finds secrets already committed (GitHub secret scanning, and the same tools in CI).
Canary tokens / honeytokens — deliberately planted fake credentials that alert you if anyone uses them, revealing a breach.
Cloud provider alerts — AWS, GitHub, and others notify you (and sometimes auto-quarantine) when they detect an exposed key.

⚖️ Trade-offs

Choice	Gain	Cost
Env vars	Simple, universal	Leak into logs/child processes; weak
Secrets manager	Encryption, access control, audit, rotation	A system to run; a dependency
Dynamic secrets	No long-lived secret to leak; per-use credentials	Requires supporting infrastructure
Workload identity	No stored root secret at all	Runtime-dependent; cloud/K8s specific
Automatic rotation	Limited exposure window	Coordination or dynamic-secret infrastructure
Secret scanning	Catches leaks early	False positives; must be enforced

In the real world

Leaked AWS keys in public GitHub repos are exploited within minutes — automated bots scan for them continuously and spin up crypto-mining on the victim’s account, producing five- and six-figure bills. This is why “never commit secrets” and “rotate anything that touched a repo” are drilled so hard.
HashiCorp Vault’s dynamic secrets changed the model for many organizations — moving from long-lived shared database passwords to per-session short-lived credentials that are automatically revoked, so there’s simply no durable secret to steal.
Cloud OIDC federation for CI/CD (GitHub Actions assuming an AWS role via OIDC, no stored AWS key) is the current best practice for deployment credentials, eliminating the long-lived cloud key that used to sit in every CI system.

🚨 Interview traps

Secrets in code or committed config. The most basic and most common mistake.
Not knowing git never forgets — a deleted secret is still in history and must be rotated.
Treating env vars as secure. They leak into logs and child processes.
Baking secrets into container images.
Not knowing Kubernetes Secrets are base64, not encrypted.
No rotation strategy, or a secret that can’t be rotated.
Not mentioning workload identity / dynamic secrets as the modern approach.

🎙️ Soundbites

“Secrets never go in code or committed config — git never forgets, so anything that touches a repo must be treated as compromised and rotated. I’d use a secrets manager, not environment variables, which leak into logs and child processes.”
“The strong version is dynamic secrets: the app requests a short-lived database credential per session rather than everyone sharing one long-lived password. Then there’s no durable secret to leak.”
“The bootstrap problem — how does the app authenticate to Vault without a stored secret — is solved by workload identity. The app proves who it is by where it runs, a Kubernetes service account or a cloud IAM role, and exchanges that for short-lived credentials. No root secret is stored anywhere.”
“Never bake secrets into container images — layers are cached and inspectable. Inject at runtime. And Kubernetes Secrets are base64-encoded, not encrypted, so I’d enable etcd encryption or use an external secrets operator.”
“I’d add secret scanning in pre-commit hooks and CI to catch leaks before they ship, and rotate immediately whenever someone with access leaves or a secret might have leaked.”

🛠️ Try it

1. Find secrets in git history. Run trufflehog or git-secrets against a repo (a test one, or a public one — they’re full of leaked keys). Seeing real secrets that were “deleted” but live on in history makes the “git never forgets” point permanent.

2. Prove env vars leak. Set a secret in an env var, then run cat /proc/<pid>/environ for the process, and trigger an error that dumps the environment. Watch the secret appear. That’s why env vars aren’t strong.

3. Use dynamic secrets. Run Vault locally with the database secrets engine. Request a database credential and watch Vault create a short-lived database user on the fly, valid for a few minutes, then auto-revoked. This makes the “no long-lived secret” model concrete and is genuinely impressive to see.

4. Set up workload identity. In a cloud environment, give a VM/pod an IAM role and have it access a resource with no stored credentials — the cloud provides short-lived ones automatically. Note that there’s no secret anywhere in your code or config.

Check yourself

1. Why must you rotate any secret that has ever been committed to git?

Because git history is permanent and distributed. Deleting the secret in a later commit removes it from the current files but not from the repository's history — the old commit still contains it, and `git log` or a history rewrite tool retrieves it instantly. Worse, anyone who ever cloned the repo has a full copy of the history including the secret, every fork has it, and if the repo was ever public, automated scanners have almost certainly harvested it (leaked cloud keys are exploited within minutes). So the moment a secret touches a repo, you must assume it's compromised and rotate it — generating a new one and revoking the old — regardless of whether you "deleted" it. Truly removing it from history requires rewriting all history and coordinating with everyone who has a clone, which is disruptive and still doesn't recall the copies already taken.

2. Why are environment variables not a strong way to store secrets?

Because they're broadly readable and prone to leaking. A process's environment is exposed at `/proc//environ` to anyone who can read it, child processes inherit the parent's environment (so a spawned subprocess or shell sees the secrets), and — most commonly — they leak into logs: crash-dump handlers, error-reporting tools, and debug endpoints frequently dump the full environment, sending secrets to logging systems and third-party error trackers with weaker access controls. They're better than hardcoding (the secret isn't in the artifact), but a dedicated secrets manager adds encryption at rest, per-secret access control, audit logging, rotation, and dynamic short-lived secrets — none of which env vars provide. </details>

3. What are dynamic secrets and why are they better than shared static ones?

Dynamic secrets are credentials generated on demand, unique per request or session, and short-lived — for example, Vault creating a fresh database user valid for one hour when your app asks, then automatically revoking it. They're better because they change the security model from "protect this long-lived shared password forever" to "there is no durable secret to protect." A leaked dynamic credential is useful only until it expires (minutes to an hour); every access is attributable to a specific requester rather than lost in a shared password; there's no rotation to coordinate because credentials continuously expire and regenerate; and revoking access for one consumer doesn't disrupt others. The long-lived shared secret — the thing that leaks and can't easily be rotated — simply ceases to exist.

4. What is the secrets bootstrap problem, and how is it solved?

The chicken-and-egg problem: to fetch secrets from a secrets manager, the application must authenticate to it — which requires *some* credential. If that bootstrap credential is itself a stored secret, you've just moved the problem rather than solved it. The solution is **workload identity**: the application proves who it is by *where and how it runs*, not by a stored secret. On Kubernetes, the cluster injects a service account token that the secrets manager is configured to trust. On cloud platforms, an instance/pod/function has an IAM role and the provider supplies short-lived credentials automatically. SPIFFE/SPIRE gives cryptographic identity tied to the runtime. In all cases the app holds no long-lived secret at all — its identity is intrinsic to its environment, and it exchanges that identity for short-lived credentials, eliminating the root secret entirely.

5. Why is it dangerous to bake secrets into a container image?

Because container images are built in layers that are cached, shared, and inspectable. A secret added in a `Dockerfile` layer — even if a later layer deletes the file — remains in the earlier layer and is recoverable, and secrets passed as build arguments appear in `docker history`. Anyone who can pull the image (from a registry, a CI cache, or another team) can extract the secret, and images are frequently shared far more widely than the source code. Because images are immutable artifacts, you also can't rotate a baked-in secret without rebuilding and redeploying. The correct approach is to inject secrets at *runtime* — via environment provided by the orchestrator, mounted secret volumes, or fetching from a secrets manager on startup using workload identity — so the image itself contains no credentials and the same image can run in any environment with its own secrets.

--- ## Further reading - [The OWASP Top 10](/system-design/07-security/05-owasp-top-10.html) — next - [Encryption](/system-design/07-security/03-encryption.html) - [Containers](/system-design/09-deployment-and-infra/01-containers.html) - [CI/CD](/system-design/09-deployment-and-infra/03-ci-cd.html) - HashiCorp Vault and cloud secrets-manager documentation

This site is open source. Improve this page.