system-design

Kubernetes for System Designers

When you have hundreds of containers across dozens of machines, someone has to decide what runs where, restart what dies, and route traffic. Kubernetes is that someone — and you should understand what it does, not memorize its YAML.

Prerequisites: Containers, Load Balancers Time to read: ~20 minutes

The problem

Containers solved packaging. But now you have 500 containers across 40 machines, and you need to:

🚨 Doing this by hand across 500 containers is impossible. Kubernetes (K8s) is a container orchestrator that automates all of it. For an interview, you don’t need to write YAML — you need to understand the concepts and what problems they solve.

The core idea: declarative desired state

🚨 This is the single most important thing to understand about Kubernetes, and it explains everything else.

You don’t tell Kubernetes how to do things (imperative: “start a container here, then here”). You declare the desired state (“I want 5 replicas of this service running”), and Kubernetes continuously works to make reality match.

You declare:  "5 replicas of api-service, version 2.3"
K8s controllers constantly compare desired vs actual:
  - Only 3 running? → start 2 more
  - A machine died, losing 1? → start 1 elsewhere
  - You changed to 8 replicas? → start 3 more
  - You changed to version 2.4? → roll it out

The reconciliation loop: controllers watch the actual state, compare it to the desired state, and take action to close the gap — forever. 🚨 This is why Kubernetes is self-healing: a crashed container or a dead machine is just a gap between desired and actual state that a controller automatically closes. You don’t script the recovery; the system converges toward what you declared.

🎙️ “The core idea is declarative desired state with a reconciliation loop — you say ‘I want 5 replicas,’ and controllers continuously make reality match. That’s what makes it self-healing: a crash is just a gap the controller closes automatically.”

The concepts that matter

You need to understand these building blocks, not their YAML syntax:

Pod — the smallest unit; one or more tightly-coupled containers that share a network and storage (usually just one app container, plus maybe a sidecar). 🚨 Pods are ephemeral and disposable — they get created, killed, and replaced constantly, with a new IP each time. You never depend on a specific pod. This is the container-statelessness principle made concrete.

Deployment — declares the desired state for a stateless app: “run N replicas of this pod, at this version.” Handles scaling, rolling updates, and rollback. The workhorse for stateless services.

Service — 🚨 a stable network endpoint in front of a set of ever-changing pods. Because pods come and go with new IPs, you can’t point at a pod — you point at a Service, which load-balances across the current healthy pods. This solves the service discovery problem: a stable name (payments) that always routes to healthy instances.

Ingress / Gateway — routes external HTTP traffic to Services (path/host-based), terminates TLS. The API gateway role at the cluster edge.

ConfigMap / Secret — inject configuration and secrets into pods. 🚨 (Note: Secrets are base64, not encrypted by default.)

Namespace — logical partitioning of the cluster (e.g. team-a, staging) for organization and access control.

StatefulSet — for stateful apps (databases) that need stable identity and persistent storage. 🚨 Running databases on Kubernetes is genuinely harder than stateless apps and often better delegated to a managed service.

DaemonSet — one pod per node (log collectors, monitoring agents).

Job / CronJob — batch and scheduled work. (With at-least-once semantics — jobs can run twice.)

Health checks: liveness vs readiness

🚨 A specific, important distinction that comes up (and connects to load balancers):

The distinction: liveness answers “restart?”, readiness answers “route traffic?”. Confusing them — especially a liveness check that hits the database — causes cascading restarts.

Scaling

🚨 Autoscaling has a lag — new pods take seconds to start (image pull, warm-up), new nodes take minutes. So autoscaling handles gradual load changes, not sudden spikes or cascading failures (too slow). Keep headroom for spikes.

The architecture (briefly)

Control Plane (the brain):
├─ API Server    — the front door; everything goes through it
├─ etcd          — the datastore; ALL cluster state (declarative config) → CONSENSUS
├─ Scheduler     — decides which node each pod runs on
└─ Controllers   — the reconciliation loops (make actual match desired)

Worker Nodes (where pods run):
├─ kubelet       — the agent; runs pods, reports status
├─ kube-proxy    — networking / Service routing
└─ Container runtime — actually runs containers (containerd)

🚨 etcd is critical — it holds all cluster state via Raft consensus. It’s why etcd health and backups matter enormously, and why the control plane’s availability depends on it. (This is the coordination service you learned about, in its most important deployment.)

Do you actually need Kubernetes?

🚨 The most important judgment, and a strong interview point: Kubernetes is powerful and complex, and most teams adopt it before they need it.

It’s the right tool when:

It’s over-engineering when:

🎙️ The balanced answer: “Kubernetes is the right call at scale with many services and a platform team to run it. For a handful of services or a small team, I’d use a managed platform like Cloud Run or ECS — Kubernetes’ power comes with real operational complexity, and most teams adopt it before the benefits justify the cost.”

Managed Kubernetes (EKS, GKE, AKS) removes the burden of running the control plane, which lowers the bar considerably — but the application-level complexity (YAML, networking, debugging) remains.

⚖️ Trade-offs

  Gain Cost
Kubernetes Self-healing, autoscaling, zero-downtime deploys, portability, one API for everything Steep learning curve; large operational surface; YAML complexity
Managed K8s (EKS/GKE) No control-plane ops Still app-level complexity; cost
Managed platform (Cloud Run/ECS) Much simpler; scales your container Less control; some lock-in
Declarative model Self-healing, reproducible, version-controlled Debugging “why isn’t it converging?” can be opaque

In the real world

🚨 Interview traps

🎙️ Soundbites

🛠️ Try it

1. See the reconciliation loop. In a local cluster (minikube/kind), deploy 3 replicas of a service. Then kubectl delete pod <one> and watch a new one appear within seconds. The pod you killed comes back automatically — that’s the self-healing declarative model, felt directly. Then scale to 5 and watch two more appear.

2. Break liveness vs readiness. Configure a liveness probe that hits a dependency, make the dependency slow, and watch K8s restart the pod repeatedly (a restart loop). Then move that check to readiness and watch it just stop routing traffic instead. This makes the distinction concrete and memorable.

3. Watch a rolling update. Deploy version 1, then change the image to version 2. Watch K8s roll it out pod-by-pod, keeping the service available throughout. Then deploy a broken version 3 and kubectl rollout undo — watch it roll back. → Deployment Strategies

4. Compare with a managed platform. Deploy the same container to Cloud Run (or Render/Fly.io) — one command, no YAML, auto-scaling. Compare the effort to the Kubernetes equivalent. The difference in complexity is the “do you need K8s?” argument, experienced.

Check yourself

1. What is the declarative/reconciliation model and why does it make Kubernetes self-healing? Instead of imperatively scripting *how* to run your system ("start this container, then that one"), you declare the *desired state* — "I want 5 replicas of this service at version 2.3" — and store it in the cluster. Kubernetes controllers run continuous reconciliation loops that watch the *actual* state, compare it to the desired state, and take action to close any gap, forever. This makes the system self-healing because every failure is simply a gap between desired and actual that a controller automatically closes: a crashed container means "4 running, want 5," so the controller starts one; a dead machine means its pods vanish, so they're rescheduled elsewhere; a manual scale-up changes the desired count, so pods are added. You never write recovery scripts — the system perpetually converges toward what you declared, so it recovers from crashes, node failures, and drift automatically.
2. Why do you connect to a Service rather than directly to a pod? Because pods are ephemeral and their IPs change constantly. Kubernetes creates, kills, reschedules, and replaces pods routinely — for scaling, deploys, node failures, and rebalancing — and each new pod gets a new IP. If you pointed at a specific pod's IP, your connection would break every time that pod was replaced, which is constantly. A Service provides a stable, permanent network endpoint (a stable name and virtual IP) in front of a dynamically-changing set of pods; it continuously tracks which pods are currently healthy (via readiness probes) and load-balances traffic across them. So you address the Service (`payments`), and it routes to whatever healthy pods currently back it — solving service discovery and load balancing together, and decoupling clients from the churn of individual pods.
3. What's the difference between a liveness and a readiness probe, and why must a liveness probe not check the database? A **liveness probe** answers "is this container broken and should it be *restarted*?" — if it fails, Kubernetes kills and recreates the pod. A **readiness probe** answers "is this container ready to *receive traffic*?" — if it fails, Kubernetes stops routing traffic to it (removes it from the Service) but does *not* restart it, used during startup (not ready until warm) and transient issues. A liveness probe must not depend on external services like the database because of the failure mode: liveness failure triggers a restart, so if the liveness check queries the database and the database has a brief blip, *every* pod's liveness check fails simultaneously, and Kubernetes restarts *all* of them at once — turning a momentary database hiccup into a mass restart and a self-inflicted outage. Dependency checks belong in readiness (which just pauses traffic, cached and tolerant of transient failures), while liveness should only verify the process itself is functioning.
4. Why can't autoscaling handle sudden traffic spikes or cascading failures? Because it's too slow. The Horizontal Pod Autoscaler observes metrics (CPU, queue depth), decides to add pods, and those pods take seconds to start — pull the image, boot, warm caches and connection pools before they can serve. If more capacity than the existing nodes can hold is needed, the Cluster Autoscaler must provision new machines, which takes *minutes* (cloud VM provisioning, node join, scheduling). A sudden spike or a cascading failure unfolds in seconds — the cascade completes in under two minutes as failing instances shift load to survivors that then also fail. Autoscaling simply can't react fast enough to intervene. It's designed for *gradual* load changes (traffic growing over minutes to hours), where its reaction time is fine. For spikes and cascades you need mechanisms that work instantly: running with headroom (60-70% utilization so survivors absorb load), load shedding, rate limiting, and circuit breakers — not autoscaling.
5. When should you NOT use Kubernetes? When its power isn't worth its operational complexity, which is more often than it's adopted. Specific cases: a small number of services or a monolith, where a managed platform (Cloud Run, App Engine, ECS, Render, Fly.io, Heroku) delivers scaling and deployment with a fraction of the complexity; a small team with no platform/SRE expertise, since Kubernetes has a steep learning curve and a large operational surface (upgrades, networking, RBAC, YAML sprawl, debugging), and running it badly is worse than not running it at all; and when your actual need is just "run my container and scale it," which serverless containers do directly. Kubernetes earns its cost when you have many services to orchestrate, a genuine need for fleet-wide self-healing and zero-downtime deploys, portability requirements, and a dedicated team to operate it. Managed Kubernetes lowers the bar by removing control-plane ops, but the application-level complexity remains — so the honest question is always whether the scale and team justify it, and many teams adopt it prematurely.

Further reading

This site is open source. Improve this page.