When machines must agree on something — who’s the leader, who’s alive, what the config is — you need one place that is definitively right. That place is a coordination service.
Prerequisites: CAP & PACELC, Replication Time to read: ~20 minutes
You have 20 machines. They need to agree on things:
You could store this in your main database. 🚨 But then you inherit its failure modes, you get no notifications when values change (you’d poll), and you have no way to say “this value is owned by that machine, and should vanish if it dies.”
You need a small, extremely reliable, strongly consistent store designed for exactly this. That’s a coordination service.
They are not general-purpose databases. They are deliberately small and deliberately slow:
| Property | Why |
|---|---|
| Strongly consistent (linearizable) | Two nodes must never disagree about who the leader is. → Consistency |
| CP, not AP | During a partition, the minority side refuses to answer rather than risk divergence. → CAP |
| Consensus-based | Every write goes through Raft/ZAB/Paxos and a majority quorum. → Consensus |
| Small data only | Kilobytes to a few hundred MB total. Not for application data. |
| Watches / notifications | Clients subscribe to changes instead of polling |
| Ephemeral state | Data tied to a client session, auto-deleted when the session dies. The key primitive. |
📐 They are slow by design. A write requires a majority of nodes to acknowledge — typically 1,000–10,000 writes/second for a whole cluster, versus tens of thousands for a single Postgres. That is the correct trade: you’re buying agreement, and agreement costs round trips.
🚨 Therefore: never put application data in one. A coordination service holds metadata about your system, not the system’s data. Using etcd as a key-value store for user profiles is a real mistake people make, and it will fall over.
This is the concept to actually understand, because leader election, service discovery, and distributed locks are all built from it.
A client holds a session with the cluster, kept alive by heartbeats. Data it creates as ephemeral exists only while that session lives. If the client crashes, is partitioned, or stops heartbeating, the session expires and its ephemeral data is automatically deleted.
Client A connects, session 0x123
Client A creates ephemeral node /services/api/instance-1 → "10.0.1.5:8080"
... Client A crashes ...
Session 0x123 times out after ~10 seconds
/services/api/instance-1 is automatically deleted
Everyone watching /services/api is notified immediately
🚨 This solves the hardest problem in distributed coordination: cleaning up after a machine that died without telling anyone. A dead machine can’t release its own lock or deregister itself. The session timeout does it for them.
Combine ephemeral nodes with watches (get notified when a path changes) and you can build everything else.
Every candidate creates an ephemeral sequential node under /election/:
/election/node-0000000001 ← client A (lowest sequence = LEADER)
/election/node-0000000002 ← client B
/election/node-0000000003 ← client C
The lowest sequence number wins. Each other node watches the one immediately before it (not the leader — watching the leader would cause a “herd effect” where all N nodes wake at once).
Leader dies → its ephemeral node disappears → node-2 is notified → node-2 is now lowest → node-2 is leader. Automatic, correct, and no split brain, because the cluster’s consensus guarantees only one view of the sequence.
Each service instance registers an ephemeral node with its address. Clients watch the parent path and maintain an up-to-date list.
/services/payments/
├── 10.0.1.5:8080 (ephemeral, session A)
├── 10.0.1.6:8080 (ephemeral, session B)
└── 10.0.1.7:8080 (ephemeral, session C)
An instance crashes → its node vanishes within the session timeout → clients are notified and stop routing to it. No health-check polling required — though you usually keep health checks too, since a process can be alive and heartbeating while being unable to serve.
Same pattern as leader election: create an ephemeral sequential node, lowest sequence holds the lock, watch your predecessor.
🚨 But distributed locks are genuinely dangerous, and this is important enough that it has its own chapter. The core problem: a client can hold a lock, be paused by a GC pause for 30 seconds, have its session expire, and then wake up still believing it holds the lock and write to the protected resource — while another client legitimately holds it now.
The mitigation is fencing tokens: every lock acquisition returns a monotonically increasing number, and the protected resource rejects writes with a token lower than the highest it has seen. Without fencing, your lock is advisory at best. → Distributed Locking
Store config; clients watch for changes and reload without a restart. Strong consistency means every node sees the same value at the same logical time — which matters for things like “the schema version is now 4.”
Which nodes are in the cluster, which shards live where, which node owns which partition. This is what Kafka historically used ZooKeeper for.
| ZooKeeper | etcd | Consul | |
|---|---|---|---|
| Consensus | ZAB | Raft | Raft |
| Language | Java | Go | Go |
| API | Custom (znodes) | gRPC / HTTP | HTTP + DNS |
| Data model | Hierarchical tree | Flat keys with prefixes | KV + service catalogue |
| Watches | One-shot (must re-register) | Streaming | Blocking queries |
| Killer feature | Maturity, battle-tested | Kubernetes uses it | Built-in service mesh + health checks + DNS |
| Used by | Kafka (historically), HBase, Solr, Hadoop | Kubernetes, CoreOS ecosystem | HashiCorp stack, Nomad |
How to choose:
Note on Kafka: modern Kafka has removed the ZooKeeper dependency in favour of KRaft, its own internal Raft implementation. This is part of a broader trend — systems embedding consensus rather than depending on an external coordinator. Knowing this is a good currency signal. → Kafka
Cluster size must be odd, and small. You need a majority to make progress:
| Nodes | Majority | Can survive |
|---|---|---|
| 3 | 2 | 1 failure |
| 5 | 3 | 2 failures |
| 7 | 4 | 3 failures |
🚨 Even numbers are strictly worse. 4 nodes need 3 for a majority, so they tolerate only 1 failure — the same as 3 nodes, with more coordination overhead. Use 3 or 5. Beyond 5, write latency grows (more nodes to convince) with negligible availability gain.
It becomes your most critical dependency. If the coordination service is down, service discovery stops updating, leader elections can’t happen, and locks can’t be acquired. Everything that depends on it degrades.
🎙️ “This makes etcd a hard dependency for the whole platform, so it runs as 5 nodes across availability zones, and services should cache their last-known-good view so a brief etcd outage degrades rather than halts them.”
That caching point is important: clients should tolerate the coordination service being briefly unavailable by continuing with their last known configuration. A service discovery client that stops routing entirely when etcd blips has turned a coordination outage into a total outage.
Session timeouts need care. Too short and a GC pause or a network hiccup causes a spurious failover (disruptive). Too long and a genuinely dead node holds its lock for ages (unavailable). Typical: 10–30 seconds. There’s no correct value, only a trade — and this is exactly the failure detection problem.
Watch out for the herd effect. If 1,000 clients all watch the same key and it changes, you get 1,000 simultaneous notifications and 1,000 simultaneous reads. This is why leader election has each node watch only its immediate predecessor.
⚖️ Coordination services are operationally significant. Before adding one, check whether you actually need coordination:
SELECT ... FOR UPDATE, advisory locks, or a
unique constraint provide locking with a system you already run. For modest scale, this is
frequently the right answer.🎙️ “Rather than a distributed lock, I’d partition the work by user ID so each worker owns a disjoint set — that removes the need for coordination entirely, which is more robust than making the coordination correct.”
Designing coordination away is better than designing it well. That’s a genuinely senior framing.
| Decision | Gain | Cost |
|---|---|---|
| Use a coordination service | Correct agreement, automatic failure cleanup | A critical dependency; operational burden; low write throughput |
| 3 nodes vs 5 | Faster writes, cheaper | Survives only 1 failure |
| Short session timeout | Fast failure detection | Spurious failovers from GC pauses |
| Long session timeout | Stable | Dead nodes hold locks longer |
| Database locks instead | One less system | Doesn’t scale; no automatic session cleanup; no watches |
| Partition instead of coordinate | No coordination at all | Requires the work to be partitionable |
1. Run a 3-node etcd cluster in Docker Compose. Then:
etcdctl put /config/feature_x true
etcdctl watch /config/ --prefix # in another terminal
etcdctl put /config/feature_x false # watch it fire
2. See CP behaviour directly. Pause one container (docker pause) — the cluster still works
(2 of 3 is a majority). Pause a second — writes now fail. This is the CP choice made visible, and
it’s a much better teacher than any explanation.
docker pause etcd2
etcdctl put /test hello # still works
docker pause etcd3
etcdctl put /test world # fails: no quorum
3. Build leader election. Use etcd’s lease + campaign API (or ZooKeeper’s ephemeral sequential nodes). Run three instances. Kill the leader. Watch a new one take over, and time how long it takes — that duration is your session timeout, and it’s your failover MTTR.
4. Demonstrate the lock danger. Acquire a lock, then kill -STOP the process for longer than the
session timeout, then kill -CONT it. The process wakes up still believing it holds a lock it lost.
Now implement fencing tokens and watch the stale write get rejected.