Mutual exclusion across machines. It looks like a mutex and it isn’t one — and the difference has caused a lot of corrupted data.
Prerequisites: Coordination Services, Idempotency Time to read: ~22 minutes
Ten worker instances. Exactly one should process this job / write this file / send this invoice.
On one machine, a mutex solves it. The OS guarantees that only one thread holds it, and if the thread dies the OS knows.
Across machines, there is no OS. There is a network that drops messages, processes that pause without warning, and clocks that disagree. Every guarantee a local mutex gives you is gone.
# ❌ Broken in three ways
if not redis.get("lock:job-42"):
redis.set("lock:job-42", "mine")
do_work()
redis.delete("lock:job-42")
Bug 1 — not atomic. Two clients both GET nothing, both SET, both proceed. Fix: SET NX,
which is a single atomic operation.
Bug 2 — no expiry. The holder crashes, and the lock is held forever. Nobody can ever run the job
again. Fix: SET NX EX 30.
Bug 3 — anyone can delete anyone’s lock. Client A’s lock expires, client B acquires it, then A finishes and deletes the lock — releasing B’s lock. Fix: store a unique token and only delete if it matches, atomically (which requires Lua, because check-then-delete isn’t atomic).
# Better — but still not correct, see below
token = str(uuid4())
if redis.set("lock:job-42", token, nx=True, ex=30):
try:
do_work()
finally:
# Atomic compare-and-delete
redis.eval("""
if redis.call('get', KEYS[1]) == ARGV[1]
then return redis.call('del', KEYS[1]) else return 0 end
""", 1, "lock:job-42", token)
That’s the correct implementation of a broken idea. Here’s why.
This is the part that matters, and it’s the thing most engineers don’t know.
t=0 Client A acquires the lock, TTL 30 seconds.
t=1 Client A begins the work.
t=2 Client A's process is STOPPED — a long GC pause, CPU starvation,
a VM being live-migrated, or the container being throttled.
t=35 The lock expires. Redis deletes it.
t=36 Client B acquires the lock legitimately and starts working.
t=40 Client A RESUMES. It has no idea any time passed.
It finishes its work and writes to storage.
TWO CLIENTS WROTE. The lock was correctly implemented and did not help.
🚨 A lock cannot pause a process that has already passed the check. There is no mechanism by which Redis can reach into client A and stop it. A stop-the-world GC pause of 30+ seconds is unusual but entirely real, and a throttled container can be starved for far longer.
This means: a distributed lock alone is never a correctness guarantee.
Every lock acquisition returns a monotonically increasing number. Every write carries it. The protected resource rejects any write with a token lower than the highest it has seen.
t=0 A acquires the lock → token 33
t=35 Lock expires
t=36 B acquires the lock → token 34
t=37 B writes with token 34 → storage records highest = 34 ✅
t=40 A resumes, writes with token 33 → storage REJECTS (33 < 34) ✅
🚨 The critical requirement, and the most commonly missed part: the storage layer must enforce the check. If it accepts whatever it’s given, the token is decorative. This is the same principle as epoch numbers in leader election.
In practice you often get fencing for free from something you already have:
| Mechanism | How it fences |
|---|---|
| Conditional write | UPDATE ... WHERE version = 7 — a stale writer’s version doesn’t match |
| Unique constraint | The second insert simply fails |
ZooKeeper zxid |
Monotonic transaction ID, usable as a fence |
| etcd revision | Same idea |
| S3 conditional writes | If-Match on an ETag |
🎙️ “I’d use a lock to avoid duplicate work, but I wouldn’t rely on it for correctness — a GC pause longer than the TTL means two holders. The actual guarantee comes from a conditional write with a version check, which fences the stale writer.”
Redlock is Redis’s algorithm for locking across N independent Redis instances: acquire from a majority, with the acquisition timed to ensure enough TTL remains.
🚨 Martin Kleppmann’s critique (2016) is one of the most-cited exchanges in distributed systems, and knowing the shape of the argument is a real signal:
Antirez (Redis’s creator) responded defending the algorithm under stated assumptions, and the exchange is worth reading.
⚖️ The practical takeaway, and the useful framing:
Ask what the lock is for. If it’s an optimization (don’t do the work twice), a simple lock is fine and Redlock is overkill. If it’s for correctness (must never happen twice), a lock alone is insufficient and you need fencing.
🚨 The best distributed lock is the one you didn’t need. Every one of these is a better answer than a better lock:
If running the operation twice is harmless, you don’t need mutual exclusion.
INSERT INTO invoices (customer_id, period, amount)
VALUES (42, '2026-07', 150000)
ON CONFLICT (customer_id, period) DO NOTHING;
Two workers race; one inserts, the other no-ops. No lock, no TTL, no fencing, no GC-pause problem. → Idempotency ⭐
If each worker owns a disjoint set of keys, nothing is shared and no exclusion is needed.
❌ Ten workers competing for a lock on each job
✅ Worker N handles jobs where hash(job_id) % 10 == N
If the work touches one database anyway, use its transactional facilities:
-- Atomically claim a job; other workers skip it without blocking
SELECT * FROM jobs WHERE status = 'pending'
ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT 1;
🚨 FOR UPDATE SKIP LOCKED is excellent and underused. It gives you a correct work queue with no
external lock service, and the lock is genuinely tied to the transaction — if the worker dies, the
transaction aborts and the row is released by the database, with no TTL guesswork.
Postgres advisory locks (pg_advisory_lock) are the other option, tied to a session.
Don’t lock; detect. UPDATE ... WHERE version = 7 fails if someone else changed it.
→ Transactions
Route all writes for a given entity through one owner — a Kafka partition consumer, a shard owner, an actor. No two writers means no lock.
Use a consensus-backed store, not Redis, when correctness matters: etcd, ZooKeeper, or Consul. They’re built for this, they provide sessions with automatic cleanup, and their revision numbers give you fencing tokens for free. → Coordination Services
Choose the TTL carefully.
Too short → the lock expires mid-work → two holders
Too long → a crashed holder blocks everyone for the full TTL
📐 There’s no correct value. Set it to several times your p99 work duration, and extend it via a heartbeat for long operations (a background thread renewing the lease every TTL/3). Note that a heartbeat thread can itself be starved by the same GC pause — which is precisely why fencing is still required.
Always release in a finally block, and always with the atomic compare-and-delete so you can’t
release someone else’s lock.
Set an acquisition timeout. Waiting indefinitely for a lock turns a lock-service problem into an application-wide hang.
Monitor lock wait time and contention. A lock that’s frequently contended is a scalability bottleneck — it’s a serial section, and Amdahl’s law applies.
| Approach | Gain | Cost |
|---|---|---|
Redis lock (SET NX EX) |
Fast, simple, one line | Not safe for correctness; TTL guesswork |
| Redlock | Survives a Redis instance failure | Complex; still no fencing; disputed assumptions |
| etcd / ZooKeeper lock | Session-based cleanup; fencing tokens available | Slower; a consensus cluster to operate |
FOR UPDATE SKIP LOCKED |
Correct; no external service; released on abort | Only within one database; contention on the table |
| Fencing tokens | Actual correctness | The protected resource must enforce it |
| Idempotency instead | No lock at all; no failure modes | Requires a natural key or a design change |
| Partitioning instead | No coordination whatsoever | Requires partitionable work |
SKIP LOCKED underpins a whole generation of database-backed job queues (Que, Solid
Queue, GoodJob, River). It’s a good example of the right answer being “use what you already have.”GET then SET instead of SET NX).SELECT ... FOR UPDATE SKIP LOCKED in Postgres — correct, no external
lock service, and the lock is released automatically if the worker’s transaction aborts.”1. Demonstrate the GC-pause problem. This is the exercise that makes the chapter stick:
# Terminal 1 — acquire a lock with a 10s TTL, then simulate a pause
python worker.py # acquires lock, prints PID, then sleeps
kill -STOP <pid> # freeze it — like a GC pause
# Terminal 2 — wait 11 seconds, then run a second worker
python worker.py # acquires the lock legitimately, starts working
# Terminal 1
kill -CONT <pid> # it resumes and writes, still believing it holds the lock
Two writers. Correct lock implementation. Then add a fencing token and a version check on the target and watch the stale write get rejected.
2. Break the naive implementation three ways. Implement GET-then-SET and race two clients.
Implement it without a TTL and kill the holder. Implement release without a token check and have one
client release another’s lock. Each takes two minutes and each is a real bug you’ll now recognize.
3. Build a job queue with SKIP LOCKED. Ten concurrent workers against one jobs table. Confirm
no job is processed twice. Kill a worker mid-transaction and watch its job become available again
immediately — no TTL involved.
4. Compare etcd and Redis. Implement the same lock with both. Kill the holder and observe the difference: Redis waits for the TTL; etcd’s session expires and cleans up, and you get a revision number you can fence with.
SELECT ... FOR UPDATE SKIP LOCKED often better than a Redis lock for a job queue?