Leases and fencing tokens: locks that expire safely

A distributed lock that never expires deadlocks when its holder crashes; one that does expire can be held by two processes at once — fencing tokens close that gap.

The dilemma of a distributed lock

You want exactly one node to act — one leader writing, one worker processing a job. A lock gives you that. But across a network it has a nasty dilemma:

A lock with no expiry deadlocks. If the holder crashes while holding it, the lock is never released and the resource is frozen forever.
A lock that expires can be double-held. Add a timeout so a crashed holder’s lock auto-releases — and now you risk two processes believing they hold it.

A lease is the standard answer to the first problem; fencing is the answer to the new problem the lease creates.

The lease

A lease is a lock with a time-to-live. The holder must renew it (heartbeat) before it expires to keep holding it. If the holder dies, it stops renewing, the lease lapses, and someone else can take over — no manual cleanup, no permanent deadlock. This is how leader election, lock services, and DHCP all hand out time-bounded ownership.

The danger: a paused holder

The lease assumes “not renewing” means “dead.” But a process can freeze and come back — a stop-the-world GC pause, a VM migration, a long syscall. While it’s frozen its lease expires, a second process acquires the lease and starts working, then the first one wakes up still believing it holds the lock and writes:

Holder A:  acquire(lease, TTL=10s) ──work── [GC pause 12s] ─────────── write! (thinks it owns the lock)
                                       lease expires at 10s  │
Holder B:                                        acquire ────┼── work── write
                                                             ▼
                                          both A and B write → corruption

A timeout alone can’t prevent this — A was never actually dead, just slow, and you can’t tell slow from dead (the heartbeat lesson’s core idea).

Fencing tokens

The fix lives at the resource, not the lock. Each time the lock is granted, hand out a monotonically increasing token (1, 2, 3, …). Every write to the protected resource carries its token, and the resource rejects any token less than or equal to the highest it has already seen:

A acquires -> token 33, starts writing slowly
B acquires -> token 34 (A's lease expired)
B writes with 34 -> storage records highest = 34
A wakes, writes with 33 -> 33 <= 34 -> REJECTED

The stale holder is fenced off — it physically cannot corrupt state, even though it still thinks it’s in charge. The token must be enforced by the downstream store, which is the part people forget.

Where it shows up

ZooKeeper / etcd / Chubby lock services hand out fencing via zxid or revision numbers; Raft uses a leader epoch/term as a fence; Kafka uses a leader epoch so a stale partition leader’s writes are rejected. The famous critique of naive Redlock is exactly the missing fence.

The interview cue

When your design needs mutual exclusion across nodes, don’t stop at “we take a lock.” Say: “I’d use a lease with a TTL so a crashed holder auto-releases, and hand out a fencing token on each acquisition that the resource checks — so if a GC-paused holder wakes up after its lease expired, its stale write is rejected instead of corrupting state.” Naming the paused-holder hazard and the fencing fix is a strong distributed-systems signal.