Leader election is easy, failover is where systems die

A scheduled job runs on whichever instance holds the leader lock. The lock has a 15 second TTL, renewed every 5 seconds, which is a standard and reasonable configuration.

One instance experiences a 22 second stop the world pause. The lock expires, a second instance acquires it, and starts the job.

The paused instance resumes. From its perspective nothing happened: it has a lock object in memory that says it is the leader, its last renewal succeeded, and no error was ever returned. It continues from exactly where it left off.

Both instances now run the job. Both write. Neither is wrong from the inside.

Election is the easy half

Getting a cluster to agree on one leader is genuinely solved. Raft, ZooKeeper, etcd and Kubernetes leases all do it correctly, and the algorithms have been analysed to death. If you are implementing leader election from scratch you are almost certainly making a mistake, and that is not the interesting failure.

The interesting failure is that leadership is a claim about the present, and every mechanism for revoking it is asynchronous.

A cluster decides a leader is dead by not hearing from it. That inference is correct when the process crashed and wrong when the process is merely unreachable or paused, and there is no way to distinguish the two from the outside. That is the core of it: failure detection over an asynchronous network cannot separate slow from dead.

sequenceDiagram
    participant O as Old leader
    participant Z as Coordination service
    participant N as New leader
    participant D as Database

    O->>Z: renew lease
    Note over O: 22 second GC pause begins
    Z->>Z: lease expires after 15s
    Z->>N: you are the leader now
    N->>D: write (believes it is leader)
    Note over O: pause ends
    O->>D: write (also believes it is leader)
    Note over D: two writers, no error anywhere

The old leader is not malfunctioning. It has stale information and no mechanism for learning that until it next talks to the coordination service, which might be after it has already written.

Timeouts cannot fix this from the leader’s side

The obvious remedy is for the leader to check its lease before acting.

// Looks careful. Is not sufficient.
if (lock.isStillHeld()) {
    database.write(record);
}

The gap between the check and the write is where the pause happens. Making the gap smaller makes the window smaller and never closes it, because the pause can occur at any instruction boundary including between those two lines.

Checking the remaining lease time and refusing to act when it is short helps a little and has the same structural flaw: the process cannot detect a pause that has not happened yet.

This is the same class of problem as relying on wall clock timestamps to order events across machines. Any solution phrased in terms of “if enough time has passed” depends on a clock that the pausing process does not control.

Fencing moves the check to the resource

The mechanism that works is to make the write itself carry proof of leadership, and to have the resource enforce ordering.

Every leadership grant issues a monotonically increasing token. The holder sends it with every write. The storage layer remembers the highest token it has seen and rejects anything lower.

// Granted at election time, strictly increasing across grants.
long fencingToken = lock.acquire();     // e.g. 33

// Every write carries it.
database.write(record, fencingToken);

-- The resource enforces the ordering. A stale leader fails deterministically.
UPDATE job_state
SET last_run = now(), fencing_token = :token
WHERE job_id = :id AND fencing_token < :token;

Zero rows updated means a newer leader has already written and this process is stale. That is a definitive answer available at the moment of the write, not an inference from elapsed time.

ZooKeeper gives you this for free: the zxid on a znode increases monotonically and can be used as the token. etcd exposes the revision. Kubernetes leases carry a resourceVersion. The primitive usually exists and is usually ignored, because the lock API looks sufficient on its own.

The awkward part is that the resource has to participate. A database can check a column. An object store with conditional writes can check an etag. A third party API that has no notion of your token cannot, and for those the honest answer is that fencing is unavailable and the operation needs to be idempotent instead, so a duplicate from a stale leader is harmless rather than prevented.

Failover has costs that are easy to forget

The window between the old leader stopping and the new one being useful is longer than the election.

Detection takes at least the failure timeout, which is bounded below by how tolerant you are of false positives. A 5 second timeout detects failure quickly and declares healthy leaders dead during ordinary pauses. A 30 second timeout is stable and means half a minute of no leader.

Election itself is fast, typically well under a second with a healthy quorum.

Then the new leader has to become useful: load state, rebuild caches, re-establish connections, and catch up on whatever the old leader was partway through. That last part is usually the longest and it is the part nobody measures, because the metric people watch is time to elect rather than time to serve.

Split brain is not the only failure either. Two nodes each getting half the votes means no leader at all, which is why the node count should be odd and why the definition of a majority matters as much as the election itself.

What I would verify

Test failover by pausing rather than killing. SIGSTOP on the leader process, wait past the lease TTL, then SIGCONT. A clean kill exercises the easy path where the old leader is genuinely gone; a pause exercises the path where it comes back believing it is still in charge, which is the one that produces incidents.

Check whether writes carry a fencing token at all. If the only protection is a lock object in application memory, the system is relying on the absence of pauses, and pauses are not optional.

Make leadership-guarded operations idempotent regardless. Fencing stops the stale write at the resource; idempotency means that if one slips through some path you did not fence, the damage is a duplicate rather than corruption. Belt and braces is the correct posture here because the failure is silent when it happens.

And measure time to useful rather than time to elect. A cluster that elects a new leader in 200 milliseconds and then spends 40 seconds warming up has a 40 second outage, and every dashboard will report the 200 milliseconds.