one condition, two consumers

A small service here does two jobs on one timer. It restates, every cycle, that its node is still the one in charge, because the record it maintains expires if it goes quiet. It also tells a human when that status changes. For a long time both jobs were driven by the same branch, which was invisible until the condition was widened: the node began re-announcing every cycle instead of only on a change, and the human-facing half went with it. The result was a page every few seconds, for as long as the condition held, about a system that was working exactly as intended.

The mistake is a category error rather than a coding one. Liveness and notification look like the same event because the same fact triggers both, and they have opposite requirements. The first must be frequent, repetitive and boring; the second must be rare, meaningful and impossible to ignore. Any condition serving both will eventually satisfy one at the other's expense.

The fix split the branch. Re-announcement continues every cycle and is deliberately noisy; the human alert fires only on an actual change of leadership. A timeout on the outbound call means an unresponsive hub can delay a cycle but never stall it.

There is a second, quieter lesson in the same record: the running build contained a change that had never been committed, and the changelog says so rather than rebuilding over the gap.

What it proves is that automation which pages people needs a different trigger from automation which stays alive, and that admitting which build is running is part of operating either.