the watchdog watched the wrong thing

A producer published fresh data by swapping the name that points at the current generation, and its consumers held a handle open to it. A staleness check inspected the identity of that handle in order to detect the swap. But a handle's identity is fixed when it is opened and never changes afterwards, so the check could not fire. Not rarely: never.

The consequence was not a crash. The engine stayed up, stayed polling, and kept reading a generation that had already been replaced. Decisions formed during the stall were emitted only when something else restarted the process, minutes late, and were then refused for being late. The supervisor's view said healthy throughout, because health had been defined as the process existing.

The same shape appeared later on the other side of the system: a follow-on engine stranded on a stale generation for most of a working day, emitting nothing, while the layer above re-read the same frozen stream on every pass and logged thousands of rows an hour describing work that changed nothing. Two processes, both fine, both idle, both invisible.

The fixes were small and structural: watch the name rather than the handle, and impose a hard periodic re-warm so recovery does not depend on any watcher surviving to make the call. What it proves is that the team distinguishes liveness from progress, and treats a silent stall as a defect to be found rather than weather to be waited out.