field notes KEEL

the second path, present on every machine by construction

A machine whose remote-access service stops accepting connections is not down. It is running, hosting its workloads, answering everything that asks about its health, and unreachable through the only door its operator uses. That is the shape of the problem: the shortest path in is gone, and the alternatives all cost a reboot, a console session, or a request to somebody with physical hands.

The repair used a channel that already existed on the machine. The workload scheduler delivers work to it by definition, so a single small command can be submitted to that one machine and run as it, in this case restarting the access service. The command ran, the listener came back, the machine answered a connection from a different machine inside the same session, and uptime was never interrupted.

Two conditions make that legitimate rather than lucky. The command has to be small and reversible: restart a service, do not repair a system, because there is no interactive feedback channel while you work. And the result has to be confirmed from somewhere else, a connection from another machine rather than a job status reporting success. A job that reports success is reporting on itself.

The cost is that any second path is also a second way in. This one inherits the trust the scheduler already holds over every workload on that machine, so it has to be scoped like that rather than treated as an emergency exception that quietly survives into ordinary operation. What it proves is that recovery tooling is the discipline of building paths that do not depend on the path you are trying to repair.

drawn from: out-of-band repair commits

Every note carries a codename and one idea. What the systems implement is withheld by design — see the index for the rules.