field notes KEEL

renew without becoming an outage

Certificate renewal is a periodic job that writes a new file and exits. The running service keeps the old certificate in memory and keeps serving it, so the write is not the delivery. The first version of the automation finished by restarting the service, which is correct and expensive: on a cluster member a restart is an availability event, and a restart on a schedule is the fleet taking itself down politely, on time, forever.

The fix used a capability that was already there. The runtime reloads its credentials on a signal without dropping its listeners. So the procedure is: send the reload, then connect to the service's own encrypted port and compare the fingerprint of the certificate actually being served against the fingerprint of the file just written. Retry briefly. Restart only if what is served is still the old certificate. The check is on the served artifact, because having written a file is not evidence that anything changed.

The cost showed up as a rule. A force-run was halted by the operator because the same script would still have restarted the cluster service on the machines where the reload path had not been proven. Automation holding the ability to restart a member needs an explicit preference order, written down, and someone with the authority to stop it mid-flight.

What it proves is that renewal is a delivery problem, and the last hop, the process that must actually read the new credential, is where the availability risk lives.

drawn from: renewal automation commits

Every note carries a codename and one idea. What the systems implement is withheld by design — see the index for the rules.