a scheduled job runs somewhere, whether you said where or not
A periodic job is two contracts, and only one of them gets written down. The first is what it does. The second is where it is allowed to do it, and that one stays implicit in the job's body: it reads a directory on a local disk, or a socket on the machine, or a copy of its own script that was installed on exactly one host. The scheduler does not read the script. It picks a machine.
Most of the estate's jobs carried an explicit placement constraint for precisely this reason. Three did not. One of those ran on a machine that had neither the script nor the directory, failed immediately, and produced nothing. The failure was found the next day by looking, not by being told, because a scheduled job that fails has no surface at all: its whole output was the artifact that never appeared.
The fix added the constraint the three were missing, placed by a stable machine label rather than by name so a rebuild cannot silently move the job, then dry-ran the change, then forced a single run and confirmed the artifact existed with the expected owner and mode. Not that the job completed. That the file was there.
The cost is a line per job and the near-certainty that the next job omits it. The version that holds encodes placement in whatever the jobs are generated from, so the constraint is inherited instead of remembered. What it proves is that reliability for scheduled work is not whether a job runs, it is whether the job can succeed where it lands, and the only dependable detector is something that checks for the output.
drawn from: periodic job placement commits
Every note carries a codename and one idea. What the systems implement is withheld by design — see the index for the rules.