field notes
Short notes from real engineering work: one decision, one incident, one reversal at a time. Each one names the shape of the problem, the tension that made it hard, the decision, and what it cost. The internals stay out — the codename is the hint.
If you have five minutes rather than an afternoon: the three case studies condense this material to one problem, one decision and what the decision cost. The notes below are the record those studies cite — and the projects the codenames belong to are on the front page.
A small service here does two jobs on one timer. It restates, every cycle, that its node is still the one in charge, because the record it maintains expires if it goes quiet. It also tells a human when that status changes. For a long time…
A register of written decisions, checked automatically against the machinery meant to implement them, produces one of two useless results unless you are careful. Either it reports dozens of gaps that are not defects at all — decisions taken…
The pipeline's first step took the disks out of their array. Everything else depends on that step, and it had an effect nobody had written down: the running system's root lived on those disks, so removing them from the array left the root…
Two identical disks, one operating system, and a naming scheme assigned by the order in which the hardware answers at boot. This is the ordinary condition of a machine, and it means every script that names a disk is guessing. The guesses…
One component was removed from a machine's definition outright, along with the packages that came with it. The change was small, made sense at the time, and survived about a day. Two commits later the same component was back, with two…
Reproducible builds assume an input set larger than the thing being built. Here that input set — the entire closure of what a machine needs, kept so a rebuild does not fetch the world again — lives on one host, and the build runs on that…
The last stage of building a machine image is giving it to whoever will install it. Here the installer is the hosting provider's own automation, running against hardware that has not been commissioned, holding no credentials of ours, and…
A configuration repository here contained, inside itself, a second copy of a repository. Everything looked right: the directory had files in it, the machine ran, the definitions loaded. But the history did not record the files. It recorded…
A cluster has to be founded exactly once, and the founding step cannot tell whether it is first or merely early. Here the first node would bring up the group's own membership rules and access policy before the other nodes could be reached —…
Installation scripts are written on one machine and run on another, and the difference is rarely written down. On this hardware two identical disks present themselves in an order that differs between installations, so a script that names a…
Replacing a machine's operating system means writing over the only thing that knows how to start it. The boot configuration is not part of the image being installed; it belongs to the machine and to the firmware that reads it. In this work…
An installation image is a snapshot of a system taken at a moment nobody records, and the machine that downloads it inherits that moment. The first thing the bootstrap does here is move the installed package set forward to current, which is…
method
Not a changelog entry and not a repository summary. One decision, one incident, one reversal, or one hard-won invariant — the thing an engineer would tell you at a whiteboard.
Prefer records with dates and root causes? Those are on incidents. Live numbers are on stats.