field notes
Short notes from real engineering work: one decision, one incident, one reversal at a time. Each one names the shape of the problem, the tension that made it hard, the decision, and what it cost. The internals stay out — the codename is the hint.
If you have five minutes rather than an afternoon: the three case studies condense this material to one problem, one decision and what the decision cost. The notes below are the record those studies cite — and the projects the codenames belong to are on the front page.
The first service migration looked like a copy and a connection string: take a database out of the application's container, run it as a supervised process on the host, point the application at it. The move took far longer than that, and…
A machine whose remote-access service stops accepting connections is not down. It is running, hosting its workloads, answering everything that asks about its health, and unreachable through the only door its operator uses. That is the shape…
When new machines are built to replace an old one, the old one becomes the reference, and the comparison that matters is not between files. A parity audit set the retiring member against the freshly built replacements, and the configuration…
A periodic job is two contracts, and only one of them gets written down. The first is what it does. The second is where it is allowed to do it, and that one stays implicit in the job's body: it reads a directory on a local disk, or a socket…
The root volume was filling, and the obvious reading was that the machine needed more disk. The correct reading was that the machine had been arranged badly. The root filesystem was copy-on-write with duplication enabled, which buys…
Everything else trusts the identity service, which makes it the worst place to be uncertain. An audit found two kinds of uncertainty. The service was not recording sign-in events or administrative events, so the question of who changed what…
Every other service in the estate is used through software in this estate. Mail is used through a stranger's client, which means a defect arrives as "it doesn't work" from someone who cannot explain why, while the server's own logs say…
An alerting system has one job and several dependencies: something evaluates a rule, something delivers the result, something transports it, something makes a noise. Every link can be empty while the previous link reports success, and…
The convenient shape for a secret store is one credential that reads everything, held by every machine. The cost is that any machine becomes a copy of the whole keychain, and one mistaken read becomes indistinguishable from a legitimate…
The ordinary way to obtain a certificate for a name is to let the outside world fetch a token from the address that name resolves to. It is the path with the fewest moving parts, and it assumed something that stopped being true: that the…
Certificate renewal is a periodic job that writes a new file and exits. The running service keeps the old certificate in memory and keeps serving it, so the write is not the delivery. The first version of the automation finished by…
Routing used to be a file per host, with the address of every backend written into it. That works until a service moves, and then the file is wrong in a way nothing reports. The migration replaced those entries with a directory that…
A proposal arrived to make the database highly available: two copies, automatic failover, the standard shape that appears in every design document. The review before building it asked a narrower question than the proposal did. What failure…
The declarative path existed and lagged the imperative one, in the ordinary way: a change typed by hand at the moment of need, a repository updated later, or not at all. Most changes were still typed by hand, and the written record said…
There are three ways a check earns silence without earning confidence. The first is that it does not exist. Alerting covered machine health — processor, disk, memory — and nobody had asked whether the service people actually used was up, so…
A backup job is a producer with no consumer. Nothing in the system notices when the archive is empty. The job either exits zero, and the emptiness is invisible, or it fails into a log nobody reads, and the schedule keeps firing regardless.
Per-port rules look like security and behave like a promise to remember every port. Workloads get new ports when they are rescheduled, and the rule list does not grow with them, so the estate accumulates silent holes that nobody wrote down.
Many reporters, one writer. Each machine is allowed to say what it believes about itself; exactly one process is allowed to turn those statements into records. With several writers, two orderings of the same announcements produce two…
Leadership moves, and anything that hard-codes where leadership was becomes a fact somebody has to remember at the worst possible moment. A name that always resolves is not a reliable name; it is a name that can be confidently wrong, and…
For a long time the honest name for this estate was staging, and the word licensed sloppiness: try it, leave it running, fix it later. Calling it permanent changed nothing about the hardware and a great deal about the decisions. Permanent…
method
Not a changelog entry and not a repository summary. One decision, one incident, one reversal, or one hard-won invariant — the thing an engineer would tell you at a whiteboard.
Prefer records with dates and root causes? Those are on incidents. Live numbers are on stats.