field notes
Short notes from real engineering work: one decision, one incident, one reversal at a time. Each one names the shape of the problem, the tension that made it hard, the decision, and what it cost. The internals stay out — the codename is the hint.
If you have five minutes rather than an afternoon: the three case studies condense this material to one problem, one decision and what the decision cost. The notes below are the record those studies cite — and the projects the codenames belong to are on the front page.
Every risk document has a worst-case line, and that line is usually taken at face value. One candidate's single worst day was an extreme outlier, far outside everything around it. It would have been easy either to discard it as noise or to…
Two candidates were put through the full gate set. Each had satisfied a progression condition that had been set as a required precondition, and satisfaction of that condition was widely expected to be the deciding factor. Both were rejected…
A campaign sweeps a large grid of configurations and ranks them on a computed score. Near the top of the grid, the score stops being informative, because the quantity being scored is bounded — a proportion cannot exceed one — and any design…
A candidate had been admitted into a collection on the strength of a strong measured result. Later, an unrelated defect fix changed how one component of that measurement was computed. Re-measured, the candidate did not merely weaken: the…
A large sweep produced a map of candidate configurations, and the map was produced once, on one machine, by one program. That is exactly the situation in which a subtle dependency on the machine — an ordering, a threading choice, a…
Before a decision stream is allowed to feed anything live, it has to pass a registration gate. The interesting part is not that the gate checks the output is well formed. It is that the gate tries to show the output could have been produced…
An archive had stopped at a fixed date, and a study needed it to continue past that date. The dangerous move is the obvious one: rebuild the whole series from new inputs and accept whatever comes out, because then every earlier result…
A quality filter had been in place since the archive was assembled, and it was simple: keep a value only if it does not differ from the last value kept by more than a declared bound. That filter carries an assumption nobody had written down…
A family of published results had been computed on a cost basis assumed to be the right one. A check of what was actually on disk showed otherwise: the cost surface underneath came from a third-party historical source that covered only a…
An independent reviewer was asked a deliberately unfriendly question about the decision log: not whether it recorded the right outcomes, but whether a reader could tell from the record alone why nothing happened at a given moment. The…
Restarts are where a live system loses its argument with itself. Nothing crashes; the process returns, reads its configuration, and begins the day fresh — except that the day did not begin when the process did.
A condition can be written two ways that are equivalent for every number and behave differently for the value that is not a number. The reference implementation asked whether the magnitude of a statistic had reached a threshold, written…
The requirement was absolute: a replayed stream had to be byte-identical to what the live system emitted, or nothing built on the replay could be believed. It nearly was. A full-scale comparison across millions of records showed a small…
A live path produced nothing for a stretch of hours. The obvious readings were that there were no qualifying candidates, or that the feed was broken. Both were wrong, and the way the truth was found is the useful part: rather than guessing,…
Every guard added to a live path is itself a risk. A guard that misfires under load takes the system down more reliably than the fault it was built for. So a new live cost gate was added the other way round from the obvious one: it ships…
The archive had to be pulled incrementally — a day at a time, resumable, verifiable — and the only description of the wire encoding was that it existed. Published notes covered the container and the field order. They did not cover how the…
The archive is the ground truth for every later measurement, so the way it fails matters more than the way it works. The failure found was quiet. The upstream service would occasionally answer a data request with a successful, well-formed,…
An independent reviewer went through a family of published result tables looking for internal inconsistency rather than wrong arithmetic. They found one that mattered. The worst-case column carried no sign. A minimum had been formatted…
Any interesting question in a long research programme can be asked a dozen ways. Ask it a dozen ways and one will look good; that is arithmetic, not discovery. The programme's answer costs a little time every single day: a measurement is…
The system had a number it liked. It came from a cost model assembled out of a small table of assumed per-transaction costs — reasonable when written, never checked against what the system actually paid. Every optimisation downstream…
method
Not a changelog entry and not a repository summary. One decision, one incident, one reversal, or one hard-won invariant — the thing an engineer would tell you at a whiteboard.
Prefer records with dates and root causes? Those are on incidents. Live numbers are on stats.