field notes
Short notes from real engineering work: one decision, one incident, one reversal at a time. Each one names the shape of the problem, the tension that made it hard, the decision, and what it cost. The internals stay out — the codename is the hint.
If you have five minutes rather than an afternoon: the three case studies condense this material to one problem, one decision and what the decision cost. The notes below are the record those studies cite — and the projects the codenames belong to are on the front page.
One component in the live path answered operations with a code that meant nothing. The counterparty accepted the request, carried it out, and returned a failure. The truth was visible in that counterparty's own journal, where an accepted…
A replay was run to confirm that a change behaved as intended. The replay invoked the ordinary daily routine, and the routine began with a sweep: every entry not opened by the current pass was, by that definition, a day old and due to be…
Opening a position and attaching its protection were two separate requests. The gap between them was real, of seconds order, and entirely unmeasured: nothing in the system described it, so nothing bounded it. When the question was finally…
A protection level was attached to an open position and then read back to confirm it had landed. The read-back compared the level the system had sent against the level the counterparty reported, and demanded that the two be equal. The…
A wave of work was split across parallel workers so that several implementation tasks would land at once. Both of the slots doing the primary work died on their first steps, having exhausted their working context before producing a change.…
Large generated tables had been committed alongside the code that produced them, repeatedly, over months. Nothing about any single commit was wrong. The aggregate was: the repository had grown to gigabytes, most of it result artifacts that…
Before a decision that could be taken back was taken, the taking-back was built. Two manifests were prepared, each pinning a previously committed configuration byte for byte, each applied as a single command that ran a fixed sequence: back…
A filter that decides which candidates reach the live path had been fitted offline, in one language, against historical data. It had to run in the live path, in another. Between the fit and the decision sits a translation, and a translation…
A rule was proposed that sounds like discipline: keep an execution only if its measured cost is within a small multiple of the cost the model assumed for it, and drop the rest. The premise is that costlier-than-modelled work is work nobody…
Fitting a configuration to history and then reporting what it did is a test of memory, not of method. The programme's answer is a blocked holdout: the span is cut into contiguous blocks, the tuning sees only part of it, and the remainder is…
Months of results carried a caveat that had not been checked. A stretch of the input in one season looked thinner than the rest, and thin input is a comfortable explanation for anything measured over that stretch: it explains a bad result…
A holdout run produced a headline that split cleanly by era: strong in one year, weak in the other. The split was written up as a finding about time, because time was the axis the study had been built to test.
A gate that asks whether a result is bigger than nothing is asking the wrong question when the result was chosen as the largest of many. The maximum of a set of random draws is not zero and is not small: it sits predictably some distance…
The platform reports, about each account it holds, whether that account is live. The programme had been treating the flag as a fact about the world. It is not: one of the accounts it was pointed at was the rehearsal environment with the…
Two failures landed in the same pass, and they were the same failure twice. A guard meant to permit one open item per member never blocked anything, because the member identifier was derived from a text label by stripping a prefix and the…
A live path had placed nothing since it started, and the question was not whether it was broken but whether silence was still a legitimate answer. Two explanations fit the same observation: the conditions had not occurred, or they had…
A pass that was ready to act on its decisions was given a hold it could not talk itself out of. Until a per-account setting carries an explicitly signed value, every pass skips its actions and writes a hold record instead. The value does…
A live pass depends on a reference input produced outside the system, arriving on somebody else's schedule. That dependency is the classic availability-versus-legitimacy trade: when the feed is late, the system can act on what it already…
An operator-facing specification listed a set of operating windows as fixed wall-clock spans, described as holding all year. An auditor checked the table against the engine's own per-region calendars rather than against the previous version…
A report named the peak of a long excursion: a specific date, with the worst excursion-from-peak measured against it. A later reader noticed the date could not be right. The named peak came after the trough it was supposed to belong to. A…
method
Not a changelog entry and not a repository summary. One decision, one incident, one reversal, or one hard-won invariant — the thing an engineer would tell you at a whiteboard.
Prefer records with dates and root causes? Those are on incidents. Live numbers are on stats.