field notes
Short notes from real engineering work: one decision, one incident, one reversal at a time. Each one names the shape of the problem, the tension that made it hard, the decision, and what it cost. The internals stay out — the codename is the hint.
If you have five minutes rather than an afternoon: the three case studies condense this material to one problem, one decision and what the decision cost. The notes below are the record those studies cite — and the projects the codenames belong to are on the front page.
Incident review is where a team's honesty is easiest to perform and hardest to prove. A good write-up is not the one that reads well. It is the one whose follow-up items are still recognisable a month later, each with an owner, with…
The most expensive failure in a payment path is not an outage. It is a system that believes it has been paid. A test gateway left reachable in a live configuration, a placeholder credential that authenticates nothing, a fallback that reads…
A snapshot policy is a promise about the future. It is also configuration that runs unattended, which means its failure mode is silence: nothing errors, nothing alerts, and storage grows.
Every retention policy is a promise to delete, and deletion is the one operation you cannot review by reading it. A predicate with the wrong comparison either keeps everything forever or removes a row that mattered, and both look identical…
Dependency risk rarely announces itself. It arrives as a default: a client library with its optional features left on, dragging in a platform-native cryptography library and a second encrypted-transport stack through bindings that need…
Most systems have an audit trail in the sense that logs exist somewhere. Very few can answer, for one specific record, which person changed it and what the value was beforehand. That question decides disputes, and it is nearly always…
Account recovery is usually the weakest path in an authentication system, because it is the one place where a stranger holding a link can obtain an account. The reset flow that landed treats that link as a credential, with every property a…
The cheapest way to hold a login is a signed token the client carries. It is also the hardest to take back: revocation becomes a blocklist, and a blocklist is a second store with its own lifetime problem. The platform had token plumbing…
A second factor is only as strong as the secret behind it. If the material that protects every enrolled user's authenticator state lives in configuration with a fallback, the control has a quiet ceiling, and the ceiling is whatever the…
Removing a credential from a repository is a small, satisfying act that answers almost none of the question. The question is whether the tree you are about to ship still contains one, and whether the strings that look like credentials are…
A request-forgery guard is a cheap, unglamorous control: refuse a mutating request unless it carries a header that a page on another site cannot set. It works precisely because the browser will not attach it on someone else's behalf.
A browser security policy is easy to write and expensive to switch on. Every directive that closes a hole is also a promise that the application already behaves well, and this application was written before anyone made that promise.
A credential reached a repository. The usual response is to remove it, which is necessary and mostly useless: the value is in the history, in the clones, in the mirrors, and in everyone's working copy. What was added alongside the removal…
A scheduled backup job ran green for as long as anyone cared to look. Green meant the snapshot was written and the tool's own integrity check passed. It did not mean the data could come back, because nobody had tried, and a snapshot that…
A venue's public page needed a small configuration payload — name, tagline, hours, delivery parameters — served as structured data. The route was added and did not work. It did not fail either. The static file handler had a catch-all that…
The operator console had to show everything: every venue, every deployment, the state of the cluster, the register of what exists. An aggregate view like that is the highest-value surface in the system, because one page compresses what…
Payment providers retry. Customers double-tap. Networks drop the acknowledgement, not the request. On a platform that turns a payment acknowledgement into order state, the failure mode is not an error page — it is a second order, a second…
Two facts decide what a customer is charged: whether their address falls inside a delivery zone, and whether a discount they claimed still applies. Both arrive from a browser, and both are editable in a browser. The version of this feature…
Privacy law gives any person the right to ask what a business holds about them, and to have it exported or erased. On a platform where most orders come from guests who never created an account, the intake surface for those requests cannot…
One server process serves several scopes: the public storefront, the marketplace, the authentication pages, the staff tools. Each had a host check — the request must arrive addressed to exactly the host this deployment serves. Strict,…
method
Not a changelog entry and not a repository summary. One decision, one incident, one reversal, or one hard-won invariant — the thing an engineer would tell you at a whiteboard.
Prefer records with dates and root causes? Those are on incidents. Live numbers are on stats.