A post-mortem is the part of a session that outlives it: written after the fact,
never during, and ending in a change. Each note records how the failure was
noticed, the actual root cause, and — after the fix — what it taught the method
rather than only the system.
Two certificates on the host that terminates the mesh were failing to renew. One had already expired, so the machine could no longer authenticate to its own cluster. The other expired in October and took the control plane with it: every agent that reconnected was refused until the certificate on the wire was fixed.
taught the method “we have a timer” is not a control. Every claim of renewal has to name the thing that reads the renewed artefact, and prove it on the wire.
netbirdtlsacmednspostmortem
·4 min read
The zombie handshake storm
Four hosts with expired client certificates retried against three cluster voters, indefinitely — two of them several times a second. The servers were right to reject them; the failure was that our alerting turned a certificate-hygiene bug into a leader-flap panic.
taught the method an alert that fires constantly is not monitoring, it is noise with a rota. The audit cadence checks the credential behind a timer, not the timer's status — an enabled unit failing every run looks identical to a healthy one.
nomadconsultlsalertingpostmortem
·4 min read
The cert that killed 215 allocations
A daily renewal that never told Vault to re-read the file it replaced. Hours later, 215 gitea allocations and 216 traefik allocations had failed on a TLS handshake the cluster could no longer complete — and six days later the same blast radius returned through DNS.
taught the method a lane's report separates “changed” from “verified”. A renewal is not done when the file changes; it is done when the consumer proves it read the new file.
vaultpkinomaddnspostmortem
method
How a note is written
The format is fixed so the notes stay comparable, and so the two
uncomfortable parts — the root cause and the method lesson — are hard to
skip. The last row is the one that outlives the incident.
timeline
Timestamps in UTC, from first symptom to last action.
Detection time is listed separately from start time —
the gap between them is usually the real finding.
impact
What users saw, for how long, and which services were actually affected
versus merely alarming.
root cause
The mechanism, not the trigger. “A disk filled up” is a trigger;
the missing alert or unbounded log is the cause.
change
The durable fix, plus what was deliberately left alone. A note with no
change attached is flagged as open.
method
What the failure taught the operating structure — which gate, cadence or
verdict field would have caught it, and which one now does.
A note that only fixes the system is half a note.
Looking for the method behind the fixes rather than the failures themselves?
That is how I work. Looking for the decisions? Those
are the field notes. Looking for current state?
Live dashboards are on infra; the numbers are on
stats.