codalume.com

post-mortems

What broke, and what it changed

A post-mortem is the part of a session that outlives it: written after the fact, never during, and ending in a change. Each note records how the failure was noticed, the actual root cause, and — after the fix — what it taught the method rather than only the system.

source stories/ · 4 notes · rendered server-side
all postmortem 3 dns 2 nomad 2 tls 2 acme 1 alerting 1 consul 1 docker 1 forensics 1 incident-response 1 netbird 1 pki 1 postgres 1 security 1 vault 1

1 of 4 notes

· 4 min read

The cert that killed 215 allocations

A daily renewal that never told Vault to re-read the file it replaced. Hours later, 215 gitea allocations and 216 traefik allocations had failed on a TLS handshake the cluster could no longer complete — and six days later the same blast radius returned through DNS.

taught the method a lane's report separates “changed” from “verified”. A renewal is not done when the file changes; it is done when the consumer proves it read the new file.

vaultpkinomaddnspostmortem

method

How a note is written

The format is fixed so the notes stay comparable, and so the two uncomfortable parts — the root cause and the method lesson — are hard to skip. The last row is the one that outlives the incident.

timeline
Timestamps in UTC, from first symptom to last action. Detection time is listed separately from start time — the gap between them is usually the real finding.
impact
What users saw, for how long, and which services were actually affected versus merely alarming.
root cause
The mechanism, not the trigger. “A disk filled up” is a trigger; the missing alert or unbounded log is the cause.
change
The durable fix, plus what was deliberately left alone. A note with no change attached is flagged as open.
method
What the failure taught the operating structure — which gate, cadence or verdict field would have caught it, and which one now does. A note that only fixes the system is half a note.

Looking for the method behind the fixes rather than the failures themselves? That is how I work. Looking for the decisions? Those are the field notes. Looking for current state? Live dashboards are on infra; the numbers are on stats.