codalume.com

post-mortems

What broke, and what it changed

A post-mortem is the part of a session that outlives it: written after the fact, never during, and ending in a change. Each note records how the failure was noticed, the actual root cause, and — after the fix — what it taught the method rather than only the system.

source stories/ · 4 notes · rendered server-side
all postmortem 3 dns 2 nomad 2 tls 2 acme 1 alerting 1 consul 1 docker 1 forensics 1 incident-response 1 netbird 1 pki 1 postgres 1 security 1 vault 1

1 of 4 notes

· 4 min read

The zombie handshake storm

Four hosts with expired client certificates retried against three cluster voters, indefinitely — two of them several times a second. The servers were right to reject them; the failure was that our alerting turned a certificate-hygiene bug into a leader-flap panic.

taught the method an alert that fires constantly is not monitoring, it is noise with a rota. The audit cadence checks the credential behind a timer, not the timer's status — an enabled unit failing every run looks identical to a healthy one.

nomadconsultlsalertingpostmortem

method

How a note is written

The format is fixed so the notes stay comparable, and so the two uncomfortable parts — the root cause and the method lesson — are hard to skip. The last row is the one that outlives the incident.

timeline
Timestamps in UTC, from first symptom to last action. Detection time is listed separately from start time — the gap between them is usually the real finding.
impact
What users saw, for how long, and which services were actually affected versus merely alarming.
root cause
The mechanism, not the trigger. “A disk filled up” is a trigger; the missing alert or unbounded log is the cause.
change
The durable fix, plus what was deliberately left alone. A note with no change attached is flagged as open.
method
What the failure taught the operating structure — which gate, cadence or verdict field would have caught it, and which one now does. A note that only fixes the system is half a note.

Looking for the method behind the fixes rather than the failures themselves? That is how I work. Looking for the decisions? Those are the field notes. Looking for current state? Live dashboards are on infra; the numbers are on stats.