codalume.com

post-mortems

What broke, and what it changed

A post-mortem is the part of a session that outlives it: written after the fact, never during, and ending in a change. Each note records how the failure was noticed, the actual root cause, and — after the fix — what it taught the method rather than only the system.

source stories/ · 4 notes · rendered server-side
all postmortem 3 dns 2 nomad 2 tls 2 acme 1 alerting 1 consul 1 docker 1 forensics 1 incident-response 1 netbird 1 pki 1 postgres 1 security 1 vault 1

1 of 4 notes

· 4 min read

The day the mesh died from an expired certificate

Two certificates on the host that terminates the mesh were failing to renew. One had already expired, so the machine could no longer authenticate to its own cluster. The other expired in October and took the control plane with it: every agent that reconnected was refused until the certificate on the wire was fixed.

taught the method “we have a timer” is not a control. Every claim of renewal has to name the thing that reads the renewed artefact, and prove it on the wire.

netbirdtlsacmednspostmortem

method

How a note is written

The format is fixed so the notes stay comparable, and so the two uncomfortable parts — the root cause and the method lesson — are hard to skip. The last row is the one that outlives the incident.

timeline
Timestamps in UTC, from first symptom to last action. Detection time is listed separately from start time — the gap between them is usually the real finding.
impact
What users saw, for how long, and which services were actually affected versus merely alarming.
root cause
The mechanism, not the trigger. “A disk filled up” is a trigger; the missing alert or unbounded log is the cause.
change
The durable fix, plus what was deliberately left alone. A note with no change attached is flagged as open.
method
What the failure taught the operating structure — which gate, cadence or verdict field would have caught it, and which one now does. A note that only fixes the system is half a note.

Looking for the method behind the fixes rather than the failures themselves? That is how I work. Looking for the decisions? Those are the field notes. Looking for current state? Live dashboards are on infra; the numbers are on stats.