codalume.com

post-mortems

What broke, and what it changed

A post-mortem is the part of a session that outlives it: written after the fact, never during, and ending in a change. Each note records how the failure was noticed, the actual root cause, and — after the fix — what it taught the method rather than only the system.

source stories/ · 4 notes · rendered server-side
all postmortem 3 dns 2 nomad 2 tls 2 acme 1 alerting 1 consul 1 docker 1 forensics 1 incident-response 1 netbird 1 pki 1 postgres 1 security 1 vault 1

1 of 4 notes

· 4 min read

Ransomware hunt across 11 machines

A test database on a public port, a brute-forced password, and a crypto-miner wearing postgres clothes — with an extortion note left in a database named readme_to_recover. The recovery was routine; the 11-machine sweep a month later was the work.

taught the method containment is an incident; eradication is a campaign. Closing an item requires a read-only sweep and a written verdict, not a plausible story.

incident-responseforensicsdockerpostgressecurity

method

How a note is written

The format is fixed so the notes stay comparable, and so the two uncomfortable parts — the root cause and the method lesson — are hard to skip. The last row is the one that outlives the incident.

timeline
Timestamps in UTC, from first symptom to last action. Detection time is listed separately from start time — the gap between them is usually the real finding.
impact
What users saw, for how long, and which services were actually affected versus merely alarming.
root cause
The mechanism, not the trigger. “A disk filled up” is a trigger; the missing alert or unbounded log is the cause.
change
The durable fix, plus what was deliberately left alone. A note with no change attached is flagged as open.
method
What the failure taught the operating structure — which gate, cadence or verdict field would have caught it, and which one now does. A note that only fixes the system is half a note.

Looking for the method behind the fixes rather than the failures themselves? That is how I work. Looking for the decisions? Those are the field notes. Looking for current state? Live dashboards are on infra; the numbers are on stats.