Writing Incident Reports & Postmortems

A blameless postmortem assumes everyone acted reasonably given what they knew at the time and focuses on the systems and processes that allowed the incident, not on punishing individuals. It matters because it's the only way to get the truth: if people fear blame, they hide the details that reveal the real root cause, so you fix nothing and the incident recurs. Instead of "Alex pushed bad config," you write "a config change reached production because no validation gate caught this error class and the review checklist didn't cover it" — which names fixable system gaps. Blame drives the truth underground; blamelessness surfaces it, and the goal is learning, not accountability theatre.

It opens with metadata and an executive summary — three to five sentences a VP can read covering what happened, the impact, the root cause, and current status, written last but placed first. Then a quantified impact section (users, data, downtime, cost, regulatory implications), a factual UTC timeline citing evidence sources and including time-to-detect and time-to-respond, and the root cause analysis using something like the 5 Whys to reach systemic causes — ideally identifying both a missing preventive control and a missing detective one. Then a balanced "what went well" and a blameless "what went poorly," and the most important section: specific, owned, tracked action items categorised as prevent, detect, or mitigate. It closes with lessons learned and appendices of IOCs and queries. The test is whether the same incident could happen tomorrow if every action item shipped.

I use the 5 Whys, repeatedly asking why until I reach something systemic rather than the immediate trigger. If a bucket was public, "a developer set it public" isn't the root cause — keep going: why was there no guardrail preventing public buckets, why wasn't account-level Block Public Access enabled, why didn't anything detect the drift. That surfaces the real causes: a missing preventive control and a missing detective control. I also distinguish trigger, root cause, and contributing factors, and I assume there's rarely a single cause — using the Swiss cheese model, incidents happen when holes in several layers line up, so I document the whole chain of gaps, because fixing only the last one leaves the others open.

For executives I lead with business impact and risk in plain language — what data was affected, how many customers, whether it's contained, and what we're doing — and I keep it to a short summary with no packet-level detail. For engineers I provide the full technical depth: the exact root cause, reproducible detail, the timeline with evidence, and the specific fixes. It's the inverted pyramid — conclusion and impact first, supporting detail next, deep technical appendices last — so each audience can stop reading at the right depth and still be correctly informed. The facts are identical; the framing and altitude differ. For legal I'd be especially precise and factual since the document may be discoverable.

Because they're the only part that changes the future — the rest of the postmortem documents the past, but action items prevent recurrence. A postmortem with no owned, tracked actions is theatre. A good action item is specific ("enable S3 Block Public Access org-wide," not "improve cloud security"), owned by a named person or team, tracked as a ticket with a due date and followed to completion, and categorised by whether it prevents the incident, detects it faster, or mitigates its impact. The test of the whole exercise is simple: if every action item shipped, could this exact incident happen again tomorrow? If yes, either the root cause analysis fell short or the actions are too weak.