Postmortem

A postmortem is the written review produced after an incident: what happened, in what order, what was affected, why the system allowed it, and what will change as a result. It is deliberately separate from the incident itself, because the goal during an outage is to restore service and the goal afterwards is to understand it, and those two goals want different states of mind. The discipline that makes postmortems worth the hours is the refusal to stop at a person. Almost every incident can be described as somebody ran the wrong command, and that description is both true and useless, because it explains nothing about why running it was possible, why nothing caught it, why the effect was so wide, and why recovery took as long as it did. A review that names an individual produces one quieter engineer and no change in the system. A review that names the conditions produces a list of things to fix. Three habits distinguish a document that gets used from one that gets filed. Write the timeline from evidence rather than memory, including the gaps — the twelve minutes when nobody was looking are usually the most informative part. Separate what you would do again from what you would change, because a review that recommends only more care is a review with no actions in it. And give each action an owner and a date, then check the list at the next review; the most common failure of the practice is not writing bad postmortems but writing good ones whose actions nobody performs. Finally, keep them readable and keep them where people look, because their second and larger value is as a record for whoever inherits the system after everyone in the room has moved on. A short note on scope. Not every incident needs the full document, and insisting otherwise is how the practice dies of paperwork. Agree in advance which incidents get one — usually those that reached customers, lasted beyond some duration, or surprised the people responding — and let the rest be a paragraph in a channel.