Blameless postmortems and toil

What a postmortem is

A postmortem (also called an incident review or retrospective) is a document written after an incident. It records what happened, how users were affected, what the team did, why it happened, and what will change so it does not happen again. The goal is learning, not paperwork.

Write one for every serious incident. The Google SRE book lists triggers such as: user-visible downtime or degradation beyond a set limit, any data loss, an on-call engineer having to step in (a rollback or rerouting traffic), a resolution time above a limit, and a monitoring failure, meaning a human found the problem before the alerts did.

Why blameless

Blameless means the review focuses on systems and processes, not on punishing a person. It assumes that everyone acted with good intentions, based on the information they had at the time.

If engineers fear punishment, they stay quiet about what they really did and saw, and the team learns nothing. "Human error" is never the end of the analysis; it is the start. If one mistyped command could delete a production database, the real questions are: why was that possible, why did no check stop it, and why did recovery take two hours?

A postmortem template

1. Title, date, authors, status (draft, reviewed, done).
2. Summary: three or four sentences anyone in the company can understand.
3. Impact: who was affected, for how long, how many requests, orders or rupees, and how much error budget was used.
4. Timeline: key events with times and time zone, from the first sign to full recovery.
5. Root causes and contributing factors: usually several, not one.
6. Detection: how we found out and how long it took.
7. What went well, what went badly, and where we got lucky.
8. Action items: each with an owner, a due date, a priority and a ticket link.

The five whys, and its limits

The five whys is a simple method: ask "why?" again and again until you reach something you can fix in the system.

Checkout failed for 40 minutes.
Why? The database ran out of connections.
Why? The new cart code opened a connection per item and did not close it.
Why? The code review did not catch the leak.
Why? There is no load test in the pipeline that would show rising connections.
Why? Load tests were seen as optional for small changes.
Fix: add a load test stage and a connection-pool alert; add a lint rule for unclosed connections.

Its limits: it follows one chain, but real incidents usually have several contributing factors (the leak, the missing alert, the slow rollback). It can stop at "someone made a mistake". And the answers depend on who is asking. Use it as a starting tool, then also ask "what else made this worse?" and "what made recovery slow?".

Action items that actually happen

• Make each one specific and checkable: "Add an alert when the DB pool is above 80% for 5 minutes", not "improve monitoring".
• Give each an owner and a due date, and put it in the team's normal backlog, not only in the document.
• Mix types: prevent (stop it happening), detect (find it faster), mitigate (recover faster).
• Avoid "be more careful". People cannot be careful at 2 am every time; systems can.
• Review open action items every week or two until they are done.

Sharing what you learn

Publish postmortems where the whole engineering team can read them. Some teams hold a monthly postmortem reading session, or role-play old incidents with new on-call engineers (Google calls this the "Wheel of Misfortune"). Learning from other teams' incidents is much cheaper than having your own.

Toil

The SRE book defines toil as work that is manual, repetitive, automatable, tactical (reacting rather than planning), without lasting value, and that grows as the service grows. Examples: restarting a service by hand every morning, copying certificates to servers, creating user accounts from emails, resizing disks when an alert fires.

Some toil is unavoidable, but too much leaves no time for engineering, and people burn out. Google's SRE teams aim to keep toil below 50% of each SRE's time and spend the rest on engineering work that reduces future toil. Measure it (for example, log hours spent on manual tasks for two weeks), then automate the biggest items first: scripts, self-service forms, auto-remediation and better defaults.