Expert · SRE: reliability and incidents · topic 43 of 53 · 30 min
Incident management
Explanation
What counts as an incident?
An incident is an unplanned event that disrupts or degrades a service, or threatens to (for example a security breach or data loss), and needs more than one quick fix by one person. Not every alert is an incident. But declare one early when in doubt: cancelling an incident costs a few minutes, while declaring too late costs users and trust.
Severity levels
Severity (SEV) levels tell everyone how serious an incident is and how to respond. Each company writes its own definitions; here is a common example.
| Level | Example definition | Response |
|---|---|---|
| SEV1 (critical) | A core flow (payments, login, checkout) is down for many users, or there is data loss or a security breach | Page immediately, all needed people join, leadership updates every 30 minutes |
| SEV2 (major) | A core flow is broken for some users, or a large share of users is affected | Page on-call and the owning team, updates every 30 to 60 minutes |
| SEV3 (minor) | A non-core feature is broken or a small share of users is affected; a workaround exists | Fix in working hours, track in a ticket |
| SEV4 (low) | Cosmetic or very small problems | Normal backlog |
Roles
- Incident commander (IC): runs the response. Sets priorities, gives out tasks, decides on actions such as a rollback, and keeps the big picture. The IC does not debug; their hands stay off the keyboard.
- Communications lead: writes updates for the status page, support teams, leadership and partners, on a fixed schedule.
- Operations lead and subject matter experts: the engineers who investigate and make changes, each saying out loud what they are about to do.
- Scribe: keeps the timeline in the incident document: what was seen, decided and changed, with times.
In a small team one person may hold two roles, but the IC role should always be clear, and handed over out loud ("I am now IC") when people change.
The lifecycle
- Detect: an alert fires or a customer reports a problem.
- Triage: judge the impact, set the severity, declare the incident, name the IC and open a channel.
- Mitigate: stop the damage for users as fast as possible.
- Resolve: fix the underlying problem properly and confirm the metrics are back to normal.
- Review: write a blameless postmortem and track its action items.
Mitigate first, then fix. If checkout broke ten minutes after a release, roll back now, even if you are not yet sure the release was the cause. Understanding can wait; users cannot. You can study logs and graphs calmly after service is back.
Mitigation options
| Option | When it helps |
|---|---|
| Roll back the last release | The problem started after a deploy |
| Turn a feature flag off | A new feature is misbehaving and is behind a flag |
| Scale up or out | The service is overloaded: CPU, connections or queues are full |
| Fail over | One database, zone or region is unhealthy and a standby exists |
| Rate-limit or block traffic | A bot, a retry storm or one client is flooding the service |
| Restart (carefully) | A process is stuck; first save logs or a memory dump, because a restart wipes evidence |
Communication
Open one channel per incident, such as #inc-2026-10-19-checkout, and keep all talk there. Update the public status page for customer-facing incidents. Send internal updates on a fixed schedule even when nothing has changed, because silence makes people interrupt the responders to ask.
[SEV1] Checkout errors - update 3 - 20:45 IST
Impact: about 30% of checkout attempts fail since 20:05. Browsing works.
Known: errors started after the 20:00 release of cart-service.
Doing: rolling back cart-service (IC: Priya). Payments team on standby.
Next update: 21:15 IST or sooner if anything changes.War-room tips
- One channel, one IC, one incident document.
- Say what you are about to do before you do it, and wait for the IC to agree. Change one thing at a time.
- Write facts with times ("20:12 error rate 31%"), not guesses.
- Hand over after a few hours; tired people make mistakes.
Need help?
Stuck on a word or an idea? Search this course, or every course, and jump straight to the explanation or a worked question.
Try:
Watch the lesson
Incident management · 1/7
In this lesson you will learn how teams respond when production breaks.
Prefer a video by a teacher? Find free videos on “incident management incident commander roles sre tutorial” on YouTube ↗
In real life: A hospital appointment system goes down on Monday morning
At 9:05 am on a Monday the appointment booking system of a hospital chain in Chennai starts timing out, just as patients call in. The on-call engineer declares a SEV2 and becomes IC. A developer finds that a database index was dropped by a migration on Sunday night; a second engineer handles updates to the front desk staff every 20 minutes. The IC decides to mitigate first by moving booking reads to a replica that still has the index, which brings bookings back by 9:40. The index is rebuilt in the evening, and a postmortem the next day adds a migration check to the pipeline.
Practice (0/3 done)
During a SEV1, what is the incident commander's main job?
Run a tabletop exercise (a practice incident done by talking it through, without breaking anything) for this scenario: on a big sale day at 8:00 pm, 25% of checkout attempts on your e-commerce site start failing. A new "apply coupon" feature was released at 7:45 pm behind a feature flag. Assign roles, set the severity, and write a timeline from detection to resolution, including mitigation and communication.
Write classifySeverity(impact). impact has usersAffectedPercent (a number), coreFlowDown (true if payments, login or checkout are broken) and dataLoss (true or false). Rules, checked in this order: "SEV1" if dataLoss, or if coreFlowDown and usersAffectedPercent >= 25. "SEV2" if coreFlowDown, or usersAffectedPercent >= 25. "SEV3" if usersAffectedPercent >= 1. Otherwise "SEV4".
Quick quiz
Read more (free)
Related help
Before this, learn
This topic builds on these. If something here feels new, open them first.
Related topics
- Blameless postmortems and toilSRE: reliability and incidents · same module
- Resilience patterns, capacity and chaos engineeringSRE: reliability and incidents · same module
- The incident response lifecycle (NIST SP 800-61)Cyber Security: from zero to SOC analyst · Logs, SIEM and incident response
- Monitoring, logs and alertsFull-Stack Development · Deploying and running the app
- SIEM and alert triageCyber Security: from zero to SOC analyst · Logs, SIEM and incident response
- India: CERT-In and the legal landscapeEthical Hacking and Penetration Testing (legal labs) · Bug bounties, disclosure and careers
Questions people ask about this
Open a question to see the answer and why. The first ones are this topic's own quiz, so try the quiz above before you peek.
Checkout broke 10 minutes after a release. The cause is not yet clear. What should the team do first?
Answer: Roll back the release to restore service, then investigate
Mitigate first: a rollback is quick and usually safe. Root cause comes after users are safe.
Who keeps the timeline of what was seen, decided and changed?
Answer: The scribe
The scribe records the timeline as it happens, which makes the postmortem accurate.
Why send updates on a fixed schedule even when nothing has changed?
Answer: So stakeholders do not interrupt responders to ask, and know when to expect news
Regular updates build trust and protect the responders' focus.
Which is the best example of a SEV1 under the definitions in this lesson?
Answer: Login fails for 60% of users
A core flow down for most users is critical.
Before restarting a stuck process during an incident, what should you try to do?
Answer: Save logs or a memory dump, because the restart wipes evidence
Capture evidence first, so the root cause can still be found later.
Why use an out-of-band channel during an incident?Related
Answer: Attackers may be reading the normal email system
Assume the attacker can see your normal communications.
In an incident, what comes before finding the root cause?Related
Answer: Stopping the harm, for example by rolling back
First get users working again (rollback, restart, switch a feature off). Then find and fix the cause calmly.
Ransomware is actively encrypting files on a server. Severity?Related
Answer: Critical
Active damage to important systems is critical: act immediately.
The safe assumption about CERT-In rules and timelines is that they…Related
Answer: Can change, so check official sources
Always verify the current directions.
Pass the quiz (60% or more) or mark this topic as done when you are ready.
Your progress is saved on this device only.
Next steps
- 2 more topics in “SRE: reliability and incidents” before its exam: Blameless postmortems and toil, Resilience patterns, capacity and chaos engineering.
- Module 10 exam: SRE, reliability and incidents · 13 questions · 20 min, without help. 4 topics of this module still to finish first.
- Then the next module, DevSecOps. 53 topics left in the course before the final exam.