Expert · SRE: reliability and incidents · topic 42 of 53 · 35 min
SLIs, SLOs, SLAs and error budgets
Explanation
What SRE is
Site Reliability Engineering (SRE) began at Google in 2003, when Ben Treynor Sloss was asked to run a production team. His idea was to give operations work to software engineers and let them solve it with software: automation, measurement and good design instead of more and more manual effort. Google later published its practices in free books at sre.google/books.
SRE and DevOps share the same goals: fast, safe delivery and shared ownership of production. SRE adds concrete tools: service level objectives, error budgets, a limit on manual work, and blameless postmortems. Many people describe SRE as one specific way of doing DevOps.
SLI: what you measure
A service level indicator (SLI) is a carefully chosen measurement of how well the service is working for users. The most useful form is a ratio: good events divided by valid events, shown as a percentage.
- Availability: successful requests / all valid requests.
- Latency: requests answered faster than 300 ms / all valid requests.
- Freshness: data refreshed within 1 minute / all data reads.
- Correctness: payments with the right amount in the ledger / all payments.
"Valid" matters. If a UPI payment fails because the user typed a wrong PIN or has no balance, the service worked correctly, so that request is not counted as bad.
SLO: the target
A service level objective (SLO) is a target for an SLI over a time window, for example "99.9% of payment requests succeed, measured over a rolling 30 days". A rolling window always covers the last 30 days, so it moves forward every day.
SLA: the contract
A service level agreement (SLA) is a promise to customers with consequences, usually service credits or refunds, if it is broken. Cloud providers publish SLAs for their services. The SLA should be looser than the internal SLO, for example SLO 99.9% and SLA 99.5%, so the team is warned and acts long before the company owes money. Internal services usually have SLOs but no SLA.
| Term | What it is | Example | Who cares |
|---|---|---|---|
| SLI | A measurement | 99.93% of requests succeeded this month | Engineers |
| SLO | An internal target for the SLI | 99.9% over 30 days | Engineering and product |
| SLA | A contract with penalties | 99.5% monthly, or a 10% service credit | Customers, sales, legal |
The nines
| Availability | Allowed downtime per 30 days | Per year |
|---|---|---|
| 99% | 7.2 hours | about 3.65 days |
| 99.5% | 3.6 hours | about 1.83 days |
| 99.9% ("three nines") | 43.2 minutes | about 8.76 hours |
| 99.95% | 21.6 minutes | about 4.38 hours |
| 99.99% ("four nines") | 4.32 minutes | about 52.6 minutes |
| 99.999% ("five nines") | about 26 seconds | about 5.26 minutes |
Error budgets
The error budget is 1 minus the SLO, over the window. With an SLO of 99.9% over 30 days, the budget is 0.1%: about 43.2 minutes of full downtime, or 10,000 failed requests out of 10 million. Worked example: 30 days x 24 hours x 60 minutes = 43,200 minutes, and 0.1% of 43,200 is 43.2.
The budget is meant to be spent: on releases, migrations and experiments, which all carry some risk. An error budget policy, agreed in advance by product, development and SRE, says what happens as it runs out. While budget is left, the team ships freely. When it is used up, risky releases pause (except urgent fixes) and the team works on reliability until the SLO is met again. This turns "move fast" against "be careful" from an argument into a rule based on data.
Burn rate
The burn rate is how fast you are using the budget: the observed error rate divided by the allowed error rate (1 minus the SLO). A burn rate of 1 uses the whole budget exactly by the end of the window. A burn rate of 10 uses it ten times faster, so a 30-day budget would be gone in 3 days.
Choosing SLOs users care about
Pick a few SLIs that match what users notice. For a UPI app: payment success rate (excluding user errors) and the share of payments completed within a few seconds. Not CPU or memory: users do not see those. Measure as close to the user as you can, for example at the load balancer instead of inside the app, so failures before the app are counted too.
Do not aim for 100%. Users' phones and mobile networks fail more often than that, so they would not notice the difference, and each extra nine costs far more engineering time and money. Start from how the service performs today and from what users complain about, then tighten slowly.
Need help?
Stuck on a word or an idea? Search this course, or every course, and jump straight to the explanation or a worked question.
Try:
Watch the lesson
SLIs, SLOs, SLAs and error budgets · 1/8
In this lesson you will learn how site reliability engineering measures and agrees on reliability.
Prefer a video by a teacher? Find free videos on “SLI SLO SLA error budget explained site reliability engineering” on YouTube ↗
In real life: SLOs for a UPI payments app
A payments startup in Mumbai sets two SLOs for its UPI flow. Availability: 99.9% of valid payment requests succeed over a rolling 30 days (wrong PIN and low balance are excluded). Latency: 99% of payments get a final status within 5 seconds. With about 20 million payments a month, the availability budget is about 20,000 failed payments. In week two a bad release fails 12,000 payments in one evening, using 60% of the budget. Under the agreed policy, new features wait until the following week while the team adds a canary stage to the release pipeline.
SLI = good payments / valid payments
SLO = 99.9% over rolling 30 days
Budget = 0.1% of ~20,000,000 = ~20,000 failed payments
Used = 12,000 (60%) -> policy: pause risky releases, fix the pipelinePractice (0/3 done)
Write error_budget_minutes(slo_percent, days) that returns the allowed downtime in minutes for an SLO given as a percentage (like 99.9) over a window of days, rounded to 1 decimal place. Formula: (100 - slo_percent) / 100 x days x 24 x 60. Example: error_budget_minutes(99.9, 30) returns 43.2.
Write budget_report(good, total, slo) where slo is a fraction like 0.999. Return a dict with: "sli" = good / total, rounded to 4 places; "budget_remaining" = 1 - (bad / allowed_bad), where bad = total - good and allowed_bad = (1 - slo) x total, rounded to 2 places (it goes negative when the budget is overspent); "burn_rate" = (bad / total) / (1 - slo), rounded to 2 places.
Choose SLIs and SLOs for a railway ticket booking website at its busiest moment, when tatkal booking opens. Write two or three SLIs as good / valid ratios, an SLO for each with a window, and one line of error budget policy.
Quick quiz
Read more (free)
Related help
Before this, learn
This topic builds on these. If something here feels new, open them first.
Related topics
- Incident managementSRE: reliability and incidents · same module
- Blameless postmortems and toilSRE: reliability and incidents · same module
- Resilience patterns, capacity and chaos engineeringSRE: reliability and incidents · same module
- Designing for high availabilityCloud Computing (AWS, Azure, Google Cloud) · High availability, scaling and disaster recovery
- Capstone project: from commit to monitored productionGitOps, platform engineering and DevOps careers
- Writing the API contractFull-Stack Development · Planning an app
Questions people ask about this
Open a question to see the answer and why. The first ones are this topic's own quiz, so try the quiz above before you peek.
Which is an SLI?
Answer: The share of requests answered in under 300 ms
An SLI is the measurement itself; the target is the SLO and the contract is the SLA.
Why should the SLA be looser than the SLO?
Answer: So the team is warned and can act before customers are owed money
Missing the internal SLO is a warning; missing the SLA costs money.
An SLO is 99.9% over 30 days. Roughly how much full downtime does the error budget allow?
Answer: 43.2 minutes
0.1% of 43,200 minutes is 43.2 minutes.
The error budget for the month is used up. What does a typical error budget policy say?
Answer: Pause risky releases and focus on reliability until the SLO recovers
The policy shifts effort from features to reliability while the budget is spent.
The observed error rate is 0.5% and the SLO is 99.9%. What is the burn rate?
Answer: 5
Burn rate = 0.5% / 0.1% = 5, so a 30-day budget would last only 6 days.
What does group_by: ["alertname", "service"] do in Alertmanager?Related
Answer: Sends related alerts with the same alertname and service as one notification
Grouping turns many similar alerts into one message.
From Alerting and on-call
What share of time do Google's SRE teams aim to keep toil below?Related
Answer: 50%
The SRE book states a goal of keeping toil below 50% of each SRE's time.
About how much downtime per month does 99.9% availability allow?Related
Answer: About 43.8 minutes
0.1% of a 730-hour average month is 0.73 hours, about 43.8 minutes.
Which setting stops anyone pushing straight to main in your capstone repository?Related
Answer: Branch protection (or a ruleset) requiring pull requests
Branch protection rules or rulesets require pull requests and passing checks before merging.
Pass the quiz (60% or more) or mark this topic as done when you are ready.
Your progress is saved on this device only.
Next steps
- 3 more topics in “SRE: reliability and incidents” before its exam: Incident management, Blameless postmortems and toil, Resilience patterns, capacity and chaos engineering.
- Module 10 exam: SRE, reliability and incidents · 13 questions · 20 min, without help. 4 topics of this module still to finish first.
- Then the next module, DevSecOps. 53 topics left in the course before the final exam.