Alerting and on-call

What an alert is for

An alert is an automatic message that asks a human to do something. A page is an urgent alert that rings a phone, even at 3 am. Because pages interrupt sleep and family time, each one has to earn its place. A good rule from Google's SRE practice: every page should be urgent, actionable, and about a problem users notice or will soon notice.

Symptoms, not causes

A symptom is what users feel: errors, slowness, a page that does not load. A cause is something inside: high CPU, one pod restarting, a full cache. Page on symptoms. CPU at 95% with users perfectly happy is not an emergency; a 10% checkout error rate with CPU at 30% is. Watch causes on dashboards and use them to debug.

Alert fatigue

Alert fatigue happens when people get so many alerts, mostly false or unimportant, that they stop paying attention. The phone buzzes twenty times a night, the engineer starts muting it, and the one real alert is missed. Every alert that fires without needing action is a bug in your alerting. Review pages every week and delete, tune or downgrade the noisy ones.

Thresholds and for durations

A threshold is the line that triggers the alert, such as "error rate above 5%". Real data is spiky, so a single bad minute should not page anyone. The for field in a Prometheus rule says how long the condition must stay true before the alert fires. A 5 to 15 minute for removes most flapping (an alert going on and off again and again). Choose thresholds from data: look at last month's graphs and from what users complained about.

Burn-rate alerts on SLOs (a preview)

Next module covers SLOs (targets like "99.9% of requests succeed over 30 days") and error budgets (the 0.1% you are allowed to fail). The burn rate is how fast you are using that budget: a burn rate of 1 uses it up exactly at the end of the window; a burn rate of 14.4 uses 2% of a 30-day budget in one hour. The Google SRE workbook suggests paging on fast burns over short windows (like 14.4 over 1 hour) and opening tickets for slow burns (like 1 over 3 days). This catches both sudden outages and slow leaks.

(
  sum(rate(http_requests_total{job="checkout",status=~"5.."}[1h]))
    / sum(rate(http_requests_total{job="checkout"}[1h]))
) / (1 - 0.999) > 14.4

Routing, grouping and escalation

Alertmanager decides who hears about each alert. It groups alerts that share labels (one message saying "30 pods are failing" instead of 30 messages), waits a little to collect related alerts, routes by labels (severity=page goes to the pager, the rest to chat), and supports silences during planned maintenance and inhibition (if the whole data centre is down, do not also send 500 alerts about each service in it).

route:
  receiver: team-chat              # default for everything
  group_by: ["alertname", "service"]
  group_wait: 30s                  # wait to collect related alerts
  group_interval: 5m               # gap between updates for a group
  repeat_interval: 4h              # remind if still firing
  routes:
    - matchers:
        - severity="page"
      receiver: oncall-pager

receivers:
  - name: team-chat
    slack_configs:
      - api_url: "https://hooks.slack.com/services/REPLACE_ME"
        channel: "#orders-alerts"
  - name: oncall-pager
    pagerduty_configs:
      - routing_key: "REPLACE_WITH_INTEGRATION_KEY"

Paging tools such as PagerDuty or Opsgenie hold the on-call schedule and the escalation policy: if the primary on-call does not acknowledge within, say, 15 minutes, the secondary is paged, and then the team lead.

Runbooks

A runbook (or playbook) is a short page linked from the alert that tells a sleepy engineer what to do. It should say: what the alert means, what users are feeling, which dashboard and logs to open, the first checks, safe ways to reduce the damage (roll back, turn off a feature flag, scale up), and whom to call if stuck. Put its link in the alert's runbook_url annotation.

Healthy on-call

• Share the rotation among enough people that nobody is on call most weeks, with a primary and a secondary.
• Google's SRE book suggests no more than about two incidents per 12-hour shift, so there is time to handle each one properly and follow up.
• Hand over at a fixed time with notes on what is still open.
• Give time off or pay for night pages, and fix the causes of repeated pages instead of accepting them.