Prometheus and Grafana

What Prometheus is

Prometheus is a free, open-source monitoring system and time series database. A time series is a list of values with time stamps, like "requests so far" every 15 seconds. It started at SoundCloud and is now a graduated project of the CNCF (Cloud Native Computing Foundation), the same home as Kubernetes. It is the default choice for metrics in most Kubernetes setups.

The parts

• Prometheus server: scrapes targets on a schedule, stores samples in its local TSDB (time series database) and evaluates alert rules. Its web UI listens on port 9090.
• Targets: anything that serves metrics as plain text over HTTP, usually at /metrics. Many apps expose this with a client library.
• Exporters: small programs that translate metrics from things that cannot expose them themselves. node_exporter (port 9100) gives host metrics like CPU, memory, disk and network. There are exporters for MySQL, PostgreSQL, Redis and more, and blackbox_exporter for probing URLs from outside.
• Service discovery: Prometheus can find targets automatically from Kubernetes, cloud APIs or files, so new pods are scraped without editing config by hand.
• Alertmanager: receives alerts from Prometheus, removes duplicates, groups related ones, routes them to email, chat or a paging tool, and lets you silence them during maintenance. Port 9093.
• Pushgateway: only for short batch jobs that finish before Prometheus could scrape them. The job pushes its final numbers there and Prometheus scrapes the gateway. Do not use it for normal services.

Prometheus uses a pull model: it pulls metrics from targets, the targets do not push. A bonus is that a failed scrape is itself a signal: the built-in metric up is 1 when the last scrape worked and 0 when it did not.

# HELP http_requests_total Total HTTP requests handled.
# TYPE http_requests_total counter
http_requests_total{method="GET",route="/orders",status="200"} 10234
http_requests_total{method="POST",route="/orders",status="500"} 12
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 5.24288e+07

The four metric types

Labels and the cardinality trap

Labels are key-value pairs that split a metric into separate series, such as method="GET" or status="500". Every unique combination of name and labels is its own time series. Cardinality means how many series you end up with. Labels with a few fixed values are fine. A label like user_id, email, order_id or a full URL with IDs in it creates millions of series and can run Prometheus out of memory.

A scrape config

global:
  scrape_interval: 15s       # how often to scrape every target
  evaluation_interval: 15s   # how often to check alert rules

rule_files:
  - "alerts.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: "orders-api"
    metrics_path: /metrics
    static_configs:
      - targets: ["orders-api:8080"]
  - job_name: "node"
    static_configs:
      - targets: ["10.0.1.11:9100", "10.0.1.12:9100"]

PromQL basics

PromQL is the query language of Prometheus. An instant vector is the latest value of each matching series, for example http_requests_total. A range vector is all samples in a time window, written with square brackets: http_requests_total[5m]. You cannot graph a range vector directly; you pass it to a function like rate().

rate(http_requests_total[5m])

sum by (status) (rate(http_requests_total[5m]))

sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

rate() gives the average per-second increase of a counter over the window, and it handles counter resets when a pod restarts. sum by (status) adds up all series but keeps the status label separate. =~ means "matches this regular expression", so "5.." matches 500 to 599. histogram_quantile() estimates a percentile from histogram buckets; keep the le label when you sum.

Alert rules

groups:
  - name: orders-api
    rules:
      - alert: OrdersHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="orders-api",status=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="orders-api"}[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "orders-api error rate is above 5%"
          description: "Error rate is {{ $value | humanizePercentage }} for the last 10 minutes."
          runbook_url: "https://wiki.example.com/runbooks/orders-high-error-rate"

While the condition is true but for has not passed, the alert is "pending". Only after 10 minutes in a row does it "fire" and go to Alertmanager. This stops one bad scrape from waking someone up.

Grafana

Grafana is an open-source tool for dashboards. It does not store metrics itself; it connects to data sources such as Prometheus, Loki, Elasticsearch, PostgreSQL, MySQL, Amazon CloudWatch and Azure Monitor, and shows them as panels. Variables add drop-downs (choose the service or namespace) so one dashboard serves many services. Dashboards are saved as JSON, so you can keep them in Git.

On Kubernetes: kube-prometheus-stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

With the operator, you tell Prometheus what to scrape by creating a ServiceMonitor object next to your app, instead of editing prometheus.yml.