Skip to content

Expert · Monitoring and logging · topic 38 of 53 · 30 min

Monitoring and observability: metrics, logs and traces

Explanation

Monitoring means collecting numbers and events from running systems and checking them against what you expect. Observability is how well you can work out what is happening inside a system from what it outputs, even for questions you did not plan for. The three main kinds of data are metrics, logs and traces, and good teams watch user-facing signals like latency, traffic, errors and saturation, using percentiles instead of averages.

Why watch a running system?

Your code passed every test and the deploy went green. Then, at 8 pm on a Saturday, some users start seeing a spinning wheel on the payment page. Without monitoring you learn about it from angry tweets an hour later. With monitoring a graph turns red within a minute, an alert reaches the on-call engineer, and the data shows where the problem is. Production is where software meets real users, real traffic and real failures, so you need eyes on it.

Monitoring versus observability

Monitoring is collecting signals and checking them against known expectations: "Is the error rate above 2%?", "Is the disk more than 90% full?". It answers questions you thought of in advance.

Observability is a property of the system: how well you can understand its inside from its outputs. A system is observable when you can answer a new question, such as "why are only users on one mobile network in Pune slow since 7 pm?", without shipping new code. Monitoring is something you do; observability is something you build in.

Think of it like this: Monitoring is the warning lights on a car dashboard: engine hot, fuel low. Observability is what a good mechanic has: a diagnostic port, readings from every sensor and a history of trips, so they can find a strange fault that has no warning light of its own.

The three pillars: metrics, logs and traces

PillarWhat it isGood forExample
MetricsNumbers measured over time, cheap to store and fast to queryDashboards, trends, alertshttp_requests_total, CPU use, queue length
LogsTime-stamped records of single events, with detailFinding exactly what happened and why{"level":"ERROR","message":"card declined","order_id":"A1029"}
TracesThe path of one request across services, split into timed steps called spansFinding which service or call is slowcheckout 1.9 s = cart 40 ms + payment 1.8 s + email 60 ms

They work together. A metric alert says the checkout error rate is high. The dashboard shows only the payment service is affected. A trace of one failed request shows the slow step is a call to the bank gateway. The logs for that trace, found by its trace ID, show the exact error message: "connection pool exhausted". Each pillar narrows the search for the next.

The four golden signals

Google's free Site Reliability Engineering book suggests that if you can measure only four things about a user-facing system, measure these:

  • Latency: how long requests take. Measure successful and failed requests separately, because a fast error can make latency look better than it is.
  • Traffic: how much demand there is, for example requests per second or transactions per minute.
  • Errors: the rate of requests that fail, either openly (HTTP 500) or quietly (a 200 response with the wrong content).
  • Saturation: how full the system is, for example CPU, memory, connection pool or queue use. Things usually get slow before they reach 100%.

RED and USE: two handy checklists

MethodUse it forSignals
REDServices that handle requests (APIs, web apps)Rate (requests per second), Errors (failed requests per second), Duration (how long they take)
USEResources (CPU, memory, disks, network links)Utilisation (how busy), Saturation (how much work is waiting), Errors (error events)

RED tells you how users experience a service. USE tells you whether the machines underneath are in trouble. Use RED for every service and USE for every important resource.

White-box and black-box monitoring

White-box monitoring looks from the inside: metrics and logs that the application and servers expose themselves. Black-box monitoring looks from the outside, like a user would. A synthetic check is a small robot that calls your login page or health endpoint every minute, perhaps from Mumbai and Singapore, and records whether it worked and how long it took.

You need both. Black-box checks catch problems the app cannot see in itself: an expired TLS certificate, a broken DNS record or a CDN outage. White-box metrics explain why something is slow.

Percentiles, not averages

A percentile tells you the value below which a given share of measurements fall. The 95th percentile (p95) latency is the time within which 95% of requests finished; 5% were slower. The median is p50.

Averages hide pain. Suppose 90 requests take 100 ms and 10 take 3,000 ms. The average is (90 x 100 + 10 x 3,000) / 100 = 390 ms, which sounds fine. But p50 is 100 ms and p95 is 3,000 ms: one user in ten waits three seconds. Those users are often your heaviest customers, with the biggest carts.

One simple way to compute a percentile is the nearest-rank method: sort the values, compute rank = ceil(p x n / 100), where n is the number of values and ceil rounds up, and take the value at that rank (counting from 1). For the 100 requests above, p95 is the 95th value, which is 3,000 ms.

Dashboards that answer questions

  • Start every service dashboard with one question: "Is this service healthy for users right now?" Put the golden signals in the top row.
  • Below that, add panels that explain causes: per-endpoint latency, database time, pod restarts, CPU and memory.
  • Mark deploys on the graphs (annotations), so "it broke at 7:42" can be matched with "we released at 7:40".
  • Keep one dashboard per service with the same layout everywhere, so anyone on call can read any of them.
Careful: Common mistakes: (1) Watching only averages; use p95 and p99. (2) Watching only CPU and memory but not what users feel; add error rate and latency. (3) Mixing failed and successful requests in one latency graph. (4) Having no black-box check, so an expired certificate goes unnoticed. (5) Building 40 dashboards nobody opens; build a few that answer real questions and delete the rest.

Need help?

Stuck on a word or an idea? Search this course, or every course, and jump straight to the explanation or a worked question.

Try:

Watch the lesson

Monitoring and observability: metrics, logs and traces · 1/7

Monitoring and observability

In this lesson you will learn how teams see what their software is doing in production.

Prefer a video by a teacher? Find free videos on “observability metrics logs traces golden signals explained” on YouTube ↗

In real life: Salary day at a bank's UPI service

On the first of the month, a bank's UPI traffic triples at 10 am as salaries arrive. The SRE team's dashboard shows traffic rising as expected and the error rate steady at 0.2%, but p99 latency jumps from 600 ms to 4 seconds while the average moves only from 180 ms to 260 ms. A trace of a slow payment shows most of the time is spent waiting for a free database connection. Saturation on the connection pool is at 100%. The team raises the pool size and adds read replicas before the evening peak. If they had watched only the average, they would have missed it.

output
Top row of the payments dashboard (the four golden signals)
Traffic     4,800 req/s     (normal salary-day peak)
Errors      0.2 %           (within target)
Latency     p50 140 ms   p95 900 ms   p99 4.1 s   <- tail is hurting
Saturation  DB pool 100/100 in use   <- the cause
A dashboard that answers "are users OK?" first, then "why not?".

Practice (0/3 done)

Task 1 · Code · Python

Write error_rate(statuses) that takes a list of HTTP status codes and returns the share of server errors (codes 500 and above), rounded to 4 decimal places. 4xx codes such as 404 are client errors and do NOT count. An empty list returns 0.0. Example: [200, 200, 500, 404, 503] returns 0.4.

Task 2 · Code · Python

Write percentile(values, p) using the nearest-rank method: sort the values, compute rank = ceil(p * n / 100) where n is the number of values, and return the value at that rank counting from 1 (so index rank - 1). Then write latency_summary(values) that returns a dict with keys "p50", "p95" and "p99". The starter wrongly uses the average.

Task 3 · Practice

Design the top two rows of a dashboard for a food delivery "place order" API. Row 1 must answer "are users OK right now?"; row 2 must help find the cause. For each panel, name the signal, the unit and why it is there.

Quick quiz

Question 1

1.Which of these is a trace?

Question 2

2.Which list is the four golden signals?

Question 3

3.90 requests take 100 ms and 10 take 3,000 ms. What is the p95 latency (nearest rank)?

Question 4

4.Your app looks healthy from its own metrics, but users cannot reach it because the TLS certificate expired. What would have caught this?

Question 5

5.The USE method is best for checking what?

Read more (free)

Open a question to see the answer and why. The first ones are this topic's own quiz, so try the quiz above before you peek.

Which of these is a trace?

Answer: The path of one checkout request through cart, payment and email services with the time spent in each

A trace follows one request across services, split into timed spans.

Which list is the four golden signals?

Answer: Latency, traffic, errors, saturation

The SRE book names latency, traffic, errors and saturation.

90 requests take 100 ms and 10 take 3,000 ms. What is the p95 latency (nearest rank)?

Answer: 3,000 ms

Rank ceil(95 x 100 / 100) = 95. The 95th sorted value is one of the 3,000 ms requests.

Your app looks healthy from its own metrics, but users cannot reach it because the TLS certificate expired. What would have caught this?

Answer: A black-box synthetic check from outside

Black-box checks test the service the way a user does, so they see certificate and DNS problems.

The USE method is best for checking what?

Answer: Resources like CPU, disks and network links

USE (utilisation, saturation, errors) is for resources; RED is for request-driven services.

p95 latency is 1.8 seconds. What does that mean?Related

Answer: 95% of requests took 1.8 s or less; 5% were slower

p95 is the time under which 95% of requests finished. It shows what the slower users feel.

From Logging and monitoring

19 requests take 50 ms and one takes 5000 ms. Which number shows the slow request best?Related

Answer: The p95 or a higher percentile

Averages hide the slow tail. Percentiles like p95 and p99 show what the slowest users feel.

From Monitoring, logs and alerts

Your EC2 dashboard shows CPU but no memory use. Why?Related

Answer: EC2 cannot see inside the OS; install the CloudWatch agent

The hypervisor sees CPU, network and disk I/O, but not memory used inside the OS. The CloudWatch agent sends it as a custom metric.

From Monitoring with CloudWatch, CloudTrail and Config

Besides answer quality, what should an LLM app track?Related

Answer: Latency, cost per request and safety failures

A correct answer that is too slow or too costly can still fail the product.

From Evaluating LLM apps

Pass the quiz (60% or more) or mark this topic as done when you are ready.

Your progress is saved on this device only.

Next: Prometheus and Grafana →

Next steps

  1. 3 more topics in “Monitoring and logging” before its exam: Prometheus and Grafana, Logging: structured, safe and centralised, Alerting and on-call.
  2. Module 9 exam: Monitoring and logging · 13 questions · 20 min, without help. 4 topics of this module still to finish first.
  3. Then the next module, SRE: reliability and incidents. 53 topics left in the course before the final exam.

← Previous: Roles, templates, Galaxy and Vault