Logging: structured, safe and centralised

Why logs?

A metric tells you that 3% of payments failed. A log tells you which order failed, for which reason, at which second. Logs are the detailed diary of your system. You read them when debugging, when a customer complains about a specific transaction, and during security investigations.

From plain text to structured logs

An old-style log line is free text: "Payment failed for order A1029 after 3 tries". A human can read it, but a program has to guess where the order ID is. A structured log writes the same event as fields, usually one JSON object per line. Machines can then filter and count by any field.

{"timestamp":"2026-10-19T20:15:02.481+05:30","level":"ERROR","service":"payments","message":"Payment failed","order_id":"A1029","attempts":3,"reason":"gateway_timeout","request_id":"req-7f3a9c","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736"}

Useful fields in every line: timestamp (ISO 8601 with the time zone), level, service name, message, and a request_id or trace_id so you can follow one request. Add business IDs like order_id when they help, but think before adding anything personal.

Log levels

Choose the level by asking: "Does someone need to act on this?" Normal traffic is INFO, not ERROR. If every 404 is logged as ERROR, real errors drown. Set the minimum level by configuration, so you can turn DEBUG on for ten minutes while investigating without a new deploy.

Never log secrets or personal data

Logs are copied to many places and read by many people: developers, support staff, vendors. Anything in a log should be treated as widely visible. Never log passwords, OTPs, API keys, tokens, full card numbers or CVVs, Aadhaar or PAN numbers, or full request bodies of login and payment calls. India's Digital Personal Data Protection Act, 2023 makes companies responsible for personal data they hold, and that includes their logs.

Defend in layers: do not put the data in the message at all; mask it in the logging library (show only the last 4 digits); and scan logs for patterns like 12-digit numbers or email addresses before they leave the server.

Centralised logging

On one server you can run tail -f. With 40 pods that come and go, a crashed pod takes its logs with it. Centralised logging means an agent on each machine collects logs and ships them to one store, where you can search everything together.

{app="payments", namespace="prod"} | json | level="ERROR" | reason="gateway_timeout"

Correlation IDs and OpenTelemetry

A correlation ID (request ID) is a unique ID created when a request enters your system, passed to every service in a header, and written in every log line. Searching for request_id="req-7f3a9c" then shows the whole story across services. Tracing systems do this with a trace ID, carried in the standard W3C traceparent header.

OpenTelemetry (OTel) is a vendor-neutral, open standard from the CNCF for producing traces, metrics and logs. You add its SDK to your code once; the OpenTelemetry Collector then sends the data to whichever backend you choose. Because log lines and spans share the same trace ID, you can jump from a slow trace to its logs.

Retention and cost

Logs can cost more than the servers they describe. Keep DEBUG off in production, sample very noisy INFO logs, and drop health-check logs. Set retention: for example, 15 days searchable and 1 year in cheap object storage. Regulated businesses such as banks may have to keep some logs longer, so check with your compliance team.