Monitoring with CloudWatch, CloudTrail and Config

Why monitoring matters

Your users will find problems before you do unless you watch your systems. Observability means being able to answer "what is happening inside right now, and why?" from the data your system gives out. That data comes in three main kinds: metrics (numbers over time, like CPU percent), logs (text records of events, like an error message) and traces (the path of one request through many services).

CloudWatch metrics

A metric is a series of numbered datapoints over time. Each metric lives in a namespace, a group name such as AWS/EC2, AWS/Lambda or your own FoodApp. Dimensions are name and value pairs that say which resource the number belongs to, for example InstanceId=i-0abc or FunctionName=hello-api. A statistic summarises the datapoints in each period: Average, Sum, Minimum, Maximum, SampleCount, or a percentile such as p99.

EC2 sends basic metrics every 5 minutes for free (standard monitoring). Detailed monitoring sends them every 1 minute for an extra charge. Note that EC2 does not report memory use or disk space used by default, because AWS cannot see inside your operating system. Install the CloudWatch agent to send those.

Custom metrics are numbers you send yourself, such as orders placed per minute. Standard resolution is 1 minute; high-resolution metrics can go down to 1 second. Each custom metric has a monthly cost, so check the pricing page before sending hundreds of them.

aws cloudwatch put-metric-data --namespace FoodApp \
  --metric-name OrdersPlaced --unit Count --value 1 \
  --dimensions City=Pune

CloudWatch Logs

Logs are organised into log groups (usually one per application or function, like /aws/lambda/hello-api). Inside a group, each source writes to its own log stream, and each line is a log event. By default a log group keeps data forever, which slowly grows your bill. Set a retention period, such as 30 days for dev and 1 year for production, or whatever your rules require.

CloudWatch Logs Insights lets you search and count log lines with a small query language. The first query finds the latest errors; the second uses the REPORT lines Lambda writes after each run to show duration and memory per 5 minutes.

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 20

filter @type = "REPORT"
| stats avg(@duration), max(@duration), max(@maxMemoryUsed / 1000 / 1000) as maxMemoryMB by bin(5m)

Alarms

An alarm watches one metric (or a math expression) and is in one of three states: OK, ALARM or INSUFFICIENT_DATA. You set the statistic, the period (how long each datapoint covers), the threshold, and the comparison, such as GreaterThanThreshold.

Evaluation periods (N) is how many recent periods the alarm looks at. Datapoints to alarm (M) is how many of those must breach the threshold. "2 out of 3" ignores a single spike but still catches a real problem. Worked example: period 5 minutes, threshold 80% CPU, M = 2, N = 3. Recent datapoints 85, 70, 90: two breach, so the alarm goes to ALARM. With 85, 70, 60 only one breaches, so it stays OK.

You also choose how to treat missing data: as missing, as not breaching, as breaching, or ignore it. When the state changes, the alarm can notify an SNS topic (email, SMS, chat), run an Auto Scaling policy, or stop, reboot or recover an EC2 instance.

aws cloudwatch put-metric-alarm --alarm-name web-high-cpu \
  --namespace AWS/EC2 --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0abc1234def567890 \
  --statistic Average --period 300 \
  --evaluation-periods 3 --datapoints-to-alarm 2 \
  --threshold 80 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:ap-south-1:111122223333:ops-alerts

Dashboards

A CloudWatch dashboard puts graphs, numbers and alarm states on one screen. Build one per service with the few numbers that matter: requests, errors, latency (p99) and saturation such as CPU or queue length. Keep it simple enough that a tired engineer understands it at 2 am.

CloudTrail, CloudWatch and Config: who, what and how

CloudTrail keeps 90 days of management events in its event history at no charge. To keep them longer, or to record data events such as S3 object reads, create a trail that delivers to an S3 bucket (extra charges apply). AWS Config uses rules, such as one that flags public S3 buckets, to check resources against your standards.

X-Ray: tracing requests

AWS X-Ray follows one request through API Gateway, Lambda, DynamoDB and other services, and shows a map of where the time went. AWS now recommends instrumenting new code with OpenTelemetry (the AWS Distro for OpenTelemetry); check the current docs before you start.