Load balancing and auto scaling

Vertical versus horizontal scaling

Vertical scaling (scaling up) means moving to a bigger machine: more CPU and memory. It is simple and needs no code changes, but there is a largest size, resizing often needs a restart, and it is still one machine, so still a single point of failure. Horizontal scaling (scaling out) means adding more machines of the same size. It has almost no upper limit and improves availability, but needs a load balancer and stateless servers.

Elastic Load Balancing

Elastic Load Balancing (ELB) is AWS's managed load balancer service. It spreads incoming traffic across targets (EC2 instances, containers, IP addresses or Lambda functions) in several AZs, and it scales itself.

Three words to know. A listener checks for connections on a port and protocol, such as HTTPS on 443, and has rules that decide where requests go. A target group is a set of targets that receive traffic, plus its health check settings. The health check is the regular test (for example GET /health must return 200) that decides which targets are healthy.

aws elbv2 create-target-group --name results-web-tg \
  --protocol HTTP --port 80 --vpc-id vpc-0abc1234 \
  --health-check-path /health --health-check-interval-seconds 15 \
  --healthy-threshold-count 2 --unhealthy-threshold-count 3

Auto Scaling groups

An Auto Scaling group (ASG) keeps a fleet of EC2 instances at the right size. It starts instances from a launch template (which machine image, instance type, security groups and start-up script to use). You set three numbers: minimum (never fewer), desired (how many right now) and maximum (never more, which also caps cost). The ASG spreads instances across the subnets you give it in different AZs, and replaces any instance that fails its health check.

aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name results-web-asg \
  --launch-template LaunchTemplateName=results-web,Version='$Latest' \
  --min-size 2 --desired-capacity 2 --max-size 20 \
  --vpc-zone-identifier "subnet-0aaa1111,subnet-0bbb2222" \
  --target-group-arns arn:aws:elasticloadbalancing:ap-south-1:111122223333:targetgroup/results-web-tg/0123456789abcdef \
  --health-check-type ELB --health-check-grace-period 120

Scaling policies

Target tracking works roughly like this: desired = ceil(current × metric ÷ target). With 4 instances at 80% CPU and a target of 50%, you need 4 × 80 ÷ 50 = 6.4, so 7 instances. The result is always kept between the minimum and maximum.

# cpu50.json
# {
#   "PredefinedMetricSpecification": { "PredefinedMetricType": "ASGAverageCPUUtilization" },
#   "TargetValue": 50.0
# }
aws autoscaling put-scaling-policy --auto-scaling-group-name results-web-asg \
  --policy-name cpu50 --policy-type TargetTrackingScaling \
  --target-tracking-configuration file://cpu50.json

# Scale up before a 10 am booking window, every day at 9:30 India time
aws autoscaling put-scheduled-update-group-action --auto-scaling-group-name tatkal-web-asg \
  --scheduled-action-name before-tatkal --recurrence "30 9 * * *" --time-zone "Asia/Kolkata" \
  --min-size 30 --desired-capacity 30 --max-size 120

Warm-up and cooldown

A new instance needs time to boot, start the app and fill its caches. If its metrics are counted too early, the group may think it is still overloaded and add even more servers. Instance warm-up tells the ASG how long to wait before a new instance's metrics count. A cooldown (used with simple scaling policies) is a pause after one scaling action before the next. Scaling in (removing servers) is usually done more slowly than scaling out, so a brief dip does not remove capacity you need a minute later.

Scaling the database

App servers are easy to multiply; the database is usually the real limit. Common steps, from simple to complex: a bigger database instance (vertical); read replicas, copies that serve read-only queries such as "show my result"; a cache such as ElastiCache in front of the database for hot data; connection pooling (for example RDS Proxy), so thousands of app processes do not open thousands of connections; and finally sharding, which splits data across several databases by a key such as state or user ID. Sharding is powerful but complex, so treat it as a last step.

Planning for a known peak

Some peaks are on the calendar. On a ticket booking site, Tatkal booking for AC classes opens at 10 am, and huge traffic arrives in the first minutes. On a state board results day, millions of students open the site at the announced hour. Reactive scaling alone is too slow for a wall of traffic that arrives in seconds. Teams scale up on a schedule before the peak, load test beforehand, cache everything that can be cached (the landing page, static files on a CDN), use read replicas for result lookups, and put a queue or a virtual waiting room in front of the heaviest step.