Scaling: handling more users

Two ways to grow

Vertical scaling means a bigger machine: more CPU, more RAM. It is simple, because the code does not change, but there is a top limit and one machine is still one point of failure. Horizontal scaling means more machines running the same app, with traffic shared between them. It can grow much further and survives one machine dying, but it needs some design work.

Stateless servers

A server is stateless when it keeps nothing important in its own memory between requests. If user sessions or uploaded files live in one server's memory or disk, the next request may go to a different server that knows nothing about them. So move state out: sessions to Redis, files to object storage (like Amazon S3), data to the database. Then any server can answer any request.

import { createClient } from 'redis';
const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

async function saveSession(sessionId, data) {
  // expires after 24 hours
  await redis.set('session:' + sessionId, JSON.stringify(data), { EX: 60 * 60 * 24 });
}

async function loadSession(sessionId) {
  const raw = await redis.get('session:' + sessionId);
  return raw ? JSON.parse(raw) : null;
}

Load balancers and health checks

A load balancer sits in front of your servers and passes each request to one of them. Round robin sends requests to each server in turn: A, B, C, A, B, C. Least connections sends each request to the server that is busy with the fewest requests right now, which is better when some requests are slow. Examples are nginx, HAProxy and the managed load balancers of every cloud.

upstream notes_api {
  least_conn;
  server 10.0.0.11:3000;
  server 10.0.0.12:3000;
  server 10.0.0.13:3000;
}
server {
  listen 80;
  location / { proxy_pass http://notes_api; }
}

A health check is a small endpoint, like GET /healthz, that the load balancer calls every few seconds. If a server stops answering 200, the load balancer stops sending it traffic until it recovers. Keep the check cheap, but make it test what matters, for example that the database answers SELECT 1.

Scaling the database

Most apps read far more than they write: people view products many times for each order. A read replica is a copy of the database that receives every change from the main (primary) database and answers read queries. Writes go to the primary; reads can go to replicas. The copy is not instant: the small delay is called replication lag.

Sharding splits one big table across several databases, for example users with ids 1 to 1 crore in shard 1 and the next crore in shard 2. Each shard holds part of the data, so writes spread out too. It is powerful but complex: queries across shards and moving data between them are hard. Most apps never need it; try indexes, caching and replicas first.

CDNs and autoscaling

A CDN (Content Delivery Network) keeps copies of static files, like images, CSS, JavaScript and videos, on servers in many cities. A user in Guwahati gets the image from a nearby CDN server instead of your server in Mumbai: faster for them, less load for you. Autoscaling adds servers automatically when CPU or request count goes up, and removes them when traffic falls, so you pay for busy hours only.