Resilience patterns, capacity and chaos engineering

Failure is normal

In a system of many services, something is always a little broken: a slow database, a network blip, a pod being replaced. Resilience means the system keeps serving users, perhaps with less functionality, when parts fail. The danger is a cascading failure: one slow service makes its callers wait, they run out of threads, their callers wait too, and soon the whole site is down.

Timeouts

A timeout is the longest you will wait for a reply. Without one, a hung dependency keeps your threads or connections busy forever. Every network call needs a timeout. Choose it from the dependency's normal p99 latency plus a margin, and keep it shorter than the timeout of whoever is calling you, or your work is wasted.

Retries with exponential backoff and jitter

Many failures are brief, so one more try often works. But careless retries make outages worse: if a struggling service is hit by every client retrying at once, it never recovers. Rules:

• Retry only temporary errors (timeouts, 503, 429 Too Many Requests) and only operations that are safe to repeat.
• Use exponential backoff: wait base x factor to the power of the attempt, with a cap. For base 1 s, factor 2 and cap 30 s: 1, 2, 4, 8, 16, 30 seconds.
• Add jitter: a random part of the wait, so thousands of clients do not retry at the same moment.
• Limit attempts (often 3) and retry at one layer only. If three layers each try 3 times, the bottom service can get 27 calls for one click.

import random
import time

def call_with_retries(call, attempts=4, base=0.5, factor=2, cap=8):
    for attempt in range(attempts):
        try:
            return call()
        except TimeoutError:
            if attempt == attempts - 1:
                raise
            delay = min(cap, base * factor ** attempt)
            time.sleep(random.uniform(0, delay))   # "full jitter"

Idempotency keys

An operation is idempotent if doing it twice has the same effect as doing it once. Reading a balance is idempotent; "debit 500 rupees" is not. If the reply to a payment request is lost and the app retries, the customer could be charged twice. The fix is an idempotency key: the client creates a unique ID for each payment and sends it with every attempt, often in an Idempotency-Key header. The server stores the result for that key and, on a repeat, returns the saved result instead of charging again.

Circuit breakers

A circuit breaker wraps calls to a dependency and stops calling it when it keeps failing. It has three states:

Bulkheads, rate limits, load shedding, graceful degradation

• Bulkheads: separate pools of threads or connections per dependency, like the watertight compartments of a ship. A slow recommendations service can then use up only its own pool, not the threads that payments need.
• Rate limiting: cap how many requests each client may send, for example 100 per minute per API key, and answer 429 Too Many Requests above that.
• Load shedding: when the server is overloaded, reject low-priority work early (with 503) so the important work stays fast, instead of everything becoming slow.
• Graceful degradation: when a dependency fails, serve a reduced version. Show cached prices, hide "people also bought", but keep checkout working.

Capacity planning

Capacity planning means making sure you have enough resources for future demand. Combine normal growth with known peaks: a sale day, an IPL final, tatkal opening at 10 am, the last day for filing tax returns. Load test to find where the system breaks, keep headroom (many teams aim to run below about 60 to 70% of capacity at peak), keep spare capacity so one zone can fail (N+1), and scale up before a known event, because autoscaling takes minutes to react.

Chaos engineering

Chaos engineering is running careful experiments that inject failures on purpose, to learn how the system behaves before a real failure teaches you. Netflix made it popular with Chaos Monkey, which randomly stopped servers.

1. Define the steady state: a metric that shows normal behaviour, such as payment success rate.
2. Write a hypothesis: "If one of three payment pods is killed, success stays above 99.5%."
3. Inject a realistic failure: kill a pod, add 300 ms of network delay, fill a disk.
4. Keep the blast radius small: start in staging, then a small slice of production, with a clear abort switch.
5. Compare results with the hypothesis, fix weaknesses, and repeat.

Tools include Chaos Mesh and LitmusChaos (both CNCF projects for Kubernetes), AWS Fault Injection Service, Azure Chaos Studio and Gremlin.