Pods, ReplicaSets and Deployments

Pods: the smallest unit

A Pod is a wrapper around one or more containers that always run together on the same node. The containers in a pod share one network identity (one IP address, so they can talk to each other on localhost) and can share storage volumes. Most pods have one main container. Sometimes a helper container runs next to it, called a sidecar, for example to ship logs or act as a network proxy.

Pods are ephemeral, which means temporary. If a node dies, its pods are gone; they are not moved. A new pod is created elsewhere with a new name and a new IP. So you should never depend on a pod's name or IP, and you should almost never create bare pods for an app. Let a controller such as a Deployment create them.

apiVersion: v1
kind: Pod
metadata:
  name: hello
  labels:
    app: hello
spec:
  containers:
    - name: web
      image: nginx:1.27
      ports:
        - containerPort: 80

Labels and selectors

Labels are key-value tags you stick on objects, such as app: payments-api, tier: backend or env: prod. A selector is a query that picks objects by their labels. This is how Kubernetes objects find each other: a Deployment finds its pods by label, and a Service finds the pods to send traffic to by label. You can use them too: kubectl get pods -l app=payments-api.

ReplicaSets and Deployments

A ReplicaSet keeps a set number of identical pods running. A Deployment sits on top: it creates and manages ReplicaSets, and adds rolling updates and a history you can roll back to. In practice you write Deployments and let them handle ReplicaSets for you.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: payments
  labels:
    app: payments-api
spec:
  replicas: 4
  selector:
    matchLabels:
      app: payments-api          # must match the template labels below
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%               # up to 1 extra pod during an update (25% of 4)
      maxUnavailable: 25%         # at most 1 pod below 4 during an update
  template:                       # the blueprint for every pod
    metadata:
      labels:
        app: payments-api
    spec:
      containers:
        - name: api
          image: registry.example.com/payments-api:1.4.2
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080

Rolling updates

When you change the pod template, for example a new image tag, the Deployment creates a new ReplicaSet and moves pods over a few at a time: start some new pods, wait until they are ready, stop some old ones, repeat. Users keep being served throughout. Two settings control the speed and safety:

• maxSurge: how many pods above the desired replicas may exist during the update. A percentage is rounded up.
• maxUnavailable: how many pods below the desired replicas may be unavailable. A percentage is rounded down.
• Both default to 25%. They cannot both be 0, or the update could never make progress.

Worked example: replicas is 10 with the defaults. Surge is 25% of 10 = 2.5, rounded up to 3, so at most 13 pods exist. Unavailable is 2.5 rounded down to 2, so at least 8 pods stay available. With replicas 4 you get at most 5 pods and at least 3 available.

The other strategy is Recreate: stop all old pods first, then start the new ones. It causes downtime, but it is needed when two versions must never run at the same time, for example when the app takes an exclusive lock on a file or database.

kubectl apply -f deployment.yaml                    # after changing the image tag in the file
kubectl rollout status deployment/payments-api -n payments    # wait and watch
kubectl rollout history deployment/payments-api -n payments   # list revisions
kubectl rollout undo deployment/payments-api -n payments      # back to the previous revision
kubectl rollout undo deployment/payments-api -n payments --to-revision=3
kubectl scale deployment/payments-api -n payments --replicas=8

Scaling automatically

kubectl scale changes replicas by hand. A HorizontalPodAutoscaler (HPA) changes them for you, for example to keep average CPU use near 70% of the requested CPU, between a minimum and a maximum. The HPA needs a metrics source, usually metrics-server, installed in the cluster.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payments-api
  namespace: payments
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-api
  minReplicas: 4
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Pod statuses and how to debug them

The STATUS column of kubectl get pods shows either the pod phase (Pending, Running, Succeeded, Failed) or, when a container is stuck, the reason it is waiting, such as CrashLoopBackOff.

kubectl get pods -n payments
kubectl describe pod payments-api-6c9f-zt4qp -n payments    # Events at the bottom
kubectl logs payments-api-6c9f-zt4qp -n payments --previous # output of the crashed run
kubectl get events -n payments --sort-by=.metadata.creationTimestamp