GitOps and platform engineering

What is GitOps?

GitOps is a way of running systems where Git is the single source of truth. The desired state (what should be running: Kubernetes manifests, Helm values, image tags) lives in a Git repository. A software agent compares that desired state with what is actually running and fixes any difference. Nobody changes production by hand; they change Git, and the agent does the rest.

The OpenGitOps project describes four principles:

1. Declarative: you describe what you want ("3 replicas of image canteen-api:3f9c2ab"), not the steps to get there.
2. Versioned and immutable: the desired state is stored in Git, so every change has an author, a review and a history.
3. Pulled automatically: an agent fetches the desired state; nobody needs to push it into the cluster.
4. Continuously reconciled: the agent keeps comparing and correcting, not just once at deploy time.

Push versus pull

Argo CD and Flux are the two best-known GitOps tools, and both are graduated CNCF (Cloud Native Computing Foundation) projects. Argo CD has a web dashboard that many teams like; Flux is a set of controllers that you mainly drive through Git and the command line.

Reconciliation, sync status and drift

Reconciliation is the loop that compares desired state (Git) with live state (the cluster). In Argo CD an application is Synced when they match and OutOfSync when they differ. Drift is a difference caused by something other than Git, for example someone running kubectl scale at midnight to "fix" a slow service. With self-heal turned on, Argo CD changes the cluster back to what Git says.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: canteen-api-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/acme/canteen-deploy.git
    targetRevision: main
    path: apps/canteen-api/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: canteen
  syncPolicy:
    automated:
      prune: true      # delete resources that were removed from Git
      selfHeal: true   # undo manual changes (drift)

Rollbacks with git revert

If a release goes wrong, you roll back by reverting the commit that changed the image tag. The agent sees the new commit and syncs the cluster back to the previous version. Git still tells the whole story: what changed, who rolled it back and when.

git log --oneline -3 apps/canteen-api/overlays/prod
# 8c41d2e prod: canteen-api 3f9c2ab -> 7b0e915
git revert 8c41d2e
git push origin main      # Argo CD syncs back to 3f9c2ab within minutes

Environment promotion through pull requests

A common layout keeps application code in one repository and deployment configuration in another. CI builds the image, tags it with the commit SHA and updates the dev environment automatically. Moving the same image to staging and then production is a pull request that changes one line: the image tag in that environment's folder. The review and approval of that pull request is your change record.

canteen-deploy/
  apps/canteen-api/
    base/                 # Deployment, Service, shared settings
    overlays/
      dev/                # image tag updated by CI bot
      staging/            # promoted by pull request
      prod/               # promoted by pull request, needs 2 approvals

Platform engineering and internal developer platforms

When a company has 5 teams, each can build its own pipeline. With 50 teams, everyone re-solving CI, Kubernetes, secrets and dashboards wastes time and creates 50 different security levels. Platform engineering is the practice of a platform team building an internal developer platform (IDP): a set of self-service tools and paved roads that other teams use to build and run their services.

• Golden path: the supported, recommended way to do a common job, such as "create a new Node.js service". It is the easy road, not a forced one.
• Self-service templates: one form or command creates a repository with a Dockerfile, a CI pipeline, a Helm chart, dashboards and alerts already wired up.
• Backstage: an open-source framework for building developer portals, created at Spotify and now a CNCF project. It offers a software catalogue (who owns which service), templates and documentation in one place.

Measuring developer experience

Treat the platform as a product with internal customers, and measure whether it helps them. Useful measures: the DORA metrics (deployment frequency, lead time for changes, change failure rate and time to recover from a failed deployment), time for a new service to reach production, time for a new joiner to make a first deploy, how many teams use the golden path, and short regular surveys asking developers what slows them down.