Skip to content

Intermediate · CI/CD pipelines · topic 29 of 53 · 35 min

Deployment strategies, feature flags and GitOps

Explanation

How you release matters as much as what you release. Recreate and rolling updates are simple; blue-green switches all traffic at once between two environments; canary sends a small share of users to the new version first and promotes it only if its metrics look healthy. Feature flags separate deploying code from releasing a feature, database changes use expand and contract, and GitOps lets an agent in the cluster pull the desired state from Git.

Why strategy matters

Every release is a risk. A deployment strategy decides how the new version replaces the old one, which controls how many users a bad release can hurt and how fast you can undo it. There is no single best strategy; each trades cost, speed and safety.

StrategyHow it worksDowntimeRollbackExtra cost
RecreateStop all old instances, then start new onesYes, brieflyRedeploy the old versionNone
RollingReplace instances a few at a timeNoRoll back graduallyA few extra instances
Blue-greenRun the new version (green) next to the old (blue), then switch all trafficNoSwitch traffic back at onceDouble capacity during the release
CanarySend a small share of traffic to the new version, watch metrics, then increaseNoSend all traffic back to the old versionSmall, plus good monitoring

Blue-green

You keep two identical environments. Blue serves users today. You deploy the new version to green, test it there, then switch the router or load balancer so all traffic goes to green. If something is wrong, you switch back to blue in seconds. The cost is running two full environments, and both versions must work with the same database.

Canary

The name comes from miners who carried a canary into the mine as an early warning. A canary release sends a small share of traffic, say 5%, to the new version. You compare its metrics with the old version (the baseline): error rate, latency and business numbers such as payment success rate. If they look as good, you promote in steps (5%, 25%, 50%, 100%). If not, you send all traffic back. Tools such as Argo Rollouts and Flagger can run this analysis automatically.

Latency is often measured as p95: the time within which 95% of requests finish. It shows the slow experience of real users better than an average does.

Feature flags

A feature flag is a switch in the code, controlled from outside without a new deploy. It separates deploying (the code is on the servers) from releasing (users can see the feature). With flags you can do a dark launch (the code runs in production but no user sees it yet), turn a feature on for staff, then 5% of users, then everyone, and keep a kill switch to turn a broken feature off in seconds.

Tip: Every flag is a small piece of debt. Once a feature is fully released, delete its flag and the old code path. Old flags that nobody remembers make code hard to read and test.

Rollback or roll forward

A rollback goes back to the previous version. A roll-forward ships a quick fix as a new version. Rollback is usually the safest first move when users are affected. Roll forward when a rollback is not possible, for example after a database change the old code cannot handle.

Database changes: expand and contract

During a rolling or canary release, old and new code run at the same time against one database. So every database change must work with both versions. The expand and contract pattern (also called parallel change) splits a risky change into safe steps. Example: renaming the column phone to mobile_number.

  1. Expand: add the new column mobile_number. Old code ignores it.
  2. Deploy code that writes to both columns and reads from the old one.
  3. Migrate: copy existing data from phone to mobile_number in batches.
  4. Deploy code that reads from mobile_number.
  5. Contract: in a later release, stop writing phone and then drop the column.

GitOps, briefly

In GitOps, a Git repository holds the desired state of your environments (Kubernetes YAML, Helm values). An agent running inside the cluster, such as Argo CD or Flux, keeps pulling from Git and applying any difference: the reconciliation loop again. CI builds and publishes the image, then only updates the image tag in Git. This pull model means the CI system needs no cluster credentials, every change is a reviewed commit, and rollback is a git revert.

Careful: Common mistakes. (1) A canary without metrics to compare is just a slow rollout. Decide the promote and rollback rules before you start. (2) Dropping or renaming a column in the same release as the code change, which breaks the old pods still running. (3) Flags that never get removed. (4) Blue-green without testing green under real-like traffic first. (5) No practised rollback: try your rollback in staging before you need it at 2 am.

Need help?

Stuck on a word or an idea? Search this course, or every course, and jump straight to the explanation or a worked question.

Try:

Watch the lesson

Deployment strategies, feature flags and GitOps · 1/7

Deployment strategies

In this lesson you will compare ways to release a new version safely, and learn feature flags, database migrations and GitOps.

Prefer a video by a teacher? Find free videos on “blue green vs canary deployment feature flags explained” on YouTube ↗

In real life: A UPI payments app rolls out "split the bill" to 5% of users

A UPI payments app with millions of daily users builds a new "split the bill" feature. The code is deployed dark: it is on the servers behind a feature flag that is off. Staff test it for a week. Then the flag is turned on for 5% of users, chosen by a stable hash of the user ID so each user keeps the same experience. At the same time, the new backend version runs as a canary taking 5% of traffic. The team compares the canary with the baseline every 10 minutes: error rate, p95 latency and payment success rate. On day one the canary's p95 latency is 40% higher because of a missing database index, so the analysis says rollback, the flag is turned off, and no money is affected. After the fix, the rollout goes to 25%, 50% and 100% over a week.

python
baseline = {"requests": 200000, "errors": 300, "p95_ms": 420}   # 0.15% errors
canary   = {"requests": 10000,  "errors": 16,  "p95_ms": 590}   # 0.16% errors
# error rate is fine, but 590 > 420 * 1.2 = 504, so the decision is "rollback"
The day-one numbers: latency, not errors, triggered the rollback.

Practice (0/3 done)

Task 1 · Question

Which strategy lets you switch ALL traffic back to the old version almost instantly, at the cost of running two full environments?

Task 2 · Code · Python

Write canary_decision(baseline, canary, max_error_increase=0.01, max_latency_ratio=1.2). Each side is a dict {"requests": int, "errors": int, "p95_ms": number}, with at least one request. Error rate is errors / requests. Return "rollback" if the canary's error rate is more than the baseline error rate plus max_error_increase, or if the canary's p95 is more than the baseline p95 times max_latency_ratio. Otherwise return "promote".

Task 3 · Code · Python

Percentage rollouts must be stable: the same user should get the same answer every time. Write in_rollout(user_id, flag, percent): hash the text flag + ":" + user_id with hashlib.sha256, turn the hex digest into an int, take it modulo 100 to get a bucket from 0 to 99, and return True if the bucket is below percent. (Python's built-in hash() changes between runs for strings, so it is not suitable here.)

Quick quiz

Question 1

1.What does a feature flag let you do?

Question 2

2.You must rename a database column while old and new pods run together. What is the safe approach?

Question 3

3.In a canary release, what decides whether to promote?

Question 4

4.What is the "pull model" in GitOps?

Question 5

5.What does p95 latency of 400 ms mean?

Read more (free)

Open a question to see the answer and why. The first ones are this topic's own quiz, so try the quiz above before you peek.

What does a feature flag let you do?

Answer: Release a feature to some users, or turn it off, without a new deploy

Flags separate deploying code from releasing features, enabling dark launches, percentage rollouts and kill switches.

You must rename a database column while old and new pods run together. What is the safe approach?

Answer: Expand and contract: add the new column, write both, migrate, switch reads, drop the old column later

Each step works with both code versions, so a rolling or canary release never breaks.

In a canary release, what decides whether to promote?

Answer: Comparing the canary's metrics, such as error rate and p95 latency, with the baseline

Agreed metrics and thresholds, checked before each step, decide promote or rollback.

What is the "pull model" in GitOps?

Answer: An agent in the cluster, such as Argo CD or Flux, pulls the desired state from Git and applies it

The cluster pulls from Git, so CI needs no cluster credentials and Git history is the deployment history.

What does p95 latency of 400 ms mean?

Answer: 95% of requests finished within 400 ms

p95 is the 95th percentile: only 5% of requests were slower.

In a GitOps setup, how do you usually roll back a bad production release?Related

Answer: git revert the commit that changed the image tag and let the agent sync

Git stays the source of truth, and the rollback is reviewed and recorded like any change.

From GitOps and platform engineering

What is a preview deployment?Related

Answer: A temporary copy of the app built from a pull request, with its own URL

Previews let reviewers try a change for real before it is merged. They should use preview data, not production.

From CI/CD: shipping every change safely

Which strategy stops all old pods before starting new ones?Related

Answer: Recreate

Recreate causes a short outage but guarantees that two versions never run together.

From Pods, ReplicaSets and Deployments

In trunk-based development, how do teams merge unfinished features safely?Related

Answer: They hide unfinished work behind feature flags

Feature flags let half-built code be merged and deployed while switched off, so branches stay short-lived.

From Collaboration: remotes, pull requests and branching strategies

Pass the quiz (60% or more) or mark this topic as done when you are ready.

Your progress is saved on this device only.

Next: Why infrastructure as code →

Next steps

  1. This is the last topic of “CI/CD pipelines”. The module exam comes next.
  2. Module 6 exam: CI/CD pipelines · 12 questions · 20 min, without help. 5 topics of this module still to finish first.
  3. Then the next module, Infrastructure as code with Terraform. 53 topics left in the course before the final exam.

← Previous: Testing, quality gates and artefacts