Operations / Field guide

Safe releases, canaries & rollback

Expose a change gradually and recover without confusing shared state.

Traffic rolls back; data stays

ONE CUSTOMER · TWO CODE VERSIONSREQUEST TRAFFIC95% stable5% candidateStable computeCandidate computeShared data persists

A small, stable audience uses candidate code while both versions share durable data.

01 / CONTEXT

Why it matters

A canary limits the first audience for a release. Blue and green keep old and new compute available during a switch. Neither approach is safe by traffic percentage alone: both versions may touch the same data and trigger the same external effects.

A rollback can restore earlier code or routing. It cannot erase a payment, email, migration, or write already committed by the candidate. Design data compatibility and a forward-fix boundary before the release starts.

02 / DESIGN

Decisions to make

  • Choose the unit of exposure: one customer, a stable user cohort, a request percentage, or a whole deployment.
  • Decide which entry points must honor the same traffic split and session affinity.
  • Keep durable data and identity owned outside replaceable compute, with compatible versions on both sides.
  • Choose one fenced owner for scheduled jobs, queues, streams, and outbound effects during the release.
  • Set health metrics, a bake period, promotion criteria, and a clear stop or rollback action.
  • Mark the point after which a data change requires a forward fix instead of old-code rollback.
03 / IMPLEMENTATION

Build checklist

  • Publish an immutable candidate version and keep a known-good version available.
  • Apply additive schema changes before shifting traffic; remove old fields only after all readers move.
  • Use a stable cohort so one user does not bounce between incompatible versions.
  • Send a small audience to the candidate and compare errors, latency, and business outcomes.
  • Test that a zero-percent candidate stops its background workers and outbound effects.
  • Rehearse promotion and reversal while checking shared data remains readable by both versions.
04 / WATCH FOR

Common mistakes

  • Calling an operator fleet wave a canary for one customer’s own users.
  • Splitting HTTP requests while both versions independently run the same background job.
  • Promising instant rollback after an incompatible migration or external side effect.
05 / GO DEEPER

Tools & references

3 starting points to compare or study. Check fit, maintenance, terms and current capabilities before choosing one.

BUILDING THIS NOW?

Explore the SaaSLayer approach.

See the services and developer guides behind our AWS backed SaaS building blocks.

Read SaaSLayer docs
NEXT FEATURERegional authentication continuity