When a deploy breaks production, you have roughly ten minutes before error budgets burn and customers notice. Rolling back restores the last known-good artifact—container image, Compose stack, or git manifest—not debugging forward under fire. This DarDevOps runbook covers git revert, kubectl rollout undo, Compose pull, health verification, and stakeholder comms—the sequence we use on Kubernetes clusters and dardev-vps Compose stacks. Rollback is faster when staging matched production and deployment history still holds the previous image tag.
Decide rollback in 60 seconds
- Error rate or 5xx spiked within minutes of deploy
- Readiness probes failing on new pods
- Critical smoke test fails (login, checkout, /health)
- No config fix you can apply in under two minutes
Fix forward only when the bug is a bad env var you can patch without rebuilding, or when rollback would leave a destructive migration half-applied. When in doubt, roll back—customers prefer yesterday's features over today's outage. We assume GitLab CI pushed immutable SHA-tagged images as in our GitLab CI to Kubernetes guide.
The ten-minute timeline
- 0–1 min: Freeze
Post in incident channel. Cancel in-flight CI deploy jobs. Lock production deploy in GitLab.
- 1–3 min: Revert git
git revert the merge commit or reset manifest to previous image tag. GitOps reconciles within one to two minutes.
- 3–6 min: Roll back runtime
kubectl rollout undo on Deployments, or docker compose pull + up with previous tag on VPS stacks.
- 6–8 min: Verify health
Pods ready, smoke checks green, error-rate PromQL below threshold for five minutes.
- 8–10 min: Communicate
Status update: what broke, what you did, what is still degraded. Schedule post-mortem within 48 hours.
Git revert the bad change
git log --oneline -5 main
git revert -m 1 <merge-commit-sha> --no-edit
git push origin main
# GitOps / Helm: pin previous image in deployment.yaml
# image: registry.example.com/api:abc123previousSource control is the audit trail—do not force-push unless policy allows it during incidents. With Flux or Argo CD, reverting git often triggers cluster rollback automatically; see GitOps with Flux for our bootstrap pattern.
Kubernetes rollback commands
kubectl rollout history deploy/api -n production
kubectl rollout history deploy/api -n production --revision=3
kubectl rollout undo deploy/api -n production
kubectl rollout undo deploy/api -n production --to-revision=2
kubectl rollout status deploy/api -n production --timeout=120s
kubectl get pods -n production -l app=apiDeployment objects keep ReplicaSet history—undo reverts without rebuilding. For multiple services, undo in dependency order: API before workers, edge last. kubectl describe pod shows which sidecar blocks readiness on the whole pod.
Docker Compose rollback on VPS
cd /opt/my-stack
git checkout HEAD~1 -- docker-compose.dardev-vps.yml
docker compose -f docker-compose.yml -f docker-compose.dardev-vps.yml pull api
docker compose -f docker-compose.yml -f docker-compose.dardev-vps.yml up -d --no-deps api
docker compose ps
curl -sf http://127.0.0.1:3001/healthzOur internal mailer stack on dardev-vps uses pinned tags in docker-compose overlays—Twenty, Listmonk, sync worker—same discipline as K8s prod. Use --no-deps so Postgres and Redis keep running. Pull before up. Retain previous registry tags at least seven days; garbage-collected images make Compose rollback impossible.
Verify health before closing the incident
- kubectl get pods — all Running, READY matches desired
- curl -sf /health plus one business-critical path
- Grafana: 5xx rate at baseline for five minutes
- Logs: no new panics since rollback timestamp
Green CI does not mean green production. Run the same checks your pipeline should have blocked before promote. Use PromQL queries for deploy health and wire alerts to dashboards that link here—see Grafana dashboards for small teams. Wait for metrics to stabilize, not a single successful curl.
Communicate clearly
Subject: [Resolved] Brief service disruption
We deployed an update at 14:05 UTC that caused elevated API errors.
We rolled back to the previous version at 14:12 UTC.
All systems are operating normally. Summary within 48 hours.Assign an incident commander so one person coordinates comms while others execute commands. Internal channel gets deploy time, rollback time, and current status. External update when user-visible—no jargon, no blame. Update every fifteen minutes until metrics stabilize, then once at resolution.
After rollback
- Reproduce in staging that matches production
- Add the missing CI health gate
- Post-mortem with bad revision and root cause
- Migrations: plan forward-only repair—do not blindly re-roll
DarDevOps engagements include runbooks tied to your cluster and Compose layouts, plus staging drills before production cutover—dardev.net/products.
Rollback or fix forward?
Roll back when errors spiked right after deploy and you cannot patch in two minutes. Fix forward for ConfigMap typos, feature flags, or non-destructive env vars. When database migrations ran, assess reversibility before undoing app pods—schema rollback is a separate problem.
How many revisions does rollout undo keep?
Default ten old ReplicaSets per Deployment (revisionHistoryLimit). Increase on critical services. Beyond history depth, redeploy a known-good tag from git.
What if a database migration already ran?
Container rollback does not undo schema. If backward-compatible, roll back the app and fix forward. If not, engage DBA before touching pods.
Same runbook for Compose and Kubernetes?
Same flow: freeze, revert git, restore artifact, verify, communicate. Commands differ—rollout undo versus compose pull and up.
Where does DarDev apply this?
On client Kubernetes clusters and internal stacks we operate—with GitLab promote paths and Prometheus health gates. See dardev.net/products.
