Prometheus deploy monitoring means every production release leaves a visible mark in your metrics stack: a Grafana annotation, a version label bump, or a short-lived counter from CI. Without that correlation, a 5xx spike ten minutes after Friday deploy looks like random noise—on-call greps logs, debates whether the app or the network broke, and rollback takes twice as long. With deploy hooks wired correctly, the dashboard shows a vertical line at 18:04 UTC, tag v1.8.3, pipeline #4821, and the error-rate panel jumps in the same scrape window.
DarDevOps runs this pattern on dardev-vps and client Kubernetes clusters: GitLab CI emits deploy events, kube-state-metrics exposes rollout state, and Grafana ties graphs to runbook-rollback-bad-deploy. This guide covers the hooks we actually maintain—not a generic observability wish list.
What a deploy event should capture
- Timestamp (UTC) when production traffic switched to the new artifact
- Service or namespace affected
- Image tag or git SHA from CI
- Pipeline or job URL for audit
- Who approved manual production gates, when applicable
Store human-readable context in annotations; keep numeric signals in metrics. Annotations do not replace alerts—they explain them. When someone pages at 2 a.m., the first question is always what changed; deploy metadata answers that before anyone opens kubectl.
Grafana annotations from GitLab CI
The lowest-friction hook is a curl to Grafana's HTTP annotations API at the end of your production deploy job. GitLab masked variables hold GRAFANA_URL and a service account token with Editor role on the folder that owns production dashboards. Post JSON with tags deploy, production, and the service name; include pipeline ID and commit SHA in the text field.
#.gitlab-ci.yml — after kubectl rollout status succeeds
curl -sS -X POST "$GRAFANA_URL/api/annotations" \
-H "Authorization: Bearer $GRAFANA_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"time\": $(date +%s000),
\"tags\": [\"deploy\",\"production\",\"api\"],
\"text\": \"Pipeline $CI_PIPELINE_ID — $CI_COMMIT_SHA — $CI_COMMIT_TITLE\"
}"Enable annotation overlays on RED panels in Grafana—error rate, latency, saturation. We reuse dashboard layout from grafana-dashboards-small-teams; deploy markers belong on the same row as HTTP 5xx, not buried on a CPU graph nobody watches during incidents. If you run multiple environments, tag staging deploys separately so production on-call is not distracted by MR review-app noise.
Kubernetes rollout metrics
kube-state-metrics exposes kube_deployment_status_replicas_unavailable, kube_replicaset_owner, and related series. Alert when unavailable replicas stay above zero five minutes after a rollout—often a bad readiness probe or config mount, not a traffic spike. Pair deployment labels with your app metrics: expose build_info{version="$CI_COMMIT_SHA"} as a gauge set to 1 at startup so PromQL can group error rates by version.
For GitOps clusters, the deploy event is the merge to main that changes the image tag in git. Flux or Argo CD reconciliation time is when production actually shifts—emit the annotation from a post-sync hook Job or from CI after the sync health check passes, not only when the pipeline builds the image. See gitlab-ci-kubernetes-pipeline for the stage order we use before production promotion.
Post-deploy alert windows
Static thresholds miss deploy regressions because baseline shifts. Use short-window comparisons: rate(http_requests_total{status=~"5.."}[5m]) after deploy versus the same query offset 1h, or increase() on a deploy counter scraped from CI. promql-deploy-health documents queries we attach as optional GitLab deploy gates—fail the job when error rate doubles within fifteen minutes of rollout.
- Baseline: 5xx rate under 0.5% for thirty minutes pre-deploy
- Gate: block or warn if 5xx rate exceeds 2× baseline for ten minutes post-deploy
- Rollback trigger: page on-call if SLO burn continues twenty minutes; link runbook-rollback-bad-deploy in the alert annotation
Route deploy-related alerts to the engineer who merged, not the entire team. Logs still matter for root cause; metrics tell you whether to roll back first. logs-metrics-traces explains when Loki search earns its disk after the PromQL spike fires.
Pushgateway vs application /metrics
CI jobs are batch processes—Prometheus cannot scrape a finished GitLab runner job. Use Pushgateway for deploy counters and last_deploy_timestamp gauges pushed once per pipeline, then expire stale groups with --web.enable-admin-api or TTL hygiene. Do not route long-lived service metrics through Pushgateway; that path is for ephemeral batch metadata only.
Application pods expose /metrics on a stable ServiceMonitor target. On dardev-vps Compose stacks, push the same Grafana annotation from deploy scripts after docker compose up --wait—the missing piece is usually the marker, not another exporter.
Checklist before the next production release
- Verify annotation API
Dry-run curl from a GitLab runner with production credentials; confirm markers appear on the right dashboard folder.
- Label version once per pod
Expose build_info or equivalent at startup; confirm Prometheus targets scrape after deploy.
- Test post-deploy query
Run promql-deploy-health gate queries against staging with an intentional bad deploy.
- Link runbook in alerts
Every deploy-regression alert includes rollback steps and pipeline URL.
DarDevOps engagements start with this baseline on clusters we operate alongside Twenty CRM, Listmonk, and product APIs. Platform details live at dardev.net/products; more DarDevOps guides on news.dardev.net.
Minimum setup for a two-person team?
Grafana annotations from your production deploy script plus one 5xx-rate panel with markers. Add kube-state-metrics when you run Kubernetes with more than one Deployment.
Should CI fail if post-deploy error rate rises?
Warn on staging always; on production prefer alert plus manual rollback unless you have reliable smoke tests and stable traffic. A false-positive pipeline block during a marketing spike hurts more than a delayed auto-rollback.
How is this different from promql-deploy-health?
This article covers event hooks and correlation—annotations, labels, CI integration. promql-deploy-health lists specific PromQL queries and thresholds for deploy gates.
Pushgateway on a small VPS—worth it?
Only if you need numeric deploy counters in PromQL and cannot expose them from the app. Annotations alone satisfy most SME incident workflows without another component to patch.
Where does DarDev run this?
On production clusters and dardev-vps stacks we operate for DarDev products and selected client DarDevOps engagements—same GitLab, Prometheus, and Grafana reference set documented on dardev.net.
