PromQL queries for deploy health

A deploy is healthy when error rate stays flat, new pods pass readiness, and p95 latency does not drift more than fifty percent above the pre-release baseline. PromQL expresses those checks as time-series math Prometheus evaluates every scrape—then Alertmanager pages you, or GitLab CI blocks the pipeline until numbers recover. You need six to eight queries tied to deploy events, not another dashboard with thirty panels nobody reads after week two.

DarDev attaches the same PromQL bundle to GitLab deploy jobs on dardev-vps and client Kubernetes clusters. Pair these queries with runbook-rollback-bad-deploy so the engineer who merged knows when to undo versus when to wait out a slow rollout. If golden signals are new to your team, read logs-metrics-traces first—deploy health is metrics-first work.

What deploy health means in PromQL

Deploy health is a short window—typically fifteen to thirty minutes after the pipeline finishes—where you compare current signals to a baseline from before the change. Baselines use offset (same time yesterday or one hour ago) or recording rules that snapshot pre-deploy values. Avoid comparing Monday traffic to Friday night; annotate Grafana with deploy timestamps so on-call can see correlation without guessing.

• Error rate — 5xx or application error counter versus traffic
• Readiness — kube_pod_status_ready or equivalent health checks
• Rollout progress — unavailable replicas or stale deployment generation
• Latency — histogram p95 drift versus offset baseline
• Saturation — CPU, memory, or connection pool pressure after new code ships

Query 1: HTTP 5xx rate after deploy

Page when user-visible failures spike. Use rate over five minutes to smooth scrape jitter; divide 5xx by all requests so a quiet service does not false-positive on a single error.

sum(rate(http_requests_total{status=~"5..", namespace="production"}[5m]))
/
sum(rate(http_requests_total{namespace="production"}[5m]))
> 0.02

Tune threshold to your SLO and add for: 5m. nginx_ingress_controller_requests_total works if the app lacks RED metrics—see prometheus-metrics-every-saas.

Query 2: Pods not ready in the target namespace

Rolling updates leave old pods until new ones pass probes. This query fires when any pod in production reports ready=false—often a crashed container, bad config mount, or migration lock.

sum(
 kube_pod_status_ready{condition="false", namespace="production"}
 * on(pod, namespace) group_left()
 kube_pod_labels{label_app_kubernetes_io_name="api"}
) > 0

Query 3: Deployment rollout stalled

Unavailable replicas staying above zero for ten minutes usually means image pull failure, probe misconfiguration, or resource limits—not a slow JVM warm-up.

kube_deployment_status_replicas_unavailable{
 namespace="production",
 deployment="api"
} > 0

Pair with observed_generation != metadata_generation for stuck reconciles. Pass DEPLOYMENT_NAME from gitlab-ci-kubernetes-pipeline to template one rule per service.

Query 4: Latency drift versus pre-deploy baseline

Error rate can stay green while p95 doubles—users feel that before your counter moves. Compare current histogram quantile to the same quantile one hour ago with a multiplier guard band.

histogram_quantile(
 0.95,
 sum(rate(http_request_duration_seconds_bucket{namespace="production"}[5m])) by (le)
)
>
1.5 * histogram_quantile(
 0.95,
 sum(rate(http_request_duration_seconds_bucket{namespace="production"}[5m] offset 1h)) by (le)
)

Requires histogram buckets; summary metrics need avg latency with offset until you migrate type.

Query 5: Queue or worker backlog after release

API deploys often break workers silently. Alert when depth grows while consumers run—the pattern we use for Listmonk campaign workers and Hesabi async jobs.

(
 sum(rabbitmq_queue_messages{queue="jobs", vhost="production"})
 -
 sum(rabbitmq_queue_messages{queue="jobs", vhost="production"} offset 30m)
) > 100

Swap metric names for Redis, Sidekiq, or BullMQ. Compare delta versus thirty minutes ago, not absolute depth.

Query 6: Saturation on nodes running the new version

100 - (
 avg by (instance) (
 rate(node_cpu_seconds_total{mode="idle"}[5m])
 ) * 100
) > 85

Add memory with MemAvailable ratio below ten percent. grafana-dashboards-small-teams shows RED plus node panels on one row for deploy review.

Wire queries to GitLab deploy gates

Label alert rules severity=deploy-gate. After apply, CI polls Prometheus /api/v1/query or Alertmanager until soak completes—fail if any expression is true after ten minutes.

1. Annotate Grafana at pipeline start with CI commit SHA
2. Apply only deploy-gate rules in staging first for one sprint
3. Route deploy-gate failures to the merge author via alert-fatigue-routing patterns
4. Link rollback runbook in alert annotation description

Common mistakes

• Alerting on pod restart count during normal rolling updates without for: duration
• Using instant queries in CI without rate() on counters
• High-cardinality labels in deploy rules—never put commit SHA in metric labels
• No baseline—absolute thresholds that fire every Monday morning
• Twenty deploy queries when rollback decision needs three numbers

How DarDev helps

DarDev Services observability engagements include PromQL review, deploy-gate wiring in GitLab, and alert budgets matched to team size. Request a scoped assessment at dardev.net/products.