Building a Kubernetes-native product pipeline

A Kubernetes-native product pipeline moves every release through the same path: GitLab CI builds and tests code, publishes an immutable container image, deploys to staging with automated health gates, then promotes to production only when probes, smoke checks, and metrics say the rollout is safe. That is how DarDev ships internal products—Twenty CRM, Listmonk, sync workers, and client-facing services—without treating production as a special-case SSH session.

We organize delivery around four steps—Authorize, Develop, Deploy, Observe—mapped to Workspace IAM, DarDevCDE dev environments, DarDevOps clusters, and Prometheus/Grafana observability. Kubernetes is the shared runtime; GitLab is the orchestration layer that connects git commits to running pods.

Pipeline flow: build, image, staging, production

1. Authorize: merge requests and protected branches gate who can trigger deploy jobs
2. Develop: feature branches run unit tests; DarDevCDE or local Docker validates integration
3. Build: GitLab CI produces a tagged image in the registry—never reuse floating latest in prod
4. Staging: automated deploy to a namespace that mirrors production topology
5. Health gates: readiness probes, HTTP smoke tests, and PromQL checks must pass
6. Production: manual or canary promote with rollback-ready previous image tag

Skipping staging because the diff looks small is how regressions reach customers. Our mailer stack on dardev-vps follows the same discipline even when only one service changes: staging namespace first, metrics quiet, then production overlay update.

GitLab CI: where build meets cluster

GitLab CI stages split concerns. Test jobs run on every push. Build jobs use BuildKit cache, push $CI_COMMIT_SHA tags, and optionally semver tags on git releases. Deploy jobs use namespace-scoped kubeconfig tokens—staging runners cannot delete production resources. For a concrete.gitlab-ci.yml skeleton and runner hardening notes, see our GitLab CI to Kubernetes guide.

Image immutability matters: once an image tag ships to staging, production promotes that exact digest. Rebuilding on the prod job introduces untested layers. Database migrations run as Kubernetes Jobs using the same image tag as the app deploy—not from a developer laptop.

Staging that earns trust

Staging is not a smaller prod with fake data only—it should share ingress class, TLS termination pattern, Postgres major version, and resource limit tiers with production. We document fidelity requirements in staging environments that match production. When staging uses SQLite and prod uses Postgres, you are testing fiction.

Merge request review apps add ephemeral namespaces per MR with URLs posted to GitLab. For long-lived staging, Flux or Helm overlays pin image tags until a human approves promotion.

Health gates before production

Health gates are the contract between CI green and actually safe. Kubernetes readiness probes must match real dependencies—if your app needs Postgres, the probe should fail when Postgres is unreachable, not return 200 from a static page.

• Readiness and liveness probes aligned with app startup time—not 1s timeout on a 40s JVM boot
• Post-deploy smoke script hitting /health and one critical API path
• PromQL query: error rate below threshold for five minutes after deploy
• Optional: compare p95 latency to pre-deploy baseline
• Manual approval job on protected production environment in GitLab

We tie deploy events to Prometheus annotations so on-call can correlate error spikes with pipeline IDs. Useful PromQL patterns for deploy health are covered in PromQL queries for deploy health. If metrics go red after staging promote, production never starts—fix forward in staging, do not bypass gates because Friday is busy.

Production promotion and rollback

Production deploys are deliberate. GitLab environment with deployment tier production, when: manual on the prod job, and protected branch rules on main. Canary or blue-green is optional; kubectl rollout undo with a pinned previous image tag is mandatory knowledge for every platform engineer on the team.

When teams outgrow kubectl set image, GitOps with Flux separates build (CI pushes image) from desired state (git declares which tag runs). We share bootstrap lessons in GitOps with Flux—CI finishes at registry push; the cluster reconciler applies manifests.

Internal patterns we run today

DarDev dogfoods this pipeline on the unified mailer stack—Twenty CRM, Listmonk, Stalwart MTA, sync worker—on dardev-vps with Compose for local dev and Kubernetes overlays for staging. Subdomain boundaries (crm.dardev.net, mail.dardev.net, send.dardev.net for outbound) live in git. Observability closes the loop: Grafana dashboards, deploy-tied alerts, logs retained for rollback forensics.

Failure modes to avoid

• Floating latest tag in production manifests
• Staging namespace missing NetworkPolicies present in prod
• CI deploy success while pods CrashLoopBackOff—missing probe validation in pipeline
• Secrets baked into images instead of Kubernetes Secrets or SOPS
• No documented rollback path until after the first outage

Start with one service, one staging namespace, one manual production gate. DarDevOps applies these patterns on engagements before recommending enterprise add-ons—see dardev.net/products for platform overview and contact.