DarDev adopted Flux GitOps on internal and select client clusters after too many where is the yaml that is running incidents. Git became the desired state; Flux reconciliation became the loop that applies and reports drift—when configured honestly, not as magic.
These lessons come from dardev-vps overlays, lab clusters, and SME migrations—not from a conference talk demo that hides secrets in bootstrap scripts.
Why Flux over push deploys
- Audit trail: every prod change ties to git commit
- Rollback: revert merge request instead of hunting kubectl history
- Drift visibility: manual kubectl edit surfaces as out-of-sync
- Multi-env: Kustomize overlays per staging and prod
Push-based CI still builds images; Flux pulls manifests and updates image tags via ImageUpdateAutomation when we enable it—split responsibility avoids CI holding cluster-admin.
Repo layout that survived review
Monorepo folder clusters/ with base and overlays/staging, overlays/production. Flux Kustomization points at overlay path; GitRepository watches main branch with tag or semver when we promote releases.
Bootstrap pains we hit
- CRD race
Install Flux CRDs before Kustomizations referencing them—bootstrap script order matters.
- Secret leakage temptation
Plain secrets in git rejected; use SOPS or SealedSecrets from day one.
- Health checks
Flux health waits on Deployment ready—missing probes causes false green.
- NetworkPolicy surprise
Flux controller could not reach GitLab until egress allowed.
When GitOps is overkill
Single VPS Docker Compose with one developer does not need Flux—GitLab CI ssh deploy is fine if documented. We say no to GitOps theater for Tunis startups with three containers and no staging.
Adopt when you have staging plus prod clusters, multiple contributors, and audit questions about who changed prod last Tuesday.
Drift and emergency patches
Incident kubectl scale is allowed—then immediately commit equivalent manifest or Flux reverts your fix. Teams that skip the commit recreate mystery state.
Suspend Kustomization temporarily for disaster recovery only—with ticket and time box.
Observability for reconciliation
Alert on Kustomization not Ready and GitRepository fetch failures. Small teams miss this and blame application when Flux stopped syncing three days ago.
Pair with gitlab-ci-kubernetes-pipeline for image build stages and staging-environments-match-prod for overlay discipline.
DarDevLab teaches Flux in gitops-lab-first-deploy before client projects assign GitOps migrations.
Helm versus Kustomize debate settled per repo pragmatism—Flux supports both; we standardize per client to reduce cognitive load.
Document bootstrap commands in runbook stored outside cluster—total loss recovery starts from empty k3s plus git clone.
Image automation cautiously
ImageUpdateAutomation commits tag bumps to git when CI pushes new semver—enable only after team trusts test gates. Automatic prod tag bump without staging sync caused our most memorable Friday incident; now staging must reconcile first.
Multi-cluster future
Some clients will run Tunis DR cluster later—Flux supports multiple Kustomizations with same repo different paths. Design overlays before second cluster exists to avoid forked repos.
RBAC: Flux service account needs only namespaces it manages—not cluster-admin in production.
Compare notes with compose-to-kubernetes-migration before GitOps on week-one cluster—sequence matters.
Windows operators use WSL for flux CLI; reconciliation still cluster-side.
Testing manifest changes safely
Use flux diff before merge when CLI version supports it—or render locally with kustomize build. Staging Kustomization reconciles within one minute; prod waits on manual promotion PR merging overlay tag bump.
Network policies tested in staging first—Flux must reach Git, application must reach DB; reconcile order documented.
Helm charts wrapped via HelmRelease when upstream maintains chart; Kustomize patches when we own plain YAML.
On-call runbook lists flux suspend command with approval requirement—emergency only.
Client handoff includes thirty-minute GitOps walkthrough recorded for their team—knowledge transfer not mystery tooling.
Upgrade Flux on schedule—lagging major versions breaks bootstrap compatibility with new Kustomize features.
GitOps does not replace backups—Postgres PVC snapshots continue regardless of manifest elegance.
Tunis clients on OVH often start GitOps after compose-to-kubernetes migration completes—do not parallelize two learning curves.
Flux notification controllers can post reconcile failures to Slack—wire after first month stable sync.
Keep kubectl cheatsheet for debugging pods—GitOps operator still needs runtime skills.
Store cluster bootstrap terraform or script in same org as manifests—disaster recovery spans repos if you split infrastructure.
Review Flux release notes quarterly—security patches apply to controller running inside your cluster.
SME success metric: zero unexplained prod drift for thirty days—Flux status all green.
Document who approves production manifest merges—GitOps without CODEOWNERS repeats old ssh-as-root politics.
Pair reading with staging-environments-match-prod so overlay diffs stay minimal and reviewable.
When in doubt, reconcile staging first and sleep before merging prod overlay at midnight.
GitOps maturity is measured in boring Tuesdays—not flashy launch day demos alone.
Contact DarDev Services if bootstrap exceeds internal capacity—we scope GitOps adoption separately from app development.
Treat every prod merge as a release note entry your future self will read.
Flux or Argo CD?
DarDev standardized Flux for lighter ops footprint; both valid—pick one per org.
Private GitLab?
Deploy key or token in Kubernetes secret; rotate on offboarding.
Monorepo or multi?
SME clients: monorepo clusters/ folder; enterprise may split by team.
Windows dev machines?
Edit manifests locally; Flux runs in cluster—WSL fine for kubectl checks.
DarDev implement GitOps for us?
DarDev Services scopes bootstrap plus training—contact via dardev.net.
