End-to-End Delivery Workflow

What: how a commit in an application repo becomes a running pod — the two-repo GitOps model, the GAP wrapper-chart deployment artifact, and the Kargo-driven promotion path. Why it's its own page: this is the current end-to-end standard for custom apps (wrapper charts + single ArgoCD Helm source), verified on platform-demo. Kargo Promotion goes deep on promotion mechanics; Third-Party Charts covers upstream Helm tools (Grafana/Vault/…), which follow a different (no-promotion) path. Chart internals: the Helm golden-path ADR and .claude/rules/gitops.md (Helm golden path — GAP wrapper-chart).

Two repositories, two responsibilities

Delivery is split across application repos and the single deployment repo (minicloud-gitops). This mirrors an enterprise Application Platform (wrapper-chart) model.

	Application repo (e.g. `platform-demo`, `ktayl-policy-service`)	Deployment repo (`minicloud-gitops`)
Contains	source, `Dockerfile`, tests, CI	ArgoCD `Application`s, per-app wrapper Helm charts, the shared library chart, platform manifests
Answers	"what the software is"	"how/where it runs"
Produces	a signed, SBOM'd image	the declarative desired state of the cluster
Owner	application developer	platform/DevOps (here: same person; the split is structural, enforced by CODEOWNERS)
Holds deploy config?	no — zero Helm/k8s config in the app repo	yes — all of it

The application developer never edits infrastructure to ship a version; the deployment repo is the auditable source of truth for what is running.

The GitOps system at a glance — humans and Kargo only ever write to Git; ArgoCD is the only thing that writes to the cluster, reconciling desired state (Git) ↔ live state and self-healing drift (the red loop). (full SVG · also on Architecture → C4)

Start here: what change are you making?

Most delivery confusion comes from opening the right repo but the wrong layer. Use this routing table before editing anything:

Change	Edit where	Why
Application code, tests, API behavior, UI behavior	the application repo (`retrieva`, `platform-demo`, `ktayl-policy-service`, …)	this changes what the software is
Image version promotion from dev to prod	normally Kargo; review the PR it opens in `minicloud-gitops`	promotion is an auditable Git change, not a local edit
Replicas, resources, probes, env vars, ingress hosts, KEDA, Vault/ESO wiring	`minicloud-gitops/services/<svc>/helm/values*.yaml` or wrapper `templates/`	this changes how the app runs
ArgoCD app source, namespace, project permissions	`minicloud-gitops/apps/` and `manifests/argocd-project/`	this changes what ArgoCD is allowed to reconcile
Third-party platform tool config	`minicloud-gitops/helm-values/`	upstream chart plus local values is the platform-tool contract
Cluster-wide policy, quota, RBAC, network rule	`minicloud-gitops/manifests/`	shared platform controls belong outside one service
Bootstrap, OS, node prep, MAAS/k3s install	`minicloud-ansible` or `minicloud-opentofu`	these are below the GitOps application layer

The rule of thumb: source repos produce artifacts; minicloud-gitops decides how artifacts and platform services run; ArgoCD is the only cluster writer for managed workloads.

The pipeline, end to end

System-design view — the four stages colour-coded: grey build & prove · cyan Kargo promote · green = dev (continuous deployment, automatic) · amber = prod (continuous delivery, human-gated). The CODEOWNERS reviewer is the visible manual gate. (full SVG · also on the Architecture → C4 Delivery flow)

The same flow in text detail:

 APPLICATION REPO                         DEPLOYMENT REPO (minicloud-gitops)
 ────────────────                         ─────────────────────────────────
 git push main
   │
   ▼
 CI: test → build → scan → sign (cosign) → SBOM
   │  (build + PROVE only — no gitops write)
   ▼
 image → Harbor (dev tag) + ghcr (prod SHA)
   │
   ▼
 ┌── KARGO ──────────────────────────────────────────────────────────┐
 │ Warehouse detects the new image  →  Freight (immutable artifact)   │
 │   │                                                                │
 │   ▼  auto-promote (ProjectConfig: stage dev, autoPromotionEnabled) │
 │  Stage dev  → yaml-update  services/<svc>/helm/values-dev.yaml     │
 │                            key minicloud-app-deployment.image.tag  │
 │   │                                                                │
 │   ▼  opens PR (label: automerge)                                   │
 │  kargo-automerge.yml  → merges (dev-only path guard)  ───────────► main
 │   │                                                                │
 │   ▼  Stage dev verification (AnalysisRun <svc>-dev-verify)         │
 │  Freight "Verified in dev"  (the dev→prod gate)                    │
 │   │                                                                │
 │   ▼  promote to Stage prod  (manual / gated)                       │
 │  Stage prod → yaml-update  services/<svc>/helm/values-prod.yaml    │
 │   │                                                                │
 │   ▼  opens PR (NO automerge)                                       │
 │  CODEOWNERS review on services/*/helm/  ────────────────────────► main
 └────────────────────────────────────────────────────────────────────┘
   │
   ▼
 ArgoCD renders the wrapper chart (helm dependency build → library chart from OCI)
   │
   ▼
 cluster: Deployment/Rollout + Service + Ingress + KEDA + Cert + … (dev / prod)

Division of labour: CI builds and proves the artifact; Kargo promotes it (the one thing ArgoCD does not do); ArgoCD deploys it. Nothing writes to the cluster except ArgoCD.

Continuous Delivery vs Continuous Deployment

These two terms are constantly confused. They share the entire pipeline above — build, test, scan, sign, make release-ready — and differ in exactly one place: the last step into production.

	Continuous Delivery	Continuous Deployment
Every change is built, tested, and made release-ready automatically	✅	✅
Release to production is…	triggered by a human (an approval / a PR merge)	fully automatic — no human in the loop
A manual gate before prod	yes, by design	no

The one-liner: continuous delivery means every change is always ready to ship but a person decides when; continuous deployment means every change that passes the pipeline ships itself to production with no approval. So continuous deployment is continuous delivery with the final manual gate removed — every continuous deployment is also continuous delivery, but not the reverse.

Mnemonic: delivery → a human enables the release · deployment → the machine releases it.

Where this platform sits — both, per environment

This stack is deliberately mixed, and that is the point of a two-stage Kargo pipeline:

Environment	Model	Why
dev	Continuous Deployment	Kargo auto-promotes new Freight to dev (`ProjectConfig: stage dev, autoPromotionEnabled: true`), the dev PR auto-merges (`kargo-automerge.yml`), ArgoCD syncs — no human between a green `main` build and a running dev pod.
prod	Continuous Delivery	Kargo opens a CODEOWNERS-gated PR on `services//helm/`; it needs `@AndreLair` review and* the Freight must be verified in dev and the live QA gate must pass. A human decides when prod ships.

That human prod gate is a feature, not friction: for an insurance-domain system it's the governance / DORA-change-management sign-off (one auditable approval per prod change). Crossing fully into continuous deployment would just mean letting the prod PR auto-merge on green checks — a one-line policy change we choose not to make. The dev stage already gives the fast feedback of continuous deployment where the blast radius is safe.

Workflow for GitOps changes in `minicloud-gitops`

For a GitOps-only change, the workflow is shorter than a full application release:

Open minicloud-gitops and run git status.
Identify the smallest owned path: services/<svc>/helm/ for a custom app, helm-values/ for a third-party chart, or manifests/ for a shared platform concern.
Make the declarative change in Git. Do not patch the live cluster to make the desired state true.

Render or diff locally when possible:

cd services/<svc>/helm
helm dependency update .
helm template <svc> . -f values-dev.yaml

Open a PR. Dev-only Kargo PRs may auto-merge; prod paths and platform controls require CODEOWNERS review.
After merge, ArgoCD reconciles the change. Verify the relevant app is Synced and Healthy.

Direct kubectl apply, helm upgrade, or manual ArgoCD sync is reserved for bootstrap, recovery, or documented break-glass work. The normal path is always Git commit -> PR -> ArgoCD reconciliation.

The deployment artifact — a wrapper Helm chart

Each app is its own thin Helm chart under services/<svc>/helm/ that declares a dependency on the shared minicloud-app-deployment library chart and carries its service-specific extras in its own templates/:

services/<svc>/helm/
  Chart.yaml        # dependencies: [minicloud-app-deployment @ X.Y.Z, oci://ghcr.io/andrelair-platform]
  Chart.lock        # committed — ArgoCD runs `helm dependency build` to fetch the library
  values.yaml       # common: the "minicloud-app-deployment:" subchart block + wrapper-local keys
  values-dev.yaml   # dev overlay  (image.tag, hosts, replicas)        ← Kargo yaml-updates this
  values-prod.yaml  # prod overlay (image.tag, hosts, replicas, gates) ← Kargo yaml-updates this
  templates/        # service-specific extras (DB, AnalysisTemplate, SSO Ingress, ExternalSecret…)

One Helm render, one ArgoCD source — no kustomize, no multi-source $values, no separate satellites source. The app's values.yaml is the deployment contract; the library chart is the platform golden path. See the ADR for the chart's config surface and hardened defaults.

The three control points

Control	Where	Guards
dev auto-merge	`.github/workflows/kargo-automerge.yml`	merges only dev-only PRs — paths matching `services//minicloud-1/dev/` or `services//helm/values-dev.yaml`. Everything else is refused.
dev→prod gate	Kargo Stage prod `sources.stages: [dev]` + `<svc>-dev-verify` AnalysisRun	prod only accepts Freight already verified in dev
prod merge gate	`.github/CODEOWNERS` on `services//helm/`, `services//base/`, `services/*/kargo/`, `apps/`	prod promotion PR needs `@AndreLair` review

Instant sync — GitHub webhook + reconciliation backoff

ArgoCD reconciles on a timer (timeout.reconciliation), so a merge to main used to take up to ~3 minutes to start syncing. A GitHub push webhook to ArgoCD's /api/webhook collapses that to seconds (observed ~11s push→refresh; argocd-server logs refreshing app <x> from webhook).

Only the webhook path is public. ArgoCD is Tailscale-only, so the Cloudflare tunnel exposes a path-scoped argocd.devandre.sbs/api/webhook only (the endpoint is unauthenticated-by-design but HMAC-validated against argocd-secret webhook.github.secret); the UI/API stay off the tunnel.
Host rewrite is required. ingress-nginx routes on the HTTP Host header and the ArgoCD Ingress declares only argocd.10.0.0.200.nip.io, so the tunnel rule sets httpHostHeader to that name (originServerName only sets the TLS SNI — not the Host; that was a second, silent 404).
The poll stays as a backstop at 1h (not 0). The webhook is best-effort (one HTTP POST, no retry) and fires only on a git push — so the periodic reconcile is what still catches missed deliveries, live cluster drift (selfHeal — out-of-band mutation is not a push), and non-matching sources. 180s → 1h cuts routine polling ~20× while guaranteeing worst-case convergence ≤ 1 hour. Event for speed, poll for guarantee.

Full decision, the SSA configs.cm gotcha (a kubectl patch + controller restart is needed to apply a reconciliation-timeout change), and verification: the ArgoCD instant-sync ADR.

Verified end-to-end

Proven on platform-demo (two promotion cycles): a Kargo Promotion yaml-updated services/platform-demo/helm/values-dev.yaml image.tag → opened the dev PR → kargo-automerge merged it → ArgoCD synced the wrapper chart → the Rollout picked up the promoted tag. The same mechanics drive the CODEOWNERS-gated prod PR.

Invariants & gotchas (learned the hard way)

Invariant	Why
Commit `Chart.lock`; don't list `charts/` in `.helmignore`	ArgoCD needs the lock to `helm dependency build`; listing `charts/` in `.helmignore` makes helm treat the dep as missing. `charts/` is gitignored instead.
Set `helm.releaseName: <svc>` on the ArgoCD app	the library uses the release name for `fullname`; without it, the workload takes the app's name.
Escape non-Helm `{{ }}` in `templates/`	ESO output-templates and Argo-Rollouts args (and even YAML comments) are parsed by Helm — wrap them in backtick strings.
Migrating an existing workload: match the immutable selector	set subchart `selectorLabels` to the live Deployment/Rollout selector for a zero-downtime in-place flip. A changed `spec.selector` → `SyncFailed`.
Keep a service's own SA-token / Vault needs in mind	`automountServiceAccountToken: false` is the hardened default, but Vault agent injection requires the token mount — disabling it denies pod creation (latent behind scale-to-zero).
A canary Rollout at scale-to-zero can't self-complete	with 0 replicas the canary analysis has no pod/traffic → it sits `Progressing`/`Degraded` until real traffic (or the dev-verify smoke) wakes it via the KEDA interceptor.
Images must be env-agnostic	the same artifact runs dev and prod; read env at runtime. Prod pins an immutable ghcr SHA.

Operational runbook

# Render/validate a service's wrapper chart locally
cd services/<svc>/helm && helm dependency update . && helm template <svc> . -f values-dev.yaml

# Promote a specific Freight to dev (normally automatic) — Kargo CLI or a Promotion CR
kubectl -n <svc> create -f - <<'EOF'
apiVersion: kargo.akuity.io/v1alpha1
kind: Promotion
metadata: { generateName: manual-, namespace: <svc> }
spec: { stage: dev, freight: <freight-name> }
EOF

# Promote dev-verified Freight to prod → opens the CODEOWNERS-gated PR (approve with --squash)

# Roll back: re-promote the previous Freight (or revert the values PR); ArgoCD reconciles.

# Wake a scale-to-zero app (e.g. to let a canary finish) via the KEDA interceptor:
#   curl -H 'Host: <dev-host>' http://keda-add-ons-http-interceptor-proxy.keda.svc:8080/<readyz>

Third-Party Charts · Kargo Promotion · Helm vs Kustomize · Argo Rollouts · KEDA scale-to-zero
Deployment repo: services/_template-helm/ (scaffold) · Helm golden-path ADR · ArgoCD instant-sync ADR

Two repositories, two responsibilities​

Start here: what change are you making?​

The pipeline, end to end​

Continuous Delivery vs Continuous Deployment​

Where this platform sits — both, per environment​

Workflow for GitOps changes in minicloud-gitops​

The deployment artifact — a wrapper Helm chart​

The three control points​

Instant sync — GitHub webhook + reconciliation backoff​

Verified end-to-end​

Invariants & gotchas (learned the hard way)​

Operational runbook​

Related​