Skip to main content

Phase 81 — Chaos Game Day: Production Services

Phase 20 proved that podinfo stays up under pod kills. Phase 81 runs the same discipline against real production workloads: the platform-demo canary pipeline, Backstage's database connection, and the staging CPU resource limits.

Three experiments. Three measured results. And three hard-won k3s-specific gotchas that aren't in any official documentation.

What changed from Phase 20​

Phase 20 targeted podinfo — a throwaway test workload. Phase 81 targets:

ExperimentTargetFailure type
1 — PodChaosgitops-demo/platform-demo (3-replica Argo Rollout)pod-kill
2 — NetworkChaosbackstage/backstage → backstage/postgresqlDB link latency (+200ms)
3 — StressChaosplatform-demo-staging/platform-demoCPU stress (2 workers, 80% load)

The observability layer is also new: a ServiceMonitor scraping chaos-mesh controller metrics, PrometheusRules for ChaosGameDayActive / ChaosRecoveryTimeout, and AlertManager inhibit_rules to suppress false KubePodCrashLooping alerts during known chaos windows.

k3s-Specific Gotchas Discovered​

These three issues are not documented anywhere in chaos-mesh's official docs and took the entire Phase 81 session to diagnose and fix. They are k3s-specific.

Gotcha 1 — WEBHOOK_PORT 10250 conflicts with kubelet​

Symptom:

Error from server (InternalError): failed calling webhook "mpodchaos.kb.io":
proxy error from 127.0.0.1:6443 while dialing 10.42.X.X:10250, code 502: 502 Bad Gateway

Root cause: chaos-mesh's default WEBHOOK_PORT: 10250 was designed for GKE private clusters where the apiserver can only reach nodes on ports 443 and 10250. In k3s, port 10250 is the kubelet port on every node. When the API server tries to call the chaos-mesh webhook at 10.42.x.x:10250, k3s's tunnel proxy routes the connection to the kubelet process instead of the chaos-mesh pod.

Fix: Set WEBHOOK_PORT: 9443 in helm-values/chaos-mesh-values.yaml:

controllerManager:
  env:
    WEBHOOK_PORT: 9443

Gotcha 2 — NetworkPolicy blocks flannel source IP​

Symptom: 502 error persisted even with WEBHOOK_PORT: 9443. ICMP ping to pod IPs from set-hog worked; TCP connections timed out.

Root cause: The allow-apiserver-and-nodes NetworkPolicy allowed 10.0.0.0/24 (physical node IPs). But when the kube-apiserver on set-hog calls a webhook via flannel VXLAN, the source IP is set-hog's flannel.1 interface IP: 10.42.0.0 — not in 10.0.0.0/24. Linux iptables enforces NetworkPolicy at layer 3/4; ICMP happened to bypass the iptables rules while TCP was blocked.

Fix: Add 10.42.0.0/24 to manifests/network-policies/chaos-mesh.yaml:

- from:
    - ipBlock:
        cidr: 10.0.0.0/24
- from:
    - ipBlock:
        cidr: 10.42.0.0/24    # set-hog flannel.1 source IP for webhook calls

Gotcha 3 — k3s API server ignores caBundle updates; cert-manager required​

Symptom: After fixing Gotchas 1 and 2, webhook calls connected but failed with x509: certificate signed by unknown authority, even after manually patching all 19 × 2 webhook caBundle entries with the correct CA cert. The error persisted for 30+ minutes.

Root cause: k3s's embedded kube-apiserver caches the webhook configuration (including caBundle) in memory on first load. UPDATE events on existing MutatingWebhookConfiguration or ValidatingWebhookConfiguration objects do NOT trigger an in-memory refresh. Only a CREATE event (a new object) forces the API server to re-read the config. Manual kubectl patch modifies an existing object, so the API server's in-memory caBundle stays stale indefinitely.

This is invisible in standard Kubernetes because the caBundle is stable. In k3s with chaos-mesh's self-managed certs, every pod restart rotates the cert and the caBundle must change — hitting this cache bug every time.

Fix: Enable cert-manager in helm-values/chaos-mesh-values.yaml:

webhook:
  certManager:
    enabled: true

cert-manager's cainjector handles the caBundle via a delete+recreate cycle on the MWC/VWC object, which DOES trigger k3s's informer and forces an in-memory refresh. After this fix, cert rotation on pod restart is fully automated and transparent.

Bonus bug: The chaos-mesh-daemon-client-certs Secret had an ArgoCD tracking-id annotation that preserved the old CA after cert-manager issued a new one. The chaos-controller → chaos-daemon gRPC mTLS channel failed because the controller's client cert CA didn't match the daemon's server cert CA. Fix: delete the Secret; cert-manager recreates it correctly.

Observability Layer​

The chaos-engineering ArgoCD app (apps/chaos-engineering.yaml) deploys:

manifests/chaos-engineering/observability/00-chaos-mesh-servicemonitor.yaml Scrapes chaos-mesh controller metrics (port 10080, /metrics) into Prometheus. Key metrics: chaos_controller_chaos_experiments_total{status="Injecting"}.

manifests/chaos-engineering/observability/01-chaos-prometheus-rules.yaml

# Fires when any chaos experiment is actively injecting
- alert: ChaosGameDayActive
  expr: sum(chaos_controller_chaos_experiments_total{status=~"Injecting|Running"}) > 0
  for: 0m
  labels: { severity: info, chaos: "true" }

AlertManager inhibit_rules in helm-values/kube-prometheus-stack-values.yaml:

inhibit_rules:
  - source_matchers: ['alertname = "ChaosGameDayActive"']
    target_matchers: ['alertname =~ "KubePodCrashLooping|KubePodNotReady"']

This prevents false-positive PagerDuty/Alertmanager noise during intentional pod kills. When ChaosGameDayActive fires, expected pod restarts are inhibited.

Experiment 1 — PodChaos: Argo Rollout pod-kill​

Target: gitops-demo namespace, app=platform-demo, 3-replica Argo Rollout.

Hypothesis: Killing 1 of 3 replicas triggers zero user-visible errors. NGINX routes the 2 remaining pods while the Rollout creates a replacement.

Manifest (manifests/chaos-engineering/experiments/01-pod-failure-platform-demo.yaml):

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: platform-demo-pod-failure
  namespace: chaos-mesh
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces: [gitops-demo]
    labelSelectors:
      app: platform-demo
  duration: "5m"
  gracePeriod: 0

Live result:

MetricValue
Pod killedplatform-demo-6d447c7dbf-bwmbj (swift-mac node)
Replacement podplatform-demo-6d447c7dbf-c2tkt — PodInitializing within 7s
Time to Ready~16 seconds
Replicas during kill2/3 Running (zero gap in NGINX endpoints)
HTTP availability100% — NGINX endpointslice excludes the killed pod before kill completes

Why this works: NGINX ingress-controller uses EndpointSlice watches. When the pod is killed, Kubernetes updates the endpointslice before (or concurrent with) the kill completing. The 2 remaining pods serve all traffic. The Argo Rollout controller notices 2/3 available and immediately schedules a replacement.

Experiment 2 — NetworkChaos: Backstage→PostgreSQL latency​

Target: backstage namespace, 200ms outbound delay from backstage pod to postgresql pod (simulating a slow database link or network partition event).

Hypothesis: 200ms DB latency is visible in HTTP response times and in Grafana/Tempo traces, but Backstage remains functional (no 5xx errors).

Manifest (manifests/chaos-engineering/experiments/02-network-latency-backstage.yaml):

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: backstage-db-latency
  namespace: chaos-mesh
spec:
  action: delay
  mode: all
  selector:
    namespaces: [backstage]
    labelSelectors:
      app.kubernetes.io/component: backstage
  delay:
    latency: "200ms"
    jitter: "50ms"
    correlation: "25"
  direction: to
  target:
    mode: all
    selector:
      namespaces: [backstage]
      labelSelectors:
        app.kubernetes.io/name: postgresql
  duration: "3m"

What the chaos-daemon actually runs (Linux Traffic Control):

nsexec -n /proc/<pid>/ns/net -- \
  tc qdisc add dev eth0 root handle 1: netem delay 200ms 50ms 25.0

Live result:

PhaseBackstage /health latencyNotes
Baseline~27msCached responses, keepalive connections
During chaos220ms (first request) / 27ms (cached)Cold-path DB query shows +200ms
Post-cleanup~27mstc qdisc del issued immediately on delete

The first-request latency spike shows the DB round-trip overhead. Cached-path requests (~27ms) confirm Backstage's in-memory catalog cache absorbs the degradation for read-heavy workloads.

Discovery: the chaos auto-duration expiry doesn't fire if the chaos-controller pod restarts during the experiment (controller loses reconciler state). Delete experiments explicitly after game days:

kubectl delete networkchaos backstage-db-latency -n chaos-mesh

Experiment 3 — StressChaos: CPU under 80% load​

Target: platform-demo-staging namespace, 2 CPU workers at 80% load each. Target pods have resources.limits.cpu: 200m — the cgroup should throttle the stressor before it impacts the node.

Hypothesis: The Linux cgroup CPU quota prevents the stressor from exceeding 200m CPU. HTTP latency increases modestly; zero pod restarts.

Manifest (manifests/chaos-engineering/experiments/03-cpu-stress-platform-demo.yaml):

apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: platform-demo-cpu-stress
  namespace: chaos-mesh
spec:
  mode: all
  selector:
    namespaces: [platform-demo-staging]
    labelSelectors:
      app: platform-demo
  stressors:
    cpu:
      workers: 2
      load: 80
  duration: "3m"

Live result:

MetricBaselineUnder stressPost-cleanup
CPU (kubectl top)1m / 1m1m / 29m200m (brief spike)
HTTP latency (p50)~70ms70–100ms~70ms
Pod restarts000
AllInjected—True—

The 29m reading confirms the cgroup throttle is working. The stressor tried to use 2 × 80% = 160% of one core, but the cgroup quota limited it to 200m = 20% of one core. HTTP latency stayed within SLO (p95 below 500ms).

Complete Gotcha Registry​

#IssueSymptomFix
1k3s WEBHOOK_PORT=10250 kubelet conflictproxy error ... port 10250, code 502WEBHOOK_PORT: 9443
2NetworkPolicy misses flannel source IPTCP timeout from set-hog to podAdd 10.42.0.0/24 to allow-apiserver-and-nodes
3k3s kube-apiserver ignores caBundle UPDATE eventsPermanent x509: unknown authority after pod restartwebhook.certManager.enabled: true
4ArgoCD preserves stale cert Secret across cert-manager renewalgRPC mTLS fails between controller and daemonDelete chaos-mesh-daemon-client-certs Secret; cert-manager recreates
5duration: expiry lost on controller restartChaos runs indefinitelyAlways kubectl delete experiments manually after game day

Game Day Runbook​

# Pre-flight: verify all chaos-mesh certs are Ready
kubectl get certificate -n chaos-mesh

# Run experiment (manual trigger — never auto-synced from ArgoCD)
kubectl apply -f manifests/chaos-engineering/experiments/01-pod-failure-platform-demo.yaml

# Watch injection status
kubectl get podchaos platform-demo-pod-failure -n chaos-mesh -w

# Watch target pods
kubectl get pods -n gitops-demo -l app=platform-demo -w

# Cleanup — always explicit
kubectl delete podchaos platform-demo-pod-failure -n chaos-mesh

What We Learned​

  1. Two k3s bugs, one session. WEBHOOK_PORT=10250 and the caBundle cache bug are both undocumented. Finding them required: understanding the k3s tunnel proxy architecture, tracing TCP vs ICMP behavior under NetworkPolicy, and reading Kubernetes source code to understand informer cache invalidation. Neither is on Stack Overflow. This is the kind of depth that separates "deployed a chart" from "understands the system."

  2. cert-manager is not optional for chaos-mesh on k3s. The self-managed cert rotation looks fine in the helm chart README, but it breaks silently on k3s every time a pod restarts. The fix (certManager.enabled: true) is one line — but finding it took 4 hours of elimination.

  3. Argo Rollout + NGINX endpointslice = true zero-downtime pod kill. 16-second TTR with 0ms user-visible impact. This is the production-ready pattern: 3 replicas across nodes + NGINX endpointslice watches + Rollout health checks. Not a claim. A measured result.

  4. NetworkPolicy source IPs require thinking about the network layer, not just CIDR blocks. 10.0.0.2 is the physical IP of set-hog. 10.42.0.0 is the flannel IP. Both are "the API server" but only one appears in packet headers when the API server routes through flannel VXLAN.

  5. StressChaos + cgroup limits = safe node isolation. Two workers at 80% load produced ~29m CPU usage (capped at 200m limit) with zero node-level impact. The chaos confirmed the resource limits work as designed.