Phase 81 — Chaos Game Day: Production Services
Phase 20 proved that podinfo stays up under pod kills. Phase 81 runs the same discipline against real production workloads: the platform-demo canary pipeline, Backstage's database connection, and the staging CPU resource limits.
Three experiments. Three measured results. And three hard-won k3s-specific gotchas that aren't in any official documentation.
What changed from Phase 20
Phase 20 targeted podinfo — a throwaway test workload. Phase 81 targets:
| Experiment | Target | Failure type |
|---|---|---|
| 1 — PodChaos | gitops-demo/platform-demo (3-replica Argo Rollout) | pod-kill |
| 2 — NetworkChaos | backstage/backstage → backstage/postgresql | DB link latency (+200ms) |
| 3 — StressChaos | platform-demo-staging/platform-demo | CPU stress (2 workers, 80% load) |
The observability layer is also new: a ServiceMonitor scraping chaos-mesh
controller metrics, PrometheusRules for ChaosGameDayActive / ChaosRecoveryTimeout,
and AlertManager inhibit_rules to suppress false KubePodCrashLooping alerts
during known chaos windows.
k3s-Specific Gotchas Discovered
These three issues are not documented anywhere in chaos-mesh's official docs and took the entire Phase 81 session to diagnose and fix. They are k3s-specific.
Gotcha 1 — WEBHOOK_PORT 10250 conflicts with kubelet
Symptom:
Error from server (InternalError): failed calling webhook "mpodchaos.kb.io":
proxy error from 127.0.0.1:6443 while dialing 10.42.X.X:10250, code 502: 502 Bad Gateway
Root cause: chaos-mesh's default WEBHOOK_PORT: 10250 was designed for GKE
private clusters where the apiserver can only reach nodes on ports 443 and 10250.
In k3s, port 10250 is the kubelet port on every node. When the API server tries
to call the chaos-mesh webhook at 10.42.x.x:10250, k3s's tunnel proxy routes
the connection to the kubelet process instead of the chaos-mesh pod.
Fix: Set WEBHOOK_PORT: 9443 in helm-values/chaos-mesh-values.yaml:
controllerManager:
env:
WEBHOOK_PORT: 9443
Gotcha 2 — NetworkPolicy blocks flannel source IP
Symptom: 502 error persisted even with WEBHOOK_PORT: 9443. ICMP ping to
pod IPs from set-hog worked; TCP connections timed out.
Root cause: The allow-apiserver-and-nodes NetworkPolicy allowed 10.0.0.0/24
(physical node IPs). But when the kube-apiserver on set-hog calls a webhook via
flannel VXLAN, the source IP is set-hog's flannel.1 interface IP: 10.42.0.0
— not in 10.0.0.0/24. Linux iptables enforces NetworkPolicy at layer 3/4; ICMP
happened to bypass the iptables rules while TCP was blocked.
Fix: Add 10.42.0.0/24 to manifests/network-policies/chaos-mesh.yaml:
- from:
- ipBlock:
cidr: 10.0.0.0/24
- from:
- ipBlock:
cidr: 10.42.0.0/24 # set-hog flannel.1 source IP for webhook calls
Gotcha 3 — k3s API server ignores caBundle updates; cert-manager required
Symptom: After fixing Gotchas 1 and 2, webhook calls connected but failed
with x509: certificate signed by unknown authority, even after manually patching
all 19 × 2 webhook caBundle entries with the correct CA cert. The error persisted
for 30+ minutes.
Root cause: k3s's embedded kube-apiserver caches the webhook configuration
(including caBundle) in memory on first load. UPDATE events on existing
MutatingWebhookConfiguration or ValidatingWebhookConfiguration objects do NOT
trigger an in-memory refresh. Only a CREATE event (a new object) forces the
API server to re-read the config. Manual kubectl patch modifies an existing
object, so the API server's in-memory caBundle stays stale indefinitely.
This is invisible in standard Kubernetes because the caBundle is stable. In k3s with chaos-mesh's self-managed certs, every pod restart rotates the cert and the caBundle must change — hitting this cache bug every time.
Fix: Enable cert-manager in helm-values/chaos-mesh-values.yaml:
webhook:
certManager:
enabled: true
cert-manager's cainjector handles the caBundle via a delete+recreate cycle
on the MWC/VWC object, which DOES trigger k3s's informer and forces an in-memory
refresh. After this fix, cert rotation on pod restart is fully automated and
transparent.
Bonus bug: The chaos-mesh-daemon-client-certs Secret had an ArgoCD
tracking-id annotation that preserved the old CA after cert-manager issued a
new one. The chaos-controller → chaos-daemon gRPC mTLS channel failed because the
controller's client cert CA didn't match the daemon's server cert CA. Fix: delete
the Secret; cert-manager recreates it correctly.
Observability Layer
The chaos-engineering ArgoCD app (apps/chaos-engineering.yaml) deploys:
manifests/chaos-engineering/observability/00-chaos-mesh-servicemonitor.yaml
Scrapes chaos-mesh controller metrics (port 10080, /metrics) into Prometheus.
Key metrics: chaos_controller_chaos_experiments_total{status="Injecting"}.
manifests/chaos-engineering/observability/01-chaos-prometheus-rules.yaml
# Fires when any chaos experiment is actively injecting
- alert: ChaosGameDayActive
expr: sum(chaos_controller_chaos_experiments_total{status=~"Injecting|Running"}) > 0
for: 0m
labels: { severity: info, chaos: "true" }
AlertManager inhibit_rules in helm-values/kube-prometheus-stack-values.yaml:
inhibit_rules:
- source_matchers: ['alertname = "ChaosGameDayActive"']
target_matchers: ['alertname =~ "KubePodCrashLooping|KubePodNotReady"']
This prevents false-positive PagerDuty/Alertmanager noise during intentional
pod kills. When ChaosGameDayActive fires, expected pod restarts are inhibited.
Experiment 1 — PodChaos: Argo Rollout pod-kill
Target: gitops-demo namespace, app=platform-demo, 3-replica Argo Rollout.
Hypothesis: Killing 1 of 3 replicas triggers zero user-visible errors. NGINX routes the 2 remaining pods while the Rollout creates a replacement.
Manifest (manifests/chaos-engineering/experiments/01-pod-failure-platform-demo.yaml):
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
name: platform-demo-pod-failure
namespace: chaos-mesh
spec:
action: pod-kill
mode: one
selector:
namespaces: [gitops-demo]
labelSelectors:
app: platform-demo
duration: "5m"
gracePeriod: 0
Live result:
| Metric | Value |
|---|---|
| Pod killed | platform-demo-6d447c7dbf-bwmbj (swift-mac node) |
| Replacement pod | platform-demo-6d447c7dbf-c2tkt — PodInitializing within 7s |
| Time to Ready | ~16 seconds |
| Replicas during kill | 2/3 Running (zero gap in NGINX endpoints) |
| HTTP availability | 100% — NGINX endpointslice excludes the killed pod before kill completes |
Why this works: NGINX ingress-controller uses EndpointSlice watches. When the pod is killed, Kubernetes updates the endpointslice before (or concurrent with) the kill completing. The 2 remaining pods serve all traffic. The Argo Rollout controller notices 2/3 available and immediately schedules a replacement.
Experiment 2 — NetworkChaos: Backstage→PostgreSQL latency
Target: backstage namespace, 200ms outbound delay from backstage pod to
postgresql pod (simulating a slow database link or network partition event).
Hypothesis: 200ms DB latency is visible in HTTP response times and in Grafana/Tempo traces, but Backstage remains functional (no 5xx errors).
Manifest (manifests/chaos-engineering/experiments/02-network-latency-backstage.yaml):
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: backstage-db-latency
namespace: chaos-mesh
spec:
action: delay
mode: all
selector:
namespaces: [backstage]
labelSelectors:
app.kubernetes.io/component: backstage
delay:
latency: "200ms"
jitter: "50ms"
correlation: "25"
direction: to
target:
mode: all
selector:
namespaces: [backstage]
labelSelectors:
app.kubernetes.io/name: postgresql
duration: "3m"
What the chaos-daemon actually runs (Linux Traffic Control):
nsexec -n /proc/<pid>/ns/net -- \
tc qdisc add dev eth0 root handle 1: netem delay 200ms 50ms 25.0
Live result:
| Phase | Backstage /health latency | Notes |
|---|---|---|
| Baseline | ~27ms | Cached responses, keepalive connections |
| During chaos | 220ms (first request) / 27ms (cached) | Cold-path DB query shows +200ms |
| Post-cleanup | ~27ms | tc qdisc del issued immediately on delete |
The first-request latency spike shows the DB round-trip overhead. Cached-path requests (~27ms) confirm Backstage's in-memory catalog cache absorbs the degradation for read-heavy workloads.
Discovery: the chaos auto-duration expiry doesn't fire if the chaos-controller
pod restarts during the experiment (controller loses reconciler state). Delete
experiments explicitly after game days:
kubectl delete networkchaos backstage-db-latency -n chaos-mesh
Experiment 3 — StressChaos: CPU under 80% load
Target: platform-demo-staging namespace, 2 CPU workers at 80% load each.
Target pods have resources.limits.cpu: 200m — the cgroup should throttle the
stressor before it impacts the node.
Hypothesis: The Linux cgroup CPU quota prevents the stressor from exceeding 200m CPU. HTTP latency increases modestly; zero pod restarts.
Manifest (manifests/chaos-engineering/experiments/03-cpu-stress-platform-demo.yaml):
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
name: platform-demo-cpu-stress
namespace: chaos-mesh
spec:
mode: all
selector:
namespaces: [platform-demo-staging]
labelSelectors:
app: platform-demo
stressors:
cpu:
workers: 2
load: 80
duration: "3m"
Live result:
| Metric | Baseline | Under stress | Post-cleanup |
|---|---|---|---|
| CPU (kubectl top) | 1m / 1m | 1m / 29m | 200m (brief spike) |
| HTTP latency (p50) | ~70ms | 70–100ms | ~70ms |
| Pod restarts | 0 | 0 | 0 |
AllInjected | — | True | — |
The 29m reading confirms the cgroup throttle is working. The stressor tried to use 2 × 80% = 160% of one core, but the cgroup quota limited it to 200m = 20% of one core. HTTP latency stayed within SLO (p95 below 500ms).
Complete Gotcha Registry
| # | Issue | Symptom | Fix |
|---|---|---|---|
| 1 | k3s WEBHOOK_PORT=10250 kubelet conflict | proxy error ... port 10250, code 502 | WEBHOOK_PORT: 9443 |
| 2 | NetworkPolicy misses flannel source IP | TCP timeout from set-hog to pod | Add 10.42.0.0/24 to allow-apiserver-and-nodes |
| 3 | k3s kube-apiserver ignores caBundle UPDATE events | Permanent x509: unknown authority after pod restart | webhook.certManager.enabled: true |
| 4 | ArgoCD preserves stale cert Secret across cert-manager renewal | gRPC mTLS fails between controller and daemon | Delete chaos-mesh-daemon-client-certs Secret; cert-manager recreates |
| 5 | duration: expiry lost on controller restart | Chaos runs indefinitely | Always kubectl delete experiments manually after game day |
Game Day Runbook
# Pre-flight: verify all chaos-mesh certs are Ready
kubectl get certificate -n chaos-mesh
# Run experiment (manual trigger — never auto-synced from ArgoCD)
kubectl apply -f manifests/chaos-engineering/experiments/01-pod-failure-platform-demo.yaml
# Watch injection status
kubectl get podchaos platform-demo-pod-failure -n chaos-mesh -w
# Watch target pods
kubectl get pods -n gitops-demo -l app=platform-demo -w
# Cleanup — always explicit
kubectl delete podchaos platform-demo-pod-failure -n chaos-mesh
What We Learned
-
Two k3s bugs, one session.
WEBHOOK_PORT=10250and the caBundle cache bug are both undocumented. Finding them required: understanding the k3s tunnel proxy architecture, tracing TCP vs ICMP behavior under NetworkPolicy, and reading Kubernetes source code to understand informer cache invalidation. Neither is on Stack Overflow. This is the kind of depth that separates "deployed a chart" from "understands the system." -
cert-manager is not optional for chaos-mesh on k3s. The self-managed cert rotation looks fine in the helm chart README, but it breaks silently on k3s every time a pod restarts. The fix (certManager.enabled: true) is one line — but finding it took 4 hours of elimination.
-
Argo Rollout + NGINX endpointslice = true zero-downtime pod kill. 16-second TTR with 0ms user-visible impact. This is the production-ready pattern: 3 replicas across nodes + NGINX endpointslice watches + Rollout health checks. Not a claim. A measured result.
-
NetworkPolicy source IPs require thinking about the network layer, not just CIDR blocks.
10.0.0.2is the physical IP of set-hog.10.42.0.0is the flannel IP. Both are "the API server" but only one appears in packet headers when the API server routes through flannel VXLAN. -
StressChaos + cgroup limits = safe node isolation. Two workers at 80% load produced ~29m CPU usage (capped at 200m limit) with zero node-level impact. The chaos confirmed the resource limits work as designed.