Anatomy of a Cascading Kubernetes Outage: When the First Symptom Is Three Layers From the Cause
A monitoring alert fired: the platform watchdog was DOWN. Within minutes the picture looked ugly — cluster DNS was refusing connections, and shortly after, distributed-storage volumes stopped rebuilding. A textbook cascade.
This is the full post-mortem: how the outage propagated, why the obvious cause turned out to be a symptom three layers from the root, the fix, and — more importantly — the prevention that shipped so it can't recur the same way. Every change referenced here is a public, reviewable pull request.
The platform in question — minicloud — is a production-grade Kubernetes environment running on refurbished ThinkPads, built and operated solo as a simulation of an enterprise information system. Bare metal, no managed control plane, no safety net. Which is exactly why it's a good teacher.