5 articles tagués avec « security »

When you build a production Kubernetes platform from scratch, you accumulate non-human identities faster than you expect. By the time the minicloud platform reached operational maturity — a 5-node bare-metal k3s cluster running 70+ workloads — it had more than 35 distinct non-human identities spanning six different categories. Most of them are invisible during normal operations. You only notice them when one breaks.

This post maps every non-human identity type in use on the platform, explains how they differ, and documents the operational lessons learned from the ones that caused incidents.

Every Kubernetes cluster that uses cert-manager for TLS has the same quiet risk buried in it: the CA private key that signs all your internal certificates is sitting in a Kubernetes secret, stored in plaintext in your cluster's datastore.

On managed clusters with etcd encryption at rest, this is adequately mitigated. On k3s with kine and SQLite — which is how many bare-metal clusters run — the secrets table is plaintext. Anyone who can read state.db from the control plane node can extract your CA private key and forge certificates your entire cluster trusts.

This post covers how we migrated the minicloud root CA private key into HashiCorp Vault's PKI secrets engine, with the same CA cert so nothing else needed to change — no re-trust, no downtime, no changes to any of our 43 Certificate resources.

Most enterprise identity architectures did not start as what they are today. They started in 1999 with Active Directory, accumulated LDAP integrations over the following decade, and are now partway through a migration toward cloud-based identity via Microsoft Entra ID — carrying the weight of every layer that came before.

We built our platform from scratch. We never had an on-premises domain. We never configured LDAP. We skipped directly to the protocol stack that enterprises are spending years and significant money trying to reach. This post explains what we chose, why, and how the resulting identity architecture compares to the enterprise standard.

OpenShift Container Platform is an opinionated enterprise Kubernetes distribution. My minicloud cluster is a 5-node k3s stack assembled component by component from CNCF projects. After going through the full build — GitOps, observability, secrets, registry, OIDC, ingress, storage replication, chaos testing, security patching, upgrades — I can say with some precision what the difference actually is.

It is not that OpenShift does more. It is that OpenShift has already made every choice you would have to make yourself, packaged those choices as a versioned, tested, supported unit, and enforced them at the architecture level. Whether that is a benefit or a constraint depends entirely on what you are trying to do.

Managed Kubernetes providers make security patching look simple. You enable auto-upgrade on GKE, you click "update node group" on EKS, and the CVE goes away. What actually happens is that the provider patches the OS image, replaces the node, validates the binary, and restores your workloads — all in the time it takes to refresh the AWS console.

On a self-managed cluster, none of that is automatic. You own the OS. You own the runtime. You own the base images. You own the Helm chart versions. And when a CVE drops, you own the decision about which layer it lives in and which mechanism will close it.

This post documents all five patching layers on minicloud — a 5-node k3s cluster on bare-metal ThinkPads — what's automated, what requires a PR, and where the gaps were and how they were closed.