I run the information system of a simulated insurer on a six-node Kubernetes cluster, and the identity story is clean: Authentik is the single sign-on provider, every app is behind it, one login opens everything. That's the whole pitch of SSO — stop sprinkling passwords across forty apps.

So it was a little uncomfortable when, for the second time, I found myself standing up an LDAP server right next to Authentik. First for mail. Then, this week, for the ITSM. Twice I reached past the shiny OIDC door and opened an older, less fashionable one. The honest question a good engineer asks at that point isn't "how do I wire LDAP?" — it's "why does my beautiful SSO keep failing me here?"

The answer turned out to be the same both times, and it's worth writing down so I never re-derive it a third time.

A real industrial insurer does not operate by hiring employees and letting each department improvise. It runs on a structured framework — policies, standards, guidelines, procedures and controls — that defines how decisions are made, who is authorized to make them, what controls apply, and what evidence is kept. For an insurer this isn't bureaucracy: the company carries financial responsibility for risks worth millions or billions of euros.

Picture five situations:

• An underwriter accepts a €200 million industrial risk without proper assessment.
• A claims handler approves a €2 million claim without checking coverage or getting the required approval.
• A salesperson promises cover the company legally cannot provide.
• An IT administrator grants access to confidential claims data without authorization.
• A manager signs a contract with a supplier under financial sanctions.

Each one is a potential financial loss, legal problem, regulatory breach or reputational hit. Guidelines exist precisely to stop these — by defining how decisions must be made, who may make them, what controls are required, and what evidence must survive.

I rebuilt this framework end-to-end for ktayl-solution, a simulated insurer I run as an engineering laboratory. Here's what I learned.

There's a reflex in our industry, and I had it too: a new business domain shows up, and the hand reaches for "…so that's a new microservice." It feels modern. It feels scalable. It feels like what serious engineers do.

On the ktayl-solution information system — a six-node Kubernetes platform running a simulated insurer's entire IS — I'd built four domains exactly that way: policy, claims, underwriting, identity, each its own repo, CI pipeline, database, promotion track. Then I stopped and asked the question that matters more than any framework choice: is this the right architecture, or just the fashionable one?

This post is the answer I arrived at — modular-monolith-first — and, honestly, the more valuable half: the decision I deliberately didn't make.

The frontier conversation in AI has quietly shifted. For two years it was "which model is smartest?" Now the people actually shipping agentic systems are asking a different question: "what do you put around the model?"

That surrounding machinery has a name — the harness. The model is the engine. The harness is the chassis, the steering, the seatbelt and the brakes. A brilliant engine bolted to nothing kills you at the first corner; a modest engine in a well-built car gets you home every time. On the ktayl-solution information system — a six-node Kubernetes platform running a simulated insurer's entire IS — I've spent months building the harness that lets an AI agent do real engineering work against live infrastructure without me holding my breath.

This post is that harness, concept by concept. Not the theory — the actual rules I put in place, why each one exists, and the incident that usually forced it.

While hardening disaster recovery on the ktayl-solution information system, a fair question came up: we already mirror backups to Cloudflare R2 — so why keep a MinIO server running on the controller at all? Couldn't we retire it and go cloud-only?

R2's free tier is generous, cloud object storage is durable, and one less service to operate is always appealing. So it's a reasonable instinct. It's also wrong — and about an hour after I talked myself through why, I proved it the hard way by deleting the wrong bucket prefix. This post is that reasoning, and the honest incident that validated it.

ArgoCD reconciles everything inside my cluster: 95 applications, from Vault to the AI gateway, all declared in Git and continuously synced. But ArgoCD cannot format a disk, install open-iscsi, or fix a default route. There is a layer below GitOps — the operating system on each bare-metal node — and if that layer isn't codified, "reproducible infrastructure" is a half-truth.

For the ktayl-solution information system, that layer is owned by one small repo: minicloud-ansible. It's about 200 lines of task code across four roles, and it does exactly one job well: make the node OS prerequisites for a 6-node k3s cluster reproducible and auditable. This post is how I use it to operate the organisation's platform.

A monitoring alert fired: the platform watchdog was DOWN. Within minutes the picture looked ugly — cluster DNS was refusing connections, and shortly after, distributed-storage volumes stopped rebuilding. A textbook cascade.

This is the full post-mortem: how the outage propagated, why the obvious cause turned out to be a symptom three layers from the root, the fix, and — more importantly — the prevention that shipped so it can't recur the same way. Every change referenced here is a public, reviewable pull request.

The platform in question — minicloud — is a production-grade Kubernetes environment running on refurbished ThinkPads, built and operated solo as a simulation of an enterprise information system. Bare metal, no managed control plane, no safety net. Which is exactly why it's a good teacher.

If you run a Kubernetes cluster seriously, you eventually hit the Docker Hub wall. One morning your CI pipeline starts failing with toomanyrequests: You have reached your pull rate limit. Or worse — it fails during a cluster recovery at 2 AM, when your nodes are pulling images to restart critical workloads and Docker Hub decides you've exceeded your 100-pulls-per-6-hours anonymous quota.

The standard advice is to add authentication. But that only raises the limit — it doesn't eliminate the dependency on an external service during your most vulnerable moments. The production answer is a pull-through proxy cache, and on a self-hosted k3s cluster, Harbor + k3s mirrors makes every node behave as if Docker Hub were local.

When you build a production Kubernetes platform from scratch, you accumulate non-human identities faster than you expect. By the time the minicloud platform reached operational maturity — a 5-node bare-metal k3s cluster running 70+ workloads — it had more than 35 distinct non-human identities spanning six different categories. Most of them are invisible during normal operations. You only notice them when one breaks.

This post maps every non-human identity type in use on the platform, explains how they differ, and documents the operational lessons learned from the ones that caused incidents.

Every Kubernetes cluster that uses cert-manager for TLS has the same quiet risk buried in it: the CA private key that signs all your internal certificates is sitting in a Kubernetes secret, stored in plaintext in your cluster's datastore.

On managed clusters with etcd encryption at rest, this is adequately mitigated. On k3s with kine and SQLite — which is how many bare-metal clusters run — the secrets table is plaintext. Anyone who can read state.db from the control plane node can extract your CA private key and forge certificates your entire cluster trusts.

This post covers how we migrated the minicloud root CA private key into HashiCorp Vault's PKI secrets engine, with the same CA cert so nothing else needed to change — no re-trust, no downtime, no changes to any of our 43 Certificate resources.