minicloud — Production Stack Architecture

On-premises 5-node k3s cluster · 4 ThinkPads + MacBook Pro 2012 · GitOps · Observability · AI Platform · Phase 80 complete
INTERNET USERS browsers · mobile devandre.sbs · *.nip.io DEVELOPER MacBook Air (local) kubectl · ArgoCD CLI GITHUB ACTIONS 8 repos · CI/CD Tailscale + CA trust TAILSCALE VPN 100.88.123.8 controller tag:ci for GitHub Actions CLOUDFLARE CDN · WAF · DNS devandre.sbs zone AWS SES eu-west-1 · outbound relay inbound: SQS → S3 → pod GROQ API llama-3.1-8b-instant phi3-financial fallback k3s CLUSTER · v1.36.2+k3s1 · 5 NODES · 71 ARGOCD APPS ACCESS LAYER MetalLB LB IP: 10.0.0.200 NGINX Ingress community 4.15.1 · TLS Authentik SSO OIDC · TOTP · All apps cert-manager Let's Encrypt · minicloud CA Cloudflared Tunnel public → internal routing Vault + ESO PKI · secrets · KMS · AWS Gatekeeper + Falco OPA policies · runtime sec Longhorn + MinIO distributed storage · backup GITOPS & AUTOMATION ArgoCD app-of-apps · 71 apps Harbor Registry Cosign · SBOM · CVE scan Argo Rollouts Canary · BlueGreen AnalysisTemplate Prometheus gates · k6 KEDA event-driven scale VPA right-sizing · Auto mode Reloader ConfigMap hot-reload system-upgrade k3s upgrades · Plans Velero + Kopia daily backup → MinIO Vaultwarden SSO passwords · Timshel NetworkPolicies · default-deny-ingress Gatekeeper · Falco runtime · kube-bench CIS DEVELOPER PLATFORM Backstage IDP TechDocs · Templates NATS JetStream · messaging platform-demo Go svc · Canary 50% minicloud-plane Go svc · BlueGreen Plane CE project management Homer dashboard · all links COLLABORATION Nextcloud files · docs · OnlyOffice ERPNext HR source of truth Matrix + Element Synapse · SSO · OIDC Jitsi Meet star-kitten · JVB host Stalwart Mail v0.16.13 devandre.sbs · SMTP:587 · IMAP:993 → SES relay outbound AI / ML PLATFORM LiteLLM Proxy virtual keys · routing Ollama phi4-mini · nomic-embed Open WebUI chat UI · RAG · SSO Langfuse LLM traces · evals markitdown-proxy PDF/Office → Markdown rag-ingest chunk → embed → store pgvector / ragdb 768-dim nomic-embed-text vectors · PostgreSQL AI OBSERVABILITY METRICS TTFT · Tokens/s Langfuse traces Cost/Request LiteLLM Prometheus LLM Error Rate success/fail counters Embed Latency nomic-embed-text p95 CI / CD PIPELINE Build → Trivy scan → Sign docker buildx · Cosign · syft SBOM Push → Harbor pre-flight harbor.10.0.0.200.nip.io via Tailscale bump-gitops → ArgoCD sync GPG-signed PR · kustomize set image PARALLEL — ANALYSIS WINDOW canary-load k6 · 5 min · p95<500ms rollout-gate polls Healthy/Degraded promote-staging needs: [rollout-gate, canary-load] · GPG PR REGRESSION DETECTION Prometheus Gates success-rate ≥ 95% RolloutAutoRollback Degraded → email alert STORAGE & INFRA Longhorn distributed block storage · swift-mac PVCs MinIO controller · S3 compat Velero + Kopia daily → MinIO TTL 7d MAAS Controller ThinkPad X390 · NAT · Tailscale endpoint · node_exporter EMAIL PIPELINE SES Inbound (AWS eu-west-1) MX → S3 → SNS → SQS → ses-ingest pod SES Outbound Relay Stalwart → ses-relay → SES:587 → recipient minicloud CA root CA · TLS all svcs OpenTofu / MAAS node provisioning IaC OBSERVABILITY STACK · 64/64 PROMETHEUS TARGETS UP · 17 CUSTOM APP TARGETS Prometheus (kps) kube-state-metrics Node Exporter (cluster + ctrl) cAdvisor · 33 recording rules SLO rules · Regression rules Argo Rollouts metrics :8090 scheduler · ctrl-mgr socat proxy 64 targets · 1 min eval · 30d retention Grafana 35+ dashboards Workload Health App Logs Explorer Error Rate by Service SLO Dashboard · Traces Regression Detection dataLinks → Loki → Tempo Alertmanager SMTP → Stalwart:587 → SES → kanmegnea@ RolloutAutoRollback LatencyRegression ErrorRateRegression SLO burn-rate alerts 6 LogQL ruler alerts (Loki) Loki log aggregation OTTL transforms level label extraction kopia cardinality fix ruler: 6 LogQL alerts controller via Promtail 30-day retention · OTLP native OTel Collector DaemonSet 0.156.0 filelog receiver (pods) OTTL transformprocessor → Loki (otlphttp) → Tempo (otlp gRPC) runAsUser:0 for /var/log replaces Promtail in-cluster Tempo 2.9.0 distributed tracing OTLP gRPC:4317 TraceQL · TraceToLogs tracesToLogsV2 → Loki 14-day retention Go services instrumented Gaps A–E complete (2026-07-25) SLO + Regression 7 SLO objectives 11 recording rules burn-rate 2-tier offset 1h baseline 5 regression alerts Argo Rollouts gates Prometheus PrometheusRules HARDWARE · MAAS PROVISIONED · UBUNTU 22.04 · AUTO-RESTART ON POWER FAILURE (except swift-mac) MAAS Controller ThinkPad X390 · Ubuntu 22.04 100.88.123.8 · 98G NVMe NAT · MinIO · Promtail · Cloudflared node_exporter :9100 set-hog k3s CONTROL PLANE · 10.0.0.2 ThinkPad · Kine/SQLite etcd scheduler + ctrl-mgr socat proxy hostNetwork metrics :10269 :10267 fast-skunk k3s worker · 10.0.0.4 ThinkPad X390 general workloads fast-heron k3s worker · 10.0.0.7 ThinkPad Vault pinned (nodeSelector) star-kitten k3s worker · 10.0.0.8 ThinkPad Jitsi JVB pinned (hostNetwork) swift-mac k3s worker · 10.0.0.10 MacBook Pro 13" 2012 · Ubuntu 22.04 Longhorn storage preferred no Apple SMC auto-restart
GITOPS & AUTOMATION
  • ArgoCD 3.x — app-of-apps, 71 apps, SSA=false for Rollout apps
  • Harbor — registry, Cosign signing, SBOM attestation, Trivy CVE scan
  • Argo Rollouts — platform-demo (Canary 50%), minicloud-plane (BlueGreen)
  • KEDA — event-driven scale on NATS JetStream queue length
  • VPA — Off mode: langfuse-clickhouse, Prometheus; Auto: backstage, litellm
  • Reloader — auto-restart: homer, litellm, searxng, backstage on ConfigMap change
  • system-upgrade-controller — k3s v1.36.2+k3s1 all 5 nodes via Plans
  • Velero + Kopia — daily 03:00, TTL 7d, MinIO bucket, itemOperationTimeout 8h
SECURITY
  • Authentik SSO — OIDC + TOTP for all apps, Embedded Outpost, forward-auth
  • Vault — HashiCorp CE, AWS KMS auto-unseal, fast-heron pinned (RWO)
  • ESO — ExternalSecrets, ClusterSecretStore, Vault backend
  • cert-manager — Let's Encrypt (public), minicloud CA (internal TLS)
  • Gatekeeper OPA — 9 constraints: BlockNetRaw, BlockCapabilities, RequireSeccomp
  • Falco — runtime syscall monitoring, DaemonSet
  • Vaultwarden — Timshel fork 1.34.1-6, SSO via Authentik, 15 creds migrated
  • kube-bench — k3s-CIS-1.7: 16/16 PASS on all 4 workers
AI / ML PLATFORM
  • LiteLLM — proxy, virtual keys, routing, Prometheus metrics, cost tracking
  • Ollama — phi4-mini (chat), nomic-embed-text 768-dim (RAG embeddings)
  • Open WebUI — custom image, sitecustomize.py Prometheus instrumentation
  • Langfuse — LLM traces, TTFT, tokens/s, prompt eval, ClickHouse + PG backend
  • markitdown-proxy — PDF/images→Docling, Office→MarkItDown, port 8000
  • rag-ingest — convert→chunk→embed→pgvector pipeline, port 8001
  • pgvector (ragdb) — PostgreSQL 768-dim vectors for nomic-embed-text
  • Groq — llama-3.1-8b-instant fallback for phi3-financial (25/25 CI eval)
CI / CD & REGRESSION DETECTION
  • 8 custom repos — all → harbor.10.0.0.200.nip.io via Tailscale tag:ci
  • Org-level secrets — 7 secrets on andrelair-platform, all repos inherit
  • Golden path — build→Trivy→Cosign→SBOM→pre-flight→bump-gitops→GPG PR
  • canary-load — parallel k6 5min, covers AnalysisRun window, initialDelay:60s
  • rollout-gate — polls Rollout phase, HTTP smoke probe, 8min timeout
  • Prometheus gates — success-rate≥95%, p95≤500ms, or vector fallback
  • RolloutAutoRollback — argo_rollout_info{phase="Degraded"}→email in 1min
  • promote-staging — needs: [rollout-gate, canary-load], GPG-signed PR
OBSERVABILITY
  • Prometheus — 64 targets, 17 custom app targets, 33 recording rules, SLO
  • Grafana — 35+ dashboards: Workload, App Logs, Error Rate, Crash, SLO, Traces
  • Loki — OTLP native, 30d retention, ruler 6 LogQL alerts, kopia cardinality fix
  • OTel Collector — DaemonSet, OTTL transforms, runAsUser:0, replaces Promtail
  • Tempo — 2.9.0, distributed tracing, TraceQL, tracesToLogs→Loki, 14d
  • Alertmanager — SMTP via Stalwart:587→SES, regression alerts, SLO burn-rate
  • Node Exporter — cluster nodes + MAAS controller (Docker+systemd)
  • SLO rules — 7 objectives, 11 recording rules, 9 alert rules (burn-rate 2-tier)
COLLABORATION & EMAIL
  • Stalwart Mail v0.16.13 — devandre.sbs, RocksDB, JMAP API, STARTTLS
  • SES eu-west-1 — production access 2026-07-26, 50k msg/day, SPF+DKIM+DMARC
  • SES Inbound — MX→S3→SNS→SQS→ses-ingest pod→Stalwart→mailbox
  • Matrix Synapse — v1.156.0, Element Web, OIDC via Authentik, 10Gi PVC
  • ERPNext — v16.28.0, HR source of truth, MariaDB+Valkey, employee onboard
  • Jitsi Meet — v2.21.0, JVB pinned star-kitten hostNetwork, Authentik forward-auth
  • Nextcloud — OnlyOffice DocumentServer, SSO, Longhorn PVC
  • Backstage onboard template — scaffolder→Stalwart JMAP→Authentik→mail
STORAGE & NETWORKING
  • Longhorn — distributed block storage, swift-mac preferred, multipathd IET blacklist
  • MinIO — controller Docker container, S3-compat, Velero backup bucket
  • MetalLB — L2 mode, LB IP 10.0.0.200, NGINX Ingress Service
  • NGINX Ingress — community 4.15.1, latency histogram, annotations-risk-level:Critical
  • Cloudflared — systemd Restart=always, public *.devandre.sbs tunnel
  • Tailscale — controller endpoint 100.88.123.8, CI tag:ci OAuth
  • minicloud CA — root CA, macOS System Keychain, Vault PKI, cert-manager ClusterIssuer
  • MAAS — OpenTofu IaC, bare-metal provisioning, NAT 10.0.0.0/24
HARDWARE
  • MAAS Controller — ThinkPad X390, 98G NVMe, Tailscale, NAT, MinIO, Cloudflared
  • set-hog — k3s control-plane, 10.0.0.2, Kine/SQLite, socat proxy for metrics
  • fast-skunk — k3s worker, 10.0.0.4, general workloads
  • fast-heron — k3s worker, 10.0.0.7, Vault nodeSelector pinned
  • star-kitten — k3s worker, 10.0.0.8, Jitsi JVB hostNetwork pinned
  • swift-mac — k3s worker, 10.0.0.10, MacBook Pro 13" 2012, Longhorn preferred, no SMC auto-restart
  • Boot order — controller (30s) → cluster nodes (2min) → Tailscale on Mac
  • Power failure recovery — iptables-restore + FORWARD rules manually on controller
GITOPS DESIGN PATTERNS
  • Two-repo split — source code repo + deployment config repo, always separate
  • Source repo — application code, Dockerfile, CI pipeline (builds image, bumps tag)
  • Deployment repo — Kustomize overlays / Helm values only, zero application code
  • Monorepo pattern — one deployment repo for all services; best for single team or solo operator
  • Per-project pattern — one deployment repo per service/team; required when access control or per-app audit trails are needed
  • Decision rule — start monorepo, split at ~5–10 teams when access control becomes concrete
  • minicloud — monorepo (minicloud-gitops) for all services; correct at current scale
→ Full doc: GitOps Repository Patterns