AI Gateway — LiteLLM + PostgreSQL

Phase complete: 2026-07-05
Issues closed: #34 (LiteLLM proxy), #41 (inference optimization)
GitOps: manifests/ai/ in minicloud-gitops — ArgoCD Application litellm Synced/Healthy

Architecture

Open WebUI ──► LiteLLM Gateway (:4000)
                     │
                     ├── Ollama primary   (fast-heron,  :11434)  ← local-first
                     ├── Ollama secondary (star-kitten, :11434)  ← local-first
                     │
                     ├── Groq             (llama-3.1-8b-instant) ← auto-fallback #1
                     ├── DeepSeek         (deepseek-chat)        ← auto-fallback #2
                     ├── Mistral          (large, small)         ← premium/standard
                     ├── Anthropic        (claude-sonnet, haiku) ← premium
                     ├── OpenAI           (gpt-4o, gpt-4o-mini)  ← premium/standard
                     ├── Gemini           (gemini-2.5-flash)     ← premium/standard
                     ├── HuggingFace      (Qwen/Gemma via featherless-ai) ← standard
                     └── NVIDIA NIM       (nemotron-70b, llama-8b, deepseek-r1) ← premium/standard

LiteLLM ──► Valkey cache   (:6379)    ← exact-match prompt dedup, 10 min TTL
LiteLLM ──► Langfuse       (:3000)    ← trace every call with token/cost/model metadata

All components in the ai namespace. LiteLLM is the single OpenAI-compatible endpoint — Open WebUI talks only to LiteLLM, never to Ollama directly.

Components

Component	Image	Storage	Node
LiteLLM gateway	`harbor.10.0.0.200.nip.io/library/litellm:1.90.3-guardrails-v1`	stateless	any
PostgreSQL (ai)	`harbor.10.0.0.200.nip.io/library/postgresql:18.4.0`	Longhorn 5Gi	any
Ollama primary	`ollama/ollama`	local-path 10Gi	fast-heron
Ollama secondary	`ollama/ollama`	local-path 10Gi	star-kitten
Valkey cache	`valkey/valkey:8.1-alpine`	local-path 1Gi	any

Models

Local models (data stays on-premise)

Model name	Size	Capability	Tier access
`phi3-financial`	2.2 GB	Finance-domain text, restricted	premium, standard
`qwen3.5:4b`	3.4 GB	General text, native tool calling	premium, standard
`phi4-mini`	2.5 GB	Instruction following, technical	all
`deepseek-r1:7b`	4.7 GB	Reasoning — math, code, logic	premium
`llama3.2:1b`	1.3 GB	Ultra-fast, simple queries	all
`moondream`	1.7 GB	Vision — fast OCR, image description	premium, standard
`llava-phi3`	2.9 GB	Vision — detailed image analysis, document OCR	premium
`bge-m3`	1.2 GB	Embeddings — 1024-dim vectors, 100+ languages, French RAG	premium, standard

phi3-financial and vision models (moondream, llava-phi3) have no cloud fallback — sensitive content stays on-premise.

bge-m3 is available via /v1/embeddings. Open WebUI calls Ollama directly (bypassing LiteLLM) for RAG embeddings to avoid polluting cost tracking — see RAG Pipeline.

Cloud models (premium and standard tiers)

Model name	Backend	Tier access
`groq-fallback`	`groq/llama-3.1-8b-instant`	automatic fallback only
`deepseek-chat`	`deepseek/deepseek-chat`	automatic fallback + standard
`mistral-large`	`mistral/mistral-large-latest`	premium
`mistral-small`	`mistral/mistral-small-latest`	premium, standard
`claude-sonnet`	`anthropic/claude-3-5-sonnet-20241022`	premium
`claude-haiku`	`anthropic/claude-3-haiku-20240307`	premium
`gpt-4o`	`openai/gpt-4o`	premium
`gpt-4o-mini`	`openai/gpt-4o-mini`	premium, standard
`gemini-2.0-flash`	`gemini/gemini-2.5-flash`	premium, standard
`gemini-1.5-pro`	`gemini/gemini-2.5-flash`	premium
`hf-qwen`	`Qwen/Qwen2.5-1.5B-Instruct` via featherless-ai	premium, standard
`hf-gemma`	`google/gemma-2-2b-it` via featherless-ai	premium, standard
`nvidia-nemotron-70b`	`nvidia/llama-3.1-nemotron-70b-instruct` via NVIDIA NIM	premium
`nvidia-llama-8b`	`meta/llama-3.1-8b-instruct` via NVIDIA NIM	premium, standard
`nvidia-deepseek-r1`	`deepseek-ai/deepseek-r1` via NVIDIA NIM	premium, standard

gemini-2.0-flash and gemini-1.5-pro are model_name aliases kept for department key compatibility — both route to gemini/gemini-2.5-flash behind the scenes.

qwen3.5:4b thinking mode: qwen3 generates reasoning tokens (think phase) before the answer. Set max_tokens ≥ 500 — small values exhaust the token budget on reasoning, leaving content empty. For simple fast queries without thinking overhead, use phi4-mini or llama3.2:1b.

HuggingFace routing note: api-inference.huggingface.co was retired in 2025. HF models use https://router.huggingface.co/featherless-ai/v1 (OpenAI-compatible, free-tier provider). LiteLLM config uses openai/ provider type with explicit api_base and HUGGINGFACE_API_KEY.

NVIDIA NIM routing note: NVIDIA's inference API at https://integrate.api.nvidia.com/v1 is OpenAI-compatible. LiteLLM uses the openai/ provider type with explicit api_base and NVIDIA_API_KEY. Model IDs use the full org/model-name format (e.g. openai/nvidia/llama-3.1-nemotron-70b-instruct). Key registered at build.nvidia.com, stored in Vault at secret/platform/cloud-providers as nvidia-api-key, added 2026-07-07.

Routing strategy

router_settings.routing_strategy: least-busy — LiteLLM picks the backend with the fewest in-flight requests.

Cloud fallback — Groq then DeepSeek activate only when both Ollama backends exhaust num_retries: 2:

router_settings:
  routing_strategy: least-busy
  num_retries: 2
  timeout: 120
  # Circuit breaker: after 3 consecutive failures a backend is marked
  # degraded for 60 s — prevents a hung Ollama draining the retry budget.
  cooldown_time: 60
  allowed_fails: 3
  fallbacks:
    - qwen3.5:4b:
        - groq-fallback
        - deepseek-chat
    - phi4-mini:
        - groq-fallback
        - deepseek-chat
    - llama3.2:1b:
        - groq-fallback
        - deepseek-chat

phi3-financial has no cloud fallback — financial prompts must stay on-premise.

Circuit breaker behaviour: if one Ollama node becomes slow or unresponsive, LiteLLM stops routing to it after 3 failures and retries on the other Ollama node (or the Groq/DeepSeek fallback for llama models) for 60 seconds. Without this, a hung backend absorbs retries from every incoming request until timeout fires.

PostgreSQL databases

Database	Owner	Used by
`openwebui`	`aiplatform`	Open WebUI session/user data
`litellm`	`aiplatform`	LiteLLM virtual keys, spend tracking

Credentials: ESO ExternalSecret ai-postgresql-secret ← Vault KV secret/platform/ai-postgresql.

Department virtual key tiers

15 keys pre-provisioned (stored at ~/.litellm-department-keys on controller, mode 600).

Tier	Departments	TPM	RPM	Budget / 30 d	Allowed models
premium	IT, Data Analytics, Actuariat, Transformation	200k	500	$100	all models (local + cloud + vision)
standard	Cybersecurity, Audit, Finance, Reinsurance, Juridique, Souscription, Commercial	100k	200	$30	phi3-financial, qwen3.5:4b, phi4-mini, llama3.2:1b, moondream + groq + deepseek + mistral-small + gpt-4o-mini + gemini-2.0-flash + hf-qwen + hf-gemma
basic	Sinistres, Operations, RH, Services Généraux	50k	100	$5	phi3-financial, phi4-mini, llama3.2:1b

budget_duration: 30d resets automatically. When max_budget is reached LiteLLM returns HTTP 429 (BudgetExceededError) for that key until the next reset.

Update a key's rate limits or budget:

MASTER_KEY=$(kubectl get secret -n ai litellm-credentials -o jsonpath='{.data.master-key}' | base64 -d)
kubectl port-forward -n ai svc/litellm 4000:4000 &

curl -X POST http://localhost:4000/key/update \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "key": "sk-<dept-key>",
    "tpm_limit": 200000,
    "rpm_limit": 500,
    "max_budget": 100.0,
    "budget_duration": "30d",
    "models": ["phi3-financial", "llama3.2:3b", "gpt-4o"],
    "metadata": {"department": "direction-it", "tier": "premium"}
  }'

API field name gotcha: the budget cap field is max_budget (not budget_limit) — the LiteLLM API docs sometimes show both names but only max_budget is persisted.

Verify current spend and budget for all keys:

curl -s -H "Authorization: Bearer $MASTER_KEY" http://localhost:4000/key/list \
  | python3 -m json.tool | grep -E 'key_alias|spend|max_budget'

Prompt caching (Valkey)

Exact-match Redis cache. Identical prompts (same model + messages) return the cached response without hitting inference — typically 80ms vs 2500ms for a cold call.

litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: litellm-cache.ai.svc.cluster.local
    port: 6379
    ttl: 600      # 10 minutes
    namespace: litellm

Check cache stats: kubectl exec -n ai litellm-cache-0 -- valkey-cli info stats | grep hit

PII / DLP guardrail (Presidio)

Microsoft Presidio runs as two in-cluster services (presidio-analyzer + presidio-anonymizer, both in the ai namespace, mcr.microsoft.com/presidio-*:2.2.362). LiteLLM's pre_call guardrail presidio-pii-masking intercepts every prompt before it reaches any model.

What it does: Presidio analyzer detects PII entities in user prompts and the anonymizer replaces them with typed placeholders before the prompt is forwarded to any cloud model:

"Client jean.dupont@acme.com phone +33612345678 wants a quote"
     ↓ Presidio pre_call guardrail (input only)
"Client <EMAIL_ADDRESS> phone <PHONE_NUMBER> wants a quote"
     ↓ sent to cloud model (GPT-4o, Claude, Gemini, etc.)

Why this matters: Local models (phi3-financial, phi4-mini) keep data on-cluster. Cloud models (GPT-4o, Claude, Groq, DeepSeek, Mistral, HF) send the prompt to an external API — Presidio ensures the external API never receives raw PII. Covers models with cloud fallbacks too (if Ollama is down, PII is still anonymized before Groq receives it).

Scope: input only. output_parse_pii is intentionally NOT set. Anonymizing LLM responses would replace country names, public figures and sports results with <LOCATION>/<PERSON> placeholders, making general-knowledge responses unreadable. The security goal is protecting user data sent to cloud providers — not transforming the model's output.

Fail mode: default_on: true — guardrail is active on all requests by default.

guardrails:
  - guardrail_name: "presidio-pii-masking"
    litellm_params:
      guardrail: presidio
      mode: pre_call          # input protection only — output_parse_pii NOT set
      presidio_analyzer_api_base: "http://presidio-analyzer.ai.svc.cluster.local:3000"
      presidio_anonymizer_api_base: "http://presidio-anonymizer.ai.svc.cluster.local:3000"
      default_on: true

Verify Presidio is active:

MASTER=$(kubectl get secret -n ai litellm-credentials -o jsonpath='{.data.master-key}' | base64 -d)
kubectl port-forward -n ai svc/litellm 4000:4000 &
curl -s -H "Authorization: Bearer $MASTER" http://localhost:4000/guardrails/list | python3 -m json.tool

Presidio gotcha: Both presidio-analyzer and presidio-anonymizer MCR images listen on port 3000 (gunicorn default) — not 3000/5001 as the Presidio docs suggest for local dev.

output_parse_pii gotcha: Setting output_parse_pii: true anonymizes the LLM's response text. Presidio treats country names ("France", "Cameroun"), historical figures and dates as LOCATION/PERSON/DATE_TIME entities. A sports question gets back "<LOCATION> a battu <LOCATION> <DATE_TIME>" — unreadable. Only set output_parse_pii if your use case specifically requires redacting model outputs (e.g., a system that echoes back user-submitted documents).

Secret scanning guardrail (detect-secrets)

general_settings.detect_secrets_on_all_keys: true — LiteLLM scans every prompt using the detect-secrets library before forwarding to inference. Prompts containing high-confidence credential patterns (real API keys, tokens, connection strings) are rejected with HTTP 400.

The detect-secrets library is pre-installed in the base Wolfi venv (/app/.venv). Known-safe documentation example keys (e.g., AWS AKIAIOSFODNN7EXAMPLE) pass through — only high-entropy real credentials are blocked.

Both guardrails (presidio-pii-masking and detect_secrets_on_all_keys) run in sequence on every request.

Langfuse tracing

Every call produces a Langfuse trace with token counts, cost estimate, model used, department (from virtual key metadata), and latency.

general_settings:
  success_callback: ["langfuse", "prometheus"]
  failure_callback: ["prometheus"]
  langfuse_host: "http://langfuse-web.langfuse.svc.cluster.local:3000"

LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY injected from ESO-synced litellm-credentials secret.

Prometheus note: success_callback: ["prometheus"] is configured and the ServiceMonitor (manifests/ai/06-litellm-servicemonitor.yaml) is in git. However, LiteLLM 1.90.3's PrometheusLogger._mount_metrics_endpoint() registers the /metrics route via app.mount() after Starlette startup — the route does not take effect. Prometheus scraping from LiteLLM directly requires a future image rebuild. The Grafana cost dashboard below uses the PostgreSQL spend tables instead and is fully functional.

Cost dashboard (Grafana)

Live at https://grafana.10.0.0.200.nip.io/d/litellm-cost-dept/ (also available at https://grafana.devandre.sbs/d/litellm-cost-dept/).

Datasource: litellm-postgres (PostgreSQL, UID litellm-postgres) — points directly at the litellm database in the ai namespace PostgreSQL pod. Provisioned via Grafana API; credentials NOT in git.

# Re-provision the datasource if the Grafana pod is replaced:
MASTER=$(kubectl get secret -n ai litellm-credentials -o jsonpath='{.data.master-key}' | base64 -d)
GRAFANA_PASS=$(ssh controller cat ~/.grafana-admin)
curl -X POST -u "admin:$GRAFANA_PASS" https://grafana.10.0.0.200.nip.io/api/datasources \
  --cacert ~/minicloud-ca.crt -H "Content-Type: application/json" -d '{
    "name": "litellm-postgres",
    "type": "postgres",
    "url": "ai-postgresql.ai.svc.cluster.local:5432",
    "user": "aiplatform",
    "database": "litellm",
    "uid": "litellm-postgres",
    "secureJsonData": {"password": "<aiplatform-password>"},
    "jsonData": {"sslmode": "disable"}
  }'

The dashboard ConfigMap (manifests/ai/07-litellm-grafana-dashboard.yaml) is in the monitoring namespace with grafana_dashboard: "1" — the Grafana sidecar auto-loads it.

8 panels:

#	Type	Query
1	Stat	Total spend last 24 h (USD)
2	Stat	Total tokens last 24 h
3	Stat	Requests last 24 h
4	Stat	Active departments (30 d)
5	Table	Department budget utilisation — spend, budget, used% (colour thresholds: green < 70%, yellow < 90%, red ≥ 90%)
6	Bar gauge	Token usage by model last 30 d
7	Table	Requests by model last 30 d
8	Table	Recent requests (last 50) — audit trail with department column

Key SQL joins LiteLLM_SpendLogs (per-request records) with LiteLLM_VerificationToken (key metadata including max_budget) on api_key = token:

SELECT key_alias as "Department",
       ROUND(spend::numeric, 6) as "Spend (USD)",
       max_budget as "Budget (USD)",
       budget_duration as "Period",
       CASE WHEN max_budget > 0
            THEN ROUND((spend / max_budget * 100)::numeric, 1)
            ELSE 0 END as "Used %"
FROM "LiteLLM_VerificationToken"
WHERE key_alias LIKE 'direction-%'
ORDER BY spend DESC

Cloud provider secrets

API keys in Vault at secret/platform/cloud-providers:

VAULT_TOKEN=$(cat ~/.vault-root-token)
kubectl exec -n vault vault-0 -- \
  env VAULT_ADDR=http://127.0.0.1:8200 VAULT_TOKEN="$VAULT_TOKEN" \
  vault kv get secret/platform/cloud-providers

ESO ExternalSecret litellm-credentials (file manifests/eso-platform-secrets/10-ai-postgresql.yaml) syncs 8 cloud provider keys from secret/platform/cloud-providers into ai/litellm-credentials: groq-api-key, openai-api-key, gemini-api-key, deepseek-api-key, mistral-api-key, anthropic-api-key, hf-token, nvidia-api-key.

NetworkPolicy allow-litellm-cloud-egress permits port 443 egress only from app=litellm pods — Ollama and Open WebUI remain internet-blocked.

Custom LiteLLM image (`1.90.3-guardrails-v1`)

Three fixes over the upstream ghcr.io/berriai/litellm-database Wolfi OS base:

Fix	Reason
`chmod -R 755 /root/.cache/prisma-python`	Prisma `BINARY_PATHS` hardcodes root-owned paths; UID 1000 gets PermissionError before override env vars load
`apk add --no-cache libatomic`	Node.js binary (used by Prisma CLI) needs `libatomic.so.1`, absent on newer nodes
Install `google-generativeai>=0.8.0` via venv pip	Missing from base image; required for `gemini/` provider

Source: ~/Developer/cloudplateform/litellm-custom/Dockerfile

Rebuild:

cd ~/Developer/cloudplateform/litellm-custom
DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build --platform linux/amd64 \
  -t "litellm:1.90.3-guardrails-v1-amd64" .
docker save "litellm:1.90.3-guardrails-v1-amd64" -o /tmp/litellm.tar
crane push /tmp/litellm.tar "harbor.10.0.0.200.nip.io/library/litellm:1.90.3-guardrails-v1"
rm /tmp/litellm.tar

Important Wolfi gotchas:

Package manager is apk (not apt) — Wolfi is Alpine-compatible
No global pip/pip3 — use /app/.venv/bin/python -m ensurepip && /app/.venv/bin/python -m pip install
callbacks: ["detect_secrets"] in litellm_settings is NOT a valid proxy callback in 1.90.3 — causes ImportError. Use general_settings.detect_secrets_on_all_keys: true instead

Inference tuning (both Ollama instances)

OLLAMA_NUM_PARALLEL: "4"
OLLAMA_NUM_THREADS: "6"
OLLAMA_MAX_LOADED_MODELS: "2"
OLLAMA_FLASH_ATTENTION: "1"
OLLAMA_KV_CACHE_TYPE: "q8_0"
OLLAMA_NUM_CTX: "4096"

CPU governor set to performance on all 4 cluster nodes (persistent via cpu-performance.service).

Health check

kubectl port-forward -n ai svc/litellm 4000:4000 &
MASTER=$(kubectl get secret -n ai litellm-credentials -o jsonpath='{.data.master-key}' | base64 -d)

# Health
curl http://localhost:4000/health/readiness
# Expected: {"status": "healthy", "db": "connected"}

# Model list
curl -H "Authorization: Bearer $MASTER" http://localhost:4000/v1/models | python3 -m json.tool

# Test local
curl -H "Authorization: Bearer $MASTER" -H 'Content-Type: application/json' \
  http://localhost:4000/v1/chat/completions \
  -d '{"model":"llama3.2:1b","messages":[{"role":"user","content":"Reply OK"}],"max_tokens":5}'

# Test Gemini
curl -H "Authorization: Bearer $MASTER" -H 'Content-Type: application/json' \
  http://localhost:4000/v1/chat/completions \
  -d '{"model":"gemini-2.0-flash","messages":[{"role":"user","content":"Reply OK"}],"max_tokens":5}'

Adding models to Ollama

SECONDARY_POD=$(kubectl get pods -n ai -l app.kubernetes.io/instance=ollama-secondary \
  -o jsonpath='{.items[0].metadata.name}')

# Temp egress for model pull (delete after)
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: temp-allow-ollama-pull
  namespace: ai
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/instance: ollama-secondary
  egress:
  - ports:
    - port: 443
  policyTypes:
  - Egress
EOF

# Pull via REST (avoids TTY requirement)
kubectl port-forward -n ai pod/$SECONDARY_POD 11434:11434 &
curl -X POST http://localhost:11434/api/pull -d '{"model":"phi3.5","stream":false}'

kubectl delete networkpolicy temp-allow-ollama-pull -n ai

All secrets in Vault

VAULT_TOKEN=$(cat ~/.vault-root-token)
VE() { kubectl exec -n vault vault-0 -- env VAULT_ADDR=http://127.0.0.1:8200 VAULT_TOKEN="$VAULT_TOKEN" vault kv get "$1"; }

VE secret/platform/ai-postgresql    # DB credentials
VE secret/platform/litellm          # master key, Langfuse keys
VE secret/platform/cloud-providers  # Groq, OpenAI, Gemini, DeepSeek, Mistral, Anthropic, HuggingFace, NVIDIA API keys

Architecture​

Components​

Models​

Local models (data stays on-premise)​

Cloud models (premium and standard tiers)​

Routing strategy​

PostgreSQL databases​

Department virtual key tiers​

Prompt caching (Valkey)​

PII / DLP guardrail (Presidio)​

Secret scanning guardrail (detect-secrets)​

Langfuse tracing​

Cost dashboard (Grafana)​

Cloud provider secrets​

Custom LiteLLM image (1.90.3-guardrails-v1)​

Inference tuning (both Ollama instances)​

Health check​

Adding models to Ollama​

All secrets in Vault​