RAG Pipeline — pgvector + bge-m3 + SearXNG Web Search

Implemented: 2026-07-05 (RAG + web search + re-ranking), 2026-07-06 (Docling OCR + French BM25 + bge-m3 1024-dim), 2026-07-07 (smaller chunks + markitdown-proxy + rag-ingest cluster services) Status: Production

Retrieval-Augmented Generation (RAG) enables Open WebUI to answer questions grounded in uploaded documents. Every component runs on-cluster — no external embedding API, no new services.

Knowledge sources

Open WebUI provides two complementary grounding mechanisms that can be used independently or together:

Source	What it knows	Latency	Activation
RAG (pgvector)	Documents users have uploaded (PDFs, text, CSV)	0.3–7 ms (pgvector) + 2–8 s (inference)	Paperclip icon → upload file
Web Search (SearXNG)	Real-time internet — current events, latest docs, live data	~1–3 s	Globe icon in toolbar

Architecture

RAG path (uploaded documents)

Two ingestion paths feed the same retrieval pipeline:

Path 1 — Open WebUI paperclip upload:
  File upload → markitdown-proxy:8000
      → PDF/image → Docling (OCR + layout)
      → DOCX/XLSX/PPTX/HTML → MarkItDown (structure-aware)
      → Open WebUI chunker (500 chars, 100 overlap)
      → bge-m3 via Ollama (1024-dim) → pgvector HNSW (ragdb)

Path 2 — rag-ingest cluster service (POST /ingest):
  File POST → rag-ingest:8001
      → markitdown-proxy:8000 (same conversion routing)
      → French heading parser (Article/Annexe/Chapitre/Section)
      → structural chunks + metadata (document_type, article, section, source)
      → bge-m3 via Ollama (1024-dim) → ragdb.document_chunk INSERT

Retrieval (both paths feed the same index):
  User question → bge-m3 embeds query (1024-dim)
      → pgvector cosine search (TOP_K=10) + Python BM25Retriever (French stemmer)
      → RRF merge → cross-encoder re-ranker → top 3 chunks → LLM answer

Web search path (real-time internet)

User question (web search toggle ON)
    → Open WebUI → SearXNG /search?q=<query>&format=json
    → SearXNG aggregates: Google + Bing + DuckDuckGo + Wikipedia
    → top 5 results (title + URL + snippet)
    → Open WebUI fetches full page content (via internet egress NP)
    → results injected into LLM context → LLM answer with citations

Components

Component	Image / Version	Role
bge-m3	1.2 GB (Ollama)	1024-dim embedding model — 100+ languages, state-of-the-art multilingual retrieval
postgresql-ai	pgvector 0.8.4 (noavx512)	Vector store — ragdb database on the existing pod
markitdown-proxy	harbor.../markitdown-proxy:fd8b5f7	Docling-compatible HTTP proxy — routes PDF→Docling, Office/HTML→MarkItDown
rag-ingest	harbor.../rag-ingest:58e9bc1	Cluster-native ingestion service — POST /ingest → structure chunker → bge-m3 → ragdb
Docling	docling-serve-cpu:v1.26.0	PDF OCR + layout conversion (called by markitdown-proxy)
Open WebUI	0.9.4	RAG orchestrator + web search frontend (app data → PostgreSQL openwebui DB)
LiteLLM	1.90.3	AI Gateway — chat completions only; embeddings bypass LiteLLM (direct Ollama)
SearXNG	2026.7.3	Self-hosted meta-search engine — real-time internet results

Why these choices

bge-m3 over nomic-embed-text: BAAI/bge-m3 (1024-dim) outperforms nomic-embed-text on French insurance vocabulary — nomic is primarily English-trained and has weaker morphological coverage for inflected French forms ("indemnités", "réassureurs"). bge-m3 supports 100+ languages with consistent quality and produces richer semantic representations for domain-specific terms. The tradeoff is size (1.2 GB vs 274 MB) and slightly higher embedding latency; both are acceptable given inference latency (2–8 s) dominates end-to-end response time. The 1024-dim output requires vector(1024) in PostgreSQL — the ragdb schema was migrated when this switch was made.

bge-m3 over OpenAI ada-002: No external API, no per-token cost, no data leaving the cluster.

pgvector (existing pod) over a new Weaviate/Qdrant: No new services, no new PVCs. postgresql-ai already runs in the cluster; just adding a second database (ragdb) inside the same pod.

HNSW, with a custom pgvector build: pgvector 0.8.4 in the upstream Bitnami image contains AVX-512 EVEX-encoded instructions in its HNSW and IVFFlat index-update routines. All four cluster CPUs (i7-8565U and i7-10510U) lack AVX-512 — every INSERT that triggers an HNSW or IVFFlat graph update terminates the PostgreSQL backend with SIGILL (signal 4, Illegal Instruction). The fix is a custom pgvector build compiled with -mno-avx512f -mno-avx512bw -mno-avx512vl -mno-avx512dq, keeping the AVX2/SSE4 code paths intact. The new image (harbor.../postgresql:18.4.0-noavx512) is maintained in minicloud-ansible/docker/postgresql-noavx512/. Sequential-scan vector distance queries and GIN FTS are unaffected — only the index maintenance code path contained the AVX-512 instructions.

Direct Ollama for embeddings (bypasses LiteLLM): RAG_EMBEDDING_ENGINE=ollama + RAG_OLLAMA_BASE_URL points Open WebUI directly at ollama.ai.svc:11434. This keeps internal embedding calls out of LiteLLM's spend-tracking database — no phantom costs on the cost dashboard.

Measured latency profile

All values measured on the live cluster (2026-07-06) with 500 rows in ragdb.document_chunk.

Operation	p50 latency	Notes
HNSW vector search	0.31 ms	SET enable_seqscan=off forces index path; planner prefers seqscan below 1k rows
Sequential scan vector search	1.29 ms	Planner-selected at 500 rows; faster than HNSW at this scale
GIN FTS query (benchmark only)	2.71 ms	The GIN index exists but is not queried by Open WebUI's hybrid search — BM25 is Python in-process (see Phase C)
INSERT + HNSW update	6.8 ms/row	Dominated by HNSW graph maintenance, not the INSERT itself
nomic-embed-text embedding (historical)	393 ms	Measured pre-bge-m3 migration; in-cluster latency was ~30–80 ms. bge-m3 latency not yet benchmarked (larger model, expect ~2–4× higher)
Ollama inference	2,000–8,000 ms	Model-dependent; CPU-only; dominates end-to-end latency

Key takeaway: pgvector contributes less than 1% of total RAG latency. Ollama inference (2–8 s) is the bottleneck at every scale. Optimizing HNSW parameters or switching index types has no user-visible impact. The right levers for latency reduction are model size (phi4-mini vs deepseek-r1:7b) and cloud routing (DeepSeek API at 5–8 s vs local CPU at 40+ s).

HNSW is auto-selected by PostgreSQL's cost model at ~1,000+ rows. Below that threshold the planner uses sequential scan, which is faster at small table sizes and produces identical quality — the planner is correct. To force HNSW for benchmarking, run SET enable_seqscan=off in the session.

Scaling roadmap

The embedding model and the vector store are matched components — you cannot improve one without also addressing the other, because they share the same bottleneck.

Current ceiling: bge-m3 (CPU, 1024-dim) + pgvector HNSW (single pod, 512 Mi RAM) → suitable for tens of thousands of chunks, sub-second search latency at demo scale.

Why switching embedding models alone changes nothing meaningful: if you replaced bge-m3 with OpenAI ada-002 (1536-dim) without replacing pgvector, the HNSW graph would grow in RAM, search latency would increase, and you'd still hit the pgvector ceiling before the embedding quality gap matters. You'd have added cost and external API dependency while the actual constraint — pgvector on a 512 Mi pod — remained unchanged.

The right upgrade path is a full stack replacement:

Scale	Action
~100k chunks (current)	Current stack is correctly matched — tune HNSW ef_search if needed
~1M chunks	Replace pgvector with a dedicated Qdrant or Weaviate pod with more RAM
~10M chunks	Managed vector DB (Pinecone, Weaviate Cloud) + OpenAI/Cohere embeddings

The lesson: optimize the bottleneck, not adjacent components. At demo scale, the current stack is the right choice. When document volume grows to the point where pgvector search latency degrades, replace both the vector store and the embedding model together.

Document ingestion quality roadmap

Compared to RAGFlow (the leading self-hosted enterprise RAG platform), three gaps exist in the current stack. All three are additive improvements on top of the existing deployment — no stack replacement required, because Open WebUI 0.9.4 already has native hooks for each.

Phase A — Re-ranking (cross-encoder) ✅ Live (2026-07-05)

The gap: after HNSW cosine-distance search returns the top-K chunks, they are ranked by embedding similarity — which measures topic proximity, not answer quality. A chunk that mentions the query terms in passing ranks the same as one that directly answers the question.

What re-ranking adds: a cross-encoder model reads the query and each candidate chunk together (not as independent vectors) and scores the pair holistically. Top-K after re-ranking is meaningfully better than top-K from cosine distance alone.

Deployed config (minicloud-ansible commit 2fc03f7):

# open-webui-values.yaml extraEnvVars
- name: RAG_RERANKING_ENGINE
  value: "sentence_transformers"
- name: RAG_RERANKING_MODEL
  value: "cross-encoder/ms-marco-MiniLM-L-6-v2"   # 80 MB, CPU-friendly
- name: RAG_TOP_K_RERANKER
  value: "3"                                         # keep top 3 after reranking

The model runs in-process inside the Open WebUI pod. No new Deployment or Service. Smoke test: "Paraguay 0-1 France FIFA World Cup 2026" scored 5.587; irrelevant "History of football in South America" scored -11.295 — correct ordering confirmed. For a larger reranker (BGE-Reranker-v2-M3, 568 MB), use RAG_EXTERNAL_RERANKER_URL pointing to a dedicated pod to keep Open WebUI pod RAM under control.

---

Phase B — OCR & advanced document parsing (Docling) ✅ Live (2026-07-06)

The gap: the current stack handles text-based PDFs only. Scanned insurance documents, contract images, and mixed-format files (PDF with embedded tables + scanned pages) produce empty or garbled chunks.

What Docling adds: IBM Docling (open-source, self-hostable) converts scanned PDFs, images, and complex layouts into clean text before chunking. It handles tables, multi-column layouts, headers/footers, and embedded images with text — producing extraction quality close to commercial solutions.

Deployed config:

• manifests/ai/08-docling.yaml (minicloud-gitops commit 116d5e4) — Deployment + Service in the ai namespace
• open-webui-values.yaml (minicloud-ansible commit d8a4bf6) — CONTENT_EXTRACTION_ENGINE=docling + DOCLING_SERVER_URL

# 08-docling.yaml (key fields)
image: ghcr.io/docling-project/docling-serve-cpu:v1.26.0   # CPU-only, 4.4 GB, models bundled
resources:
  requests: {cpu: 200m, memory: 1500Mi}
  limits:   {cpu: 2000m, memory: 3Gi}
readinessProbe: GET /ready   initialDelaySeconds: 30
livenessProbe:  GET /health  initialDelaySeconds: 120

# open-webui-values.yaml extraEnvVars
- name: CONTENT_EXTRACTION_ENGINE
  value: "docling"
- name: DOCLING_SERVER_URL
  value: "http://docling.ai.svc.cluster.local:5001"

Open WebUI calls POST /v1/convert/file and reads response.document.md_content. Text-based PDFs still work — Docling handles both. NetworkPolicy: allow-same-namespace in the ai namespace already covers Open WebUI → docling port 5001 (cluster-internal, no internet egress needed).

Gotcha — image size: docling-serve-cpu:v1.26.0 is 4.4 GB (AMD64). First pull on a node takes 10–20 minutes. Models are bundled, so no runtime downloads occur after that. Use the CPU-only variant (-cpu suffix); the base image is 8.7 GB (includes GPU libs for no reason on CPU-only nodes).

---

Phase C — Hybrid search ✅ Live (2026-07-06)

When ENABLE_RAG_HYBRID_SEARCH=true, Open WebUI fetches the entire collection from pgvector and runs two retrieval legs in Python, then merges with Reciprocal Rank Fusion:

Component	Implementation	Where it runs
Vector search	HNSW cosine similarity (vector_cosine_ops)	PostgreSQL (idx_document_chunk_vector)
BM25 keyword search	langchain_community.retrievers.BM25Retriever (rank_bm25)	Python in-process

Architecture note: the BM25 leg is entirely in Python — BM25Retriever loads all document chunks from the collection into memory and scores them using the rank_bm25 library. There is no native PostgreSQL FTS query in the hybrid path. The GIN index (idx_document_chunk_text_search) exists in the database but is not called by Open WebUI's hybrid search — it is retained as a foundation for a future native-FTS migration.

Deployed config (minicloud-ansible commit 7a2e75e):

# open-webui-values.yaml extraEnvVars
- name: ENABLE_RAG_HYBRID_SEARCH
  value: "true"
- name: RAG_HYBRID_BM25_WEIGHT
  value: "0.5"   # 50% BM25 + 50% vector; raise for keyword-heavy queries

Weight tuning guide:

• 0.5 — balanced (default, good for mixed semantic + keyword queries)
• 0.3 — lean toward vector (better for abstract questions: "what is our risk exposure?")
• 0.7 — lean toward BM25 (better for exact-term lookup: "article 42", "IBNR", specific policy numbers)

---

Phase D — French BM25 tokenizer ✅ Live (2026-07-06)

The problem: rank_bm25's default tokenizer is str.split() — whitespace only, no lowercasing, no stemming, no stop-word removal. French inflected forms are distinct tokens: "sinistres" ≠ "sinistre", "réassurance" ≠ "réassurer". A query for "sinistre" misses every chunk containing "sinistres".

The fix: a preprocess_func injected into BM25Retriever.from_texts that applies NLTK's Snowball French stemmer and removes common French stop words (le, de, est, pour, dans, etc.).

How it is deployed: an init container (patch-bm25-french) copies utils.py from the image filesystem, runs a patch script from a ConfigMap, and writes the result to an emptyDir. The main container mounts that file via subPath, shadowing the original. No custom image required — the patch re-applies on every pod restart.

minicloud-gitops/manifests/ai/09-bm25-french-patchscript.yaml   ← ConfigMap (patch script)
minicloud-ansible/helm-values/open-webui-values.yaml             ← init container + volumes

Verified stems (insurance vocabulary):

Query token	Corpus token	Shared stem	Match?
sinistre	sinistres	sinistr	✅
réassureur	réassurance	réassur	✅
indemnité	indemnités	indemnit	✅
rémunérer	rémunération	rémuner	✅

Stop words filtered from "le contrat de réassurance est soumis aux conditions de la police" → ['contrat', 'reassur', 'soum', 'condit', 'polic'].

Limitation: accent-insensitive matching requires the query to use the same accent form as the corpus. "remuneration" (no accent) and "rémunération" (with accent) produce different stems. This is a known limitation of the Snowball stemmer — the unaccent PostgreSQL extension would solve it for native FTS, but not for Python BM25. Workaround: ensure uploaded documents use proper French accents; LLM queries typically include accents via Authentik keyboard.

---

Phase E — bge-m3 1024-dim + wider retrieval net ✅ Live (2026-07-06)

Three constraints addressed together:

Constraint 1 — cross-referenced clauses (chunk retrieval breadth): Insurance contracts frequently reference other sections ("see section 12.3.a for the definition of indemnity"). With RAG_TOP_K=5, section 12.3.a might be the 7th-most-similar chunk by embedding distance and never be retrieved, so the re-ranker never sees it. Raising RAG_TOP_K to 8 casts a wider net — the re-ranker has 8 candidates to choose from and can promote the referenced section if it judges it relevant. The final context sent to the LLM is still capped at RAG_TOP_K_RERANKER=3.

Constraint 2 — multilingual embedding quality: nomic-embed-text is primarily English-trained. French insurance vocabulary — "sinistres", "réassurance", "indemnités" — is represented in a weaker region of its embedding space, producing lower cosine similarity scores for semantically related French terms. bge-m3 (BAAI, 1024-dim) is trained on 100+ languages with consistent quality and significantly better French morphological coverage. The larger embedding space (1024 vs 768 dims) also allows more nuanced semantic distinctions between legal concepts.

Constraint 3 — GIN FTS dictionary: The GIN index idx_document_chunk_text_search used the simple dictionary, which strips stop words but does not stem. "indemnités" and "indemnité" were distinct tokens. Updated to french dictionary (PostgreSQL built-in French Snowball stemmer). This is future-proofing — the GIN index is not currently queried by Open WebUI's hybrid search path, but will be if a native FTS migration is undertaken.

Deployed config changes (minicloud-ansible commit 0696615):

# open-webui-values.yaml — changed lines
- name: RAG_EMBEDDING_MODEL
  value: "bge-m3"                  # was: nomic-embed-text
- name: RAG_TOP_K
  value: "8"                       # was: 5
- name: VECTOR_LENGTH
  value: "1024"                    # was: 768
- name: PGVECTOR_INITIALIZE_MAX_VECTOR_LENGTH
  value: "1024"                    # was: 768

ragdb schema migration (applied in-cluster):

-- Run as aiplatform on ragdb
TRUNCATE document_chunk;
DROP INDEX idx_document_chunk_text_search;
DROP INDEX idx_document_chunk_vector;
ALTER TABLE document_chunk DROP COLUMN vector;
ALTER TABLE document_chunk ADD COLUMN vector vector(1024);
CREATE INDEX idx_document_chunk_vector
  ON document_chunk USING hnsw (vector vector_cosine_ops)
  WITH (m=16, ef_construction=64);
CREATE INDEX idx_document_chunk_text_search
  ON document_chunk USING GIN(to_tsvector('french', COALESCE(text, '')));

Gotcha — sentence_transformers is not a valid RAG_EMBEDDING_ENGINE in Open WebUI 0.9.4. The value sentence_transformers is only valid for RAG_RERANKING_ENGINE. Setting it as the embedding engine causes Open WebUI to crash at startup with ValueError: Unknown embedding engine: sentence_transformers. The only valid values for RAG_EMBEDDING_ENGINE are ollama and openai.

After this migration: all previously uploaded documents were wiped from the vector store (existing 768-dim embeddings are geometrically incompatible with the 1024-dim space). Documents must be re-uploaded through the Open WebUI knowledge base interface to be re-embedded with bge-m3.

---

Phase F — Smaller chunks + structure-aware ingestion ✅ Live (2026-07-07)

Two complementary improvements to document ingestion quality, deployed as minicloud-ansible commit b9b71e9.

Phase F.1 — Reduced chunk size (Open WebUI built-in chunker)

The problem: the previous CHUNK_SIZE=1500 (chars) tends to merge multiple insurance clauses into one chunk. Cosine similarity then retrieves the right clause bundle but dilutes the re-ranker's precision — the LLM sees paragraphs containing the answer mixed with unrelated clauses from the same section.

The fix: CHUNK_SIZE=500, CHUNK_OVERLAP=100, RAG_TOP_K=10. Smaller chunks isolate individual clauses; the wider retrieval net (10 instead of 8) compensates by giving the re-ranker more candidates. RAG_TOP_K_RERANKER=3 still caps final LLM context.

# open-webui-values.yaml — Phase F.1 changes
- name: CHUNK_SIZE
  value: "500"          # was: 1500
- name: CHUNK_OVERLAP
  value: "100"          # was: 200
- name: RAG_TOP_K
  value: "10"           # was: 8

Trade-off: more vectors in ragdb (roughly 3× the chunk count for the same document corpus). HNSW graph grows proportionally but search latency is unaffected at demo scale (below 1k rows uses sequential scan; HNSW auto-selected above that).

Phase F.2 — Structure-aware ingestion pipeline (rag-ingest.py)

The problem: Open WebUI's built-in chunker is RecursiveCharacterTextSplitter — it splits on \n\n → \n → . → space before hard character count. It is structure-blind: an "Article 12.3 — Exclusions" heading and its body may be split across chunks, losing the structural context that makes metadata-filtered retrieval possible.

The solution: scripts/rag-ingest.py (minicloud-ansible commit b9b71e9) — a standalone Python script that:

1. Routes by file type — PDF → Docling HTTP API, Office formats → MarkItDown (local), plain text/MD → direct read
2. Splits at structural boundaries — French insurance heading regex (Article|Annexe|Chapitre|Section|Garantie|Disposition), with a 2,000-char fallback paragraph split for large sections
3. Attaches rich metadata — document_type, article (extracted reference), section (full heading), source (document name)
4. Embeds via bge-m3 — POST to Ollama /api/embeddings for each chunk
5. Inserts directly into ragdb — document_chunk table with ON CONFLICT DO NOTHING

File type routing:

Format	Converter	Why
.pdf	Docling HTTP (5001)	Layout analysis, multi-column, scanned OCR — Docling excels at PDFs
.docx, .doc, .pptx, .ppt, .xlsx, .xls, .html	MarkItDown (local)	Native format parsing — Word styles, Excel tables, PowerPoint slides, HTML structure
.txt, .md	Direct read	No conversion needed

Metadata example:

{
  "document_type": "policy",
  "article": "Article 12",
  "section": "Article 12 — Exclusions de garantie",
  "source": "Contrat RC Pro 2026",
  "doc_type": "policy"
}

Usage:

# Prerequisites: port-forwards active, PG_PASS exported
kubectl port-forward -n ai svc/docling    5001:5001 &
kubectl port-forward -n ai svc/ollama     11434:11434 &
kubectl port-forward -n ai postgresql-ai-0 5432:5432 &

PG_PASS=$(kubectl get secret -n ai ai-postgresql-secret \
  -o jsonpath='{.data.password}' | base64 -d)

pip install requests psycopg2-binary markitdown

# Get the Knowledge Base UUID from Open WebUI:
# Workspace → Knowledge → New Knowledge Base → copy UUID from browser URL

python3 ~/Developer/cloudplateform/minicloud-ansible/scripts/rag-ingest.py \
  --file contrat_rc_pro.pdf \
  --collection <open-webui-kb-uuid> \
  --doc-type policy \
  --source "Contrat RC Pro 2026" \
  --pg-pass "$PG_PASS"

---

Phase G — markitdown-proxy + rag-ingest cluster services ✅ Live (2026-07-07)

The problem with Phase F.2 (rag-ingest.py on the Mac): the CLI script required local markitdown and psycopg2 installs, three active port-forwards, and a Mac with Python. It was not repeatable from any other machine and had duplicate routing logic vs the paperclip path.

Phase G replaces it with two always-on cluster services:

markitdown-proxy — unified document converter

A FastAPI service that mimics the Docling API (POST /v1/convert/file) so Open WebUI needs no config changes — it still uses CONTENT_EXTRACTION_ENGINE=docling but DOCLING_SERVER_URL now points at the proxy:

Client → markitdown-proxy:8000/v1/convert/file
    .pdf / images  → proxied to Docling:5001 (OCR + layout)
    .docx .xlsx
    .pptx .html    → converted locally via MarkItDown (mammoth, openpyxl, python-pptx)
    .txt .md       → converted locally via MarkItDown
    → {"document": {"md_content": "..."}}

Both the paperclip path (Open WebUI → proxy) and the ingest service (rag-ingest → proxy) use this same converter. One conversion implementation, two callers.

Image: harbor.10.0.0.200.nip.io/library/markitdown-proxy:fd8b5f7-amd64 (~300 MB)
Source: minicloud-ansible/docker/markitdown-proxy/
Config: DOCLING_URL=http://docling.ai.svc.cluster.local:5001

rag-ingest — structure-aware ingestion service

A FastAPI service that ingests a document into ragdb with full structural metadata:

POST /ingest
  file        — multipart upload (PDF/DOCX/XLSX/PPTX/HTML/MD/TXT)
  collection  — Open WebUI Knowledge Base UUID
  source      — document name ("Contrat RC Pro 2026")
  doc_type    — policy | endorsement | annexe | regulatory | tariff | internal

Response:
  {"status": "ok", "chunks_stored": 42, "collection": "...", "metadata_sample": {...}}

Pipeline inside the pod (all in-cluster, no port-forwards):

1. Send file to markitdown-proxy → markdown
2. Split at French heading boundaries (Article/Annexe/Chapitre/Section regex)
3. Fallback paragraph split for sections exceeding 2,000 chars
4. Attach metadata: document_type, article (extracted), section, source
5. Embed each chunk via ollama.ai.svc:11434 with bge-m3
6. INSERT into ragdb.document_chunk via postgresql-ai.ai.svc:5432

Image: harbor.10.0.0.200.nip.io/library/rag-ingest:58e9bc1-amd64 (~100 MB)
Source: minicloud-ansible/docker/rag-ingest/
Dependencies (injected via env): PROXY_URL, OLLAMA_URL, PG_HOST, PG_PASSWORD (from ai-postgresql-secret)

NetworkPolicy: both services are in the ai namespace and covered by the existing allow-same-namespace ingress/egress policy — no new NetworkPolicy rules needed.

How to ingest a document

# One-time port-forward (only needed for external access from Mac)
kubectl port-forward -n ai svc/rag-ingest 8001:8001 &

# Get your Knowledge Base UUID from Open WebUI:
# Workspace → Knowledge → New Knowledge Base → copy UUID from browser URL

curl -X POST http://localhost:8001/ingest \
  -F "file=@contrat_rc_pro.pdf" \
  -F "collection=<open-webui-kb-uuid>" \
  -F "doc_type=policy" \
  -F "source=Contrat RC Pro 2026"

The inserted chunks are immediately queryable from Open WebUI — the hybrid search queries ragdb.document_chunk by collection_name, so chunks appear alongside those uploaded via the paperclip.

From another pod inside the cluster (no port-forward):

curl -X POST http://rag-ingest.ai.svc.cluster.local:8001/ingest \
  -F "file=@/data/contrat.pdf" \
  -F "collection=<uuid>" \
  -F "doc_type=policy" \
  -F "source=Contrat RC Pro 2026"

---

Implementation

Step 1 — Create the database

# Connect as postgres superuser to create DB and enable extension
PG_SUPER=$(kubectl get secret -n ai ai-postgresql-secret \
  -o jsonpath='{.data.postgres-password}' | base64 -d)

kubectl exec -n ai postgresql-ai-0 -- \
  env PGPASSWORD="$PG_SUPER" psql -U postgres -c "
    CREATE DATABASE ragdb;
    GRANT ALL PRIVILEGES ON DATABASE ragdb TO aiplatform;
  "

kubectl exec -n ai postgresql-ai-0 -- \
  env PGPASSWORD="$PG_SUPER" psql -U postgres -d ragdb -c "
    CREATE EXTENSION IF NOT EXISTS vector;
    GRANT ALL ON SCHEMA public TO aiplatform;
  "

Step 2 — Pull bge-m3 on both Ollama instances

# Temporary egress NetworkPolicy for the Ollama pod (ai namespace has default-deny-egress)
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: temp-allow-ollama-pull
  namespace: ai
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: ollama
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except: [10.0.0.0/8, 192.168.0.0/16]
      ports:
        - port: 443
          protocol: TCP
EOF

kubectl port-forward -n ai svc/ollama 11434:11434 &
PF1=$!
sleep 3
curl -s -X POST http://localhost:11434/api/pull \
  -d '{"model":"bge-m3","stream":false}'
kill $PF1

# Repeat for secondary (bge-m3 is 1.2 GB — allow enough timeout)
kubectl port-forward -n ai svc/ollama-secondary 11435:11434 &
PF2=$!
sleep 3
curl -s -X POST http://localhost:11435/api/pull \
  -d '{"model":"bge-m3","stream":false}'
kill $PF2

kubectl delete networkpolicy temp-allow-ollama-pull -n ai

# Verify dimension (should print dim=1024)
kubectl port-forward -n ai svc/ollama 11434:11434 &
sleep 2
curl -s http://localhost:11434/api/embeddings \
  -d '{"model":"bge-m3","prompt":"test"}' \
  | python3 -c "import sys,json; e=json.load(sys.stdin)['embedding']; print(f'dim={len(e)}')"
kill %1

Step 3 — Update Open WebUI Helm values

Add to open-webui-values.yaml in the extraEnvVars section (before SSL_CERT_FILE):

# ── RAG / Vector Search (pgvector) ──────────────────────────────────
- name: VECTOR_DB
  value: "pgvector"
- name: PGVECTOR_DB_URL
  value: "postgresql://aiplatform:$(PG_PASSWORD)@postgresql-ai.ai.svc.cluster.local:5432/ragdb"
- name: VECTOR_LENGTH
  value: "1024"
- name: PGVECTOR_INITIALIZE_MAX_VECTOR_LENGTH
  value: "1024"
- name: PGVECTOR_INDEX_METHOD
  value: "hnsw"                  # requires custom no-AVX512 image — see pgvector SIGILL fix below
- name: PGVECTOR_HNSW_M
  value: "16"
- name: PGVECTOR_HNSW_EF_CONSTRUCTION
  value: "64"
- name: RAG_EMBEDDING_ENGINE
  value: "ollama"                # bypass LiteLLM — keep embeddings out of cost tracking
- name: RAG_EMBEDDING_MODEL
  value: "bge-m3"                # 1024-dim, 100+ languages
- name: RAG_OLLAMA_BASE_URL
  value: "http://ollama.ai.svc.cluster.local:11434"
- name: CHUNK_SIZE
  value: "500"
- name: CHUNK_OVERLAP
  value: "100"
- name: RAG_TOP_K
  value: "10"                    # wider retrieval net; re-ranker filters to TOP_K_RERANKER=3
- name: ENABLE_RAG_HYBRID_SEARCH
  value: "true"

scp ~/Developer/cloudplateform/minicloud-ansible/helm-values/open-webui-values.yaml \
  controller:/home/ktayl/minicloud-ktaylorganisation/ansible/helm-values/

ssh controller "helm upgrade open-webui open-webui/open-webui \
  --namespace ai \
  --values /home/ktayl/minicloud-ktaylorganisation/ansible/helm-values/open-webui-values.yaml \
  --timeout 4m --wait"

Open WebUI creates the document_chunk table and HNSW index automatically on first start.

Step 4 — Add bge-m3 to LiteLLM ConfigMap

Add to manifests/ai/00-litellm-configmap.yaml:

- model_name: bge-m3
  litellm_params:
    model: ollama/bge-m3
    api_base: http://ollama.ai.svc.cluster.local:11434
- model_name: bge-m3
  litellm_params:
    model: ollama/bge-m3
    api_base: http://ollama-secondary.ai.svc.cluster.local:11434

Update department key allowlists in LiteLLM PostgreSQL:

DB_URL=$(kubectl get secret -n ai litellm-credentials \
  -o jsonpath='{.data.database-url}' | base64 -d)
kubectl exec -n ai postgresql-ai-0 -- psql "$DB_URL" -c "
  UPDATE \"LiteLLM_VerificationToken\"
  SET models = array_append(models, 'bge-m3')
  WHERE 'bge-m3' != ALL(models)
    AND key_alias NOT IN ('direction-operations','direction-rh','direction-services-generaux','direction-sinistres');
"

After merging ConfigMap to git (ArgoCD syncs), restart LiteLLM:

kubectl rollout restart deployment/litellm -n ai

Verification

Embedding chain

# 1. bge-m3 via Ollama (direct) — Open WebUI uses this path
kubectl port-forward -n ai svc/ollama 11434:11434 &
curl -s -X POST http://localhost:11434/api/embeddings \
  -d '{"model":"bge-m3","prompt":"test sentence"}' | \
  python3 -c 'import sys,json; e=json.load(sys.stdin)["embedding"]; print(f"dims={len(e)}")'
# Expected: dims=1024
kill %1

# 2. bge-m3 via LiteLLM (API users)
MASTER=$(kubectl get secret -n ai litellm-credentials \
  -o jsonpath='{.data.master-key}' | base64 -d)
kubectl port-forward -n ai svc/litellm 4000:4000 &
curl -s -X POST http://localhost:4000/v1/embeddings \
  -H "Authorization: Bearer $MASTER" \
  -H "Content-Type: application/json" \
  -d '{"model":"bge-m3","input":"test"}' | \
  python3 -c 'import sys,json; r=json.load(sys.stdin); print(f"dims={len(r[\"data\"][0][\"embedding\"])}")'
# Expected: dims=1024

pgvector table

PG_PASS=$(kubectl get secret -n ai ai-postgresql-secret \
  -o jsonpath='{.data.password}' | base64 -d)
kubectl exec -n ai postgresql-ai-0 -- \
  env PGPASSWORD="$PG_PASS" psql -U aiplatform -d ragdb \
  -c "\d document_chunk"
# Expected: vector(1024) column, HNSW index, GIN index with 'french' dictionary

End-to-end RAG test

1. Go to https://chat.devandre.sbs
2. Login via Authentik (Authentik credentials or demo.it/data/sinistres @devandre.sbs)
3. Start a new chat
4. Upload any PDF or text file via the paperclip icon
5. Ask: "What does this document say about X?" — Open WebUI retrieves top 5 matching chunks and generates a grounded answer

Operational notes

Adding more documents to the knowledge base

Option A — Open WebUI paperclip (simple): upload via UI. Open WebUI sends the file to markitdown-proxy, chunks at 500 chars, embeds with bge-m3. No metadata attached.

Option B — rag-ingest service (recommended for structured insurance documents): POST /ingest with the file and metadata. Splits at Article/Section boundaries, attaches document_type/article/section/source, inserts directly into ragdb. See Phase G above.

For direct embedding via LiteLLM API:

import openai

client = openai.OpenAI(
    base_url="http://localhost:4000",
    api_key="sk-direction-it-..."
)

response = client.embeddings.create(
    model="bge-m3",
    input="Your document text here"
)
vector = response.data[0].embedding  # 1024-dim float list
# INSERT INTO ragdb.document_chunk (id, vector, collection_name, text, vmetadata) VALUES (...)

pgvector AVX-512 SIGILL fix (critical for ThinkPad i7-8565U / i7-10510U)

Symptom: PostgreSQL backend processes terminate with signal 4: Illegal Instruction whenever a document INSERT triggers HNSW or IVFFlat index maintenance. Open WebUI shows "server closed the connection unexpectedly". RAG document uploads fail silently.

Root cause: pgvector 0.8.4 in the upstream Bitnami postgresql image contains EVEX-encoded AVX-512 instructions in vector.so (in the HNSW graph traversal and IVFFlat K-means code paths). All four cluster CPUs (i7-8565U Whiskey Lake, i7-10510U Comet Lake) lack AVX-512 support. Sequential-scan vector distance queries are not affected — only index maintenance triggers the SIGILL.

Diagnosis:

# Confirm SIGILL in PostgreSQL pod logs
kubectl logs -n ai postgresql-ai-0 | grep -i "signal 4\|Illegal instruction\|terminated by signal"
# Expected: "client backend (PID ...) was terminated by signal 4: Illegal instruction"

Fix — custom pgvector build:

The fix is a custom Docker image that rebuilds vector.so from pgvector 0.8.4 source with AVX-512 disabled:

# Dockerfile at: minicloud-ansible/docker/postgresql-noavx512/Dockerfile
# Build and push (on controller):
docker build -t harbor.10.0.0.200.nip.io/library/postgresql:18.4.0-noavx512 \
  /home/ktayl/docker-builds/pgvector/
docker save harbor.10.0.0.200.nip.io/library/postgresql:18.4.0-noavx512 \
  -o /tmp/pgvec-noavx512.tar
crane push --insecure /tmp/pgvec-noavx512.tar \
  harbor.10.0.0.200.nip.io/library/postgresql:18.4.0-noavx512

Key Dockerfile excerpt:

RUN cd /tmp/pgvector-0.8.4 && \
    make OPTFLAGS="-mno-avx512f -mno-avx512bw -mno-avx512vl -mno-avx512dq" \
         PG_CONFIG=/opt/bitnami/postgresql/bin/pg_config && \
    cp vector.so /tmp/vector-nosimd.so

The deployed image is harbor.10.0.0.200.nip.io/library/postgresql:18.4.0-noavx512 (set in postgresql-ai-values.yaml, minicloud-ansible commit 9f2c2ab). This image is required — any upgrade to a newer Bitnami postgresql image must be tested for AVX-512 usage before deploying.

After rebuilding image, if the HNSW index was corrupted during a failed INSERT session:

PG_PASS=$(kubectl get secret -n ai ai-postgresql-secret \
  -o jsonpath='{.data.password}' | base64 -d)

kubectl scale statefulset open-webui -n ai --replicas=0

kubectl exec -n ai postgresql-ai-0 -- \
  env PGPASSWORD="$PG_PASS" psql -U aiplatform -d ragdb \
  -c "DROP TABLE IF EXISTS document_chunk;"

kubectl scale statefulset open-webui -n ai --replicas=1
# Open WebUI recreates the table and HNSW index cleanly on startup

If pgvector extension is missing after a DB rebuild

PG_SUPER=$(kubectl get secret -n ai ai-postgresql-secret \
  -o jsonpath='{.data.postgres-password}' | base64 -d)
kubectl exec -n ai postgresql-ai-0 -- \
  env PGPASSWORD="$PG_SUPER" psql -U postgres -d ragdb \
  -c "CREATE EXTENSION IF NOT EXISTS vector; GRANT ALL ON SCHEMA public TO aiplatform;"

---

Part 2 — Web Search (SearXNG)

Implemented: 2026-07-05

SearXNG is a self-hosted privacy-preserving meta-search engine. It queries Google, Bing, DuckDuckGo, and Wikipedia simultaneously and returns aggregated results without tracking users.

Why SearXNG over hosted search APIs

Option	Self-hosted	Cost	Rate limit	Privacy
SearXNG	✅	Free	None	On-cluster
Brave Search	❌	$0 / 2k/mo	2,000 req/mo	External
Tavily	❌	$0 / 1k/mo	1,000 req/mo	External
DuckDuckGo	❌	Free	Unreliable	External

Component

Resource	Namespace	Image	Service
searxng Deployment	searxng	docker.io/searxng/searxng:2026.7.3-747cec4c2	ClusterIP port 8080

NetworkPolicy design

The searxng namespace has the tightest egress policy on the cluster:

default-deny-ingress + default-deny-egress
    ↓
allow-from-open-webui   — ingress from ai ns, port 8080 only
allow-dns-egress        — UDP/TCP 53 to kube-system (CoreDNS)
allow-internet-egress   — TCP 443/80 to 0.0.0.0/0 (only searxng pod)

Open WebUI (in ai ns) gets its own internet egress policy (allow-open-webui-internet-egress) to fetch full page content from result URLs.

Configuration (open-webui-values.yaml)

- name: ENABLE_WEB_SEARCH
  value: "true"
- name: WEB_SEARCH_ENGINE
  value: "searxng"
- name: SEARXNG_QUERY_URL
  value: "http://searxng.searxng.svc.cluster.local:8080/search?q=<query>&format=json&language=auto"
- name: WEB_SEARCH_RESULT_COUNT
  value: "5"
- name: WEB_SEARCH_CONCURRENT_REQUESTS
  value: "10"

Gotcha: Open WebUI config.py reads ENABLE_WEB_SEARCH and WEB_SEARCH_ENGINE — not the ENABLE_RAG_* prefixed variants. Using the wrong names silently has no effect (web search appears configured in the UI settings but does not actually activate).

Deployment (GitOps)

All resources in minicloud-gitops/manifests/searxng/. ArgoCD Application searxng auto-syncs.

# Check SearXNG status
kubectl get pods -n searxng
kubectl logs -n searxng -l app=searxng --tail=20

# Verify it returns results
kubectl port-forward -n searxng svc/searxng 8181:8080 &
curl -s 'http://localhost:8181/search?q=kubernetes+latest&format=json' | \
  python3 -c 'import sys,json; r=json.load(sys.stdin); print("results:", len(r.get("results",[])))'

# Verify Open WebUI can reach it
kubectl exec -n ai open-webui-0 -- python3 -c "
import urllib.request, json
r = json.loads(urllib.request.urlopen(
    'http://searxng.searxng.svc.cluster.local:8080/search?q=test&format=json'
).read())
print('ok, results:', len(r.get('results',[])))
"

How to use web search in Open WebUI

1. Go to https://chat.devandre.sbs
2. Start a new chat
3. Click the globe icon in the toolbar to enable web search
4. Ask any question — Open WebUI queries SearXNG, retrieves real-time results, and injects them into the LLM context
5. The answer will cite the sources

Example queries that benefit from web search:

• "What's new in Kubernetes 1.36?"
• "Latest CVEs affecting containerd"
• "Current EUR/USD exchange rate"
• "Changelog for Helm 3.17"

gotcha: enableServiceLinks must be false

Kubernetes automatically injects SEARXNG_PORT=tcp://<cluster-ip>:8080 into all pods in the searxng namespace when a service named searxng exists. SearXNG uses SEARXNG_PORT as its listen port — the injected value is not a valid integer and crashes granian on startup.

Fix: enableServiceLinks: false in the pod spec. This disables k8s service env var injection for the pod.

gotcha: use ENABLE_WEB_SEARCH, not ENABLE_RAG_WEB_SEARCH

Open WebUI config.py reads ENABLE_WEB_SEARCH and WEB_SEARCH_ENGINE. The ENABLE_RAG_* prefixed variants do not exist and are silently ignored — the globe icon may appear in the UI but web search will not trigger.

Wrong (silent no-op)	Correct
ENABLE_RAG_WEB_SEARCH=true	ENABLE_WEB_SEARCH=true
RAG_WEB_SEARCH_ENGINE=searxng	WEB_SEARCH_ENGINE=searxng
RAG_WEB_SEARCH_RESULT_COUNT=5	WEB_SEARCH_RESULT_COUNT=5
RAG_WEB_SEARCH_CONCURRENT_REQUESTS=10	WEB_SEARCH_CONCURRENT_REQUESTS=10

gotcha: BYPASS_WEB_SEARCH_WEB_LOADER must be true

By default, Open WebUI fetches the full HTML content of each search result URL to give the LLM richer context. Most news and sports sites (FIFA.com, BBC Sport, Le Monde, ESPN) block scraping — the content loader gets an empty response or a captcha page. With no content extracted, the LLM responds "Aucune source trouvée" even though SearXNG returned valid results.

BYPASS_WEB_SEARCH_WEB_LOADER=true skips the content-fetching step and uses SearXNG's snippets directly (title + URL + excerpt). The snippets are sufficient for the LLM to answer factual questions like match results, news headlines and exchange rates.

- name: BYPASS_WEB_SEARCH_WEB_LOADER
  value: "true"

Scaling and replacement

SearXNG is a stateless meta-search proxy — it holds no data. If you need more throughput, scale replicas:

kubectl scale deployment searxng -n searxng --replicas=3

The allow-from-open-webui NetworkPolicy uses a namespace selector, not a pod selector — all replicas receive traffic automatically.