A Cheap, Safe, High-Precision RAG Pipeline: Architecture Notes

How we wired the AI layer for opexmx, a maintenance CMMS (Corrective + Preventive maintenance, asset management, knowledge base). The goal: a copilot + retrieval pipeline that is cheap per request, safe against prompt injection, and precise at retrieval — without paying for a frontier model on every turn.

Everything runs on Cloudflare Workers AI (via an OpenAI-compatible AI Gateway). Same model ids apply on cloud and on-prem.

---

TL;DR — the pipeline

flowchart TD
    U([user message]) --> R1["1. Regex guard — free, catches ~80–90% of prompt injection"]
    R1 -->|clean / uncertain| G["2. Granite classifier — intent + safety + routing, JSON-only (only if uncertain)"]
    G -->|allow| E["3. Qwen3-Embedding 0.6B — query to vector, 768-dim (Matryoshka from 1024)"]
    E --> V["4. Vector search — over-fetch top-K x 3, Vectorize / pgvector (recall)"]
    V --> B["5. BGE reranker base — cross-encoder re-scores, keep top-K (precision)"]
    B --> Q["6. Qwen3-30B-A3B (MoE) + tool-calling agent — one generation call, best context. Manager Mode to Llama-3.3-70B"]

The expensive model (30B/70B) runs once, only after the request is validated and the best context is gathered. Most of the per-request spend is a tiny classifier + a cheap rerank, not generation.

---

Why each layer exists

1. Guard — regex first, then a cheap classifier

A copilot that ingests free user text needs a prompt-injection defense. We run two layers, cheapest first:

• Layer 1 — regex (free). A phrase list: ignore previous instructions, reveal your system prompt, DAN, jailbreak, developer message:, etc. Catches the overwhelming majority of attacks at $0. Phrase-based, not bare keywords — api key / token alone are too common in real maintenance text, so they're left to the classifier.
• Layer 2 — LLM classifier (only when Layer 1 is clean and the message isn't trivial chitchat). A small model returns strict JSON:

  { "intent": "maintenance", "safe": true, "requires_rag": true,
    "requires_tools": false, "confidence": 0.9 }
  

  This does three jobs in one cheap call: safety gate, intent classification, and a routing hint. The hint is surfaced to the UI; the agent's tool-calling remains the real router (no agent rewrite).

Failure policy: fail-open. If the classifier errors or is unconfigured, the message is allowed — Layer 1 regex still blocks the obvious cases. Chat never hard-breaks on a guard outage.

2. Embeddings — Qwen3-Embedding-0.6B at 768 dims

We index manuals + knowledge-center docs (PDF, rich-text, image captions) into chunks and embed them. We picked @cf/qwen/qwen3-embedding-0.6b for quality at low cost.

The dimension gotcha (this bit us): Qwen3-Embedding-0.6B is natively 1024-dim, but our vector store (a Cloudflare Vectorize index + a pgvector column) was already pinned to 768 from the previous model (bge-base-en-v1.5). Mixed dimensions are illegal in one index. Rather than migrate the whole store to 1024 (new index + new column + full reindex), we exploit Matryoshka representation learning: Qwen3 supports output truncation across 32–1024 dims, so we request dimensions: 768 on the /embeddings call and get 768-dim vectors that stay compatible with the existing store.

⚠️ The OpenAI dimensions field must be forwarded by the gateway to the model. If your gateway strips it, you silently get 1024 and inserts fail. Verify with:

curl -s $GW/embeddings -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"@cf/qwen/qwen3-embedding-0.6b","input":["test"],"dimensions":768}' \
  | jq '.data[0].embedding | length'   # expect 768

Also: only send dimensions for Matryoshka-capable models. bge-base is fixed-768 and 400s on the field.

3. Retrieval — vector search (recall)

Embed the query, search the index, over-fetch (e.g. top-K×3) with filter fan-out (global docs + asset-scoped + equipment-type-scoped, merged by best score per chunk). Embeddings are recall-optimized — they surface the right neighborhood, not the exact right chunk.

4. Reranker — BGE-Reranker-Base (precision)

A cross-encoder is not another chatbot. Its only job: "of these retrieved chunks, which are actually most relevant?" It takes (query, passage) pairs and returns a relevance score, then we re-sort and keep the top-K.

This is the single biggest retrieval-quality lever for ~zero cost:

chunk	embedding sim	reranker score
Pump manual	0.84	0.99
Pump troubleshooting	0.82	0.95
Bearing spec	0.81	0.91
Conveyor SOP (off-topic)	0.80	0.20

Fails open: any reranker error → keep embedding-similarity order. Retrieval never breaks.

5. Generation — Qwen3-30B-A3B + tool-calling

A 30B Mixture-of-Experts (only ~3B active per token) gives large-model quality at small-model latency/cost, with native function calling for the agent loop. Tools: search_assets, get_asset_status, search_documents, search_problems, create_ticket, PM/parts/metrics, etc. The model routes itself.

Manager Mode escalates only the synthesis step to Llama-3.3-70B (chatModelPro) when the user asks for a cross-domain review. Tool rounds stay on the cheap base model; only the final briefing pays for the big model.

6. Autocomplete / fill-with-AI — a separate fast lane

Inline ghost-text autocomplete and "fill this field" need low latency, not reasoning. They run on a small fast model (chatModelFast, Llama-3.1-8B-instruct-fp8-fast) and fall back to the chat model if it returns nothing.

⚠️ Gotcha: if your provider is an OpenAI-compatible gateway, make sure it honors distinct per-role model ids. A common bug: the gateway collapses every @cf/ id to a single configured chat model, so your "fast model" setting is silently ignored and every autocomplete pays for the 30B. Pass the requested id through when the gateway speaks the same model namespace; only collapse @cf/→local-name on a single-model local endpoint (Ollama/mlx).

⚠️ Gotcha #2: a small reasoning model (e.g. a hybrid MoE micro) is the wrong pick for generation tasks like field-drafting. With thinking disabled it may emit nothing. Use it for classification; use a plain instruct model for autocomplete/fill.

---

Models & cost (Cloudflare Workers AI, ~$0.011 / 1k Neurons)

Role	Model	Notes
Chat / agent	@cf/qwen/qwen3-30b-a3b-fp8	cheap MoE, multilingual, tool-calling
Pro / Manager	@cf/meta/llama-3.3-70b-…-fp8-fast	synthesis step only
Autocomplete	@cf/meta/llama-3.1-8b-…-fp8-fast	separate fast lane
Embeddings	@cf/qwen/qwen3-embedding-0.6b	768-dim (Matryoshka-truncated)
Reranker	@cf/baai/bge-reranker-base	cross-encoder, post-retrieval
Guard/classifier	@cf/ibm-granite/granite-4.0-h-micro	intent + prompt-injection, JSON-only

Headline cost: ~$0.045 / light user / month, ~$0.27 / heavy user. The guard + reranker add negligible spend relative to the one 30B generation per turn.

---

Design principles

1. Cheapest-first. Regex before LLM; small classifier before 30B; embedding+rerank before generation. Pay for the big model once, last.
2. Fail-open on the safety/classifier path; fail-open on the rerank path. Neither must break chat or retrieval. The regex still blocks the obvious injection even if the LLM guard is down.
3. One cheap model, three jobs. The guard returns safety + intent + routing hint in one JSON call instead of three separate models.
4. Dimension discipline. Pick one embedding dimension and pin the whole store to it. When swapping models, either match the dimension or use Matryoshka truncation — then reindex so every vector lives in the same space.
5. Recall at retrieval, precision at rerank, quality at generation. Don't ask one stage to do another's job.

---

Lessons we paid for

• Verify live, don't trust the docs page. A doc/search source claimed Qwen3-Embedding was 768-dim "probably." The live gateway returned 1024 and inserts failed. Always curl the endpoint and check .data[0].embedding | length.
• Gateways must forward dimensions. Without passthrough, Matryoshka truncation is invisible and you get the native dim silently.
• Per-role model ids must be honored. A gateway that rewrites all @cf/ ids to one chat model silently defeats your fast/pro model settings (and Manager-Mode escalation).
• Small ≠ always cheap. A micro reasoning model on a complex generation prompt can return empty and waste seconds before falling back. Match the model class to the task.

---

Reproducing this shape

If you're building something similar, the minimal pieces:

• a guard module: regex list + a JSON-classifier call (fail-open);
• an embed step pinned to one dimension (Matryoshka-truncate if needed);
• a retrieve step that over-fetches;
• a rerank step (cross-encoder) that re-scores and trims (fail-open);
• a generate step on one good model, with a pro-model escalation path;
• a fast lane (small instruct model) for autocomplete/fill, with chat-model fallback.

The whole thing fits in a request budget where the dominant cost is the single generation call — which is exactly what you want.

Talk to us about deploying a RAG copilot on your maintenance data →