Production RAG
Freshness and re-indexing, document lifecycle, per-user ACLs, cost, latency, failure modes and degradation.
Prerequisites
- The RAG Pipeline — the thing being productionized.
- Ingestion & Chunking — the metadata designed there is what makes ACL filtering possible here.
- RAG Evaluation — you cannot operate what you cannot measure; this page assumes that harness exists.
The intuition
A backend engineer who has never touched RAG already knows how caches fail. They go stale. Deletes must propagate. A miss needs a defined behaviour. Serving cached data to the wrong user is an incident. Production RAG is those problems with vectors:
Key insight
The index is a cache of your corpus — not a source of truth. Everything hard about production RAG is a cache problem: invalidation, staleness, deletes, permission-aware reads, and what to serve when the cache is empty or the backing store is down.
A demo is a script with one happy path. Production is a data platform (ingestion, lifecycle, ACL tags) plus a serving path (filtered retrieve, latency budget, degradation). That gap is this page.
Why it exists
Prototypes ignore four things that are not optional with real users.
The corpus moves. A one-shot index is correct until someone edits a document. Then the bot quotes a rescinded policy — and nothing reports an error, because every component is healthy on its local inputs.
Not everyone may see everything. With two permission classes, retrieval becomes a per-caller authorization decision. Retrieved text enters the prompt, so a retrieval bug is a data-leak bug, not a quality bug.
Every stage can fail. Vector DB down, reranker timeout, provider rate-limit, empty retrieval — each needs a designed behaviour. The default (fluent answer from parametric memory) is the worst one.
It bills per token, per query, forever. Prototype cost is noise. At tens of QPS the same architecture is a budget line you will defend.
The core idea
Production RAG is the pipeline you already know, operated as infrastructure. Online is still retrieve → rerank → generate. What changes is everything around it: continuous ingestion, full document lifecycle (updates and deletes), entitlement filters on every retrieve, a latency budget, cost levers on the LLM tokens that dominate the bill, a degradation ladder when stages die, and request tracing wired into the eval harness.
Name the decision points, not the tools: how you keep the cache correct, who may see a chunk, what you do when evidence is missing, and how you know before users do.
How it actually works
Freshness and re-indexing
Full re-index is O(corpus) in embedding cost — reserve it for chunking or embedding-model changes. Steady state is incremental: change event → parse → content-hash compare → re-chunk only changed docs → embed → atomic swap of that doc's chunks. Prefer event-driven ingest (webhooks, CDC) plus a periodic reconciliation crawl for missed events.
Embedding-model upgrades force a full re-embed: vectors from different models live in
incompatible spaces and must never share one index. Build blue/green, run the golden set, flip a
serving alias, keep the old index warm for rollback. Deterministic chunk IDs (doc_id, chunk_seq)
make swaps and deletes exact.
Document lifecycle
On update, write the new chunk set, then delete every chunk of that doc_id. When a document
shrinks (v2 has 8 chunks, v1 had 12), upsert-only leaves orphans — still embedded, still
retrievable, still cited as current. That silent index drift is the classic "we deleted that
policy but the bot still quotes it" bug.
Deletes must hide fast. For GDPR erasure, a tombstone filter excludes chunks immediately while physical deletion and compaction catch up. Invalidate any answer cache that cited the deleted doc.
Common misconception
"Upsert handles updates." Upsert handles chunks that still exist. When a document shrinks
from 12 chunks to 8, upserting the 8 new ones leaves chunks 9–12 of the old version fully
retrievable. Delete by doc_id, then write — never rely on ID overlap.
Per-user ACLs
Permissions are enforced at retrieval, as filters — never in the prompt. At ingest, resolve
each document's readers into tags (allowed_groups, tenant_id) on every chunk. At query time,
resolve the caller's groups into a mandatory metadata filter ANDed into the vector and BM25 search.
Modern stores support filtered ANN natively — pre-filter or filter-during-traversal, not post-filter after top-k (post-filter empties candidates when most hits are unauthorized). Prompt text like "only discuss documents the user may see" is not a control: anything in context can be extracted by injection or ordinary helpfulness. If it never enters the context window, it cannot leak. (Mechanics: filtering and metadata.)
Two sub-problems bite hard. Permission changes must re-tag existing chunks — subscribe to ACL-change events, not only content events. Group explosion (thousands of groups per user) may need a permissions service that resolves identity to a compact filter at query time. Multi-tenant SaaS: tenant filter is the floor; namespace- or collection-per-tenant isolates blast radius.
Latency budget
Work backwards from a chat SLO (illustrative: ~2–3 s to first token). Stream so perceived latency is TTFT (time-to-first-token), not full completion.
| Stage | Typical | Notes |
|---|---|---|
| Query rewrite (optional) | 100–500 ms | Small fast model; skip when confidence is high |
| Query embedding | 10–50 ms | Cacheable for repeated queries |
| Vector + BM25 search | 10–100 ms | Parallel arms; fuse with RRF |
| Rerank ~50 candidates | 50–300 ms | Cross-encoder; often best quality-per-ms buy |
| LLM TTFT | 500–2000 ms | Grows with prompt size — fewer, better chunks help |
Measure p95 per stage. The tail is where SLOs die.
Cost levers
Cost per query ≈ embeddings + vector reads + rerank + LLM tokens. LLM tokens dominate. Ranked levers: (1) prompt-size discipline — rerank hard, send ~3–5 chunks not 20; (2) prompt caching — static system prompt and stable prefixes first so only the variable tail bills full price; (3) model routing — small model for easy queries, large for hard ones; (4) semantic response cache — serve a prior answer when query embedding similarity exceeds a conservative threshold (FAQ-heavy traffic hits hard; false positives are wrong answers — sample-audit hits and key by tenant); (5) embedding cost is one-time per corpus change — usually noise next to generation.
Failure modes and degradation
| Failure | Detection | Designed behaviour |
|---|---|---|
| Empty / low-score retrieval | top score below threshold, or empty after ACL filter | Honest "no relevant docs" + escalation — never freestyle from parametric memory |
| Garbage retrieval | relevance grader; online faithfulness sampling | Gate generation; one rewrite retry; then refuse |
| Stale docs served | freshness lag (source updated_at vs index time) | Timestamps on citations; alert on lag SLO |
| Vector DB down | health checks, timeouts | BM25/keyword fallback or cache hits; circuit-break |
| Reranker down | timeouts | Skip stage; serve stage-1 order |
| LLM down / rate-limited | 429/5xx | Failover model/provider; reduced-capacity mode |
| Prompt overflow | token count pre-call | Drop lowest-ranked chunks; trim history |
Principle: every stage after hybrid retrieval is skippable; generation without evidence is refusable. Quality degrades along a gradient instead of failing binary.
The flows
| Flow | Sequence | When | What breaks it |
|---|---|---|---|
| Incremental lifecycle | change event (or crawl) → content-hash → re-process if changed → write new chunks → delete all old by doc_id → invalidate caches; deletes: tombstone + physical reclaim | Steady-state churn; GDPR erasure | Missed delete events; upsert without delete-by-doc; no freshness-lag metric; mixed embedding models in one index |
| ACL-filtered serve | resolve groups → mandatory filter on hybrid retrieve → score threshold → rerank → generate + cite or refuse | Every multi-permission query | Post-filter only; unfiltered code path; prompt-level "don't reveal X"; ACL changes not re-tagging chunks |
| Degradation ladder | circuit-break failing stage → skip reranker / BM25 fallback / semantic cache / honest "no grounding" | Dependency down; empty or garbage retrieval | No skip path → freestyle hallucination; binary outage; policy invented during the incident |
A worked example
Two incidents on the governed enterprise platform. Both returned HTTP 200 and a fluent answer. Both are outages by any measure that matters.
Incident 1 — the rescinded policy. An employee asks about remote-work eligibility and gets a
confident answer citing a policy withdrawn six weeks ago. The wiki emitted page.deleted; the
pipeline only subscribed to page.updated. Twelve chunks remained fully retrievable. Retrieval did
its job — those chunks were the best match. Latency was normal. Users liked the answer.
Nothing erred. Only an explicit freshness lag SLO — max(source updated_at − indexed_at) with
an alert — turns this from "a user complained in September" into "an alert fired in July." Fix the
event set, delete by doc_id on delete, and reconcile with a periodic crawl.
Incident 2 — the draft nobody should have seen. A contractor asks about compensation bands and the assistant quotes an unreleased HR draft. Ingest tagged the doc correctly. The main path filtered. A second path — temporary upload into a shared collection — queried without the filter. The system prompt said "only discuss authorised documents."
Two lessons transfer everywhere. First, prompt instructions are not security — the restricted text was already in context. Second, the filter must be structurally unbypassable: a property of the retrieval client (no identity → no retriever), not an argument each call site remembers to pass. One unfiltered path is a breach.
Production concerns
Security beyond ACLs. Retrieved documents are untrusted input — a poisoned page can carry prompt-injection payloads. Treat context as data (delimiters, instruction hierarchy), sanitize at ingest, and never give the chat loop write-capable tools without human confirmation. (security)
Observability minimum. Per-request trace: query → rewritten query → chunk IDs + scores → prompt → answer → feedback, with IDs that join to eval. Dashboards: retrieval-score distribution, refusal rate, freshness lag, p95 per stage, cost per query. Score-distribution shift is early warning for index rot. (observability)
Rollout discipline. Prompt, chunking, embedding model, reranker, and LLM version each pass golden-set eval → shadow or canary → full rollout. Index versions are blue/green so rollback is an alias flip.
Capacity. Embedding-provider rate limits throttle bulk re-indexes. HNSW is largely RAM-resident, so vector memory dominates cost at tens of millions of vectors. Reranker GPU throughput often caps QPS first. Know which of the three saturates. (scaling and operations)
Common drill-downs
How would you productionize a RAG prototype? Six decision points: (1) event-driven incremental ingest with delete-by-doc; (2) ACL tags + mandatory retrieval filters; (3) hybrid retrieve + rerank inside a latency budget, streaming; (4) score thresholds, fallbacks, honest refusal; (5) context discipline, caching, model routing; (6) golden-set CI gate, full tracing, sampled online judging.
A document is updated — walk through what must happen. And deleted?
Update: re-parse → re-chunk → re-embed → write new → delete all old by doc_id → invalidate
caches citing that doc. Delete: tombstone for instant hide, then physical reclaim; for GDPR, verify
reclamation including backups per policy.
Retrieval returns nothing — or garbage. What does the system do? Nothing: threshold → honest refusal + escalation. Garbage: relevance grader gates generation; one rewrite retry; then refuse. Rising refusal/grader rates flag index gaps or drift.
Where does the money go, and how do you cut it? LLM tokens dominate. Cut with fewer better chunks, prompt caching on static prefixes, small-model routing for easy queries, tenant-keyed semantic cache for repeats, and trimmed history.
Set a latency budget for a ~2 s TTFT chat SLO. Reserve ~1–1.5 s for LLM TTFT. Rewrite ≤300 ms (skippable), embed ≤50 ms, hybrid search ≤100 ms, rerank ≤300 ms. Stream. Measure p95 per stage.
Vector DB is down. Does the product die? No. Circuit-break to BM25, serve semantic-cache hits, and if no grounding remains, say so and offer labelled ungrounded help or escalation. Reranker down → skip. LLM down → provider failover. Decide the ladder before the incident.
How do you ship an embedding-model upgrade safely? Full re-embed offline into a second index; golden-set compare (recall@k, MRR); shadow live queries; atomic alias flip; keep old index warm. Budget against provider rate limits.
Test yourself
Why is 'the index is a cache' more than a nice metaphor? Name three consequences it predicts.
A document shrinks from 12 chunks to 8 on update. Your pipeline upserts chunks 1–8. What's live in the index, and what does a user see?
Someone proposes enforcing permissions by instructing the model: 'only discuss documents the user is authorised to see.' Give the two-sentence rebuttal.
Your vector DB is down. Sketch the degradation ladder rather than a single fallback.
You ship a new embedding model by re-embedding 30% of the corpus, testing, then finishing the rest. What breaks?
Go deeper
- Building A Generative AI Platform — Chip Huyen — production architecture: context construction, guardrails, routing, caching, observability.
- Building Performant RAG Applications for Production — LlamaIndex — framework-level techniques for retrieval quality at corpus scale.
- RAG with Access Control — Pinecone / AuthZed — pre-filter vs post-filter authorization against a vector index (SpiceDB / ReBAC).
- Patterns for Building LLM-based Systems & Products — Eugene Yan — caching, guardrails, defensive UX, and feedback loops around the core pipeline.
- Systematically Improving Your RAG — Jason Liu — the operational flywheel: metrics, feedback, and prioritizing by measured failure class.
- Introducing Contextual Retrieval — Anthropic — ingest-time quality investment with explicit cost math (illustrative: ~$1/M doc tokens with prompt caching) — a model for reasoning about RAG cost tradeoffs.
Where this connects
- Security — retrieved text is untrusted input; prompt injection and data exfiltration are the RAG-specific attack surface.
- Reliability — retries, circuit breakers, and provider failover behind the degradation ladder above.
- Scaling & Operations — sharding, index rebuilds, and multi-tenancy for the store underneath.
- Design: Enterprise RAG — this checklist delivered as a full system design walkthrough.