Production RAG

The intuition

Here is the sentence that organizes the whole page:

That framing does real work. A backend engineer who has never touched RAG already knows that caches go stale, that deleting the source must delete the cached copy, that a cache miss needs a defined behaviour, and that serving cached data to the wrong user is an incident. Every item in this page is one of those, specialized.

The second framing that helps: a demo RAG is a script; a production RAG is a data platform plus a serving path. The script has one code path. The platform has an ingestion pipeline with its own SLOs, a permission model, a latency budget, a cost model, and a defined behaviour for every stage being down.

Why it exists

Prototypes ignore four things that are not optional with real users:

1. The corpus moves. A one-shot indexing script is correct for exactly as long as nobody edits a document. Then it starts confidently quoting a policy that was rescinded last month — and nothing anywhere reports an error.
2. Not everyone may see everything. The moment there are two users with different permissions, retrieval becomes a per-caller authorization decision. And because retrieved text enters the prompt, a retrieval bug is a data-leak bug, not a quality bug.
3. Every stage can fail. Vector DB down, reranker timing out, provider rate-limiting you, retrieval returning nothing. Each needs a designed behaviour, because the default behaviour — a fluent answer invented from parametric memory — is the worst possible one.
4. It bills per token, per query, forever. A prototype's cost is a rounding error. The same architecture at 50 QPS is a budget line someone will ask you to defend.

The core idea

A demo RAG is a script; a production RAG is a data platform plus a serving path, and the gap between them is the whole subject. Walk it as a checklist with mechanisms:

1. Index freshness — the index is a derived cache of the corpus and decays the moment sources change. You need a continuous ingestion pipeline (event-driven from source webhooks/CDC where possible, scheduled crawls elsewhere), incremental by content hash: re-parse, re-chunk, re-embed only changed docs.
2. Document lifecycle — updates and deletes, not just inserts. Upsert = write new chunks, then delete every chunk of the old version (orphaned stale chunks are the classic "we deleted that policy but the bot still quotes it" bug). Deletes must propagate fast — for compliance (GDPR erasure) a tombstone filter can hide chunks instantly while physical deletion catches up.
3. Per-user ACLs enforced at retrieval, as filters — never in the prompt. Attach permission/tenant tags to chunks at ingest; every vector query carries a mandatory filter from the caller's identity. "Please don't reveal documents the user can't see" in the prompt is not security: anything in context can be extracted by injection. If it can't enter the context window, it can't leak.
4. Latency budget per stage — set an end-to-end SLO and allocate it; stream tokens so perceived latency is time-to-first-token.
5. Cost control — cost per query = embeddings + vector reads + rerank + LLM tokens; the LLM tokens dominate, so context size discipline and caching are the levers.
6. Failure modes and graceful degradation — retrieval returns nothing, returns garbage, or a dependency is down; every case needs a designed behavior other than "hallucinate confidently."
7. Observability and eval in the loop — log query, transformed query, retrieved chunk IDs + scores, prompt, and answer for every request; sample into LLM-judge scoring; gate releases on a golden set (see rag-evaluation.md).

How it actually works

Freshness / re-indexing mechanics. Full re-index is O(corpus) in embedding cost — reserve it for chunking/embedding-model changes. Steady state is incremental: change event → parse → hash-compare → re-chunk changed docs → embed → atomic swap of that doc's chunks. Embedding-model upgrades force a full re-embed (vectors from different models aren't comparable) — do it blue/green: build the new index alongside, run the golden-set eval against it, flip an alias, keep the old index for rollback. Deterministic chunk IDs (doc_id, chunk_seq) make swaps and deletes exact.

Lifecycle details that bite: a shrinking document (v2 has 8 chunks, v1 had 12) leaves 4 stale chunks unless you delete by doc_id, not by ID overlap. Vector DB deletes are often soft (tombstoned until segment merge/compaction) — fine for serving, but check compaction actually reclaims and that filters hide tombstones immediately.

ACL mechanics. Ingest: resolve each document's readers into tags (allowed_groups, tenant_id) stored as chunk metadata. Query: resolve the user's groups → mandatory metadata filter ANDed into the vector search (modern vector DBs do filtered ANN natively — pre-filter or during-traversal, not post-filter, so top-k isn't emptied by permission drops). Two hard sub-problems: permission changes must re-tag existing chunks (subscribe to ACL-change events too, not just document events); group explosion (thousands of groups per user) may need a permissions service resolving to a compact tag set at query time. Multi-tenant SaaS: tenant filter minimum, ideally namespace/collection-per-tenant for blast-radius isolation. (Mechanics of filtered ANN: filtering-and-metadata.md.)

Latency budget (typical chat SLO ~2–3 s to first token):

Cost levers, ranked: (1) prompt-size discipline — rerank hard and send 3–5 chunks, not 20; (2) prompt caching — put static system prompt and stable prefixes first so only the variable tail bills full price; (3) model routing — small model for easy/frequent queries, big model for hard ones; (4) semantic response cache — embed incoming query, if similarity to a previously answered query exceeds a threshold, serve the cached answer (FAQ-heavy workloads see high hit rates; invalidate on index updates touching the cached answer's sources); (5) embedding costs are one-time-per-corpus-change — usually noise next to per-query LLM tokens.

Failure modes and designed degradation:

The design principle: every stage after hybrid retrieval is skippable, and generation without evidence is refusable — the system degrades along a quality gradient instead of failing binary.

The flows

Three operational paths: keep the cache correct, serve under identity, and degrade when a stage dies.

The operational model: the index is a cache; permissions are retrieval filters, never prompt instructions; quality degrades along a gradient instead of failing binary.

A worked example

Two incidents on the governed enterprise platform. Both are "the system returned a 200 and a fluent answer." Both are outages by any measure that matters.

Incident 1 — the rescinded policy.

An employee asks about remote-work eligibility and gets a confident answer citing a policy withdrawn six weeks ago. Trace the failure backwards:

The lesson that matters: there was no error anywhere. Every component behaved correctly on its own inputs. Only an explicit freshness SLO turns this class of bug from "a user complained in September" into "an alert fired in July."

Incident 2 — the draft nobody should have seen.

A contractor asks about compensation bands and the assistant quotes an unreleased HR draft.

Walk the chain and note where it could have been stopped:

ingest      → doc tagged visible_to: ["hr-core"]           ✓ correct
query       → user's groups resolved to ["contractors"]    ✓ correct
retrieval   → filter applied AFTER the ANN search (post-filter)
              ↳ top-50 fetched, then permission-filtered to 3
              ↳ …but a *different* code path (the "temporary
                document upload" feature) queried a shared
                collection without the filter at all          ✗ HERE
generation  → system prompt says "only discuss documents the
              user is authorised to see"                      ✗ not a control

Two lessons that transfer to every governed deployment:

1. The prompt instruction was never security. Anything that reaches the context window can be extracted from it — by injection, by a rephrased question, or by the model simply being helpful. The only real boundary is never retrieving it. "If it can't enter the context, it can't leak."
2. The filter must be structurally unbypassable, not conventionally applied. One un-filtered code path is a breach. The design fix is to make the filter a property of the retrieval client — a caller must pass an identity to get a retriever at all — rather than an argument each call site remembers to include.

Production concerns

This whole file is production concerns; the extra layer on top:

• Security beyond ACLs: retrieved documents are untrusted input — a poisoned document can carry prompt-injection payloads ("ignore your instructions…"). Treat context as data (delimiters, instruction hierarchy), sanitize at ingest, and never give the RAG chat loop write-capable tools without human confirmation. (Full treatment: security.md.)
• Observability minimum: per-request trace (query → rewritten query → chunk IDs + scores → prompt → answer → feedback) with IDs joining to eval; dashboards for retrieval-score distribution, refusal rate, freshness lag, p95 per stage, cost per query. Distribution shift in retrieval scores is your early-warning for index rot. (Full treatment: observability.md.)
• Rollout discipline: every change class (prompt, chunking, embedding model, reranker, LLM version) goes through golden-set eval → shadow or canary traffic → full rollout, with index versions blue/green so rollback is an alias flip.
• Capacity realities: embedding-provider rate limits throttle bulk re-indexes (queue + backoff); vector-DB memory sizing (HNSW is RAM-resident) drives cost at 10M+ vectors; reranker GPU throughput caps QPS — know which of the three saturates first in your design. (See scaling-and-operations.md.)

Common drill-downs

How would you productionize a RAG prototype? Structure it as decision points, not tools: (1) ingestion as a pipeline — event-driven incremental indexing with upsert/delete handling; (2) security — ACL tags on chunks, mandatory retrieval-time filters; (3) serving — hybrid retrieval + rerank inside a latency budget, streaming; (4) reliability — thresholds, fallbacks, honest refusal; (5) cost — context discipline, caching, model routing; (6) eval + observability — golden-set CI gate, full request tracing, sampled online judging. Naming the decision points beats naming tools.

How do you keep the index fresh? Treat the index as a derived cache with an explicit staleness SLO. Subscribe to source change events (webhooks/CDC), fall back to scheduled diff crawls; content-hash to skip unchanged docs; re-embed only changed chunks; atomic per-doc chunk swap. Full rebuilds only for chunking/embedding-model changes, executed blue/green with eval before the alias flip. Monitor freshness lag as a first-class metric.

A document is updated — walk through what must happen. And deleted? Update: change event → re-parse → re-chunk → re-embed → write new chunks → delete all old chunks by doc_id (count may differ — delete by doc, not by ID match) → invalidate any cached answers citing that doc. Delete: remove chunks everywhere + tombstone filter for instant hiding while physical deletion and cache invalidation complete; for GDPR, verify reclamation, including from backups per policy.

How do you enforce per-user permissions in RAG? At retrieval time, as metadata filters — never via prompt instructions. Chunks carry ACL/tenant tags from ingest; the query layer resolves the caller's identity to a filter ANDed into the ANN search (pre-filter, so top-k isn't gutted post-hoc). Sync permission changes as well as document changes. Prompt-level "don't show X" fails to injection; if it's not in the context, it can't leak.

Retrieval returns nothing — or garbage. What does your system do? Nothing: score threshold detects it → honest "I couldn't find relevant documents" + escalation path; never generate from parametric memory in a grounded product. Garbage: a cheap relevance grader between retrieval and generation gates the context; retry once with a rewritten query; then refuse. Track refusal and grader-trigger rates — rising rates flag index gaps or drift.

Where does the money go per query and how do you cut it? LLM prompt+completion tokens dominate. Cut: send fewer, better chunks (rerank hard); prompt caching on static prefixes; route easy queries to a small model; semantic cache for repeated questions (with invalidation on index updates); trim chat history. Embedding and vector-read costs are usually second-order.

Set a latency budget for a 2-second-to-first-token chat SLO. Work backwards from LLM TTFT (~1–1.5 s at production prompt sizes): rewrite ≤300 ms (small model, skippable), embed ≤50 ms, hybrid search ≤100 ms (parallel dense+BM25), rerank ≤300 ms. Stream generation. Reserve headroom for queueing; measure p95 per stage, not averages — the tail is where SLOs die.

Vector DB is down. Does your product die? No — degrade along the quality gradient: circuit-break to BM25/keyword search (different store or a search engine you already run), serve semantic-cache hits, and if all retrieval is gone, say so and offer non-grounded help clearly labeled or escalate. Reranker down → skip it. LLM down → provider failover. Each stage needs a defined skip/fallback behavior decided before the incident.

How do you safely ship an embedding-model upgrade? Vectors across models aren't comparable, so it's a full re-embed: build the new index offline, run the retrieval golden set against both (recall@k, MRR), shadow live queries to compare, then flip a serving alias; keep the old index warm for instant rollback. Budget the embedding run against provider rate limits; schedule off-peak.

Test yourself

Go deeper

• Building A Generative AI Platform — Chip Huyen — the full production architecture: context construction, guardrails, routing, caching, observability.
• Building Performant RAG Applications for Production — LlamaIndex — framework-level production techniques for retrieval quality at scale.
• Introducing Contextual Retrieval — Anthropic — ingest-time quality investment with explicit cost math (~$1.02/M doc tokens with prompt caching) — a model for how to reason about RAG cost tradeoffs.
• Patterns for Building LLM-based Systems & Products — Eugene Yan — caching, guardrails, defensive UX, and feedback loops around the core pipeline.
• Systematically Improving Your RAG — Jason Liu — the operational flywheel: metrics, feedback, and prioritizing improvements by measured failure class.
• A Survey of Techniques for Maximizing LLM Performance — OpenAI — DevDay talk: when to reach for RAG vs prompting vs fine-tuning, with production framing.

Where this connects

• Security — retrieved text is untrusted input; prompt injection and data exfiltration are the RAG-specific attack surface.
• Reliability — retries, circuit breakers, and provider failover behind the degradation ladder above.
• Scaling & Operations — sharding, index rebuilds, and multi-tenancy for the store underneath.
• Design: Enterprise RAG — this checklist delivered as a full system design walkthrough.