The RAG Pipeline

The intuition

An LLM sitting alone is a closed-book exam. It memorized a huge amount during training, it cannot look anything up, and it will not say "I didn't study that." It writes a fluent, confidently wrong paragraph instead.

RAG turns it into an open-book exam. Before the model answers, you go to the library, find the few pages that matter, staple them to the question, and say: answer from these pages, and cite which one you used.

That is the whole idea. Everything else in this section is engineering for one question: how do you reliably find the right pages?

Why it exists

Three constraints force the architecture:

1. The weights are frozen. Parametric knowledge stops at the training cutoff. Yesterday's policy change is not in there.
2. Your data was never in the training set — and should not be. Wikis, contracts, customer records. You cannot fine-tune on them cheaply, and often you cannot ship them outward at all.
3. The model cannot tell knowing from guessing. Next-token prediction always produces a next token. Without grounding, unsupported and supported answers look the same.

Fine-tuning bakes knowledge in statically: re-running for every document change is hours and dollars, with no citations and no per-user permissions. Stuffing a long context fails past a corpus that fits, bills tokens on every query, and still degrades when the answer sits mid-context ("lost in the middle").

RAG moves knowledge out of the weights into a store you can update in seconds, filter per user, and cite.

The core idea

Retrieval-Augmented Generation (RAG) grounds an LLM in documents fetched at query time instead of relying only on memorized weights. Lewis et al. (2020) called this parametric memory (the frozen model) plus non-parametric memory (an external index you can edit). The production recipe is fixed: retrieve relevant chunks, put them in the prompt, answer only from that context — and refuse when you cannot.

Two phases, two clocks:

OFFLINE (ingest):  documents → parse → chunk → embed → index (vector + keyword)
ONLINE  (query):   query → [rewrite] → embed → retrieve top-k → rerank → augment → generate → cite

Offline is batch: parse, chunk (typically a few hundred tokens), embed, write a dense vector index and a BM25 keyword index. Online is the request path: embed the query, retrieve, rerank, pack the prompt, generate a cited answer.

Almost every RAG failure is a retrieval failure. If the right chunk never reaches the prompt, no wording or model upgrade recovers it. That is where evaluation and engineering effort go first.

How it actually works

Parse and chunk create retrieval units. Garbage parse (shredded tables, interleaved columns) poisons every later stage. Chunks must be specific enough for one clean vector and rich enough for the LLM to answer — size, overlap, and structure live on ingestion and chunking.

Embed and index. An encoder maps each chunk to a dense vector (illustratively ~768–3072 dims). HNSW (or similar ANN) makes nearest-neighbour search sub-linear over millions of vectors. A BM25 inverted index sits beside it for exact tokens. Vectors from different embedding models are not comparable — change the model and re-embed the corpus.

Retrieve, then rerank. Stage 1 maximizes recall (k often 20–100): dense search catches paraphrase ("vacation policy" ↔ "PTO rules"); BM25 catches IDs, SKUs, error codes. Reciprocal Rank Fusion (RRF) merges the lists with ranks only — score(d) = Σ 1/(k_rrf + rank(d)) — so cosine never has to be calibrated against BM25. Stage 2 is a cross-encoder that scores each (query, chunk) pair jointly and keeps top 5–10 for precision. Bi-encoders are fast and lossy; cross-encoders are accurate but O(candidates) — too slow over the whole corpus. Same funnel as web search: cheap and wide, then expensive and narrow. (Depth: hybrid search, rerankers.)

Anthropic's 2024 contextual-retrieval numbers show the stages stack on their evals: contextual embeddings cut top-20 retrieval failures ~35%; + contextual BM25 ~49%; + reranking ~67%. Measure on your golden set; the stacking order is the lesson.

Augment and generate. Pack winners with instructions and the question. Models attend more reliably to the start and end of context than the middle (Liu et al., 2023). Force grounding, require citations, and leave an "I don't know" hatch. Without the hatch, a retrieval miss still yields a fluent answer from parametric memory — the worst failure mode, because it looks grounded.

Long context does not kill RAG. A million-token window still cannot hold millions of docs, still costs tokens per query, and still loses mid-context facts. It does relax chunking (bigger chunks, larger k). For a knowledge base under ~200k tokens, a cached full-corpus prompt can win. For entitlement-filtered enterprise corpora, retrieval selects and context carries.

The flows

Offline builds the cache. Online reads it under a latency budget. Refusal is a product path, not an error.

A worked example

On the governed enterprise LLM platform, an employee asks:

"Does the new expense policy still cover international travel?"

Offline already ran. The policy was parsed into 14 chunks of ~400 tokens with 50-token overlap (illustrative), embedded at 1024 dims, stored with {doc_id, version, effective_date, visible_to: ["all-staff"]} and a BM25 entry per chunk.

Online (illustrative timings):

Total ≈ 2.6 s; retrieval ~175 ms — under 8% when generation dominates.

Production concerns

Retrieval is the KPI. Track recall@k and MRR separately from answer quality. Bad answer + good retrieval → prompt/model. Bad answer + bad retrieval → index/chunking. Without the split you "fix" generation while the gold chunk never entered the prompt (RAG evaluation).

Latency and cost. Illustrative: embed ~10–50 ms, ANN+BM25 ~10–100 ms, rerank ~50 ~50–300 ms; generation is seconds and usually >80% of wall time. Stream first, shrink the prompt second (reranking helps), then consider a faster generator — do not retune HNSW for a 90 ms win users will not see (latency). Per query you pay embed + vector read + optional rerank + LLM tokens; 10×500-token chunks are 5k prompt tokens before the question, so fewer better chunks cut the bill (cost).

Freshness and failures. The index is a cache — continuous ingest with updates/deletes, not a one-shot script (production RAG). Under traffic: empty retrieval without a threshold (fluent invention); orphaned chunks after doc updates; prompt overflow from oversized k; information blindness (right chunk present, generator ignores it — fix packing and instructions, not only stage-1); ACL misses that look like quality bugs until someone reads a restricted draft.

Common drill-downs

Explain RAG to a backend engineer. Read-through cache: the LLM is stateless compute; the vector DB is queryable state. Load relevant records into the prompt and compute. Updating knowledge is a write to the store, not a retrain.

RAG vs fine-tuning? RAG for knowledge (facts, freshness, private data, citations, ACLs). Fine-tuning for behavior (format, tone, jargon). Compose them — style in weights, facts from retrieval. Fine-tuning is a poor knowledge store. Depth: fine-tuning.

Why chunk instead of embedding whole documents? A fixed-size vector averages a 50-page doc into one point; specific queries cannot hit specific passages. Chunks match question granularity and keep the prompt on the relevant slice.

Is RAG dead with huge context windows? No for large or entitlement-filtered corpora. For a few hundred pages a cached full-corpus prompt can win. Long context relaxes chunking; it does not remove scale, cost, or mid-context degradation.

System hallucinates — debug path? (1) Retrieved chunks irrelevant → fix retrieval (chunking, embeddings, hybrid). (2) Relevant → fix generation (grounding, refuse hatch, citations). (3) Chunk split mid-sentence? (4) Add the case to a golden set so the fix is measured.

Highest-leverage upgrade to naive RAG? Hybrid search + cross-encoder reranker (no re-ingest). Second: structure-aware chunking. Measure before stacking patterns from retrieval patterns.

Test yourself

Go deeper

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020 — original paper; parametric vs non-parametric memory.
• Introducing Contextual Retrieval — Anthropic — stacked embeddings → BM25 → rerank failure-rate cuts, plus when a full-corpus prompt can replace RAG.
• Lost in the Middle — Liu et al., 2023 — context position matters; long context ≠ reliable retrieval.
• Patterns for Building LLM-based Systems & Products — Eugene Yan — RAG among evals, caching, and guardrails.
• What is Retrieval-Augmented Generation (RAG)? — IBM Technology — short end-to-end pipeline walkthrough.
• A Survey of Techniques for Maximizing LLM Performance — OpenAI — RAG vs prompting vs fine-tuning as one decision frame.

Where this connects

• Ingestion & Chunking — offline half in detail; decisions there are the most expensive to reverse.
• Retrieval Patterns — rewrite expanded: HyDE, multi-query, parent-child, contextual retrieval.
• RAG Evaluation — how you know retrieval works, rather than believing it.
• Production RAG — freshness, ACLs, degradation before real users.
• Advanced RAG — when this fixed pipeline is not enough.