The RAG Pipeline

The intuition

An LLM sitting alone is a closed-book exam. It memorized an enormous amount during training, it cannot look anything up, and — crucially — it will never say "I didn't study that." It will write a fluent, well-structured, confidently wrong paragraph instead.

RAG turns it into an open-book exam. Before the model answers, you go to the library, find the three pages that matter, staple them to the question, and say: answer from these pages, and cite which one you used.

That's the whole idea. Everything else in this section is engineering in service of one question: how do you reliably find the right three pages?

Why it exists

Three hard constraints make RAG necessary, and it's worth being able to name them individually because each one independently forces the architecture:

1. The weights are frozen. A model's parametric knowledge stops at its training cutoff. Your Q3 policy change, yesterday's incident, this morning's ticket — none of it is in there.
2. Your data was never in the training set, and shouldn't be. Internal wikis, contracts, customer records. You can't train on them cheaply, and for most regulated data you can't ship them to a third party at all.
3. The model can't distinguish knowing from guessing. Next-token prediction always produces a next token. Without grounding, an unsupported answer looks exactly like a supported one — same fluency, same confidence, no signal.

The alternatives lose on all three. Fine-tuning bakes knowledge in statically: re-running it for every document change is hours and dollars, it produces no citations, and it can't express "user A may see this document, user B may not." Stuffing everything into a long context window doesn't scale past a corpus that fits, costs tokens linearly on every single query, and degrades in the middle of long inputs.

RAG sidesteps all of it by moving knowledge out of the weights and into a queryable store you can update in seconds, filter per user, and cite.

The core idea

RAG (Retrieval-Augmented Generation) grounds an LLM's answer in documents retrieved at query time instead of relying on what the model memorized during training. The model's weights are frozen at a training cutoff and contain none of your private data; RAG fixes both problems without touching the weights: fetch the relevant text, put it in the prompt, instruct the model to answer only from it.

Four reasons teams choose RAG over the alternatives:

1. Knowledge cutoff — retrieval serves current data; the model's parametric knowledge is months-to-years stale.
2. Private data — your wiki, tickets, and contracts were never in the training set and shouldn't be shipped to a fine-tune.
3. Hallucination reduction — grounding plus "answer only from the context" cuts fabrication and, critically, enables citations, so users can verify claims.
4. Cost and freshness vs fine-tuning — updating an index is minutes and cents; fine-tuning is hours and dollars, must be re-run for every data change, and is bad at adding facts anyway (it shifts behavior/style more reliably than knowledge).

The pipeline in one breath — two phases:

OFFLINE (ingest):  documents → parse → chunk → embed → index (vector DB + keyword index)
ONLINE  (query):   query → [rewrite] → embed → retrieve top-k → rerank → augment prompt → generate → cite

Offline, documents are parsed to text, split into chunks (~a few hundred tokens each), embedded into vectors, and stored in a vector index alongside a keyword (BM25) index. Online, the query is embedded, nearest-neighbor search returns the top-k candidate chunks, a reranker reorders them by true relevance, the winners are packed into the prompt with the question, and the LLM generates a grounded, cited answer.

The load-bearing fact: almost every RAG failure is a retrieval failure. If the right chunk isn't in the prompt, no model can answer correctly — so the engineering effort goes into retrieval quality and its evaluation, not prompt wording.

How it actually works

The two phases run on completely different clocks — one is a batch job measured in hours, the other is a request path measured in milliseconds. Keeping them separate is what makes the design legible:

What each stage contributes, and what breaks if you skip it:

Two details worth understanding one level down:

• Why retrieve-then-rerank? Embedding retrieval is fast but lossy (independent encodings, single dot product). Cross-encoders are accurate but O(candidates) LLM-ish inference — far too slow to run over the whole corpus. So the cheap stage maximizes recall, the expensive stage buys precision on a shortlist. Same funnel design as web search. (Mechanics: rerankers.md.)
• Why hybrid search? Dense vectors blur exact tokens: "error TS2345" or a part number embeds near its neighbors but BM25 nails it exactly. Reciprocal Rank Fusion (RRF) merges the two ranked lists without needing score calibration. Anthropic's contextual-retrieval benchmarks show each added stage compounds: contextual embeddings cut retrieval failures ~35%, +BM25 ~49%, +reranking ~67%. (Mechanics: hybrid-search.md.)

RAG vs long context: stuffing a 1M-token window doesn't replace RAG at corpus scale (10M docs don't fit), costs linearly per query in tokens, is slower, and models still retrieve unreliably from mid-context. In practice long context relaxes chunking pressure (bigger chunks, more of them) rather than eliminating retrieval.

The flows

Two clocks, two paths — plus the refusal branch when retrieval has nothing useful to give.

The operational model: offline builds the cache; online reads it under a latency budget; refusal is a first-class path, not an error.

A worked example

Trace one query end to end through the governed enterprise LLM platform. An employee asks:

"Does the new expense policy still cover international travel?"

Offline, this already happened. The policy document was parsed to text, split into 14 chunks of ~400 tokens with 50-token overlap, embedded with a 1024-dimension model, and written to the vector index along with metadata — {doc_id, version, effective_date, visible_to: ["all-staff"]} — plus a BM25 entry per chunk.

Online, per request:

Total ≈ 2.6 s, of which retrieval is ~175 ms — under 8%.

Now break it deliberately, one stage at a time — each omission has a distinctive production fingerprint:

Production concerns

• Retrieval quality is the KPI. Track retrieval metrics (recall@k, MRR) separately from answer metrics — a bad answer with good retrieval is a prompt/model problem; with bad retrieval it's an index problem. (Full treatment: rag-evaluation.md.)
• Latency budget: embed query ~10–50 ms, ANN search ~10–100 ms, rerank 50 candidates ~50–300 ms, generation dominates (seconds). Retrieval overhead is usually <20% of end-to-end; don't micro-optimize ANN while streaming tokens is the bottleneck. (Full treatment: latency.md.)
• Cost: per-query = 1 embedding call + vector DB read + rerank + LLM tokens. Context stuffing is the silent cost driver — 10 chunks × 500 tokens = 5k prompt tokens per query before the question. Reranking pays for itself by letting you send fewer, better chunks. (Full treatment: cost.md.)
• Freshness: the index is a cache of your corpus and goes stale; you need an ingestion pipeline with updates/deletes, not a one-shot script. (Full treatment: production-rag.md.)
• Failure modes to name: empty/garbage retrieval (need thresholds + graceful "not found"), stale chunks after doc updates, prompt overflow when k is too high, and the model ignoring context in favor of parametric memory.

Common drill-downs

Explain RAG to me like I'm a backend engineer. It's a read-through pattern: the LLM is a stateless compute layer, the vector DB is the queryable state. At request time you query the state for relevant records, join them into the prompt, and the model computes the answer over them. Updating knowledge = writing to the DB, not retraining the compute layer.

RAG vs fine-tuning — when each? RAG for knowledge: facts, freshness, private data, citations, per-user access control. Fine-tuning for behavior: format, tone, domain jargon, following a house style. They compose — fine-tune the style, retrieve the facts. Fine-tuning is a poor knowledge store: expensive to refresh, no provenance, can't do per-user permissions. (Deeper: fine-tuning.md.)

Why chunk documents at all? Why not embed the whole document? An embedding is a fixed-size vector; a 50-page doc collapses into one point and its specifics average away, so specific queries can't find specific passages. Chunks make the retrieval unit match the granularity of questions. Also embedding models have context limits, and you want to fill the prompt with only the relevant slice.

Why do you need a reranker if the vector search already ranks by similarity? Bi-encoders encode query and document independently — no term-level interaction, one dot product. A cross-encoder attends across both texts jointly and is far more accurate, but too expensive to run corpus-wide. Retrieval optimizes recall over millions; reranking buys precision over ~50. Skipping it typically costs the most quality per unit of effort saved.

With 1M-token context windows, is RAG dead? No — corpus size (millions of docs), per-query token cost, latency, and mid-context recall degradation ("lost in the middle") all remain. What changes: chunking can get coarser and k larger. Long context and RAG are complements — retrieval selects, context carries.

Your RAG system hallucinates. Walk me through debugging. Isolate the stage: (1) inspect retrieved chunks for a failing query — if irrelevant, it's retrieval (chunking, embeddings, missing hybrid search); (2) if relevant, it's generation — tighten grounding instructions, add "say I don't know", cite-or-refuse; (3) check the chunk itself is complete — mid-sentence splits make the model interpolate; (4) build a small eval set so the fix is measured, not vibes.

Where does the latency go in a RAG request, and what would you optimize first? Generation dominates (seconds); retrieval+rerank is typically tens to a few hundred ms. Optimize perceived latency first: stream tokens, cut prompt size (fewer, better chunks — reranking helps), then parallelize embed+BM25, then consider a smaller generation model.

What's the single highest-leverage improvement to a naive RAG pipeline? Usually hybrid search + a cross-encoder reranker: no re-ingestion, immediate recall and precision gains (Anthropic measured ~67% retrieval-failure reduction stacking contextual embeddings, BM25, and reranking). Second: fix chunking to respect document structure.

Test yourself

Go deeper

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020 — the original RAG paper; skim for the parametric vs non-parametric memory framing.
• Introducing Contextual Retrieval — Anthropic — the stacked-improvements numbers (embeddings → +BM25 → +rerank) that show each stage compounds.
• Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023 — why context position matters and long context ≠ reliable retrieval.
• Patterns for Building LLM-based Systems & Products — Eugene Yan — RAG in the context of the other production patterns (evals, caching, guardrails).
• What is Retrieval-Augmented Generation (RAG)? — IBM Technology — the canonical 6-minute explainer (1M+ views); a clear end-to-end walkthrough of the pipeline.
• A Survey of Techniques for Maximizing LLM Performance — OpenAI — DevDay talk placing RAG vs prompt engineering vs fine-tuning in one decision framework.

Where this connects

• Ingestion & Chunking — the offline half in detail; decisions made there constrain everything downstream and are the most expensive to reverse.
• Retrieval Patterns — the rewrite step above, expanded: HyDE, multi-query, parent-child, contextual retrieval.
• RAG Evaluation — how you know retrieval is working, rather than believing it.
• Production RAG — freshness, ACLs, degradation: what the diagram above needs before real users touch it.
• Advanced RAG — what to reach for when this fixed pipeline isn't enough.