Retrieval Patterns

The intuition

Naive retrieval assumes the user's question already looks like the answer. It usually doesn't.

Picture a library where books are shelved by their full text and you have to find one by walking to the shelf whose contents most resemble what you said. You walk up and mumble "pto carryover?". That phrase doesn't resemble any book — it's four words, and books are prose. You end up somewhere vaguely near the HR section, holding nothing.

Two fixes exist, and every pattern on this page is one of them:

• Fix the question before you walk. Say more, say it differently, say it in the shape of an answer. ("Unused annual leave carries forward as follows…") Now you're walking towards prose that resembles prose. This is the query side.
• Fix the shelving. Label every page with the chapter it came from, and let the librarian hand you the whole chapter once one page matches. This is the index side.

Why it exists

There are two structural mismatches in naive RAG, and they're worth separating because they have different cures.

Mismatch 1 — queries don't look like documents. Embedding models place text by its overall shape and vocabulary. A five-word interrogative fragment and a 400-token declarative passage land in genuinely different regions of the space even when they're about the same thing. You are asking for nearest-neighbour matching between two different distributions.

Mismatch 2 — what you match on isn't what you want to read. A small chunk makes a clean, specific vector (good for matching) and a starved, contextless paragraph (bad for answering). A large chunk is the reverse. Naive RAG forces one object to play both roles, so it's always wrong in one direction.

Everything below is a targeted response to one of those two.

The core idea

Naive RAG embeds the user's raw query and hopes it lands near the right chunks. It often doesn't, for two structural reasons: queries don't look like documents (short, interrogative, underspecified vs long, declarative, detailed — so their embeddings live in different regions), and the chunk you match on isn't always the context you want to generate from.

The advanced patterns attack one side or the other:

Query-side (transform the query before retrieval):

• Query rewriting — an LLM cleans up the query: strips chat fluff, resolves pronouns from conversation history ("what about its pricing?" → "what is Product X's pricing?"), expands acronyms. Non-negotiable for multi-turn chat.
• Multi-query — LLM generates 3–5 paraphrases/perspectives of the question, retrieve for each in parallel, merge with Reciprocal Rank Fusion. Hedges against a single unlucky embedding.
• Decomposition — split a compound question into sub-questions, retrieve per sub-question, answer over the union. For "compare X and Y" questions no single chunk answers.
• HyDE — LLM writes a hypothetical answer document, and you embed that instead of the query. A fake answer looks like a real answer, so doc-to-doc similarity beats query-to-doc similarity. Fixes the query/document mismatch directly.

Index-side (change what gets matched vs what gets returned):

• Parent-child / small-to-big — match on small chunks (precise vectors), return the parent section (rich context). Decouples retrieval granularity from generation granularity.
• Sentence-window — the extreme version: match on single sentences, return the sentence ± a window of neighbors.
• Contextual retrieval (Anthropic) — at ingest, an LLM prepends a 50–100-token blurb situating each chunk in its document before embedding and BM25 indexing, repairing the context that chunking amputated.

The distinction that matters in practice: each pattern buys recall or precision at a cost — extra LLM calls and latency (query-side) or extra ingest compute and storage (index-side). Start with hybrid search + reranking, measure, then add patterns where the eval says retrieval fails.

How it actually works

Why HyDE works. A bi-encoder maps queries and documents into one space, but short interrogative text and long declarative text cluster differently. HyDE ("Hypothetical Document Embeddings", Gao et al. 2022) asks an LLM to hallucinate a plausible answer passage — factual errors are fine — because what's embedded is the shape and vocabulary of an answer, which lands near real answer passages. Mechanism: query → LLM → fake doc → embed → k-NN → real docs. The hallucination never reaches the user; it dies after retrieval. Weakness: if the LLM's fake answer is about the wrong topic (out-of-domain query), retrieval follows it off a cliff — and it adds a generation call to every query.

Why multi-query + RRF works. Each paraphrase probes a different neighborhood of embedding space; the true answer chunk tends to rank decently for several phrasings while noise ranks high for only one. RRF scores each doc Σ 1/(rank_i + 60) across lists — rank-based, so no score calibration between runs — and consistent-but-not-top docs beat one-list flukes. Retrievals run in parallel, so latency ≈ one retrieval + one cheap LLM call; cost is k× the vector-DB reads.

Parent-child mechanics. Index child chunks (e.g., ~200 tokens) each storing a parent_id to its section (e.g., ~1000–2000 tokens). Retrieve children, dedupe by parent, return parents to the prompt. If three children of one parent all match, that's one strong parent, not three prompt slots. Sentence-window is the same idea at sentence granularity — matching text ≠ returned text is the invariant. Watch that returned parents still fit the context budget.

Contextual retrieval mechanics. For each chunk, prompt an LLM with (whole document, chunk) → emit 50–100 situating tokens ("This chunk is from ACME's Q2-2023 10-Q, discussing revenue growth over Q1…"), prepend to the chunk, then embed and BM25-index the combined text. Prompt caching makes it affordable (~$1.02/M document tokens, since the document prefix is cached across its chunks). Anthropic's numbers: contextual embeddings −35% retrieval failures; + contextual BM25 −49%; + reranking −67%. It's an ingest-time cost, so query latency is untouched — the opposite tradeoff from HyDE.

When each pattern pays:

The flows

Three end-to-end paths, chosen by where you spend compute and why retrieval is failing.

The operational model: query-side bills forever per request; index-side bills once. High-QPS systems push work to ingest; corpora you cannot re-process push work to query time.

A worked example

Third turn of a conversation on the governed enterprise platform. Earlier turns discussed the 2026 travel policy. The user now types:

"and what about contractors?"

Naive retrieval. Embed those four words. The nearest chunks are about contractor onboarding, contractor invoicing, and a vendor-management SOP. Travel appears nowhere. The model answers fluently about contractor onboarding. The user assumes it answered the question they meant.

Now each pattern, applied to the same input:

Does the 2026 travel and expense policy apply to contractors?

Production concerns

• Latency stacking: every query-side LLM call adds 200–1000 ms before retrieval even starts. Rewriting+HyDE+multi-query naively chained can double time-to-first-token. Use a small/fast model for query transforms; run what you can in parallel; consider skipping transforms when the raw query already retrieves confidently (score threshold). (See latency.md.)
• Cost asymmetry: query-side patterns bill on every request forever; index-side patterns bill once per ingest. High-QPS systems push work to ingest (contextual retrieval, better chunking); low-QPS/huge-corpus systems prefer query-side.
• Failure amplification: HyDE and rewriting insert an LLM upstream of retrieval — a bad rewrite silently corrupts everything downstream and is invisible unless you log the transformed query alongside the original. Log both; eval retrieval on both. (See observability.md.)
• Merging needs discipline: multi-query/decomposition return overlapping chunk sets — dedupe by chunk ID, fuse by RRF, cap the final context; otherwise you blow the token budget with near-duplicates.
• Measure before adopting: each pattern is a hypothesis about why retrieval fails. Run a golden-set eval (recall@k, MRR — see rag-evaluation.md), find the failure class, apply the matching pattern, re-measure. Stacking all patterns by default is a cost/latency anti-pattern.

Common drill-downs

Explain HyDE and why it beats embedding the raw query. Queries and documents are different text distributions; bi-encoders embed them into imperfectly aligned regions. HyDE has an LLM generate a hypothetical answer passage and embeds that, converting query-to-doc matching into doc-to-doc matching. The fake document's factual errors don't matter — only its answer-like shape and vocabulary do; it's discarded after retrieval. Costs one LLM call of latency per query; risky when the LLM guesses the wrong topic entirely.

Multi-query retrieval — mechanism and when it's worth k× the retrieval cost. LLM writes 3–5 reformulations, retrieve for each in parallel, merge with RRF (rank-based fusion, no score calibration needed). Correct chunks rank moderately across many lists and get boosted; flukes don't. Worth it when eval shows recall failures from phrasing sensitivity; vector reads are cheap, so the real cost is the LLM call and merge complexity.

What problem does parent-child retrieval solve? The chunk-size dilemma: small chunks embed precisely but starve the generator; large chunks embed mushily. Split the roles — index small children for matching, return their parent section for generation. Dedupe children by parent so one strong section doesn't fill three context slots.

How does Anthropic's contextual retrieval work and what did it measure? At ingest, an LLM sees the full document plus each chunk and writes 50–100 tokens of situating context, prepended before both embedding and BM25 indexing — restoring referents ("the company", "this quarter") that chunking severed. Prompt caching keeps it ~$1 per million document tokens. Measured: −35% retrieval failures from contextual embeddings, −49% adding contextual BM25, −67% adding reranking.

HyDE vs contextual retrieval — how do you choose? Same disease (embedding mismatch/ambiguity), opposite ledgers: HyDE pays per query at query time, no ingest changes — good for low QPS or corpora you can't re-index. Contextual retrieval pays once at ingest, zero query latency — good for high QPS on a stable corpus. High-QPS systems generally prefer ingest-time fixes.

User asks "how does it compare to the previous version?" mid-conversation. What must happen before retrieval? Query rewriting with conversation context: resolve "it" and "previous version" into explicit entities ("compare FooLib 3.0 to FooLib 2.x performance") using the chat history, then retrieve on the rewritten query. Embedding the raw utterance retrieves noise — this is the most common production retrieval bug in chat products.

When is sentence-window better than plain small chunks? When answers hinge on pinpoint claims inside dense text (legal clauses, API notes): match on the single most precise sentence, then return it with ±2–3 neighbors so the generator sees the qualifying context. Same precision as sentence-level chunks without their context starvation; cost is many more vectors in the index.

You've added HyDE + multi-query + rewriting and latency doubled. Fix it? Audit against eval data — keep only transforms that move recall. Use a small fast model for transforms, parallelize (HyDE generation alongside BM25 on the raw query), gate transforms behind a confidence threshold on cheap first-pass retrieval, and consider moving the fix to ingest (contextual retrieval) to unload the query path.

Test yourself

Go deeper

• Precise Zero-Shot Dense Retrieval without Relevance Labels — Gao et al., 2022 — the HyDE paper; short and readable.
• Introducing Contextual Retrieval — Anthropic — mechanism, cost math, and the benchmark numbers cited above.
• Building Performant RAG Applications for Production — LlamaIndex — decoupling retrieval chunks from synthesis chunks, structured and recursive retrieval.
• RAG from scratch: Part 5 (Query Translation — Multi Query) — LangChain — short, code-first walkthrough of multi-query with RRF.
• RAG from scratch: Part 9 (Query Translation — HyDE) — LangChain — HyDE mechanism and implementation in the same series.

Where this connects

• Hybrid Search — RRF in full, and why rank fusion beats score normalization.
• Rerankers — the precision stage these patterns feed; hybrid + rerank first, then reach for these.
• RAG Evaluation — the only honest way to decide which of these patterns your system actually needs.
• Advanced RAG — when even a well-tuned fixed pipeline isn't enough, the model itself starts driving retrieval.