Retrieval Patterns
Query rewriting, HyDE, multi-query, parent-child retrieval, contextual retrieval.
Prerequisites
- The RAG Pipeline — these patterns slot into the retrieve stage.
- Ingestion & Chunking — half of them are ingest-time fixes to problems chunking created.
- Embeddings — specifically that queries and documents are embedded by the same model but are not the same kind of text.
The intuition
Naive retrieval assumes the user's question already looks like the answer. It usually doesn't.
Picture a library where books are shelved by their full text, and you find one by walking to the shelf whose contents most resemble what you said. You mumble "pto carryover?". That phrase does not resemble any book — four words against pages of prose — so you end up near HR with nothing.
Every pattern on this page is one of two fixes. Fix the question before you walk — say more, say it differently, or say it in the shape of an answer ("Unused annual leave carries forward as follows…"). Now you walk toward prose that resembles prose. That is the query side. Fix the shelving — label every page with the chapter it came from, and hand back the whole chapter once one page matches. That is the index side.
Key insight
Query-side patterns bill per request, forever. Index-side patterns bill once, at ingest. High-QPS systems push work to ingest; a corpus you cannot re-process pushes work to query time. Pick the ledger before you pick the pattern name.
Why it exists
Two structural mismatches make naive RAG fail, with different cures.
Mismatch 1 — queries do not look like documents. Embedding models place text by shape and vocabulary. A five-word interrogative and a 400-token declarative passage about the same topic still land in different regions. You are nearest-neighbour matching across two distributions.
Mismatch 2 — what you match on is not what you want to read. A small chunk makes a clean vector (good for matching) and a starved paragraph (bad for answering). A large chunk is the reverse. Naive RAG forces one object to play both roles.
Query-side patterns attack mismatch 1. Index-side patterns attack mismatch 2 and the context loss chunking leaves behind. Neither replaces hybrid search and reranking — they sit on top of a baseline that already works, where eval says retrieval still fails.
The core idea
Retrieval patterns change either the query you search with or the units you index and return, so recall and usable context improve without hoping the raw user string is already a good search key.
Query side: query rewriting (resolve pronouns, expand acronyms from chat history), multi-query (paraphrases in parallel, fuse ranks), decomposition (sub-questions for compound asks), HyDE (embed a hypothetical answer instead of the question).
Index side: parent-child (match small, return the section), sentence-window (match a sentence, return neighbours), contextual retrieval (prepend a situating blurb at ingest before embed and BM25).
Start with hybrid search + reranking. Measure. Add the one pattern whose failure mode matches yours. Each pattern is a hypothesis about why retrieval fails — not a free quality upgrade.
How it actually works
Query rewriting is the default for multi-turn chat. A small, fast model turns recent history plus the new message into one self-contained search string. Without it, follow-ups like "and the older version?" retrieve on almost nothing. One cheap call; non-negotiable for conversational products.
Multi-query (RAG-Fusion when fused with RRF) generates 3–5 paraphrases, retrieves in parallel,
and merges with Reciprocal Rank Fusion: each doc scores Σ 1/(rank_i + 60) across lists — ranks
only, no score calibration. True chunks rank decently for several phrasings; noise tops one list and
vanishes. Latency stays near one retrieval if reads run in parallel. Use when eval shows
phrasing-sensitive recall failures.
Decomposition splits "compare X and Y" into sub-questions, retrieves per sub-question, answers over the union. Reach for it when no single chunk can cover the ask — not as a general rewrite.
HyDE (Hypothetical Document Embeddings, Gao et al. 2022) has the LLM write a plausible answer
passage, then embeds that: query → LLM → fake doc → embed → k-NN → real docs. Factual errors are
fine — the vector encodes answer-like shape and vocabulary, and matching against the real corpus
filters many invented details. The fake doc never reaches the user. The real risk is topical
error: wrong subject → wrong region → wrong chunks. One generation per query on the latency path, so
keep it selective.
Why this matters
HyDE shows embedding spaces are geometry, not truth detectors. A deliberately wrong document can still improve retrieval because the vector encodes register and vocabulary.
Parent-child indexes children (~200 tokens) that store a parent_id to a section
(~1000–2000 tokens). Retrieve children, dedupe by parent, return parents. Matching stays
precise; generation gets full context. Sentence-window is the same invariant finer-grained:
match a sentence, return it with ± neighbours — useful for pinpoint claims in dense text. Watch
parent size against the context budget.
Contextual retrieval (Anthropic) runs at ingest. An LLM sees document + chunk, emits 50–100 situating tokens, and you prepend them before both embedding and BM25. Chunking had cut referents ("the company", "this quarter"); the blurb puts them back. Query latency unchanged — opposite ledger from HyDE. Anthropic's published figures (2024): ~$1.02 per million document tokens one-time with prompt caching (illustrative pricing); top-20 retrieval failure cuts of −35% (contextual embeddings), −49% (+ contextual BM25), −67% (+ reranking). Strongest on self-similar corpora — filings, contracts, versioned manuals.
| Pattern | Fixes | Extra cost | Reach for it when |
|---|---|---|---|
| Rewriting | Conversational / underspecified queries | 1 cheap LLM call | Multi-turn chat — always |
| Multi-query | Phrasing sensitivity, low recall | 1 LLM call + k parallel retrievals | Recall@k low, phrasings vary |
| Decomposition | Multi-part questions | LLM call + n retrievals | Compare / aggregate queries fail |
| HyDE | Query↔doc embedding mismatch | 1 generation per query | Terse queries; cannot re-index |
| Parent-child | Precision vs context tension | Two-level storage | Chunks too small to answer from |
| Sentence-window | Same, finer | Many more vectors | Dense factual docs, pinpoint claims |
| Contextual retrieval | Chunks ambiguous out of context | One-time LLM pass at ingest | Stable corpus; high QPS |
The flows
| Flow | Sequence | When it applies | What breaks it |
|---|---|---|---|
| Query-side transform | raw query (+ history) → rewrite and/or multi-query / decompose / HyDE → hybrid retrieve → RRF fuse → dedupe → rerank | Multi-turn chat, phrasing-sensitive recall, query↔doc mismatch | Chained LLM latency; bad rewrite silent corruption; HyDE topical error; unlogged transforms |
| Index-side enrich | at ingest: parent-child or sentence windows or contextual blurbs → index → match small / return parent | High QPS on a stable corpus; chunk-size dilemma; self-similar docs | Re-ingest cost; parent overflow; missing parent dedupe |
| Diagnose-then-apply | golden-set eval → classify failure → apply one pattern → re-measure | Deciding what to add next | Stacking every pattern — cost and latency up, quality flat |
A worked example
Third turn on the governed enterprise platform. Earlier turns covered the 2026 travel policy. The user types:
"and what about contractors?"
Naive retrieval. Embed four words. Nearest chunks: contractor onboarding, invoicing, vendor SOP. Travel never appears. The model answers fluently about onboarding. The user thinks they got what they meant.
A cheap model emits:
Does the 2026 travel and expense policy apply to contractors?Retrieval lands on Eligibility. Highest-value pattern for chat products, and the one most often missing.
Common misconception
"More patterns is better." Stacking rewriting + HyDE + multi-query can add a second before retrieval starts, often fixing the same vocabulary gap three times. Each pattern is a hypothesis about why retrieval fails. Diagnose, apply one, re-measure.
Production concerns
Latency stacking is the first tax. Every query-side LLM call adds hundreds of milliseconds before retrieval. Use a small/fast model; parallelize (HyDE alongside BM25 on the raw query); gate transforms behind a cheap first-pass confidence score so easy queries skip the expensive path. Adaptive routing — fire multi-query or HyDE only when a weakness signal trips — captures long-tail wins without paying fusion on every request. See latency.
Cost asymmetry decides architecture. Query-side bills forever at your QPS. Index-side bills once per ingest. At high QPS on a stable corpus, contextual retrieval and parent-child almost always beat HyDE for the same mismatch class. Prefer HyDE when you cannot re-index or traffic is low.
Upstream LLM failures are invisible unless you log them. A bad rewrite or off-topic HyDE doc silently poisons the whole pipeline. Log original query, transformed query, and which pattern fired; eval retrieval on both. See observability.
Merge discipline. Multi-query and decomposition return overlapping sets — dedupe by chunk ID,
fuse by RRF, cap context. For parent-child, dedupe by parent_id or you paste the same section three
times.
Measure before adopting. Golden-set recall@k and MRR classify the failure. Apply one pattern, re-measure. Stacking by default is a cost and latency anti-pattern — see RAG evaluation.
Common drill-downs
HyDE vs contextual retrieval — how do you choose? Same disease, opposite ledgers. HyDE: per query, no ingest change — low QPS or un-reindexable corpora. Contextual retrieval: once at ingest, zero query latency — high QPS on a stable corpus. Production at scale generally prefers ingest-time fixes.
When is multi-query worth k× retrieval cost? When eval shows phrasing or vocabulary mismatch (policy: "contingent workers"; users: "contractors"). Vector reads are cheap; real costs are the LLM call and latency if you do not parallelize. Skip when a single rewrite already hits recall targets.
Sentence-window vs plain small chunks? Not "smaller chunks" — match unit ≠ return unit. Sentence precision for the vector, expanded neighbours for the generator. Plain small chunks starve the model unless you also expand.
Latency doubled after HyDE + multi-query + rewriting. What next? Drop transforms that do not move recall. Fast model; parallelize; gate behind first-pass confidence; move the fix to ingest if QPS and corpus stability allow.
"How does it compare to the previous version?" mid-conversation — first step? Rewrite with history: resolve "it" and "previous version" into explicit entities, then retrieve. Embedding the raw utterance is the most common chat-retrieval bug in production.
Test yourself
HyDE deliberately generates a document that may be factually wrong. Why doesn't that corrupt the answer?
Your chat RAG works fine on the first question and poorly from the second onward. Diagnose it.
RRF sums 1/(rank + 60). Why the constant 60, and what would happen at 0?
You run 50 QPS over a stable 200k-document corpus. Rank HyDE vs contextual retrieval.
Parent-child returns three matching children from the same parent. What must you do?
Go deeper
- Precise Zero-Shot Dense Retrieval without Relevance Labels — Gao et al., 2022 — the HyDE paper; short and readable.
- Introducing Contextual Retrieval — Anthropic — mechanism, cost math, and the failure-rate numbers above.
- Building Performant RAG Applications for Production — LlamaIndex — decoupling retrieval chunks from synthesis chunks.
- RAG from scratch: Part 5 (Query Translation — Multi Query) — LangChain — code-first multi-query with RRF.
- RAG from scratch: Part 9 (Query Translation — HyDE) — LangChain — HyDE mechanism in the same series.
Where this connects
- Hybrid Search — RRF in full, and why rank fusion beats score normalization.
- Rerankers — the precision stage these patterns feed; hybrid + rerank first, then reach for these.
- RAG Evaluation — the only honest way to decide which pattern you actually need.
- Advanced RAG — when a well-tuned fixed pipeline is not enough and the model itself starts driving retrieval.