Hybrid Search

The intuition

Two librarians share the same warehouse.

The first is a linguist: she finds documents about the same idea even when the words differ — "reset my password" lands on "credential recovery procedure." She is weak when you hand her a serial number.

The second is a filing clerk: he matches exact tokens with ruthless statistical weights. SKUs, error codes, and legal citations are his superpower. He is blind to synonyms.

Hybrid search runs both in parallel and merges their ranked lists. Documents both like rise to the top; documents only one finds still surface. That is why hybrid is the robust default over real enterprise corpora — technical docs, tickets, catalogs — where the next query might be a paraphrase or an identifier.

Why it exists

Semantic search alone is not enough for production query streams.

Hard constraints:

1. Embeddings compress meaning, not tokens. Part numbers, error codes, people's names, section cites, and fresh jargon the model never saw retrieve unreliably in pure vector space.
2. Keyword search has zero paraphrase understanding. "My payment went through twice" will not match "duplicate charge refund policy" if the tokens barely overlap.
3. Raw scores live on incomparable scales. BM25 is unbounded; cosine is bounded. You cannot average them without normalization games that drift when models change.
4. User queries mix both modes. You do not get to choose whether the next request is semantic or lexical.

Alternatives lose. Dense only misses identifiers. BM25 only misses paraphrase. Learned score blending without care breaks when re-embedding shifts score distributions. Reciprocal Rank Fusion (RRF) exists specifically to merge heterogeneous ranked lists using ranks alone — no score calibration required.

The core idea

Dense (embedding) retrieval and sparse (keyword) retrieval fail in opposite ways, so production search runs both and fuses the results. Dense retrieval matches meaning — "how do I reset my password" finds "credential recovery procedure" — but is unreliable on exact tokens: part numbers, error codes, people's names, legal citations, fresh jargon the embedding model never saw. Sparse retrieval — in practice BM25, the 30-year-old ranking function behind Elasticsearch and most search engines — matches exact terms with strong statistical weighting, is cheap, explainable, and needs no training, but has zero understanding of synonyms or paraphrase.

Hybrid search runs both queries in parallel against the same corpus and merges the two ranked lists, most commonly with Reciprocal Rank Fusion (RRF): each document scores Σ 1/(k + rank) across the lists it appears in (k ≈ 60). RRF uses only ranks, never raw scores — which sidesteps the fact that BM25 scores and cosine similarities live on incomparable scales. Documents ranked well by both systems float to the top; documents found by only one still surface.

The practical payoff: hybrid is the robust default for RAG over real-world corpora — technical docs, support tickets, product catalogs — because user queries mix semantic questions with exact identifiers, and you don't get to choose which arrives next.

How it actually works

End-to-end hybrid path from query to fused candidates:

BM25 at survival depth. For query q and document d:

score(q, d) = Σ over terms t in q:  IDF(t) · ( tf(t,d) · (k1 + 1) ) / ( tf(t,d) + k1 · (1 − b + b · |d|/avgdl) )

• IDF(t) = ln( (N − df(t) + 0.5) / (df(t) + 0.5) + 1 ) — rare terms weigh more; a term in every document is worth ~nothing. This is why an exact SKU is retrieval gold to BM25: near-maximal IDF.
• tf with k1 (default ~1.2): term-frequency saturation — the 2nd occurrence of a term adds a lot, the 20th adds almost nothing. Higher k1 → slower saturation. This is BM25's main upgrade over raw TF-IDF, which grows linearly and lets keyword-stuffed documents win.
• b (default 0.75): length normalization — divides TF's effect by how long the document is relative to the corpus average. b=1: full normalization; b=0: none. Prevents long documents winning just by containing more words.

Served from an inverted index (term → posting list of documents), so scoring touches only documents containing query terms — sub-millisecond at scale. Tokenization/stemming/analyzers decide what counts as "a term," and misconfigured analyzers (e.g. splitting ERR-1042) are a classic silent failure.

Reciprocal Rank Fusion. Given result lists R₁…Rₙ:

RRF(d) = Σᵣ 1 / (k + rankᵣ(d))        k = 60 by convention

• Rank-based → no score normalization needed; robust to scale mismatch between BM25 and vector scores.
• k damps the top ranks: with k=60, rank 1 contributes 1/61 ≈ 0.0164, rank 10 → 1/70 ≈ 0.0143 — top positions matter but no single list dominates. Smaller k → top-heavy fusion.
• Missing from a list = contributes 0. Appearing at moderate rank in both lists typically beats rank-1 in only one — the consensus effect that makes RRF work.
• Generalizes to n lists: multiple query rewrites, multiple retrievers (this is also the fusion step in RAG-Fusion).

Alternative: weighted score fusion. Normalize each list's scores (min-max or z-score), then blend: score = α·dense + (1−α)·sparse. Weaviate exposes exactly this alpha (1 = pure vector, 0 = pure BM25, default 0.75, with relativeScoreFusion as the modern default algorithm). More tunable than RRF — you can bias toward semantic or lexical per use case — but sensitive to score-distribution drift; RRF is the safer zero-tuning default.

Learned sparse (worth naming). Models like SPLADE produce sparse term-weight vectors with learned expansion (adds related terms it didn't see) — sparse infra with some semantic reach; supported natively by several vector DBs as a third retrieval arm.

Pipeline shape: query → [BM25 top-k] ∥ [vector top-k] → fuse (RRF/alpha) → optional reranker → top-n to the LLM. Fetch generously (k=50–100 per arm) since fusion + reranking will re-sort.

The flows

On the governed enterprise platform, policy and ticket search run hybrid by default: employees paste error strings and ask free-form questions. Entitlement filters apply to both arms so fusion never resurfaces unauthorized chunks.

A worked example

Corpus snippets (simplified):

Query 1 — paraphrase: "how do I reset my password?"

Query 2 — identifier: "ERR_CONN_RESET 10054"

Numeric RRF sketch (k=60) for a doc at rank 1 in BM25 and rank 5 in dense:

1/(60+1) + 1/(60+5) = 1/61 + 1/65 ≈ 0.01639 + 0.01538 ≈ 0.0318

A doc at rank 1 in only one list scores ≈ 0.0164 — consensus beats a single-list champion.

Fetch k=50 per arm, fuse, then rerank top 50 → keep 5 for the LLM (~extra 50–200 ms if a cross-encoder follows).

Common drill-downs

These are the questions a careful engineer asks one level down when designing retrieval.

Why isn't semantic search alone enough? Embeddings compress meaning, not tokens — exact identifiers (SKUs, error codes, names, citations), rare jargon, and out-of-vocabulary terms retrieve unreliably. BM25 nails exactly those via IDF-weighted exact match. Real query streams mix both types.

Walk me through the BM25 formula. Per query term: IDF (rarity) × a TF term that saturates — tf·(k1+1)/(tf + k1·(1−b+b·|d|/avgdl)). k1 (~1.2) controls how fast repeated occurrences stop adding score; b (~0.75) controls how much long documents are penalized. Sum over query terms.

What do k1 and b do, concretely? k1: term-frequency saturation — 0 makes TF binary (present/absent), higher values keep rewarding repetition longer. b: document-length normalization — 0 ignores length, 1 fully normalizes by |d|/avgdl. Defaults 1.2 / 0.75 are robust; tune only with a relevance test set.

Write down RRF and explain why it beats score averaging. RRF(d) = Σ 1/(k + rank_r(d)), k≈60. It's rank-based, so it never compares BM25 scores (unbounded) with cosine scores (bounded) directly — no normalization, no sensitivity to score-distribution drift, and consensus across lists is rewarded naturally.

What does k=60 actually control in RRF? The flatness of positional weight. Small k → rank 1 dominates (1/(1+1)=0.5 vs 1/(1+10)≈0.09); k=60 compresses the gap (0.0164 vs 0.0143) so agreement across retrievers outweighs a single retriever's top hit.

RRF vs alpha-weighted fusion — when each? RRF: zero-tuning, robust, works across any number of heterogeneous lists — the default. Alpha score-fusion: when you need explicit semantic/lexical bias per query class and can maintain score normalization; it's what Weaviate exposes and it edges out RRF when tuned on a golden set.

Give queries where keyword beats semantic, and the reverse. Keyword wins: ERR_CONN_RESET 10054, invoice INV-2024-00317, "section 80C", a person's name. Semantic wins: "my payment went through twice, how do I get money back" matching a "duplicate charge refund policy" doc sharing almost no tokens.

How does hybrid search interact with a reranker? Hybrid maximizes recall into the candidate pool from two complementary retrievers; the cross-encoder reranker then supplies precision over that pool. Fusion ordering matters less when a reranker follows — its job is mainly to get the right documents into the top 50–100.

Production concerns

• Two index infrastructures (inverted + vector) unless your store does both natively (Elasticsearch/OpenSearch, Weaviate, Qdrant, pgvector + Postgres FTS). Native beats stitching two systems: one consistency domain, one filter model, in-engine fusion. Landscape: vector databases.
• Latency = max(arms) + fusion (fusion is trivial). BM25 is rarely the bottleneck; run arms in parallel, budget the reranker separately. See latency.
• Tuning: alpha / k per query class matter less than analyzer correctness and candidate depth. Evaluate on a golden set: hybrid typically buys its biggest wins on identifier-heavy and jargon-heavy queries — measure recall@k per query type, not just in aggregate. See RAG evaluation.
• Failure modes: analyzer tokenizes IDs/code away (BM25 arm silently dead for exactly the queries it exists for); score fusion drifts when re-embedding shifts score distributions (RRF immune); one arm timing out — degrade to the other arm rather than failing the query. See reliability.
• Freshness asymmetry: inverted indexes update near-instantly; vector arms wait on embedding. Newly ingested docs may be keyword-findable before they're semantically findable — usually acceptable, worth knowing. See production RAG.
• When to skip hybrid: purely conversational queries over homogeneous prose (dense alone often suffices) or pure ID lookup (keyword alone). Hybrid costs little, so most teams default to it and tune weights instead of removing arms.

Test yourself

Go deeper

• Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods — Cormack, Clarke & Büttcher (SIGIR 2009) — the original RRF paper; two pages, worth reading.
• Practical BM25 — Part 2: The BM25 Algorithm and its Variables — Elastic — the best hands-on explanation of k1 and b, from the people who serve it at scale.
• Hybrid Search Explained — Weaviate blog — dense+sparse fusion and the alpha parameter in a real engine.
• A no nonsense intro to BM25 — Abhishek Thakur — applied BM25 walkthrough.
• What is reciprocal rank fusion? — Abhishek Thakur — RRF worked through by hand.

Where this connects

• Rerankers — hybrid maximizes recall into the pool; the cross-encoder maximizes precision on that pool.
• Retrieval patterns — multi-query and HyDE add more lists; RRF fuses them the same way.
• The RAG pipeline — hybrid sits in the online retrieve stage of the full ingest/query loop.
• Filtering and metadata — both arms must honor the same entitlement filters or fusion leaks.