Hybrid Search

The intuition

Two librarians share the same warehouse.

The first is a linguist: she finds documents about the same idea even when the words differ — "reset my password" lands on "credential recovery procedure." She is weak on serial numbers.

The second is a filing clerk: he matches exact tokens with ruthless statistical weights. SKUs, error codes, and legal citations are his superpower. He is blind to synonyms.

Hybrid search runs both in parallel and merges their ranked lists. Documents both like rise; documents only one finds still surface. That is the robust default over enterprise corpora — technical docs, tickets, catalogs — where the next query might be a paraphrase or an identifier.

Why it exists

Semantic search alone is not enough for production query streams.

1. Embeddings compress meaning, not tokens. Part numbers, error codes, names, and unseen jargon retrieve unreliably in pure vector space.
2. Keyword search has zero paraphrase understanding. "My payment went through twice" misses "duplicate charge refund policy" when tokens barely overlap.
3. Raw scores live on incomparable scales. BM25 is unbounded; cosine is bounded. Naive averages drift when models change.
4. User queries mix both modes. You do not choose whether the next request is semantic or lexical.

Dense only misses identifiers. BM25 only misses paraphrase. Naive score blending breaks when re-embedding shifts distributions. Reciprocal Rank Fusion (RRF) merges ranked lists using ranks alone — no score calibration required.

The core idea

Hybrid search runs dense (embedding) and sparse (keyword) retrieval in parallel, then fuses the two ranked lists. Dense matches meaning but is unreliable on exact tokens. Sparse — in practice BM25, the ranking function behind Elasticsearch and most search engines — matches exact terms with statistical weighting, is cheap and explainable, and needs no training, but has no synonym understanding.

The usual merge is RRF: each document scores Σ 1/(k + rank) across the lists it appears in (k ≈ 60 by convention). RRF uses only ranks, never raw scores, so BM25 and cosine never need a common scale. Dual-list hits float up; single-arm hits still surface.

How it actually works

BM25 at survival depth. For query q and document d:

score(q, d) = Σ over terms t in q:  IDF(t) · ( tf(t,d) · (k1 + 1) ) / ( tf(t,d) + k1 · (1 − b + b · |d|/avgdl) )

IDF(t) is why an exact SKU is retrieval gold: rare terms weigh more (ln( (N − df(t) + 0.5) / (df(t) + 0.5) + 1 ) is the usual form).

k1 (default ~1.2) is term-frequency saturation — the 2nd occurrence adds a lot, the 20th almost nothing. That is BM25's main upgrade over raw TF-IDF, which grows linearly and lets keyword-stuffed documents win. b (default 0.75) is length normalization relative to the corpus average (b=1 full, b=0 none).

Scoring runs over an inverted index (term → posting list). Analyzers decide what counts as a term — misconfigured ones (splitting ERR-1042) are a classic silent failure.

Reciprocal Rank Fusion:

RRF(d) = Σᵣ 1 / (k + rankᵣ(d))        k = 60 by convention

Rank-based means no score normalization. k damps top ranks: with k=60, rank 1 ≈ 0.0164 and rank 10 ≈ 0.0143 — top positions matter, but no single list dominates. Smaller k is more top-heavy. Missing from a list contributes 0. Moderate rank in both lists typically beats rank-1 in only one — the consensus effect. The same formula fuses any number of lists (query rewrites, multi-retriever setups like RAG-Fusion).

Alternative: weighted score fusion. Normalize each list, then blend: score = α·dense + (1−α)·sparse. Higher alpha → more vector weight. Weaviate's modern default is relativeScoreFusion (normalize then blend; default since v1.24); rankedFusion (RRF-style) remains available — pin fusionType across upgrades. Alpha is tunable but drifts after re-embedding; RRF is the safer zero-tuning default without a golden set.

Learned sparse. Models like SPLADE produce sparse term-weight vectors with learned expansion — sparse infra with some semantic reach. Several vector DBs support this as a third arm.

Pipeline. Query → [BM25 top-k] ∥ [vector top-k] → fuse → optional reranker → top-n to the LLM. Fetch generously (k=50–100 per arm); fusion and reranking re-sort.

The flows

On the governed enterprise platform, policy and ticket search run hybrid by default — employees paste error strings and ask free-form questions. Entitlement filters apply to both arms so fusion never resurfaces unauthorized chunks.

A worked example

Paraphrase ("how do I reset my password?"): BM25 is weak if tokens barely overlap; dense ranks A first; RRF keeps A on top.

Identifier ("ERR_CONN_RESET 10054"): BM25 ranks B first (rare tokens, huge IDF); dense may return network-ish docs without guaranteeing B; RRF puts B on top via BM25.

Numeric RRF (k=60), BM25 rank 1 + dense rank 5: 1/61 + 1/65 ≈ 0.0318 vs ≈ 0.0164 for a single-list rank-1 — consensus beats a lonely champion. Fetch k=50 per arm, fuse, optionally rerank → keep ~5 for the LLM.

Production concerns

You usually need two index infrastructures (inverted + vector) unless the store does both natively — Elasticsearch/OpenSearch, Weaviate, Qdrant, or Postgres FTS + pgvector. Native beats stitching: one consistency domain, one filter model, in-engine fusion. See vector databases.

Latency ≈ max(arms) + fusion. Fusion is trivial; run arms in parallel and budget the reranker separately (latency). Tuning alpha or RRF's k matters less than analyzer correctness and candidate depth — measure recall@k per query type on a golden set (RAG evaluation).

Failure modes that bite: analyzers that tokenize IDs away; score fusion that drifts after re-embedding (RRF is immune); one arm timing out with no degrade path (reliability). Freshness is asymmetric: inverted indexes update near-instantly; the vector arm waits on embedding and ANN upsert, so new docs may be keyword-findable first (production RAG).

When to skip hybrid: homogeneous conversational prose (dense often suffices) or pure ID lookup (keyword alone). Hybrid costs little, so most teams default to it.

Common drill-downs

What do k1 and b do? k1: term-frequency saturation (0 = binary presence; higher keeps rewarding repetition). b: length normalization (0 = ignore length; 1 = full |d|/avgdl). Defaults 1.2 / 0.75 are robust.

What does k=60 control in RRF? Positional weight flatness. Small k → rank 1 dominates; k=60 compresses the gap so multi-list agreement outweighs a single top hit.

RRF vs alpha fusion — when each? RRF for zero-tuning robustness. Alpha / relative-score fusion when you need explicit semantic vs lexical bias and can re-tune after embedding changes. Pin the vendor fusion algorithm across upgrades.

Hybrid + reranker? Hybrid maximizes recall into the pool; the cross-encoder supplies precision. After a reranker, fusion order matters less than whether the right doc entered the top 50–100.

Test yourself

Go deeper

• Reciprocal Rank Fusion — Cormack, Clarke & Büttcher (SIGIR 2009) — original RRF paper; two pages.
• Practical BM25 — Part 2 — Elastic — hands-on k1 and b.
• Hybrid search (concepts) — Weaviate docs — rankedFusion vs relativeScoreFusion and alpha.
• Weaviate fusion algorithms — why relativeScoreFusion became default in v1.24.
• Hybrid search demo — Qdrant — dense + sparse prefetch with RRF/DBSF.

Where this connects

• Rerankers — hybrid maximizes recall into the pool; the cross-encoder maximizes precision on that pool.
• Retrieval patterns — multi-query and HyDE add more lists; RRF fuses them the same way.
• The RAG pipeline — hybrid sits in the online retrieve stage of the full ingest/query loop.
• Filtering and metadata — both arms must honor the same entitlement filters or fusion leaks.