Rerankers

The intuition

First-stage retrieval is a speed-dating setup: each document wrote a short bio (one vector) ahead of time; the query writes its own bio; you match bios that sound similar. That scales — bios are precomputed — but it is shallow. Two people can share topics and still be a bad answer to the actual question.

A reranker sits both people in the same room and lets them talk. The model reads the query and the document together, token by token, and only then scores relevance. That is far more accurate and far too expensive to do with every document in the warehouse.

So production systems use two stages: a cheap wide net (bi-encoder / hybrid) for recall, then an expensive careful read (cross-encoder) for precision on ~50–150 candidates. The top handful go to the LLM.

Why it exists

Embedding similarity answers "are these roughly about the same topic?" Product answers need "does this chunk actually answer this question?"

Hard constraints:

1. Single-vector compression is lossy. Negation, constraint details, and "which of these policies applies" need token-level interaction the bi-encoder never computes at query time.
2. Cross-encoders do not index. The score is a function of the pair; nothing about documents can be fully precomputed. 10M docs ⇒ 10M forward passes per query — unusable as a sole retriever.
3. LLM context is scarce. Stuffing 50 mediocre chunks wastes tokens, confuses the model, and raises cost; 5 sharp chunks usually win.
4. Stage-1 recall bounds stage-2. A reranker cannot promote a document that never entered the candidate set.

Alternatives lose. Larger bi-encoders only help but keep the independent-encoding bottleneck. Cross-encoder-only retrieval does not scale. Skipping rerank and sending raw top-5 cosine hits is the common mediocre RAG baseline — often the single highest-leverage upgrade is adding stage-2.

The core idea

Embedding retrieval is a bi-encoder: query and document are encoded separately into fixed vectors, and relevance is approximated by their similarity. That separation is what makes it scalable — documents are embedded once, offline, and a query needs one model call plus an ANN lookup. But it's also the accuracy ceiling: each side is compressed to a single vector with no knowledge of the other, so the model can only measure "generally about the same topic," not "actually answers this question."

A cross-encoder (reranker) feeds the query and a candidate document together through one transformer. Every query token attends to every document token, so the model sees exact matches, negation, and whether the document addresses the query's intent — and outputs a calibrated relevance score. It's substantially more accurate, and completely unscalable as a retriever: scoring a query against 10M documents means 10M full forward passes, and nothing can be precomputed because the input is the pair.

Production systems get both via two-stage retrieval: stage 1, a fast retriever (vector or hybrid) over-fetches ~50–150 candidates optimizing recall; stage 2, the cross-encoder rescores just those candidates for precision, and the top 3–10 go into the LLM context. Typical cost is 50–200 ms extra latency for a large relevance gain — usually the single highest-leverage upgrade to a mediocre RAG pipeline, because it both fixes ordering and lets you send fewer, better chunks to the model.

How it actually works

Two-stage funnel from corpus to prompt:

Architecture difference, precisely.

Why cross-encoders win on quality: the bi-encoder must anticipate every future query when compressing a document into one vector — an information bottleneck. The cross-encoder defers judgment until it sees the actual query and can compare token-by-token. Cross-encoders are typically trained on labeled relevance data (e.g. MS MARCO) as classifiers/regressors over pairs; listwise variants score candidates in context of each other.

Two-stage retrieval mechanics.

1. Retrieve top-K candidates (K = 50–150) with vector/hybrid search. K is a recall knob: the reranker cannot surface what stage 1 missed.
2. Score all (query, candidate) pairs with the cross-encoder — batched on GPU or a single API call taking the query + document list.
3. Take top-n (3–10) by reranker score into the prompt. Optionally threshold on the calibrated score to drop weak hits entirely — useful for "say I don't know" behavior.

Late interaction (ColBERT) — the middle ground. Encode documents to per-token vectors offline; at query time compute MaxSim: for each query token take its max similarity over the document's token vectors and sum. Token-level interaction (better than single-vector) with offline doc encoding (unlike a cross-encoder), at the price of storing hundreds of vectors per document. Worth naming as the third point on the accuracy/cost spectrum.

Current options (2026).

The flows

On the governed enterprise platform, the online path is hybrid top-100 → managed or self-hosted cross-encoder → top 5 entitlement-safe chunks into the generator, with degrade-to-hybrid-order if the rerank dependency fails.

A worked example

Employee query on the platform:

"Does the new expense policy still cover international travel?"

Stage 1 (hybrid, K=50) returns chunks including:

Stage 2 (cross-encoder) scores each (query, chunk) pair jointly (~100–150 ms for 50 pairs via API/GPU batch). Illustrative scores:

Prompt packing: top 5 by rerank (~2,000 tokens) + "answer only from context; cite chunks." Generator answers with pre-approval + per-diem change, citing the right chunks.

Latency budget sketch: embed ~20 ms + ANN/BM25 ~30 ms + fuse ~1 ms + rerank ~120 ms + generate ~2 s → retrieval still a minority of end-to-end time.

Common drill-downs

These are the questions a careful engineer asks one level down when precision stalls.

Bi-encoder vs cross-encoder — the core difference? Bi-encoder encodes query and doc independently into vectors, relevance ≈ vector similarity — precomputable, ANN-scalable, but no query-doc token interaction. Cross-encoder encodes the pair jointly with full cross-attention and outputs a relevance score — far more accurate, nothing precomputable, O(N) forward passes.

Why exactly are cross-encoders more accurate? The bi-encoder compresses each document into one vector before knowing the query — an information bottleneck that can only capture general topicality. The cross-encoder sees both texts at once, so attention can align specific tokens, catch negation, and judge whether the doc answers the actual question.

Why can't you just use a cross-encoder for retrieval? No index structure applies: the score is a function of the pair, so nothing about documents can be precomputed. 10M docs = 10M transformer passes per query — seconds-to-minutes of GPU per query vs milliseconds for ANN.

Describe two-stage retrieval and how you'd size each stage. Stage 1: fast retriever (hybrid/vector) fetches top-K for recall; stage 2: cross-encoder rescores K for precision; top 3–10 go to the LLM. Size K where stage-1 recall@K plateaus on a golden set (often 50–150); size final n by context budget and answer-quality evals.

Retrieval quality is poor. Reranker, better embeddings, or better chunking — how do you decide? Diagnose with recall@K of stage 1: if the right chunks are in the top-100 but not the top-5, a reranker fixes ordering cheaply. If they're not in the top-100 at all, reranking can't help — fix chunking, embeddings, or add hybrid search. Measure before buying.

What is late interaction / ColBERT? Store per-token embeddings offline; score by MaxSim (sum over query tokens of max similarity to doc tokens). Token-level matching without per-query doc encoding — between bi- and cross-encoder in both accuracy and cost, but storage-heavy (hundreds of vectors per doc).

When would you use an LLM as the reranker? When ranking criteria go beyond relevance (recency, source authority, "prefer official docs") or for low-QPS/offline pipelines — listwise prompting is flexible but 10–100× slower and pricier than a purpose-built cross-encoder, which wins for standard relevance at scale.

How do you evaluate whether the reranker is earning its latency? A/B on a golden set: nDCG@10/MRR uplift stage-1 vs reranked, then end-to-end answer correctness with the same context budget. Keep it if quality gain is material at acceptable p95 latency and cost; drop K until the gain starts eroding.

Production concerns

• Latency budget: reranking 100 candidates adds ~50–200 ms (API or GPU-batched). Tune candidate count K — the latency/cost lever — by measuring where recall@K plateaus; reranking 1000 candidates rarely beats reranking the right 100. See latency.
• Cost: rerank APIs bill per query×candidates; at high QPS this rivals generation cost. Levers: smaller K, "fast/lite" model tiers, caching (query, doc)-scores for repeated queries, self-hosting bge on a modest GPU. See cost.
• Document length: cross-encoders have context limits and quality degrades on very long docs; rerank at chunk granularity, or use models with explicit long-doc handling/chunk-max aggregation. Chunk design: ingestion and chunking.
• Score semantics: reranker scores are better calibrated than cosine similarity, so absolute thresholds ("drop below 0.2") are actually usable — but still validate per model/domain. Contrast with similarity metrics.
• Failure isolation: the reranker is a separate network/model dependency; on timeout, degrade gracefully to stage-1 ordering rather than failing retrieval. See reliability.
• Evaluation: measure the delta — nDCG@10 / MRR with and without reranking on a golden set, plus end-to-end answer quality. If stage-1 recall@K is low, fix retrieval first; a reranker cannot rescue candidates that never arrived. See RAG evaluation and evals and testing.

Test yourself

Go deeper

• Rerankers and Two-Stage Retrieval — Pinecone — the canonical applied writeup of why and how, with code.
• Cross Encoders — Sentence Transformers docs — reference open-source implementation and usage patterns.
• Rerank — Cohere docs — a production rerank API: models, limits, integration shape.
• Advanced RAG 04 — Reranking with Cross Encoders, and Cohere API — Sam Witteveen — popular hands-on video comparing reranking approaches in a RAG pipeline.

Where this connects

• Hybrid search — the usual stage-1 that maximizes complementary recall before you rerank.
• The RAG pipeline — where rerank sits between retrieve and augment/generate.
• RAG evaluation — how to prove the reranker earned its latency with nDCG/MRR and answer metrics.
• Latency — budgeting the 50–200 ms stage-2 tax against generation.