Rerankers
Bi-encoder vs cross-encoder, and two-stage retrieval.
Prerequisites
- Embeddings — bi-encoder retrieval is what produces the candidate pool.
- Hybrid search — production stage-1 is often dense + BM25 fused, not dense alone.
- ANN indexes — why stage-1 can scan millions of vectors in milliseconds.
The intuition
First-stage retrieval is speed-dating: each document wrote a short bio (one vector) ahead of time; the query writes its own; you match bios that sound similar. That scales — bios are precomputed — but two texts can share topics and still fail the actual question.
A reranker puts both texts in the same room. It reads the query and the document together, then scores relevance. Far more accurate, and far too expensive for the whole warehouse. So production systems use two stages: a cheap wide net for recall, then an expensive careful read of ~50–150 candidates for precision. Only the top handful reach the LLM.
Key insight
Stage 1 optimizes recall (is the answer in the set?). Stage 2 optimizes precision (is the best answer at the top?). Flip the order and you either lose the document forever or blow the latency budget.
Why it exists
Embedding similarity answers "are these roughly about the same topic?" Product answers need "does this chunk answer this question?"
Four hard constraints force the two-stage shape:
- Single-vector compression is lossy. Negation, fine constraints, and "which policy applies" need token-level interaction the bi-encoder never computes at query time.
- Cross-encoders do not index. The score is a function of the pair — ten million docs would mean ten million forward passes per query.
- LLM context is scarce. Fifty mediocre chunks waste tokens; five sharp ones usually win.
- Stage-1 recall bounds stage-2. A reranker cannot promote a document that never entered the candidate set.
Larger bi-encoders alone keep the independent-encoding bottleneck. Cross-encoder-only retrieval does not scale. Skipping stage 2 and shipping raw top-5 cosine hits is the common mediocre RAG baseline — adding a reranker is often the highest-leverage upgrade.
The core idea
A bi-encoder embeds the query and each document separately into fixed vectors, then approximates relevance with cosine or dot product. Documents are embedded once offline; a query needs one encode plus an ANN lookup. That separation makes retrieval scalable — and it is also the accuracy ceiling, because each side is compressed without seeing the other.
A cross-encoder (the usual reranker) feeds the query and one candidate together through a transformer — typically as [CLS] query [SEP] document — so every query token can attend to every document token. It outputs a single relevance score for the pair. Nothing about documents is precomputed, so you pay a full forward pass per candidate.
Two-stage retrieval buys both: stage 1 (vector, BM25, or hybrid) over-fetches K candidates for recall; stage 2 rescores those pairs for precision; the top n (often 3–10) go into the prompt. Typical cost is on the order of 50–200 ms extra.
How it actually works
| Bi-encoder | Cross-encoder | |
|---|---|---|
| Input | query and doc encoded independently | one sequence: [CLS] query [SEP] document |
| Interaction | none until similarity | full cross-attention at every layer |
| Output | two vectors → cosine/dot | single relevance logit/score |
| Precompute docs? | yes | no — score depends on the pair |
| Query cost, N docs | 1 encode + ANN | N forward passes |
Cross-encoders are typically trained on labeled relevance data (MS MARCO and similar). Pointwise models score each pair alone; listwise models see several candidates in one context and can suppress near-duplicates.
Mechanics. Retrieve top-K with vector or hybrid search (K is a recall knob — often 50–150 where recall@K plateaus). Score all (query, candidate) pairs, batched on GPU or via one API call. Keep top-n by score for the prompt. Optionally threshold a calibrated score when "I don't know" beats a weak answer.
Late interaction (ColBERT) sits between the extremes. Encode documents to per-token vectors offline; at query time compute MaxSim (for each query token, max similarity over the document's token vectors, then sum). Token-level interaction without a full joint encode — at the price of many vectors per document.
Options (as of 2026; verify current names and limits before shipping): Cohere Rerank 4 (rerank-v4.0-pro / fast, API); Voyage rerank-2.5 / 2.5-lite (API, instruction-following); BAAI bge-reranker-v2-m3 (open ~0.6B, common self-host); Jina jina-reranker-v3 (listwise; check license for commercial use); cross-encoder/ms-marco-MiniLM (tiny CPU baseline); LLM-as-reranker via listwise prompting (offline or low QPS only).
Common misconception
"Add a reranker" does not fix bad stage-1 recall. If the right chunk is not in the top-K, stage 2 only reorders failure. Measure recall@K of stage 1 before buying latency.
The flows
| Flow | Sequence | When | What breaks it |
|---|---|---|---|
| Happy path | hybrid/vector top-K → batch cross-encode → top-n to LLM | Default quality-sensitive RAG | K too small; stage-1 miss; no degrade on timeout |
| Abstain / threshold | rerank → drop below calibrated score → refuse if empty | High-stakes; weak context worse than none | Uncalibrated thresholds; treating cosine as a rerank score |
| Degraded stage-1 only | reranker error → serve stage-1 / hybrid RRF order | Partial outages | Fail-closed on rerank errors |
| Offline / LLM listwise | retrieve K → LLM orders with custom criteria | Low QPS, editorial rules, batch jobs | High QPS online; cost explosion |
| Late-interaction serve | MaxSim over token vectors → optional cross-encoder shortlist | Single-vector stage-1 too weak | Storage blowup (many vectors per doc) |
On the governed enterprise platform: hybrid top-100 → cross-encoder → top 5 entitlement-safe chunks, with degrade-to-hybrid-order if rerank fails.
A worked example
Employee query on the platform:
"Does the new expense policy still cover international travel?"
Stage 1 (hybrid, K=50) surfaces international travel pre-approval (true answer, rank 1), domestic mileage (shared token "travel"), per-diem tables, and an old draft international policy at rank 17 (semantic near-dupe, wrong version).
Stage 2 scores each pair. Illustrative scores: pre-approval 0.91 (keep), per-diem 0.74 (keep), old draft 0.48 (demote — version filters should have removed it earlier), domestic mileage 0.22 (drop from top-5).
Prompt packing: top 5 by rerank (~2,000 tokens) plus "answer only from context; cite chunks." The generator cites pre-approval and per-diem correctly.
Illustrative latency: embed ~20 ms + ANN/BM25 ~30 ms + fuse ~1 ms + rerank ~120 ms + generate ~2 s → retrieval is a minority of end-to-end time when generation dominates.
What each omission looks like in production
- No reranker → domestic mileage stays high; the model hedges or picks wrong numbers.
- K=10 only → the true chunk at stage-1 rank 17 never enters; reranking rearranges a bad set.
- Rerank 1,000 candidates → latency and cost spike with little gain past the recall plateau.
- No degrade path → a rerank API blip fails the whole answer though stage-1 was fine.
- Rerank 20k-token documents → truncation and score noise. Rerank chunks.
Production concerns
Reranking K candidates typically adds on the order of 50–200 ms. K is the main lever: measure where stage-1 recall@K plateaus and stop there — reranking 1,000 rarely beats the right 100 (latency). At high QPS, API pricing per query × candidates can rival generation: shrink K, use a fast/lite tier, cache (query, doc_id) scores, or self-host bge-reranker-v2-m3 (cost).
Score chunks, not whole long documents — cross-encoders truncate and quality degrades (ingestion and chunking). Reranker scores are usually better calibrated than cosine, so absolute thresholds can work after domain validation (similarity metrics). On timeout, degrade to stage-1 order rather than failing the request (reliability). Prove the stage earns its keep with nDCG@10 / MRR with and without reranking, plus end-to-end answer quality at a fixed context budget. If stage-1 recall@K is low, fix retrieval first (RAG evaluation).
Common drill-downs
How do you size K and final n? Size K where stage-1 recall@K plateaus (often 50–150). Size n by context budget and answer-quality evals (often 3–10). Raising n without raising quality just stuffs the context window.
Retrieval quality is poor. Reranker, better embeddings, or better chunking? Measure stage-1 recall@K. Gold chunks in top-100 but not top-5 → reranker fixes ordering. Not in top-100 at all → fix chunking, embeddings, or hybrid first; stage 2 cannot invent missing candidates.
Pointwise vs listwise — when does the difference matter? Pointwise scores each pair alone. Listwise models see candidates jointly and can demote near-duplicates. On redundant policy corpora that difference shows up; on clean disjoint candidates both often agree. Evaluate on your redundancy profile.
When is late interaction worth the storage? When single-vector stage-1 recall is weak and you need token-level matching at higher QPS than a large cross-encoder shortlist allows. Expect many vectors per document; many systems still put a small cross-encoder on the late-interaction shortlist.
When is LLM-as-reranker justified? When ranking goes beyond pure relevance (prefer final over draft, system of record, legal holds) or for offline/low-QPS jobs. For standard relevance at interactive QPS, a purpose-built cross-encoder is typically far cheaper and faster.
Test yourself
nDCG@10 barely moves after adding a reranker, but stage-1 ordering looked messy. What do you measure next?
p95 latency jumps ~300 ms after rerank-100. Product allows +100 ms. What do you change before abandoning the idea?
Why might listwise and pointwise models disagree on the same candidate set?
A teammate wants cross-encoder-only over the full corpus 'for maximum quality.' Respond with scaling math.
When is LLM-as-reranker justified on the governed enterprise platform?
Go deeper
- Rerankers and Two-Stage Retrieval — Pinecone — applied walkthrough of why two-stage exists, with code.
- Cross Encoders — Sentence Transformers — open-source usage for pair scoring and ranking.
- Rerank overview — Cohere — production API shape, models, multilingual notes.
- Rerankers — Voyage AI — API rerankers including instruction-following variants.
- ColBERT (arXiv:2004.12832) — late interaction: MaxSim over offline token vectors.
Where this connects
- Hybrid search — the usual stage-1 that maximizes complementary recall before you rerank.
- The RAG pipeline — where rerank sits between retrieve and generate.
- RAG evaluation — how to prove the reranker earned its latency with nDCG/MRR and answer metrics.
- Latency — budgeting the stage-2 tax against generation.