RAG Evaluation

The intuition

Imagine debugging a two-stage factory where a shipped product came out wrong. Measuring only the final product ("customers are unhappy") never tells you which stage to fix. Was the wrong raw material delivered, or did assembly mishandle good material?

RAG has the same shape. Retrieval fetches the material; generation assembles it. A single end-to-end quality score conflates the stages, so it can never tell you what to do next:

Free-text evaluation feels hard because there is no single correct answer string. The trick is decomposition: break the answer into atomic claims and ask "is this claim supported by this chunk?" — a near-binary question a model can answer reliably. RAGAS turns that into ratios of small judgments rather than one holistic vibe score.

Why it exists

Three properties of RAG make the evaluation you already know inapplicable.

1. No assertions are possible. The same query can produce differently worded correct answers. assertEquals has nothing to compare.
2. Failures are fluent. A broken RAG returns a confident, grammatical, wrong paragraph — with a 200 and normal latency. Nothing in existing monitoring goes red.
3. Everything upstream silently shifts retrieval. Embedding upgrades, chunk-size tweaks, parser versions — each quietly changes what reaches the prompt. Without a regression gate, you find out from users.

So evaluation is not optional. It is the only thing between you and shipping unmeasured changes to a system that fails invisibly. "We tried it and it felt better" is not a release criterion. That is why this chapter puts evaluation before Production RAG.

The core idea

RAG is two systems glued together, so evaluate the retriever and the generator separately. An end-to-end "answer looks good" score cannot tell you whether to fix chunking or the prompt.

Retrieval metrics need labeled query → relevant-chunk pairs; they are deterministic and cheap. Hit rate / recall@k — fraction of queries where at least one relevant chunk appears in the top k; fix this first, because missing evidence sinks everything downstream. MRR (mean reciprocal rank) — mean of 1/rank of the first relevant chunk; rewards putting the answer near the top, where the generator attends most. nDCG@k (normalized discounted cumulative gain) — rank-weighted gain over all relevant chunks with graded relevance; the search-team workhorse.

Generation and context metrics (the classic RAGAS quartet) each isolate one failure. Faithfulness: every claim supported by retrieved context? Low = hallucination despite evidence. Answer relevancy (RAGAS: Response Relevancy): does the answer address the question? Faithful yet off-topic is possible. Context precision: of what was retrieved, how much was relevant and ranked high? Context recall: did retrieval fetch everything needed for the reference answer?

Precision/recall diagnose the retriever; faithfulness/relevancy diagnose the generator. Low context recall → fix retrieval. High recall but low faithfulness → fix generation. Generation metrics use LLM-as-judge — scalable, biased, itself a model you must calibrate. Run everything against a golden dataset offline before ship, plus cheaper online signals (thumbs, refusal rate) in production.

How it actually works

Every score is a ratio of verifiable sub-judgments, not a 1–10 vibe — when faithfulness is 0.6 you can list the two claims that failed.

For labeled query set Q: hit rate@k = |{q : top-k contains a relevant chunk}| / |Q|. MRR = mean(1/rank of first relevant), 0 if absent. nDCG@k = DCG@k / ideal-DCG@k where DCG = Σ rel_i / log2(i+1). Use hit rate to size k, MRR when one chunk answers, nDCG when several chunks matter with graded relevance. Run these on every chunking, embedding, or reranker change.

Golden dataset. Seed with real user queries from logs. Synthesize to scale: generate questions from corpus chunks so the source chunk is automatically the retrieval label; evolve some into multi-hop variants. Human-review a stratified sample; cover hard classes (multi-hop, no-answer-exists, ambiguous, time-sensitive). Version the set and grow it from every escalated production failure. Fifty to two hundred curated examples beat five thousand noisy ones — this set is your CI gate.

LLM-as-judge failure modes (Zheng et al., MT-Bench): position bias (favor first answer in pairwise — judge both orders); verbosity bias (longer scores higher — decompose claims); self-preference (favor own model family — use a different family than the generator); limited reasoning on subtle factual/math errors (give references, keep checks atomic); non-determinism and version drift (temperature 0, version-pin the judge, re-baseline on upgrade). Meta-rule: label 50–100 examples by hand, measure judge–human agreement, trust the judge only where agreement is high.

The flows

Offline golden-set eval is the release gate. On every chunking, embedding, prompt, model, or index change: run the versioned set, compute retrieval metrics plus context precision/recall, generate, score faithfulness and relevancy, fail the merge if thresholds break. Breaks when the golden set no longer matches traffic, when you forget to re-embed after an embedding-model change, or when you optimize one metric alone (faithfulness → always refuse).

Online continuous scoring runs where there is no ground truth. Sample live requests, score reference-free metrics, watch thumbs / refusals / escalations / retrieval-score distributions, flag outliers for human review, feed confirmed failures into the golden set. Breaks when you judge every request (cost), the judge is uncalibrated, or you ignore distribution shift.

Diagnose-then-fix when quality is bad: read the 2×2 (context recall/precision vs faithfulness/relevancy), localize the stage, change one thing, re-run the golden set. Breaks when you only have end-to-end vibe scores, or you fix generation while retrieval never fetched the evidence.

A worked example

One query from the governed enterprise platform's golden set:

query: "What's the approval threshold for international travel, and who approves it?"
relevant_chunks: ["policy-2026#sec3-eligibility", "policy-2026#sec7-approvals"]
reference_answer: >
  International travel requires pre-approval above the stated threshold.
  Approval sits with the employee's department head; finance counter-signs
  above twice that amount.

Run 1 — as shipped (top-5 retrieve, then generate):

Hit rate alone would have said "pass." Context recall catches the half-answered two-part question. Given only half the evidence, the generator filled the gap from parametric memory — exactly what low faithfulness with high relevancy looks like.

Two bugs, two owners. Retrieval: k=5 returned five chunks all from section 3 — raise k, multi-query so "who approves" retrieves separately, or dedupe by section. Generation: the prompt did not stop inventing the missing half — instruct the model to name which part is missing when context does not contain it.

Run 2 — fix retrieval only: context recall 1.0, faithfulness 0.95. Run 3 — fix generation only: recall stays 0.5, faithfulness 1.0, but the answer is "I don't have information on who approves" — honest, incomplete, user still unserved.

Production concerns

Offline vs online. Offline is CI for the RAG stack: full metric suite on the golden set for every change. Online has no ground truth, so sample — typically 1–5% of traffic for LLM judging — and lean on cheaper proxies: thumbs, answer/refusal rates, retrieval-score distributions, escalation rate, citation click-through. See Observability.

Cost and gaming. Every judged answer is 1–3 extra LLM calls — sample hard, cheap judge for triage, strong judge for the flagged tail, cache identical (query, context) pairs. Optimize faithfulness alone and the system learns permanent refusal; always pair it with relevancy and refusal rate. Gate merges on golden-set thresholds the way you gate unit tests (e.g. recall@5 must not drop more than a few points — pick a threshold your product can defend). See Evals & Testing.

Drift. Corpus and query mix both move. They decay the system and the relevance of your golden set. Refresh the set from live traffic on a schedule; monitor retrieval-score distributions. Track judge–human agreement over time — the judge is a model in production too.

Common drill-downs

Faithfulness vs answer relevancy — why both? Faithfulness is anti-hallucination (claims supported by context). Relevancy asks whether the answer addresses the question. They are orthogonal: fully faithful yet off-topic is possible; a relevant answer can be fabricated. Empty or refusal answers are vacuously faithful — that is the gaming path.

Context precision vs context recall — what do you change? Low recall → cast wider (higher k, hybrid search, better chunking). Low precision → tighten (reranker, filters, smaller k). They trade off through k, so report both. Hit rate@k is the binary go/no-go; MRR cares about the first relevant rank; nDCG is for graded multi-chunk ranking.

Offline scores great, users complain? Golden set no longer matches traffic; corpus stale relative to the eval snapshot; judge miscalibrated; or users complain about latency/format/tone — dimensions your metrics never measured. Mine complaints into the eval set, re-calibrate the judge, add the missing dimensions.

Test yourself

Go deeper

• Ragas — available metrics — definitions of faithfulness, response relevancy, context precision, context recall.
• Ragas on GitHub — reference implementation; reading the metric prompts is the fastest way to understand the scores.
• Judging LLM-as-a-Judge — Zheng et al., 2023 — position, verbosity, and self-enhancement biases, with mitigations.
• Your AI Product Needs Evals — Hamel Husain — eval infrastructure as the core of LLM product iteration.
• Systematically Improving Your RAG — Jason Liu — synthetic baselines, leading vs lagging metrics, feedback loops.
• A complete guide to RAG evaluation — Evidently — offline vs production monitoring, and how test sets evolve with the product.

Where this connects

• Evals & Testing — the same discipline for every LLM feature: CI, canaries, prompt regression suites.
• Observability — the online half: tracing, sampled judging, drift detection without ground truth.
• Production RAG — what you gate with these numbers on every index change.
• Ingestion & Chunking — experiments that are meaningless without this harness.