RAG Evaluation

The intuition

Imagine debugging a two-stage factory where a shipped product came out wrong. You can measure the final product ("customers are unhappy") but that tells you nothing about which stage to fix. Was the wrong raw material delivered, or did assembly mishandle good material?

RAG has exactly this shape. Retrieval fetches the material; generation assembles it. A single end-to-end quality score conflates them, so it can never tell you what to do next:

Every metric on this page exists to fill in one cell of that table.

Why it exists

Three properties of RAG make the evaluation you already know inapplicable.

1. No assertions are possible. The same query on the same index can produce differently-worded correct answers. assertEquals has nothing to compare.
2. Failures are fluent. A traditional service fails with a 500. A broken RAG system returns a confident, grammatical, well-formatted, wrong paragraph — with a 200 and normal latency. Nothing in your existing monitoring goes red.
3. Everything upstream silently shifts retrieval. An embedding-model upgrade, a chunk-size tweak, a new parser version — each quietly changes what reaches the prompt. Without a regression gate, you find out from users.

So evaluation isn't a nice-to-have measurement layer here; it's the only thing standing between you and shipping unmeasured changes to a system that fails invisibly. "We tried it and it felt better" is not a release criterion.

The core idea

The framing that matters: RAG is two systems glued together, so evaluate the retriever and the generator separately — an end-to-end "answer looks good" score can't tell you whether to fix chunking or the prompt.

Retrieval metrics (deterministic, need labeled query→relevant-chunk pairs):

• Hit rate / recall@k — fraction of queries where a relevant chunk appears in the top k. The first number to fix: if the evidence never arrives, nothing downstream matters.
• MRR — mean of 1/rank of the first relevant chunk; rewards putting the answer near the top, where the generator attends most.
• nDCG@k — rank-weighted gain over all relevant chunks with graded relevance; the finest-grained, standard in search teams.

Generation metrics (the RAGAS quartet — each isolates one failure):

• Faithfulness — is every claim in the answer supported by the retrieved context? Low = hallucination despite the evidence.
• Answer relevancy — does the answer actually address the question? (Can be faithful yet off-topic.)
• Context precision — of what was retrieved, how much was relevant, and was the relevant material ranked high? Low = noisy context diluting the prompt.
• Context recall — did retrieval fetch everything needed for the reference answer? Low = missing evidence → incomplete answers.

Precision/recall diagnose the retriever; faithfulness/relevancy diagnose the generator. A 2×2 diagnosis grid: low context recall → fix retrieval (chunking, hybrid search, k); high recall but low faithfulness → fix generation (grounding prompt, model, context packing).

Since judging "is this claim supported?" needs language understanding, these are scored by LLM-as-judge, which is scalable but has known biases you must name (position, verbosity, self-preference) and must itself be validated against human labels. Everything runs against a golden dataset offline before ship, plus cheaper online signals (thumbs, deflection, refusal rate) in production.

How it actually works

How RAGAS computes each score (know the mechanism, not just the name):

Note the design: every metric is a ratio of verifiable sub-judgments (claim-level checks), not one holistic 1–10 vibe score — decomposition makes LLM judging far more reliable and auditable.

Retrieval metrics at survival depth. For query set Q with labeled relevant chunks: hit rate@k = |{q : top-k contains a relevant chunk}| / |Q|. MRR = mean(1/rank_first_relevant), 0 if absent. nDCG@k = DCG@k / ideal-DCG@k where DCG = Σ rel_i / log2(i+1) — graded relevance, all positions count. Use hit rate to size k, MRR when one chunk suffices, nDCG when multiple chunks with degrees of relevance matter. These are cheap and deterministic — run them on every chunking/embedding/reranker change.

Golden datasets — how you actually build one:

1. Seed with real user queries from logs (or stakeholder conversations pre-launch) — synthetic-only sets miss how users actually phrase things.
2. Synthesize to scale: LLM generates questions from corpus chunks (RAGAS test-set generation does this, evolving simple questions into multi-hop/reasoning variants); the source chunk is automatically the retrieval label.
3. Human-review a stratified sample; discard unanswerable or trivial generations.
4. Cover the hard classes deliberately: multi-hop, no-answer-exists (tests refusal), ambiguous, time-sensitive.
5. Version it and grow it from production failures — every escalated bad answer becomes a regression test. 50–200 curated examples beat 5,000 noisy ones; this is your CI gate for prompt/model/index changes.

LLM-as-judge failure modes (from Zheng et al.'s "Judging LLM-as-a-Judge", MT-Bench):

• Position bias — in pairwise A/B comparison, judges favor the first-presented answer. Mitigation: judge both orders, count only consistent verdicts.
• Verbosity bias — longer answers score higher independent of quality. Mitigation: instruct against it, control length, or score decomposed claims instead of whole answers.
• Self-preference / self-enhancement — judges favor outputs in their own model family's style. Mitigation: judge from a different family than the generator.
• Limited reasoning — judges miss subtle factual/math errors; they're weakest exactly where generators are weakest. Mitigation: reference-guided judging (give the judge the ground truth), decomposition into atomic checks.
• Plus: score non-determinism (pin temperature 0, use rubrics + few-shot anchors) and drift when the judge model version changes (version-pin judges; re-baseline on upgrade).

The meta-rule: calibrate the judge — label 50–100 examples by hand, measure judge-human agreement, and only trust the judge where agreement is high.

The flows

Three measurement paths — offline gate, online sample, and the diagnosis loop that turns scores into fixes.

The operational model: decompose into claim-level checks, separate retriever from generator, and never treat an uncalibrated LLM judge as a measurement instrument.

A worked example

One query from the governed enterprise platform's golden set. Watch how the metrics localize a failure that an end-to-end score could only call "bad".

Golden entry:

query: "What's the approval threshold for international travel, and who approves it?"
relevant_chunks: ["policy-2026#sec3-eligibility", "policy-2026#sec7-approvals"]
reference_answer: >
  International travel requires pre-approval above the stated threshold.
  Approval sits with the employee's department head; finance counter-signs
  above twice that amount.

Run 1 — the system as shipped. Retrieved top-5, generated an answer.

The diagnosis writes itself, and note that hit rate alone would have said "pass". Recall is the metric that catches a half-answered two-part question. The generator, given only half the evidence, filled the gap from parametric memory — which is exactly what low faithfulness with high relevancy looks like.

Two independent bugs, two different owners:

1. Retrieval. The question has two parts; k=5 over one section-heavy document returned five chunks all from section 3. Fixes to try: raise k, add multi-query decomposition so "who approves" retrieves separately, or dedupe by section so one section can't monopolize the top-k.
2. Generation. The prompt didn't stop the model inventing the missing half. Fix: an explicit "if the context does not contain part of the answer, say which part is missing" instruction.

Run 2 — after fixing retrieval only. Context recall 1.0, faithfulness 0.95. Both moved, because the generator stopped needing to invent. Run 3 — after fixing generation only (a useful control to isolate the stages): recall stays 0.5, faithfulness rises to 1.0, but the answer is now "I don't have information on who approves" — honest, incomplete, and the user is still unserved.

Production concerns

• Offline vs online split: offline = golden set + full metric suite on every change (CI for your RAG stack); online = sampled continuous scoring, because you have no ground truth in production. Online proxies: faithfulness spot-checks on a sample (judges cost money — 1–5% sampling), thumbs up/down, answer/refusal rates, retrieval-score distributions, escalation-to-human rate, citation-click-through. (See observability.md.)
• Cost of judging: every judged answer is 1–3 extra LLM calls (claim extraction + verification). At scale: sample, use a cheaper judge for triage and a strong judge for the flagged tail, and cache judgments for identical (query, context) pairs.
• Metric gaming: optimize faithfulness alone and the system learns to answer "I don't know" constantly (perfectly faithful, useless). Always report faithfulness with answer relevancy and refusal rate as a balancing set.
• Regression discipline: embedding-model upgrades, chunking changes, and prompt edits all silently shift retrieval; gate merges on golden-set thresholds (e.g., recall@5 must not drop >2 pts) exactly like unit tests. (See evals-and-testing.md.)
• Drift: corpus changes and user-query mix changes decay both the system and the relevance of your golden set — refresh the eval set quarterly from live traffic, and monitor retrieval-score distributions for shift.
• The judge is a model in production too: it has versions, drift, and failure modes; track judge-human agreement over time, not just system scores.

Common drill-downs

How do you evaluate a RAG system? Split the pipeline. Retriever: golden (query → relevant chunk) pairs, measure recall@k / MRR / nDCG — deterministic and cheap. Generator: faithfulness (claims supported by context) and answer relevancy, scored by a calibrated LLM judge. Context precision/recall bridge the two. Offline on a versioned golden set for every change; online via sampled judging plus behavioral signals (feedback, refusals, escalations).

Define faithfulness vs answer relevancy — why do you need both? Faithfulness: fraction of answer claims supported by retrieved context (anti-hallucination). Relevancy: does the answer address the asked question. They're orthogonal — "Paris is in France" may be perfectly faithful to context yet irrelevant to "What's the capital of Germany?"; and a relevant answer can be fabricated. Optimizing either alone is gameable (empty answers are vacuously faithful).

Context precision vs context recall? Both grade the retriever. Recall: is everything needed present in the retrieved set (computed against a reference answer's claims)? Precision: is what was retrieved relevant and ranked high (rank-weighted)? Low recall → cast wider (higher k, hybrid search, better chunking). Low precision → tighten (reranker, filters, smaller k). They trade off through k, which is why you report both.

How does RAGAS compute faithfulness without ground truth? Two LLM steps: decompose the generated answer into atomic claims; verify each claim for support against the retrieved context; score = supported/total. No reference answer needed — the retrieved context is the ground truth being checked against, which is what makes faithfulness runnable online.

What are the failure modes of LLM-as-judge? Position bias (favors first answer in pairwise — swap orders and require consistency), verbosity bias (longer scores higher — decompose or control length), self-preference (judge favors its own family — use a different family), and weak reasoning on subtle errors (give references, decompose to atomic checks). Plus non-determinism and judge-version drift. Root mitigation: measure judge-human agreement on a labeled sample before trusting any number.

How would you build the golden dataset with no users yet? Synthesize from the corpus: LLM generates questions per chunk (chunk = retrieval label), evolve some into multi-hop and ambiguous variants, add no-answer cases, then human-review a sample for answerability and realism. Post-launch, continuously swap in real logged queries and every production failure as a regression case. Keep it versioned; keep it small and curated.

Hit rate vs MRR vs nDCG — when does each matter? Hit rate@k: binary "did the evidence arrive" — sizes k and gates everything. MRR: position of the first relevant chunk — right metric when one chunk answers and prompt order matters. nDCG: multiple relevant chunks with graded relevance — the search-quality workhorse. Report hit rate for go/no-go, nDCG for ranking tuning.

Your offline scores are great but users complain. Why? Golden set doesn't match real traffic (query mix drift, phrasings, topics); corpus staleness (eval ran on an old index snapshot); judge miscalibration inflating scores; or the complaints target latency/format/tone — dimensions your metrics never measured. Fix: mine complaint logs into the eval set, re-calibrate the judge against fresh human labels, and add the missing dimensions (latency SLO, format checks) to the suite.

How do you evaluate in production where there's no ground truth? Reference-free metrics on samples (faithfulness, relevancy — both computable from query+context+answer alone), implicit signals (thumbs, regenerations, escalations, citation clicks, session abandonment), and distribution monitoring (retrieval similarity scores, refusal rate, answer length). Route flagged low-faithfulness samples to human review; feed confirmed failures back into the golden set.

Test yourself

Go deeper

• Ragas documentation — available metrics — precise definitions and prompts behind faithfulness, relevancy, precision, recall, plus test-set generation.
• Ragas on GitHub — the reference implementation; reading the metric prompts is the fastest way to really understand the scores.
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023 — the canonical judge-bias paper: position, verbosity, self-enhancement biases and mitigations.
• Your AI Product Needs Evals — Hamel Husain — the practitioner's case for eval infrastructure as the core of LLM product iteration.
• Systematically Improving Your RAG — Jason Liu — synthetic-data baselines, leading vs lagging metrics, feedback loops.
• AI Agent Evaluation with RAGAS — James Briggs — hands-on RAGAS walkthrough over a real LangChain pipeline.

Where this connects

• Evals & Testing — the same discipline generalized to every LLM feature, plus CI, canaries, and prompt regression suites.
• Observability — the online half: tracing, sampled judging, and drift detection when there's no ground truth.
• Production RAG — what you gate with these numbers before and after every index change.
• Ingestion & Chunking — the experiments that are meaningless without the harness described here.