Evals & Testing

The intuition

Traditional unit tests are a metal stamp: the part either matches the die or it does not. LLM outputs are closer to restaurant dishes from a skilled but non-deterministic chef — many plates can be "correct," the same order twice will not be byte-identical, and the kitchen sometimes changes the recipe overnight without telling you.

You cannot stamp output == expected and call it quality. You need a tasting panel: cheap mechanical checks every plate must pass (hot, on the right dish, no allergens), a curated set of reference orders with rubrics, and occasional expert judges for properties no checklist captures. Then you treat menu changes (prompts, models, retrieval config) like code changes: run the panel before merge, ship to a few tables first, and only then to the whole dining room.

The eval suite is the executable product spec. Products that improve after launch are almost always the ones whose failure modes keep turning into new test cases.

Why it exists

LLMs break the assumptions classical testing rests on. The hard constraints:

1. Nondeterminism is structural. Sampling, provider-side model updates, floating-point/batching effects, and prompt sensitivity mean the same input does not yield one stable string.
2. Most tasks have no single correct output. Summaries, drafts, and answers admit thousands of acceptable forms. Exact match fails both ways: false reds on good answers, false greens on lucky phrasings.
3. Prompt and model changes are production changes. A one-line system-prompt edit can regress faithfulness, length, cost, and tool behavior simultaneously.
4. Offline datasets rot. Live traffic drifts; a frozen suite that never absorbs production failures becomes a false sense of safety.
5. Judges have biases. LLM-as-judge is powerful and wrong in systematic ways unless calibrated and constrained.

The alternatives lose. Manual vibe checks do not scale and do not catch regressions. Exact-match only thrash on valid variance. Synthetic-only datasets miss the failure modes users actually hit. Ship and watch support is how quality plateaus after launch.

Evals exist so changes are measured, merges are gated, rollouts are reversible, and every production failure can strengthen the suite — eval-driven development rather than prompt folklore.

The core idea

Traditional unit tests assert output == expected. LLMs break both sides of that: outputs are nondeterministic (sampling, provider-side model updates, even temperature-0 isn't bit-stable), and most tasks have no single correct output — "summarize this ticket" has thousands of acceptable answers. What replaces exact-match is a stack of three eval layers. Property-based assertions: cheap, deterministic checks on properties every valid output must have — parses as JSON, matches the schema, cites a retrieved source, under N words, contains no PII, doesn't mention competitors. Golden datasets: curated input→reference examples (from real production failures, not synthetic guesses) scored by task-appropriate metrics. LLM-as-judge: a model grading outputs against a rubric for the properties code can't check — faithfulness, relevance, tone — used with open eyes about its biases and calibrated against human labels before you trust it.

The operational half is what makes it engineering rather than vibes: prompt changes are code changes, so every prompt/model/retrieval edit runs the eval suite in CI and gates the merge like any regression suite; risky changes ship as canaries (small traffic slice, compare online metrics, promote or roll back); and model swaps get A/B tests on real traffic because offline evals never fully predict online behavior. The workflow that ties it together is eval-driven development: look at real traces, do error analysis, turn each failure mode into an eval case, fix, and re-run — the eval suite is the spec, and it compounds; it's also the single strongest predictor of whether an LLM product improves after launch or plateaus.

How it actually works

Why nondeterminism breaks asserts, precisely: sampling randomness; no seed guarantees on hosted APIs; silent model updates behind aliases; floating-point/batching nondeterminism even at temperature 0; and prompt sensitivity (a one-word edit shifts distributions). Consequences: run evals multiple times or across enough cases that pass rates are stable; assert thresholds ("≥90% of cases pass"), never single-run exact outputs; and treat a flaky eval as signal about output variance, not test noise to retry away. Background: sampling and determinism.

Assertion layer (runs on every output, prod and CI): schema/format validation, regex/containment checks, length bounds, allowlist/blocklist, "all cited doc-IDs exist in retrieved set," executable checks for code (does it run? do its tests pass?). Fast, free, objective — catch a surprising share of regressions and double as production guardrails (see reliability's semantic retry).

Golden datasets: 50–200 well-chosen cases beat 5,000 unlabeled ones. Source them from production traces — especially judge-flagged and user-flagged failures (observability) — plus deliberately hard cases and edge inputs. Each case: input, context (frozen retrieved chunks, so you can eval generation independent of retrieval), reference or rubric, and labels by failure mode. Version the dataset; grow it every time error analysis finds a new failure mode. For RAG, keep separate suites: retrieval metrics (recall@k, MRR against labeled relevant chunks) and generation metrics (faithfulness, completeness given fixed context) — so a regression localizes (RAG evaluation).

LLM-as-judge, with its biases: documented failure modes — position bias (favors first answer in pairwise comparison), verbosity bias (longer looks better), self-enhancement (favors its own model family's style), leniency drift, and rubric sensitivity. Mitigations that work: binary pass/fail per narrow criterion instead of 1–10 scales (Hamel Husain's core recommendation — humans can't consistently define a "7" either, and binary judgments are auditable); one criterion per judge call ("is every claim supported by the context: yes/no") rather than one omnibus "quality" score; swap positions and average for pairwise; require the judge to quote evidence before deciding; use a judge model different from (or stronger than) the generator; and calibrate: measure judge-vs-human agreement on a labeled slice, report judge scores only while agreement stays high, re-calibrate when the rubric or model changes.

Prompt regression suites in CI: prompts live in version control (or a prompt-management store with versions); a PR touching prompt, model, temperature, or retrieval config triggers the eval run; the report shows aggregate pass-rate deltas and per-case diffs (aggregates hide "fixed 5, broke 3 different ones" — the swap matters). Gate: hard-fail below threshold on assertions and judge criteria; humans review the diff for borderline calls. Cost control: run assertions on the full suite but judge-scoring on a representative subset per PR, full judge suite nightly via batch API at 50% price (cost).

Canary rollouts and A/B tests: offline evals gate the merge; online metrics gate the rollout. Canary a prompt/model change to 1–5% of traffic (sticky by user), compare guardrail metrics (error/refusal/schema-failure rates, latency, cost) and quality signals (judge sample, feedback) against control, then promote or roll back — prompt versions make rollback instant. Model swaps get a proper A/B: fixed allocation, pre-registered success metric (task completion, acceptance rate of generations), enough runtime for significance. Offline-online gaps are routine — a model that wins the eval suite can lose online because of latency, style, or distribution shift, which is why both layers exist. Pairs with latency and reliability fallbacks when a canary goes bad.

Offline vs online in one line each: offline = frozen dataset, reproducible, pre-deploy, cheap to iterate; online = real traffic, real users, post-deploy, the ground truth. Offline for iteration speed, online for verdicts.

The flows

A worked example

The policy-QA feature on the governed enterprise platform is changing its system prompt to require citations on every claim.

Golden set (illustrative 80 cases):

Frozen context per case: retrieval is pinned so this run measures generation under the new instruction, not index churn.

CI report after the prompt PR:

Gate policy: block if faithfulness < 90% or assertions < 95%. v42 passes gates but completeness dipped and cost/latency will rise with longer outputs (cost, latency). Human review accepts the tradeoff for citation compliance, with a follow-up task to recover completeness.

Canary (3% sticky):

Promote after 48 h; keep v41 warm for instant rollback. Add the 5 completeness regressions as new golden cases with rubrics — the suite compounds.

Common drill-downs

How do you test something nondeterministic? Stop asserting exact outputs; assert properties (schema, citations, constraints) deterministically, score semantics against golden datasets with task metrics or an LLM judge, and evaluate at the suite level with pass-rate thresholds across runs rather than single-case certainty.

Design a regression suite for a prompt change. Prompts versioned in git; PR triggers evals: assertion layer on the full golden set, judge layer (binary criteria) on a representative subset; report aggregate deltas plus per-case diffs; threshold gates merge; nightly full judge sweep on batch API. Post-merge, canary to 1–5% with online guardrail metrics before full rollout.

What are LLM-as-judge's failure modes and your mitigations? Position bias (swap and average), verbosity bias (length-controlled rubrics, binary criteria), self-enhancement (different/stronger judge model), scale incoherence (binary pass/fail per criterion, not 1–10), drift (periodic human calibration; track judge-human agreement and only trust the judge while it holds).

Where does your golden dataset come from? Production traces — judge-flagged and user-flagged failures, plus edge cases from error analysis; not synthetic-only generation. Each case labeled by failure mode; dataset versioned and refreshed as traffic drifts; frozen retrieval context per case so generation evals are isolated from retrieval evals.

Offline evals passed but users hate the new model. Why? Offline-online gap: dataset no longer matches live distribution; judge rubric misses what users value (style, latency, verbosity); Goodharted prompts overfit the suite; or online factors (TTFT, formatting in the real UI) dominate. This is why rollouts are gated by canary/A-B metrics, not offline scores alone.

How do you evaluate a RAG system specifically? Decompose: retrieval quality (recall@k/MRR against labeled relevant chunks) separately from generation quality (faithfulness-to-context, answer relevance, completeness — judge-scored with fixed retrieved context). End-to-end score alone can't tell you which stage regressed; component evals localize the fix.

How do you roll out a model swap safely? Offline: run the full eval suite on the new model, re-tune prompts if needed (prompts don't port — see reliability). Online: canary small sticky slice → guardrail metrics + judge sample vs control → scale to an A/B with a pre-registered success metric → promote. Keep the old path warm for instant rollback.

What is eval-driven development? The loop: examine real traces → error analysis into a failure taxonomy → encode each failure mode as eval cases → change prompt/retrieval/model → re-run evals → ship behind a canary → new production failures feed back into the dataset. Evals are the executable spec; every iteration makes the suite — and the product — stronger.

Production concerns

• Evals cost real money at scale: judge calls are LLM calls. Tier them — assertions everywhere, judges on samples/subsets, full sweeps nightly on the batch API. Track eval spend like feature spend (cost).
• Dataset rot: golden sets drift from the live input distribution (see observability drift). Refresh from production monthly; retire cases that no longer represent traffic; keep a stable core so longitudinal comparisons stay meaningful.
• Goodhart risk: optimizing to a fixed judge rubric overfits it — outputs get longer, more hedged, more rubric-shaped. Rotate/refresh rubrics, keep human spot-checks in the loop, and watch online metrics for divergence from offline scores.
• Provider model updates invalidate baselines: pin model snapshots where offered; when a pinned model deprecates, re-baseline the whole suite before comparing anything.
• CI latency: a 500-case judge suite serially is an hour. Parallelize aggressively (evals are embarrassingly parallel), cache unchanged-case results keyed by (prompt version, case ID, model), and keep the PR-blocking subset small.
• The organizational failure mode is skipping error analysis: teams buy an eval tool, auto-generate synthetic test cases, and measure nothing real. The non-negotiable loop is human-looks-at-traces → failure taxonomy → eval cases per failure mode. Tools automate the measurement, not the noticing.

Test yourself

Go deeper

• Your AI Product Needs Evals — Hamel Husain — the canonical case that eval systems, not prompt cleverness, separate products that improve from ones that stall.
• Using LLM-as-a-Judge For Evaluation: A Complete Guide — Hamel Husain — critique shadowing: domain experts, binary judgments, building a judge you can trust.
• Evaluating the Effectiveness of LLM-Evaluators — Eugene Yan — survey of judge biases (position, verbosity, self-enhancement) and alignment techniques, grounded in the literature.
• Why AI evals are the hottest new skill for product builders — Lenny's Podcast — Hamel Husain & Shreya Shankar on the error-analysis-first workflow.
• A Survey of Techniques for Maximizing LLM Performance — OpenAI — the DevDay talk behind eval-first optimization: measure, then choose prompting vs RAG vs fine-tuning.

Where this connects

• Observability — traces and feedback are how golden sets grow and how canaries are judged online.
• Reliability — fallback models and semantic retries need the same suites before an outage exercises them.
• Cost — judge sampling, batch sweeps, and output-length gates are cost controls as much as quality controls.
• RAG evaluation — deeper component metrics when the system under test is retrieval-augmented.