Evals & Testing

The intuition

Traditional unit tests are a metal stamp: the part matches the die or it does not. LLM outputs are more like dishes from a skilled but non-deterministic chef — many plates can be "correct," the same order twice will not be byte-identical, and the kitchen sometimes changes the recipe overnight.

You cannot stamp output == expected and call it quality. You need a tasting panel: cheap mechanical checks, curated reference cases with rubrics, and expert judges for properties no checklist captures. Treat menu changes (prompts, models, retrieval) like code: run the panel before merge, canary a few tables, then the whole room. Products that improve after launch almost always turn production failures into new cases.

Why it exists

LLMs break classical testing. Sampling, silent provider updates, and batching nondeterminism mean the same input does not yield one stable string. Most tasks have no single correct output, so exact match fails both ways — false reds on good answers, false greens on lucky phrasings. A one-line prompt edit can regress faithfulness, length, cost, and tools at once. Frozen datasets rot as traffic drifts. Judges are systematically biased unless calibrated.

Alternatives lose: vibe checks do not scale, exact-match thrash on valid variance, synthetic-only sets miss real failures, and ship-then-watch-support is how quality plateaus. Evals exist so changes are measured, merges gated, rollouts reversible, and every production failure can strengthen the suite.

The core idea

What replaces exact-match is three layers plus an operational loop.

Property-based assertions are cheap deterministic checks every valid output must pass: schema, length, citation IDs in the retrieved set, no PII. Golden datasets are curated input→reference cases — ideally from production failures — scored with task metrics. LLM-as-judge grades properties code cannot check (faithfulness, relevance, tone), only after you measure agreement with humans.

Operationally, prompt/model/retrieval edits are production changes: CI gates the merge; risky changes ship as canaries (1–5% sticky traffic, compare online metrics, promote or roll back); model swaps get A/B tests because offline never fully predicts online. The compounding loop: traces → error analysis → cases per failure mode → fix → re-run → canary. The suite is the spec.

How it actually works

Nondeterminism consequences. Sampling, missing seed guarantees, silent alias updates, batching even at temperature 0, and prompt sensitivity all apply. Assert suite pass-rate thresholds ("≥90% of cases pass"), not single-run exact outputs. A flaky case is variance signal — not noise to retry away. See sampling and determinism.

Assertions run on every output in CI and often in prod: schema/format, regex/containment, length bounds, allow/block lists, "cited IDs ⊆ retrieved set," executable checks for code. Fast, free, objective — and they double as guardrails (reliability's semantic retry).

Golden datasets and error analysis. Fifty to two hundred well-chosen cases beat thousands of unlabeled ones. Source from production traces — judge- and user-flagged failures (observability) — plus hard edges. Each case needs input, frozen context (pinned retrieved chunks so generation scores independent of retrieval), a reference or rubric, and a failure-mode label. Version the set; grow it when analysis finds a new mode.

The non-negotiable loop is error analysis, not bulk synthetic generation. Practitioners (Hamel Husain, Shreya Shankar) use open coding — a domain expert notes free-form failures on traces — then axial coding — group notes into a failure taxonomy and count frequency. Encode the most common and costly modes as assertions or judges. For RAG, separate retrieval and generation suites so regressions localize (RAG evaluation).

LLM-as-judge biases and mitigations. Known modes: position bias (favors first answer in pairwise), verbosity bias (longer looks better), self-enhancement (favors its own model family), leniency drift, rubric sensitivity. Mitigations: binary pass/fail per narrow criterion instead of 1–10 scales; one criterion per call; swap positions and average for pairwise; require evidence quotes; prefer a different or stronger judge when self-enhancement shows up; calibrate with true positive and true negative rates on a human-labeled slice (raw agreement misleads when classes are imbalanced). Re-calibrate when rubric or model changes.

CI for prompts. Version prompts in git or a prompt store. A PR touching prompt, model, temperature, or retrieval runs the suite. Report aggregate deltas and per-case diffs — aggregates hide fixed-5/broke-3 swaps. Hard-fail below thresholds; humans review borderline diffs. Cost control: full assertions every PR, judge subset on PRs, full judge sweep nightly on batch API (cost).

Canaries and A/B. Offline gates the merge; online gates the rollout. Canary 1–5% sticky by user; compare errors, latency, cost, judge sample, and feedback to control; promote or roll back. Model swaps need a proper A/B: fixed allocation, pre-registered success metric, enough runtime for significance. Offline-online gaps are routine — that is why both layers exist (latency, reliability).

The flows

A worked example

The policy-QA feature on the governed enterprise platform changes its system prompt to require citations on every claim.

Golden set (illustrative 80 cases): 30 core FAQs, 25 prior faithfulness failures from judge-flagged traces, 15 adversarial/edge (multi-hop, missing policy, ACL boundary), 10 citation-format cases. Retrieval is pinned so the run measures generation, not index churn.

Gates: block if faithfulness < 90% or assertions < 95%. v42 passes but completeness dipped and cost/latency will rise (cost, latency). Review accepts the citation tradeoff with a completeness follow-up.

Canary (3% sticky, illustrative): schema-failure 2.1% → 0.4%; thumbs-down 3.0% → 2.7%; p95 TTFT 1.1 s → 1.2 s; cost/request $0.014 → $0.017; async faithfulness 0.90 → 0.92. Promote after 48 h; keep v41 warm. Add the five completeness regressions as new golden cases — the suite compounds.

Production concerns

Judge calls are LLM calls — tier assertions everywhere, judges on samples/PR subsets, full sweeps nightly on batch API, and track eval spend like feature spend (cost). Golden sets rot as distributions drift (observability): refresh from production, retire stale cases, keep a stable core for longitudinal comparison.

Goodhart risk: optimizing a fixed rubric overfits it — longer, hedged, rubric-shaped outputs. Refresh rubrics, keep human spot-checks, and watch online metrics diverge from offline scores. Rising judge scores with rising thumbs-down is the classic signal. Pin model snapshots where offered; re-baseline when a pin deprecates. Parallelize CI, cache by (prompt version, case ID, model), and keep the PR-blocking subset small. The organizational failure mode is skipping error analysis — tools measure, they do not notice.

Common drill-downs

Offline passed, users hate the new model. Why? Dataset no longer matches live traffic; rubric misses style/latency/verbosity; prompts Goodharted the suite; or online factors (TTFT, UI formatting) dominate. Gate rollouts on canary/A-B, not offline alone.

How do you roll out a model swap safely? Full offline suite + prompt re-tune if needed (prompts do not port — reliability). Sticky canary → guardrails and judge sample vs control → A/B with a pre-registered metric → promote. Keep the old path warm.

Test yourself

Go deeper

• Your AI Product Needs Evals — Hamel Husain — eval systems separate products that improve from ones that stall.
• Using LLM-as-a-Judge For Evaluation — Hamel Husain — critique shadowing: domain experts, binary judgments, judges you can trust.
• LLM Evals FAQ — Hamel Husain & Shreya Shankar — open/axial coding, binary pass/fail, calibration with TPR/TNR.
• Evaluating the Effectiveness of LLM-Evaluators — Eugene Yan — position, verbosity, and self-enhancement biases in the literature.
• Evaluation best practices — OpenAI — continuous evaluation across single-turn, workflow, and agent architectures.
• Why AI evals are the hottest new skill — Lenny's Podcast — Hamel Husain & Shreya Shankar on error-analysis-first workflow.

Where this connects

• Observability — traces and feedback grow golden sets and judge canaries online.
• Reliability — fallback models and semantic retries need the same suites before an outage exercises them.
• Cost — judge sampling, batch sweeps, and output-length gates are cost controls as much as quality controls.
• RAG evaluation — component metrics when the system under test is retrieval-augmented.