Hallucination
Why models hallucinate, grounding, and the mitigation layers.
Prerequisites
- How LLMs Work — next-token prediction with no built-in truth check.
- Tokens & Context Windows — grounding lives or dies by what you put in the window.
- Prompt Engineering — abstention and "answer only from documents" are necessary but not sufficient.
- Structured Output — schemas without escape hatches force fabrication into required fields.
The intuition
Ask a confident storyteller a question about a book they skimmed years ago. If the culture rewards a smooth full answer more than an honest "I'm not sure," you get a polished invention. That is hallucination in miniature.
An LLM is optimized to continue text that looks like good text. Where training data was dense, plausible often lines up with true. Where data was thin, contradictory, or the fact is a rare "singleton," the model still puts probability mass somewhere — and fluency does not dim when knowledge does. There is no internal library card that returns "missing." There is only the next token. Engineering does not hunt for a single switch. It builds layers: put the truth in the context, require citations, allow abstention, verify claims, and add humans when the blast radius is high.
Key insight
Hallucination is structural, not a temporary bug in one model version. You reduce and contain it with grounding and process. Any vendor claim of "zero hallucination" for open-ended generation should raise skepticism.
Why it exists
If models only answered when certain, many products would be safer — and most benchmarks would score them worse. Several pressures push toward fluent invention:
- The training objective is plausibility. Pretraining minimizes cross-entropy on next tokens — no truth oracle in that loss.
- Rare facts are statistically hard. Singleton facts stay unreliable under generative sampling even with clean data (Kalai et al., 2025).
- Evals reward guessing. Binary right/wrong with no credit for "I don't know" makes always-answer the optimal exam strategy.
- Fluency is uncalibrated. Tone stays confident when evidence is weak, so users cannot hear uncertainty unless you design for it.
- Product pressure. Empty answers feel like failure in demos; teams accidentally punish abstention in their own evals too.
Common shortcuts lose alone: "please don't hallucinate" prompts, bigger models alone, fact fine-tunes that age poorly, and unlimited context stuffing that still allows unfaithful drift.
On the governed enterprise platform, policy answers must be grounded in entitlement-filtered retrieval with citations; write actions need human approval; high-stakes intents add a verification pass. Brainstorming can stay looser — stakes tier the stack.
The core idea
Hallucination is fluent, confident content that is false: fabricated citations, invented API methods, wrong dates, plausible-but-nonexistent facts. The model is a next-token predictor trained on statistically plausible continuations. "True" and "plausible-sounding" coincide wherever data is thin or the fact is rarer than the pattern it fits. OpenAI's 2025 analysis (Why Language Models Hallucinate, Kalai et al.) made the product implication sharp: benchmarks keep rewarding lucky guesses, so confident fabrication is the learned exam strategy. Do not wait for zero-hallucination models — reward abstention in your evals, ground claims in evidence, and match process to stakes.
Two distinctions keep conversations precise. Extrinsic hallucination contradicts world knowledge; faithfulness (in-context) hallucination contradicts the provided source — the failure RAG evals score as groundedness. Separately, hallucination invents something false; staleness repeats something true at training cutoff but false today. Retrieval fixes staleness; anti-hallucination prompts alone do not.
The mitigation stack, bottom to top: grounding (RAG/tools), citations, abstention, verification, human review gated by stakes.
How it actually works
Pretraining learns the distribution of text. For well-attested facts, plausible ≈ true. For a fact seen once or never, "The paper was published in ___" still needs probability mass — and it lands on the most pattern-typical guess. Tone stays fluent either way, so confidence in the prose carries almost no signal about knowledge strength. Classic failure surfaces: citations (title–author–venue is high-plausibility, low-constraint), niche entities, numbers and dates, self-knowledge (post-hoc confabulation), API methods that should exist by naming convention, and multi-step agent chains where one invented intermediate poisons later steps.
| Layer | Mechanism | Cost | Catches |
|---|---|---|---|
| Grounding (RAG/tools) | Source text in context instead of parametric memory | Retrieval infra | Staleness + most extrinsic hallucination — if retrieval hits |
| Prompted grounding | "Answer only from the documents; quote first; say if absent" | ~Free | Ungrounded drift |
| Citations | Per-claim source spans | Small tokens | Makes fabrication detectable |
| Abstention | Permission to say "I don't know" / return not_found | ~Free | Forced guesses — biggest bang-for-buck prompt fix |
| Verification pass | Second call: quote-support each claim, or NLI/judge entailment | +1 call | Residual unfaithfulness |
| Consistency sampling | Sample N answers; disagreement flags low confidence (SelfCheckGPT) | N × cost | Unstable fabrications |
| Human-in-the-loop | Review gated by stakes or confidence | People | Residual risk on high-stakes output |
Key insight: RAG renames the problem
Retrieval does not "solve hallucination." It converts much of it into retrieval quality + faithfulness. Wrong chunk + confident answer is a new failure mode that still looks grounded. Measure both (RAG Evaluation).
Common misconception
"We added RAG, so anti-hallucination work is done." Without abstention when retrieval is empty or weak, without citations, and without groundedness evals, models still invent — sometimes by drifting from the text you retrieved.
The flows
| Flow | Sequence | When it applies | What breaks it |
|---|---|---|---|
| Grounded answer | Retrieve → pack evidence → answer-only-from-context → cite | Policy, docs, internal Q&A | Retrieval miss; no abstention; evidence buried mid-context |
| Abstain path | Evidence missing → NOT_FOUND / hedge | Sparse corpus, "not in docs" questions | Schema requiring filled fields; UX that punishes empty answers |
| Verify-then-show | Draft → check claims vs sources → repair or escalate | High-stakes (legal, finance, medical-adjacent) | Weak judge; cost explosion on all traffic |
| Agent tool-verified | Prefer executable tools over "does this look right?" | Code, tickets, systems of record | Trusting model narration of tool results |
| Low-stakes generative | Light grounding; accept creativity | Brainstorm, labeled drafts | Mis-tiering this flow for compliance answers |
A worked example
On the governed enterprise platform, an employee asks:
"What is our refund window for enterprise SaaS contracts signed in 2024?"
Retrieve MSA v3 §4.2: "14-day refund window from countersignature for enterprise SaaS, excluding custom SOWs" (entitlement all-legal-readers). The model quotes with citation [msa_v3_c4]. The UI shows the answer plus a link to the span. Cost is roughly +50–150 ms for retrieval (illustrative) and tokens for ~1–2k context — no second pass for this tier.
Illustrative verification cost: a second judge call might add ~$0.01–0.05 and +1–3 s — fine for contract intents, wasteful on brainstorming. No abstention → empty retrieval still yields a policy. No citations → reviewers cannot audit. Required non-null fields force refund_days: 30 into the schema (Structured Output). Eval only on answerable questions and models that always guess look better than models that abstain correctly.
Production concerns
Measure before mitigating. Build a hallucination eval: curated Q&A with known answers plus adversarial "answer isn't in the docs" cases. Score groundedness with an LLM judge, then spot-check the judge. Track by release (Evals and Testing, RAG Evaluation).
Stakes tier the stack. A brainstorming copilot tolerates invention; an insurance-claims extractor does not. Low-stakes: grounding + abstention. High-stakes: add citations, verification, and human review. Show sources in the UI — a wrong answer with a visible mismatched source is recoverable; a naked wrong answer is not.
Self-inflicted classes. Structured outputs without a not_found escape force fabrication (Structured Output). A fabricated tool interpretation early in an agent loop cascades — ground claims in tool return values (Agent Reliability). Run cheap layers (abstention, citations) everywhere; reserve verification and N-sample consistency for high-stakes routes (Cost). Pair grounding with Security — prompt injection can induce false statements from untrusted context.
Common drill-downs
How do you detect hallucinations at runtime? By cost, low to high: logprob signals (weak alone), citation span-checking, NLI or judge-model entailment of answer vs source, and self-consistency (sample N, flag disagreement — SelfCheckGPT). In practice: judge-scored groundedness on sampled traffic plus hard checks on citations and numeric identifiers.
What's the cheapest effective mitigation? Prompt-level abstention: explicitly allow "if the context doesn't contain the answer, say so." Anthropic's guidance lists it first; it converts forced guesses into recoverable "not found" outcomes for the cost of a sentence.
Why do models invent plausible-looking API methods? Code is pattern-dense: a method that should exist by naming convention is high-probability even when absent. Ground in docs or type stubs, then verify by execution.
User reports the bot invented a refund policy. Reproduce and classify: no grounding, retrieval miss, or unfaithful? Short-term: tighten grounding and abstention; add the case to the eval set. Structural: citations in the UI, verification on policy intents, fix retrieval if needed, regression-test on every prompt or model change.
Test yourself
Your groundedness judge scores 0.94, but users still catch invented clause numbers. What might the metric be missing?
Why is 'always answer' the rational strategy under many academic benchmarks, and how should your product eval differ?
RAG retrieves the right doc, but the answer contradicts it. Name the failure class and three mitigations.
An agent claims 'tests passed' but CI is red. Where did hallucination enter, and what architectural rule prevents it?
When would consistency sampling (SelfCheckGPT-style) be worth 5× generate cost?
Go deeper
- Why Language Models Hallucinate (Kalai et al. — OpenAI, 2025) — rare-fact statistics plus eval-rewards-guessing. Companion: OpenAI blog.
- Extrinsic Hallucinations in LLMs — Lilian Weng — causes, detection (SelfCheckGPT, FActScore), training approaches.
- Anthropic docs: Reduce hallucinations — allow "I don't know," quote-first grounding, citation checks, best-of-N.
- Deep Dive into LLMs like ChatGPT — Andrej Karpathy — where hallucinations come from and how labs mitigate them.
Where this connects
- The RAG Pipeline — primary grounding mechanism for enterprise knowledge.
- RAG Evaluation — faithfulness, context precision/recall, and judges.
- Structured Output — avoid schema-forced fabrication with null/unknown outs.
- Agent Reliability — compounding error, tool hallucination, and human-in-the-loop.