Hallucination

The intuition

Ask a confident storyteller a question about a book they skimmed years ago. If the culture rewards a smooth full answer more than an honest "I'm not sure," you get a polished invention. That is hallucination in miniature.

An LLM is optimized to continue text that looks like good text. Where training data was dense and consistent, plausible ≈ true often enough to impress. Where data was thin, contradictory, or the fact is a rare "singleton," the model still has to put probability mass somewhere — and fluency does not dim when knowledge does. There is no internal library card catalog that returns "missing." There is only the next token.

So engineering does not hunt for a single "stop hallucinating" switch. It builds layers: put the truth in the context, require citations, allow abstention, verify claims, and add humans when the blast radius is high.

Why it exists

If models only answered when certain, many products would be safer — and most benchmarks would score them worse. The pressures that create hallucination:

1. The training objective is plausibility. Cross-entropy on next tokens does not include a truth oracle.
2. Rare facts are statistically hard. Even clean data cannot make singleton facts perfectly reliable under generative sampling (the statistical inevitability argument in Kalai et al., 2025).
3. Post-training and evals reward guessing. Binary right/wrong with no credit for "I don't know" makes always-answer the optimal exam strategy.
4. Fluency is uncalibrated. Tone stays confident when evidence is weak, so users cannot hear uncertainty unless you design for it.
5. Product pressure. Empty answers feel like failure in demos; teams accidentally punish abstention in their own evals too.

Alternatives people reach for, and why they lose alone:

On the governed enterprise platform, policy answers must be grounded in entitlement-filtered retrieval with citations; write actions need human approval; high-stakes intents add a verification pass. Brainstorming modes can stay looser — stakes tier the stack.

The core idea

Hallucination is the model producing fluent, confident content that is false — fabricated citations, invented API methods, wrong dates, plausible-but-nonexistent facts. The important framing is explaining why it's structural, not a bug: the model is a next-token predictor trained to produce statistically plausible continuations. There is no internal fact database and no truth check — "true" and "plausible-sounding" are the same thing to the training objective wherever the training data is thin, contradictory, or the fact is rarer than the pattern it fits.

OpenAI's 2025 analysis ("Why Language Models Hallucinate") sharpened this: even with perfect data, the statistics of generation make some errors inevitable (rare "singleton" facts can't be reliably learned), and — crucially — post-training and benchmarks reward guessing. Most evals score binary right/wrong with no credit for "I don't know," so a model that always guesses outperforms a calibrated one that abstains. Models are optimized to be good test-takers; confident fabrication is the learned optimal strategy under that grading. Hallucination is therefore reducible (better grounding, calibration, abstention rewards) but not eliminable — which is why applied engineering treats it as a risk to be layered against, not a bug to be fixed once.

Two useful distinctions to name: extrinsic hallucination (contradicts world knowledge — fabricated facts) vs faithfulness/in-context hallucination (contradicts the provided source — the one RAG evals measure as "groundedness"); and hallucination vs staleness (a fact true at training cutoff, false today — mitigated by retrieval, not by anti-hallucination prompting).

The mitigation stack, bottom to top: grounding (RAG/tools — put the truth in the context; models are far more faithful to context than to parametric memory), citations (make every claim auditable against a source), abstention ("say 'I don't know' when unsure" — cheap and measurably effective), verification (a second model/pass checks claims against sources), and human review gated by stakes.

How it actually works

Why the objective produces confident errors. Pretraining minimizes cross-entropy on next tokens: the model learns the distribution of text. For well-attested facts, plausible ≈ true. For a fact seen once (or never), the learned distribution around "The paper was published in ___" still has to put mass somewhere — and it lands on the most pattern-typical continuation. Calibration makes it worse-looking: the model's fluency is uniform whether the underlying knowledge is solid or vapor, so confidence in tone carries no signal.

Why RLHF/benchmarks amplify it. Human raters and automated benchmarks prefer complete, confident answers; abstentions score zero. Kalai et al. frame it as an exam under negative-marking-free grading: the optimal strategy is to always answer. Fixing this at the training level means scoring schemes that reward calibrated uncertainty — an active 2025–26 shift in how labs eval (and why newer models refuse/hedge more appropriately than 2023-era ones).

Classic failure surfaces to name: citations/references (title-author-venue is a high-plausibility, low-constraint pattern), niche entities and long-tail facts, numbers and dates, self-knowledge ("why did you say that?" — post-hoc confabulation), API/library details (invented methods that should exist), and compounding errors in multi-step agent chains (one hallucinated intermediate poisons everything downstream).

How each mitigation layer works:

Note what's downstream: RAG converts the hallucination problem into a retrieval-quality + faithfulness problem — retrieval misses cause a distinct failure ("confident answer from the wrong chunk"), and faithfulness must itself be measured (groundedness metrics; section 04 covers RAG evaluation).

The flows

A worked example

On the governed enterprise platform, an employee asks:

"What is our refund window for enterprise SaaS contracts signed in 2024?"

Illustrative verification cost: second judge call ~$0.01–0.05 and +1–3 s — fine for contract intents at moderate volume, wasteful on "summarize this brainstorm."

Common drill-downs

Why do LLMs hallucinate? The training objective optimizes plausibility, not truth — there's no fact store or truth check. Rare facts can't be reliably learned even from clean data, and post-training/benchmarks reward confident guessing over abstention (binary grading gives "I don't know" zero credit). So fabrication is the learned optimal strategy, structural rather than incidental.

Can you eliminate hallucination? No — it's inherent to the objective and the statistics of rare facts. You reduce and contain it: grounding, citations, abstention, verification, human review, matched to stakes. Any claim of "zero hallucination" should trigger skepticism.

How does RAG reduce hallucination — and what new failure does it introduce? Puts the relevant truth in context, where models are much more faithful than to parametric memory, and fixes staleness. New failure: retrieval misses or wrong chunks → confidently grounded-looking answers from wrong sources, plus residual unfaithfulness to the provided text. So you must eval retrieval quality and groundedness separately.

How do you detect hallucinations at runtime? Options by cost: logprob/confidence signals (weak), citation span-checking (does the quoted source actually contain the claim?), NLI/judge model scoring answer-vs-source entailment, and self-consistency (sample N, flag disagreement). In practice: judge-scored groundedness on sampled traffic + hard checks on citations.

What's the cheapest effective mitigation? Prompt-level abstention: explicitly allow and instruct "if the context doesn't contain the answer, say so." Anthropic's own guidance lists it first; it converts forced guesses into recoverable "not found" outcomes for the cost of a sentence.

Extrinsic vs faithfulness hallucination? Extrinsic: contradicts world knowledge (fabricated fact/citation). Faithfulness: contradicts the provided source — the model had the truth in context and drifted anyway. RAG evals measure the second as groundedness; both need separate tracking.

User reports the bot invented a refund policy. Walk through your response. Reproduce and classify (no grounding? retrieval miss? unfaithful to retrieved policy?). Short-term: tighten grounding rules + abstention, add the case to the eval set. Structural: citations in the UI, verification on policy-type intents, retrieval fix if it was a miss, and regression-test the eval on every prompt/model change.

Why do models invent plausible-looking API methods? Code is pattern-dense: a method that should exist by naming convention is high-probability under the objective even if absent from the real library. Mitigation is grounding in the actual docs/type stubs (retrieval or context stuffing) plus verification by execution — compile/run beats asking the model to double-check.

Production concerns

• Measure before mitigating. Build a hallucination eval: curated Q&A with known answers + adversarial "answer isn't in the docs" cases; score groundedness (claims supported by retrieved context) with an LLM judge and spot-check the judge. Track by release — model upgrades and prompt changes both move it. See evals-and-testing.md and rag-evaluation.md.
• Stakes-tiered design. A brainstorming copilot tolerates hallucination; an insurance-claims extractor doesn't. Match layer count to blast radius: low-stakes = grounding + abstention; high-stakes = + citations + verification + human review. How much mitigation you deploy depends on cost of error.
• UX is a mitigation layer. Show sources, link citations, badge confidence, phrase uncertainty honestly. Users who can verify catch what pipelines miss; a wrong answer with a visible (mismatched) source is recoverable, a naked wrong answer is not.
• Schema-forced fabrication. Structured outputs without a not_found/nullable escape force the model to fill fields it doesn't know (see structured-output.md) — a self-inflicted hallucination class.
• Agent loops compound it. A fabricated tool result interpretation or invented file path early in a loop cascades. Mitigate with verifiable tools (run the code, don't ask if it would work), grounding claims in tool outputs, and progress claims audited against actual results. See agent-reliability.md.
• Cost realism. Verification passes and N-sample consistency multiply spend; reserve them for the high-stakes routes and let cheap layers (abstention prompts, citations) run everywhere. See cost.md.
• Security adjacent. Prompt injection can induce harmful or false statements from untrusted context; pair grounding with the controls in security.md.

Test yourself

Go deeper

• Why Language Models Hallucinate (Kalai, Nachum, Vempala, Zhang — OpenAI, 2025) — the statistical inevitability argument plus the eval-rewards-guessing thesis; the current canonical citation.
• Extrinsic Hallucinations in LLMs — Lilian Weng — deep survey: causes, detection methods (SelfCheckGPT, FactScore), and anti-hallucination training approaches.
• Anthropic docs: Reduce hallucinations — the practical playbook: allow "I don't know," quote-first grounding, citation-verified claims, best-of-N checks.
• Deep Dive into LLMs like ChatGPT — Andrej Karpathy — includes the clearest lay explanation of where hallucinations come from and how labs mitigate them in post-training.

Where this connects

• The RAG Pipeline — primary grounding mechanism for enterprise knowledge.
• RAG Evaluation — faithfulness, context precision/recall, and judges.
• Structured Output — avoid schema-forced fabrication with null/unknown outs.
• Agent Reliability — compounding error, tool hallucination, and human-in-the-loop.