Hallucination

A hallucination is when an LLM generates text that is fluent, confident, and grammatically correct but factually false, fabricated, or unsupported by any real source — inventing a citation that doesn't exist, stating a wrong statistic, or confidently describing an API method that was never built. Hallucinations happen because LLMs are next-token predictors optimized to produce plausible-sounding text, not fact-checkers with a ground-truth database; when a model doesn't "know" an answer, it doesn't reliably say "I don't know" — it often generates the most statistically likely-sounding answer instead, which can be entirely invented. This is the single biggest trust and liability risk for SaaS builders shipping AI features, especially in regulated or high-stakes contexts (legal, medical, financial, support). A well-documented real-world example: a legal team using ChatGPT to draft a court filing submitted a brief citing several court cases that the model had completely fabricated, complete with plausible-sounding case names and docket numbers — the cases did not exist, resulting in sanctions. Mitigations builders actually use in production: grounding outputs in retrieved source documents via RAG (and instructing the model to answer only from provided context, refusing when the context doesn't contain the answer); lowering temperature for factual tasks to reduce creative drift; adding citation requirements so every claim links to a source the user can verify; using guardrails or a second "verifier" LLM call to check the first model's output against retrieved facts before showing it to a user; and clear UX signaling (confidence indicators, "AI-generated, please verify" disclaimers) so users don't over-trust output. No current mitigation eliminates hallucination entirely — it's a property of how these models generate text, not a bug that gets patched out — so production systems should assume some hallucination rate and design verification/fallback paths accordingly, especially wherever wrong output has real cost. Hallucination rates vary meaningfully by task type and are worth benchmarking before shipping: open-ended creative tasks tolerate some invention by design, while factual lookup, numeric extraction, and citation tasks demand near-zero hallucination tolerance because users will act on the output as ground truth. A useful production pattern is a self-consistency check — asking the model the same question multiple times (or via slightly different phrasings) and flagging disagreement between answers as a signal of low confidence, since a model that's genuinely certain about a fact tends to answer consistently, while a model that's hallucinating often varies its fabricated answer between attempts.