What Is an AI Hallucination? Why Models Invent Things

What the word means

A hallucination is a model producing something false while sounding exactly as confident as when it is right. An invented citation, a plausible but non-existent API method, a made-up statistic, a policy your company never had. The word is a poor metaphor — nothing is being perceived — but it names something real and specific: output that is well-formed, contextually appropriate, and untrue, with no signal in the text marking it out.

Why it happens

A language model is trained to produce likely continuations of text, not to state facts it can verify. Asked for a citation, it produces something shaped like a citation: plausible authors, a plausible journal, a plausible year. Nothing in that process consults a source, because there is no source to consult — the model holds statistical regularities, not records. Fluency and accuracy are separate properties, and only one of them was optimised.

Why it cannot simply be fixed

This follows from how the models work, not from insufficient training. A model has no internal marker distinguishing a fact it absorbed thousands of times from a pattern it is completing for the first time — which is why "only answer if you are sure" helps far less than it seems it should. The model's confidence is not a measurement of its knowledge; it is a property of the text. Every mitigation that works does so by adding something outside the model, not by asking the model to try harder.

Where it is most likely

Risk concentrates predictably. Specific facts the model would have seen rarely: small companies, recent events, niche APIs, exact numbers. Anything after the knowledge cutoff, where the model has no data but no awareness of the gap either. Questions containing a false premise, which a model will often accept and build on rather than challenge. And long generations, where an early invention becomes the foundation for everything after it.

What actually reduces it

Grounding is the main lever: retrieve the relevant source material and require the model to answer only from what you supplied, ideally citing which passage supports each claim. This does not eliminate the problem — a model can still misread or over-extend a retrieved passage — but it changes the failure from invention to misinterpretation, which is far easier to catch. Second is constraint: asking for structured output against a schema, so the answer can be validated mechanically rather than read for plausibility. Third is verification: a second pass that generates checkable questions about the draft and answers them independently.

The product decision underneath

Because you cannot reduce the rate to zero, the design question is who catches the ones that get through. A drafting tool where a person reviews every output has an acceptable answer. An unattended pipeline writing to a customer-facing system does not, and no amount of prompt engineering will supply one. The most reliable protection is not a better prompt but a workflow in which a wrong answer is visible and cheap.

Measuring it

Teams that treat hallucination as a vibe never find out whether they have improved. Build a fixed set of questions with known-correct answers — including some where the correct response is "I don't know" or "that is not in the provided documents" — and score against it whenever you change model, prompt, or retrieval. Those refusal cases are the ones that matter most and the ones nearly always missing from an evaluation set.

Talking about it honestly

If your product uses a language model, users will eventually meet a confident falsehood. Saying so plainly in the interface, showing sources where you have them, and making correction easy costs less trust than being caught presenting invention as fact. The failure is expected; presenting it as impossible is the part users do not forgive.