Automation Bias

Automation bias is the tendency to trust a machine's output more than the evidence warrants, and to stop looking for the evidence at all once the machine has spoken. It has been studied for decades in aviation and clinical decision support, and it is the human factor that decides whether an AI feature makes a team better or merely faster at being wrong. It shows up in two directions. People accept a recommendation that is incorrect but plausible, because checking is effortful and the system is usually right — and people fail to act on something the system did not flag, because the absence of an alert reads as an all-clear. The second is the more dangerous, because nothing about it looks like a decision, and it leaves no trace of a judgement that was never made. Fluency makes generative systems a special case. Confidence in language is a signal humans use to judge reliability in each other, and it is nearly free for a model to produce, so the least well-grounded output can be the most persuasive. A hedged, poorly written correct answer will lose to a polished wrong one in almost any review process that does not deliberately compensate. The countermeasures are design choices rather than warnings. Show what the answer rests on — the source, the record, the calculation — close enough that checking is easier than trusting. Surface disagreement and uncertainty rather than smoothing it away. Avoid pre-filling the decision the reviewer is meant to make. And measure whether review is real: an override rate of nearly zero across a large volume means the system is being deferred to, not read. For buyers, this is why a demo proves less than it appears to. The relevant question is not whether the tool is right most of the time, but what happens to the team's own vigilance after a few months of it being right most of the time — because that is when the case it gets wrong arrives, and by then nobody is looking.