What to Do When an AI Feature Gets Something Wrong

A different kind of failure

Ordinary software fails loudly. A request throws, a page errors, a monitor fires, and someone is paged. An AI feature fails quietly: it returns a fluent, well-formatted, entirely plausible answer that happens to be wrong. Nothing in the response distinguishes it from a correct one, no exception is raised, and the first signal is usually a customer who acted on it. Designing for that difference is the whole job, because you cannot prevent every wrong answer and you can decide what happens when one occurs.

Decide in advance what a wrong answer costs

Before anything else, classify the feature by consequence. Drafting a first version of a message a human will read and edit is low stakes — errors are caught by the person already in the loop. Summarising a document someone will act on without checking is higher. Sending a message, changing a record, issuing a refund, or answering a customer directly is higher still. The design that follows should differ sharply between those categories, and the most common product mistake is applying the same interaction pattern to all of them.

Design so the mistake is catchable

Several patterns make errors visible rather than absorbed. Show the source: if an answer came from retrieved documents, link them, so a reader can check the claim in one click rather than trusting the sentence. Keep a human in the loop where the action is irreversible — proposing an action for approval is a different product from taking it. Make the output structured where possible, because a schema lets you validate mechanically what prose lets you only hope. And say what the system does not know: a feature that declines to answer outside its scope is worth far more than one that always produces something.

Expressing uncertainty deserves care. Vague hedging in every response trains people to ignore it. It is more useful to distinguish clearly between the cases the system is designed for and the ones it is not, and to fail visibly on the latter.

When one reaches a customer

Treat it as an incident, with the same steps you would use for any other. Capture the exact input, the exact output and the version of everything involved — prompt, model, retrieved context — before anything changes, because these are hard to reconstruct later and the same prompt may not reproduce the same answer. Tell the customer plainly what happened and what it affected; an AI feature getting something wrong is not surprising to anyone in 2026, but discovering that it was known and not mentioned is.

Then find the actual cause before changing the prompt. The wrong answer might be missing context, retrieval that returned the wrong passage, an ambiguous instruction, an input the feature was never meant to handle, or genuinely stale source data. Each has a different fix, and prompt-editing as a reflex tends to move the failure rather than remove it.

Turn the incident into coverage

Every real failure should end up as a permanent test case: the exact input added to your evaluation set with the correct answer written down. This is the cheapest reliable source of test data you will ever have, because it is drawn from what actually went wrong rather than what you imagined might. Over a year this set becomes the thing that lets you change models and prompts without fear, and it accumulates only if adding to it is part of closing the incident.

What to tell users up front

Say the feature is AI-generated, say what it is good for, and say plainly that it can be wrong and should be checked where it matters. This is not a legal formality — it sets the expectation that determines whether a single bad answer is a shrug or a breach of trust. Give people an obvious way to report a bad output, and make sure those reports reach someone, because a feedback control that goes nowhere is worse than none: it collects the evidence you needed and throws it away.