AI Incident Reporting

AI incident reporting is the path by which a wrong, harmful or unexpected AI output gets recorded, triaged and acted on, rather than being noticed by one person and forgotten. It is the feedback loop that turns evaluation from a pre-launch exercise into something that keeps working after launch, when the inputs stop resembling the test set. These incidents look different from outages. Nothing is down, no error rate moves, and monitoring stays green while the system confidently produces something false, discloses information the reader should not have seen, or behaves in a way the design never anticipated. Because the usual signals do not fire, the report almost always starts with a human — a support agent, a customer, an employee who noticed the answer was wrong. That makes the reporting channel the whole design problem. It has to be reachable from where the output appears, which usually means a control next to the response rather than a form somewhere in an intranet. It has to capture the context that makes the case reproducible: the input, the output, the conversation or trace identifier, the model and prompt version in use, and the time. Without those, a report reduces to "it said something odd last week", which cannot be investigated. Triage needs its own criteria, because the volume is mostly low-severity and the serious cases hide in it. Severity here is about consequence — did anyone rely on it, was anyone harmed, was data disclosed, is it reproducible, how many users are exposed to the same path — and not about how embarrassing the output was. High-severity cases need the same discipline as a security incident: containment, a rollback path, and a decision about who has to be told. The value only lands if the reports feed back. Confirmed cases belong in the evaluation set so the same failure is caught next time, and clusters of them belong in the record of the system that produced them. An organisation that collects incidents without adding them to its tests will keep rediscovering the same failure, and its evaluation scores will keep looking better than the product is.