Error Analysis

Error analysis is the practice of reading individual failures and grouping them, rather than reading the score. The score answers how much; error analysis answers what, and only the second one tells you what to build next. The method is unglamorous. Pull the cases the system got wrong, take a few dozen of them, read each one, and write a short phrase describing what actually went wrong — not a grade but a cause. Then sort the phrases into groups and count. What almost always emerges is that a small number of distinct problems account for most of the failures, and that they are not the problems the team had been discussing. The retrieval step returned nothing useful for a whole class of question. The system answered a different question from the one asked. The output was correct but in a format the downstream step rejected. Those are four different pieces of work with four different owners, and the aggregate score contained none of that information. Doing it well requires a few disciplines. Sample rather than cherry-pick, since the memorable failures are rarely the common ones. Read the input as well as the output, because a surprising share of errors turn out to be cases that were unanswerable as posed. Keep the notes in a file rather than in a meeting, so the second round can be compared with the first. And re-run the analysis after a fix rather than trusting the score to confirm it, since the score can improve while the group you targeted is untouched and a different one has quietly shrunk. The output of a good session is a ranked list of causes, each with a count and a name, which is the raw material for both a fix list and a taxonomy. It is also the cheapest way to discover that a system is failing silently. Aggregate scores are computed over cases that produced an output; the ones that timed out, returned nothing, or were dropped before scoring are frequently missing from the denominator entirely, and reading a sample of raw runs rather than a results table is usually how somebody notices.