Human Baseline

A human baseline is the score people achieve on the same evaluation set, under the same instructions and scoring rules, as the system being measured. It exists because an isolated score is uninterpretable. A result that sounds mediocre may be close to the best anyone manages on a genuinely ambiguous task; a result that sounds strong may be well below what a trained person does in a minute. Without the comparison there is no way to tell which situation you are in, and the argument about whether the system is good enough proceeds on intuition. Measuring it properly means giving people the same conditions rather than better ones. Same cases, same guidelines, same time pressure, and scored by the same rule — including the awkward part, where a human answer is graded by whatever grader you are using for the system, so that both are subject to the same idiosyncrasies. It is also worth measuring more than one person, since the spread between individuals is often wider than the gap you are arguing about, and a single expert's score is a data point rather than a baseline. Two cautions. The baseline describes performance on this set under these instructions, not human ability in general, and quoting it as the latter is how a modest measurement becomes an inflated claim. And a system that matches the human number is not therefore a replacement, because the errors will be distributed differently: people tend to fail on the tedious and the rushed, systems on the unusual and the underspecified, and a comparable average can hide the fact that one of them fails precisely where failure costs the most. Measuring it once is also not enough if the set changes. A baseline belongs to a version of the evaluation material, and comparing this quarter's system score against a human number produced on last year's cases is a comparison of two different tasks wearing the same name.