Blind Evaluation

Blind evaluation means the person or process scoring an output does not know which system produced it. It is the cheapest correction available for a bias that is extremely reliable and almost entirely unconscious: an output known to come from the new approach is read more generously than the same output attributed to the old one, particularly when the reader has spent a fortnight building the new approach. The effect does not require dishonesty and does not disappear when people are warned about it. Making it blind is mostly mechanical. Strip the labels from the outputs, shuffle the order so that one system is not consistently first, and make sure nothing in the formatting gives the source away — a distinctive way of opening a reply, a length that only one system produces, a leftover marker in the text. Where two outputs are compared side by side, randomise which side each appears on, because there is a measurable preference for one position that has nothing to do with quality. Where a model is used as the grader, the same discipline applies: put the candidates in a random order and do not tell it which is the incumbent. A few things are worth blinding beyond the system name. The team the output came from, when several are competing. Whether a case was one the system previously failed, since knowing that shifts attention. And, where it can be arranged, the identity of the customer or account, which otherwise imports its own expectations. None of this is expensive, and the usual reason it is skipped is that the result feels obvious in advance — which is exactly the condition under which the bias operates most strongly. One caveat: blinding protects the comparison, not the standard. If the grading rubric itself is lenient, blind scoring will apply that leniency evenly to everything, and both systems will look better than they are. Blind evaluation tells you which is preferred; it does not tell you whether either is good enough.