Label Noise

Label noise is error in the reference labels themselves — cases where the answer recorded as correct is not correct. It is distinct from annotators disagreeing, which is measurable and often informative. Noise includes the cases everybody agrees on and everybody has wrong: a guideline that was misread the same way by the whole team, a category that meant something different when the set was built, a batch labelled at the end of a long day, a rule that changed halfway through without the earlier rows being revisited. Its practical effect is a ceiling. If some share of the reference answers is wrong, no system can score perfectly against that set, and worse, the systems that come closest to perfect are the ones that have learned to reproduce the mistakes. Above a certain level of measured quality you are no longer comparing systems to reality but to a flawed transcript of it, and improvements stop showing up in the number while remaining real. That is one reason a metric that has stopped moving is worth investigating from the set's side rather than only from the system's. Finding it is cheaper than most teams expect. Take the cases where the system disagrees with the label most confidently and read them by hand: a meaningful proportion are usually label errors rather than system errors, and each one found is both a fix and a symptom pointing at the instruction that produced it. Fix the label, then fix the guideline, then check whether the same misunderstanding is sitting in rows nobody has looked at. And record the corrections rather than silently editing, because a set that changes underneath a metric makes two runs of the same system incomparable without anyone noticing. It is also worth accepting a floor. Beyond a certain point the labels cannot be made cleaner at reasonable cost, and the honest response is to state the level of noise you believe the set carries and to stop reporting differences smaller than it, rather than to keep publishing a number with more precision than the material supports.