Ceiling Effect

A ceiling effect is what happens when nearly every case in an evaluation set is answered correctly, so the set stops being able to tell two systems apart. The number looks excellent and carries almost no information: a change that made things meaningfully better and a change that did nothing produce the same result, because there was no room left to demonstrate a difference. The mirror image, where almost everything fails and improvements are equally invisible, is less common in practice and behaves the same way. Sets reach the ceiling by two routes. The systems improve, which is the happy version — the material was appropriate when it was written and the work has moved past it. Or the set was assembled from cases that were easy to collect, which are usually the clear, well-formed, typical ones, and it was never able to discriminate in the first place. Either way, the symptom is the same: a high average with the spread collapsing, and month-on-month movements that are within the noise. Watching the distribution rather than the mean is what makes this visible early. If the number is high, look at how many cases every system gets right, because that fraction is dead weight in the comparison. The response is to add material, not to make the grader stricter — a harsher rubric applied to easy cases produces lower numbers without producing more information. Hard cases come from the places you already know about: the failures collected since the set was built, the categories in the taxonomy that stayed stubborn, the inputs users actually send that nobody would have written down. Keep the easy cases as a regression floor, since they are what catches a bad release, and report the two groups separately so an improvement on hard material is visible instead of being averaged into a number that cannot move. The same effect explains why a set inherited from a previous project so often looks flattering on the first run. It was calibrated against a system that no longer exists, and the reassuring number it produces is a statement about that system rather than yours.