Coverage Gap

A coverage gap is a kind of case your evaluation set contains none of. It is the most under-examined weakness in any measurement programme, because a gap produces no failures — the system is never tested on the missing material, so nothing goes red, and the number reported is a confident statement about a narrower world than the one the product lives in. Gaps arrive by ordinary means. The set was built from a sample of traffic taken before a feature shipped, so nothing exercises that feature. It was drawn from one market, or one language, or the accounts that happened to be in the export. A customer segment joined afterwards and brought a document type nobody had seen. An integration began sending requests in a shape the original cases never had. Each of these is invisible from inside the results, and none of them shows up as a lower score. Finding gaps means comparing the set against something other than itself. Two comparisons do most of the work. Take a recent slice of live inputs and ask, for each rough category in it, how many cases the set has — the answer is often zero for several. And take the list of things a stakeholder would want reassurance about, by segment, language, size, sensitivity, and check which of those the set can actually speak to. Then decide honestly which gaps to fill and which to declare. Filling every one is not possible, and a set that grows without limit becomes too slow to run and too expensive to label. Declaring is the cheaper half: say in the report which categories the number does not cover, so that a reader is not left inferring coverage from the fact that a score exists. The gap that opens fastest is the one created by your own release notes. Every new capability starts life with no evaluation cases at all, so the first weeks of its existence are measured by a set that predates it, and the score stays reassuring precisely while the risk is highest.