Annotation Guidelines

Annotation guidelines are the written instructions that tell people how to label data: what each label means, how to handle the cases that sit between labels, and what to do when the right answer is genuinely unclear. They are the least glamorous artefact in a machine learning project and one of the highest-leverage, because every downstream number inherits whatever consistency they produce. The reason they matter is that labelling is judgement, and unaided judgement diverges. Two careful people given the same ambiguous support ticket and asked whether it is a complaint will disagree, not because either is careless but because the word means slightly different things to each of them. If the guideline does not resolve that, the disagreement is written into the dataset as though it were signal, and a model trained on it learns to reproduce the ambiguity while the evaluation set carries the same noise, so nothing ever reveals the problem. Good guidelines are built from disagreement rather than from first principles. The productive loop is to write a short draft, have several people label the same small batch independently, measure where they diverge, and then spend the effort exclusively on the boundaries that produced the divergence. Abstract definitions rarely help; a pair of nearly identical examples with different correct labels and one sentence explaining why is worth several paragraphs of definition. A few things belong in every set of guidelines. A decision order, so that when two labels both seem to apply there is a stated precedence. An explicit option for cases that cannot be labelled, because forcing a choice manufactures noise and hides the fact that the taxonomy is incomplete. Instructions on how much context to use — whether to read the whole thread or only the message — since annotators silently choose differently otherwise. And a rule about what to do with an unclear case, which should be to escalate it rather than to guess. Guidelines drift, and versioning them is not optional. When a rule changes, labels created under the old version and the new one are no longer the same measurement, and comparing an evaluation run across that boundary produces a change nobody made. Record which version each label was produced under, and when a rule changes materially, relabel the affected slice rather than mixing the two. The same document has a second life now. When a model is used as a judge, the rubric it is given is an annotation guideline with a different reader, and it fails for the same reasons: vague criteria, no tie-breaking order, no option to abstain. Writing the human guideline well first is the cheapest way to get a usable automated judge, and human labels produced under it are the only credible way to check that the judge agrees with anyone.