Adjudication

Adjudication is the defined procedure for settling a case where two people who labelled the same example disagreed. Agreement measurement tells you how often that happens; adjudication is what you do about it, and a labelling effort without one accumulates contested rows that are eventually resolved by whoever exports the file. The usual mechanisms are simple. A third person breaks the tie without seeing the first two answers. Or a senior reviewer decides and records why. Or the two original labellers discuss it and reach a joint answer, which is slower but produces the best explanations. Any of these works; what does not work is silently taking the first label, or the majority of three when all three are guessing, since both hide the disagreement instead of resolving it. The important output is not the settled label. It is the reason, and the change to the instructions that the reason implies. A case that two trained people read differently is almost always a case the guidelines did not cover, and writing the resolution into the guidelines is what stops the next twenty examples of the same kind from being contested too. A set that has been adjudicated without the guidelines being updated will produce the same disagreements on the next batch, and nobody will understand why the rate has not improved. Keep a record of which rows required adjudication, and treat that list as a map of where the task is genuinely hard. It is also a warning about the metric: if a large share of a set needed a tie-break, the confidence attached to scores computed on it should be lower than the arithmetic suggests, and reporting the two groups separately is more honest than folding them together. It is also worth deciding in advance what happens when adjudication cannot settle a case at all. Some examples are genuinely ambiguous, and forcing a single answer converts an honest unknown into a wrong reference. Marking such rows as excluded, with the reason, keeps them out of the score while preserving what they demonstrate about the task.