Reproducibility

Reproducibility, in evaluation, is the property that running the same measurement again gives you the same answer — or that where it does not, you can say precisely why. It matters because every claim about improvement is a comparison between two runs, and a comparison is only meaningful if the difference between the runs is the thing you changed. Where it is not, teams end up celebrating an improvement that a colleague cannot see, and the disagreement is usually resolved by whoever ran it most recently rather than by evidence. Several things have to be pinned for a run to be reproducible, and only some of them are obvious. The version of the evaluation set, including any rows quietly corrected since the last run. The version of the instructions or rubric used to score. The version of the system under test, including the prompt material and anything it retrieves, since a retrieval step reading a changing index makes yesterday's run unrepeatable even with everything else fixed. The grader, whether that is a person, a script or a model, and its own configuration. The order cases were presented in. And any randomness in generation, of which a seed is one lever rather than the whole answer. Complete determinism is often unattainable, and chasing it is the wrong goal. The achievable and useful version is that every one of those factors is recorded alongside the number, so that when two runs disagree the list of possible causes is short and inspectable. A results file with the score and nothing else is not a measurement; it is a memory of one, and it stops being trustworthy the moment anyone asks a second question about it. The cheapest habit that buys most of this is to write the run configuration into the results file automatically rather than into a message afterwards. Anything that depends on somebody remembering to note what they changed will be accurate for about a fortnight.