Run-to-Run Variance

Run-to-run variance is the spread you get when the same system is evaluated twice on the same set without anything being changed. It is not the same as sampling error, which is about the cases you chose; this is about the system and the measurement producing different answers on identical inputs. Several sources contribute. Generation may be sampled rather than fixed. Retrieval may return a different set of passages as an index updates or as ties break differently. A grading model brings its own variation. Ordering, batching and concurrency can change results in systems where earlier work influences later work. Infrastructure can time out one case in a hundred on a busy afternoon and not on a quiet one. The consequence is the practical one and it is frequently ignored: a difference between two runs that is smaller than this spread is not a result, no matter how neat the arithmetic looks or how much a team wants it to be. Weeks are lost re-explaining changes that were noise, and worse, changes are shipped on the strength of them. The remedy is cheap. Before comparing anything, run the same unchanged configuration several times and see how much the number moves on its own. That figure becomes the resolution of your instrument, and it should be reported next to the score rather than kept in somebody's head. If a proposed improvement is smaller than the spread, either measure more, reduce the variation where you can, or state honestly that the change is not detectable with the setup you have — which is a legitimate finding rather than a failure, and considerably better than an announcement that quietly reverses next month. It is also worth reporting the spread to whoever reads the results, not only to the people producing them. A stakeholder shown a single figure will treat it as exact, and the second time a number moves in the wrong direction the conversation is much easier if everyone already knows how much movement means nothing.