Benchmark Contamination

Benchmark contamination (also called data or test-set leakage) happens when the questions and answers from a public benchmark end up in a model's training data. The model then "scores" high partly by having memorized the test, not by genuinely solving it — like a student who saw the exam beforehand. Because frontier models train on huge web scrapes and popular benchmarks are all over the web, some contamination is common, which is one reason leaderboard numbers can overstate real-world ability. For builders comparing models, the lesson is not to pick one on published benchmark scores alone. A model that tops a leaderboard may not be the best on your actual task. Practical note: build a small private evaluation set from your own real inputs and expected outputs, keep it out of any prompt logs that could be trained on, and use it to compare candidate models directly. Your own held-out eval is far more trustworthy than a public benchmark the model may already have seen.