Pairwise Evaluation

Pairwise evaluation scores model outputs by comparing two of them head-to-head and asking which is better, instead of assigning each an absolute grade. It leans on a well-known fact about both humans and LLM judges: relative judgments ('A or B?') are far more consistent than absolute ones ('rate this 1–10'), where scores drift and cluster. Chatbot Arena popularized the pattern at scale — collecting pairwise votes between anonymous models and converting them into Elo-style rankings. For builders, it's the reliable way to answer 'did my new prompt or model actually get better?': show a judge (or a human) the old and new outputs for each case in your eval set and count wins, ties, and losses. Two things to control: position bias — judges tend to favor whichever answer comes first, so swap the order and average — and the many comparisons needed to rank more than two options. Report win-rate with ties, not a single fragile average score.