How to Build an Evaluation Set That Does Not Go Stale

What this is about, and what it is not

Most teams that measure an AI feature have a set of test cases somewhere. It was put together in a week, from whatever was easy to collect, by someone who has since moved to another project, and it is still producing a number every time the pipeline runs. The number is treated as a fact about quality. Usually it is a fact about a set that stopped describing the product some time ago.

This is not the guide about running an evaluation when you have no data team; how to evaluate AI output quality without a data team covers the machinery and the cheap version of the scoring. Nor is it about judging an agent across a multi-step task, which how to tell whether an AI agent is actually working takes up. This one is about the material itself — where the cases come from, what keeps them representative, and how you know when the set has expired.

Start from failures you already have

The worst way to build a set is to sit in a room and invent examples. Invented cases are well-formed, unambiguous and polite, and they measure a version of the task nobody actually sends. The best raw material is failures you have already collected: support tickets about the feature, outputs somebody complained about, the cases an engineer pasted into a channel with the word look, anything a human corrected.

Take those first, and read them before deciding anything. A hundred real failures will teach you more about what the set needs than any amount of planning, and the phrases you write while reading them become the first draft of the categories you will later count by.

Decide the groups before you decide the size

The question is never how many cases; it is which kinds, and how many of each. Choose the groups you would want to make a separate decision about — segment, language, document type, request length, difficulty, the failure categories you just found — and take enough of each that its score means something on its own.

This deliberately over-represents the rare and expensive cases relative to traffic, which is correct for finding problems and wrong for describing typical performance. So record the composition next to the set, and when you want a figure that represents live behaviour, weight the groups back to their real proportions. The commonest reporting error in this whole area is quoting the unweighted average of a set that was built to be hard.

Write down what correct means, then test the writing

Before labelling anything, write the instructions: what counts as a good answer, how to handle the cases that sit between categories, and what to do when the right answer is genuinely unclear. Then hand a sample to two people independently and compare. Where they disagree, the guidance is missing a rule, not the people are careless — and the fix goes into the instructions, not into a conversation.

Keep the ambiguous rows rather than forcing them. A case that two careful readers see differently is the most informative row you have, and marking it as contested is more honest than freezing one of the two answers as truth.

Keep an easy floor and a hard head

A set needs both halves and they do different jobs. The easy cases are a regression floor: they are what catches the release that broke something basic, and their value is precisely that they should always pass. The hard cases are where improvement is visible, and where two candidate approaches can be told apart.

Report them separately. Folded together, the easy majority dominates the average, and a real gain on hard material disappears into a number that barely moves. That is also the early sign that the set has reached a ceiling — everything passes, the spread collapses, and month-on-month differences are noise.

Feed it from production on a schedule

The gap that opens fastest is created by your own releases. Every new capability begins life with no cases at all, so its first weeks are measured by material that predates it and the score stays reassuring while the risk is highest. The remedy is a routine rather than a project: on a fixed cadence, take a sample of recent real inputs, add the ones the set has no equivalent of, and label them.

The same routine is the place to add the failures found since last time. A complaint that becomes a permanent case is a complaint that cannot recur silently.

Protect it from the system it measures

Keep the material somewhere the system cannot read. That means not pasting its cases into prompts as examples, not letting the collection the system retrieves from be refreshed with an export that contains it, and checking overlap mechanically when any training or reference data is assembled.

Accept as well that a set you have iterated against many times is partly spent, however careful you were: tuning until the score is satisfactory is fitting to the set rather than measuring against it. Hold back a portion that is looked at rarely, so there is always material the system has never been aimed at.

Know when it has expired

Three signals are worth watching. Everything passes and the spread has collapsed. A slice of recent traffic contains categories the set has no examples of. Or the instructions have been amended so often that older rows were labelled under rules that no longer apply. Any of those means the set is describing a task the product has moved past.

Retiring material is normal and should be routine rather than dramatic: keep the version, note what changed, and do not compare a score across the boundary without saying that the set changed underneath it.

What a maintained set looks like

Not large. A few hundred cases, grouped, with the composition written down beside them, a short instruction document that has been revised more than once, a note of which rows were contested and why, and a date on the last time production material was added. Somebody's name against it, and a recurring hour in a calendar.

That is the whole practice, and it is unglamorous enough that it tends to be nobody's job. The reason to give it to somebody is that every decision downstream — whether a change helped, whether a feature is ready, whether a supplier's model is better than the one you have — is only as good as this material, and a set nobody maintains degrades in a direction that always looks like success.