How to Run a Vendor Evaluation for an AI Tool

Start from the workflow, not the feature list

Feature comparisons reward breadth, and every mature product is broad. Write down instead the two or three pieces of work you actually want to change: who does them today, how long they take, how often they happen, and what a good outcome looks like. That description is the specification the evaluation measures against, and it survives the demos.

Being specific also narrows the field faster than any shortlist exercise. A tool that is excellent at drafting and mediocre at classification is the right tool for one of those workflows and the wrong one for the other, and a generic scoring grid will not tell you which you are looking at.

Agree the criteria and their weights before you see anything

Write the criteria down and weight them while nobody has yet been impressed. Fit, risk, total cost, support experience and exit are the usual families, and the weights are where the honesty lives — if everything matters equally, the broadest product wins by default.

Separate the criteria that are gates from the ones that are scores. Data residency, a required certification, or the ability to export your data are not things a strong performance elsewhere should be able to buy a pass on, so treat them as pass or fail and stop evaluating a candidate that fails one.

Build a fixed test set out of your own cases

The single highest-value artefact in an AI evaluation is a fixed set of real inputs with known good outcomes. Draw them from actual traffic, weight them towards the cases that matter commercially, and include the awkward ones deliberately: ambiguous requests, adversarial inputs, questions with no good answer, and the failures that embarrassed you last quarter.

Run the same set against every candidate. This is what converts three separate demos into one comparison, and it keeps working after the purchase, because the same set becomes the regression check when the vendor changes a model.

Score what a person still has to do

Accuracy on its own is a weak signal, because the value of the tool is the work it removes. Measure how often output is accepted with no edit, how often it needs a small correction, and how often it has to be discarded and redone by hand. The last category is the expensive one: a wrong answer that looks right costs more than no answer at all.

Where judgement is required, write a short rubric and have more than one person score independently. Disagreement between reviewers is itself a finding, because it usually means the task is less well defined than anyone assumed.

Test the operational surface, not just the output

Ask what happens when things go wrong, and then cause some of it. Look at behaviour under rate limits, the shape of error responses, how latency behaves at your concurrency, and whether there is a way to turn the feature off without a deployment. Check whether the vendor pins model versions, how deprecations are announced, and how much notice you would get.

Cost the whole thing, not the licence

Build the total from the licence, the consumption, the integration engineering, the internal time to run it, and the training. For consumption pricing, estimate a realistic monthly volume from the pilot rather than from the roadmap, and ask what controls exist — spend caps, budget alerts, per-tenant limits — because a product with no ceiling is a product with an unbounded invoice.

Then ask the exit question while you still have leverage: what comes back, in what format, how quickly, and at what price. A vendor who cannot answer that before signing will not answer it better afterwards.

End with a decision and a review date

Set the decision date before the evaluation starts, name who decides, and record the evidence in front of them. Then put a review date in the calendar for the first renewal and keep the test set, so the question at renewal is not whether everyone still likes the tool but whether it still does the work you bought it for.