Vendor Scorecard

A vendor scorecard is a written set of weighted criteria used to compare suppliers on the same terms. Each candidate is rated against every criterion, the ratings are weighted by how much the buyer actually cares, and the totals become an argument rather than a verdict. Its real value is not the arithmetic; it is that the weights were agreed before anyone saw a demo. The criteria worth including divide into a few families. Fit — does it do the job for the workflows that matter, at your volume. Risk — security posture, compliance evidence, financial stability, and what happens if the supplier disappears. Cost — not the licence but the total, including implementation, integration engineering, training and the internal time to run it. Experience — support responsiveness, documentation quality, and how the product behaves when something goes wrong. Exit — how data and configuration come back out. Weighting is where the honesty lives. If every criterion is important, none is, and a flat scorecard reliably picks the broadest product rather than the best one for the job at hand. A useful discipline is to force the weights to differ: name the two or three criteria that would make you walk away regardless of the total, and treat those as gates rather than scores, so a strong performance elsewhere cannot buy a pass on data residency or an unusable rollback story. Scores need evidence attached. A number with no note beside it is a memory of a feeling, and by the time the decision is questioned nobody can reconstruct why one vendor got a four. Record the artefact behind each rating — the trial result, the reference call, the security document, the page in the contract — and the scorecard survives contact with a procurement review months later. For AI tools the scorecard usually needs two criteria most templates lack. The first is output quality on your own data, measured on a fixed set of cases rather than on the vendor's demo, because generic capability and capability in your domain diverge sharply. The second is change management: model updates, prompt changes and deprecations arrive on the vendor's schedule, and a supplier who ships them without notice or version pinning is a different risk from one who does not. The last use of a scorecard is the one most teams skip. Keep it, and re-score the chosen vendor at renewal against the same criteria. That converts a purchase decision into a relationship that can be reviewed, and it is the cheapest way to notice that a tool you bought for one reason is now being kept for none.