[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"guide-what-to-do-when-your-evaluation-and-your-users-disagree::en":3,"guide-related-what-to-do-when-your-evaluation-and-your-users-disagree::en":17},{"slug":4,"title":5,"excerpt":6,"body":7,"meta_title":5,"meta_description":8,"keywords":9,"category":15,"published_at":16,"updated_at":16},"what-to-do-when-your-evaluation-and-your-users-disagree","What to Do When Your Evaluation and Your Users Disagree","The dashboard says the feature is fine and the people using it say it is not. That gap is a finding rather than a nuisance, and there is a short sequence of checks that usually locates it — in the rubric, in the set, in the grader, or in the score itself.","\u003Ch2>The situation\u003C\u002Fh2>\n\u003Cp>The evaluation runs weekly and the score is comfortable. Meanwhile support is collecting complaints about the same feature, an account manager has forwarded three examples, and somebody senior has started a sentence with I tried it myself and. Two sources of evidence disagree, and the instinct in the room is to defend the one that is easier to defend, which is the number, because it is precise and repeatable and has a chart.\u003C\u002Fp>\n\u003Cp>This is not the guide about a single bad output and what to do in the hour after it, which what to do when an AI feature gets something wrong covers. It is not about a business metric moving unexpectedly, which how to tell whether a number actually moved works through. This is a standing disagreement between an offline measurement and lived experience, and it is worth treating as a finding rather than a nuisance: something is being measured that is not the thing people care about, and that is a fixable and quite specific problem.\u003C\u002Fp>\n\u003Ch2>Start by assuming the users are right\u003C\u002Fh2>\n\u003Cp>Not out of politeness. It is the cheaper hypothesis. An evaluation is a model of the task, models are wrong in enumerable ways, and the list of ways is short enough to work through in an afternoon. Whereas the alternative hypothesis — that a group of people using the product daily are collectively mistaken about their own experience — is expensive to investigate and almost never true in the form it is first stated.\u003C\u002Fp>\n\u003Cp>The checks below are ordered by how often they turn out to be the answer, not by how flattering they are.\u003C\u002Fp>\n\u003Ch2>Are they complaining about what you measured?\u003C\u002Fh2>\n\u003Cp>This is the most common resolution and the easiest to miss. Read the actual complaints and write down what each one is about. Very often the rubric asks whether the output was correct, and the complaints are about something else entirely: it took too long, the tone was wrong for the situation, the format broke the workflow it was pasted into, it answered a narrower question than the one asked, it was right but useless without context the user had to go and find.\u003C\u002Fp>\n\u003Cp>Every one of those can coexist with a high correctness score, indefinitely, because nothing in the measurement asks about them. The fix is not to argue but to add what is missing to the rubric, or to add a second measurement alongside it and report both.\u003C\u002Fp>\n\u003Ch2>Does the set contain their cases at all?\u003C\u002Fh2>\n\u003Cp>Take the complaints and look for their equivalents in the evaluation material. A surprising share of the time there are none — the users complaining are in a segment, a language, an industry or a size band that the set was never built from, or they are exercising a capability that shipped after the set was assembled.\u003C\u002Fp>\n\u003Cp>A gap of this kind never appears as a failure. The system is simply not tested on that material, so the score stays high while being silent about exactly the population that is unhappy. If this is the answer, the number was not wrong; it was answering a narrower question than anyone realised.\u003C\u002Fp>\n\u003Ch2>Is the grader agreeing with people?\u003C\u002Fh2>\n\u003Cp>Take a handful of the outputs that users complained about, and run them through the scoring exactly as it runs in the pipeline. Then look at what it said. If the grader passes outputs that a person has explicitly objected to, you have found the problem, and it is a problem in the measurement rather than in the system.\u003C\u002Fp>\n\u003Cp>This happens for ordinary reasons. A rubric written early was lenient about something that has since become important. A scoring step checks for the presence of a fact and not for the presence of an invented one alongside it. An automated grader agrees with anything confident and well-formed. Whatever the cause, a grader that disagrees with your users is not a neutral instrument, and every historical number produced by it inherits the same leniency.\u003C\u002Fp>\n\u003Ch2>Is the score real?\u003C\u002Fh2>\n\u003Cp>Four quick checks, all of which have been known to explain the entire gap on their own. Is the pipeline evaluating the version that is actually deployed, or one from before the last few changes? Have any evaluation cases leaked into the system — used as examples in prompt material, or swept into a reference collection — so that the set is being fitted to rather than measured against? Has the set hit a ceiling, so that everything passes and the score has stopped being able to move? And how much does the number vary when nothing is changed at all, because a comfortable score that bounces by more than the gap you are discussing is not evidence of anything.\u003C\u002Fp>\n\u003Ch2>Is the complaint concentrated?\u003C\u002Fh2>\n\u003Cp>Check whether the unhappiness is broad or sits in one place. Concentration is not a reason to dismiss it. A segment can be a small fraction of usage and still be the one that renews the largest contracts, or the one that talks publicly, or the one where a mistake is expensive rather than annoying.\u003C\u002Fp>\n\u003Cp>What concentration does tell you is how to report from now on. If one group is having a materially different experience, an average across everyone will keep hiding it, and the honest presentation is that group's number beside the overall one rather than folded into it.\u003C\u002Fp>\n\u003Ch2>What to do with the answer\u003C\u002Fh2>\n\u003Cp>Whichever check resolves it, close the loop the same way: turn the complaints into permanent cases in the set, so that this particular failure cannot recur without the measurement noticing. Update the rubric or add the second measurement the complaints implied. Report the affected segment separately. And say out loud, once, that the previous number was measuring something narrower than everyone had assumed — because a quiet correction leaves the same misunderstanding available to the next person who reads a chart.\u003C\u002Fp>\n\u003Cp>It is also worth writing down which check resolved it, because the same gap opens again for the same reason. A team that found a coverage hole once will find one again the next time a segment or a capability arrives, and a note saying we checked coverage first and that was it turns the next investigation from an afternoon into ten minutes.\u003C\u002Fp>\n\u003Ch2>The one move to avoid\u003C\u002Fh2>\n\u003Cp>Do not resolve the disagreement by adjusting the measurement until it agrees with the score. Raising a threshold, dropping a hard category from the set, or switching to a metric that happens to look better are all available and all fast, and each converts a solvable measurement problem into an unsolvable trust problem. The point of the evaluation was to be the thing that argues with you.\u003C\u002Fp>","Your offline score says fine and your users say otherwise. A sequence of checks: the rubric, coverage of their cases, the grader, leakage, and concentration.",[10,11,12,13,14],"AI evaluation","user feedback","offline evaluation","quality metrics","product quality","how-to","2026-09-10T03:30:02+00:00",[18,23,27,31,36,41],{"slug":19,"title":20,"excerpt":21,"updated_at":22},"ai-tool-pricing-models-seat-vs-usage-vs-credits","AI Tool Pricing Models: Seat-Based vs Usage-Based vs Credits","The three common ways AI tools charge — per seat, per usage, and by credits — and how to reason about which one will actually be cheaper for the way your team works.","2026-08-05T14:32:26+00:00",{"slug":24,"title":25,"excerpt":26,"updated_at":22},"how-ai-image-generators-differ-diffusion-vs-the-rest","How AI Image Generators Differ: Diffusion vs the Rest, in Plain Terms","A non-technical explanation of how AI image generators work, why the diffusion approach became dominant, and what practical differences to expect between tools.",{"slug":28,"title":29,"excerpt":30,"updated_at":22},"how-to-automate-your-workflow-without-code","How to Automate Your Workflow Without Code","A practical sequence for building automations that survive: picking the right process, mapping it before touching a tool, and handling the failure cases that break most first attempts.",{"slug":32,"title":33,"excerpt":34,"updated_at":35},"how-to-build-a-chatbot-without-coding","How to Build a Chatbot Without Coding","A practical route to a working chatbot using no-code tools: deciding scope, connecting your own content, handling the questions it cannot answer, and knowing what it will cost.","2026-08-05T14:32:27+00:00",{"slug":37,"title":38,"excerpt":39,"updated_at":40},"how-to-build-an-evaluation-set-that-does-not-go-stale","How to Build an Evaluation Set That Does Not Go Stale","Most evaluation sets are assembled once, from whatever was easy to collect, and then quietly stop representing the work. How to build one from failures you already have, keep it fed from production, protect it from the system it measures, and know when it has expired.","2026-09-09T03:30:02+00:00",{"slug":42,"title":43,"excerpt":44,"updated_at":45},"how-to-change-a-prompt-without-breaking-production","How to Change a Prompt Without Breaking Production","Prompts get edited in a text box and shipped in seconds, which is why they break things quietly: no compiler, no stack trace, no obvious moment of failure. Give them the release discipline code gets.","2026-08-24T03:30:02+00:00"]