[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"guide-how-to-tell-whether-an-ai-agent-is-actually-working::en":3,"guide-related-how-to-tell-whether-an-ai-agent-is-actually-working::en":17},{"slug":4,"title":5,"excerpt":6,"body":7,"meta_title":5,"meta_description":8,"keywords":9,"category":15,"published_at":16,"updated_at":16},"how-to-tell-whether-an-ai-agent-is-actually-working","How to Tell Whether an AI Agent Is Actually Working","Scoring one output is a solved problem; judging a system that chooses its own steps is not. The measurements that survive production — the recorded path, the denominator, who scores, and the failures that raise no error.","\u003Ch2>A different question from scoring one output\u003C\u002Fh2>\n\u003Cp>Judging a single AI output is a solved-enough problem: fix a set of real inputs, describe what an acceptable answer looks like, score against it repeatably. An agent needs something else, because the thing being judged is not one response but a whole task carried out over many steps, where the system chose the steps itself and may not take the same route twice.\u003C\u002Fp>\n\u003Cp>That changes what counts as evidence. The final answer alone is weak evidence, the vendor's demo is no evidence at all, and impressions collected by trying it a few times are actively misleading on a system that behaves differently on identical input. What follows is the small set of measurements that survive contact with production.\u003C\u002Fp>\n\u003Ch2>Judge the path, not only the destination\u003C\u002Fh2>\n\u003Cp>Record the full run: every model call, every tool invocation with its arguments and results, and the final output, in order. Two runs can produce the same correct answer while one called three tools and the other called eleven, retried a failing endpoint six times and arrived by luck. Only one of those will still work next month, and nothing in the answer distinguishes them.\u003C\u002Fp>\n\u003Cp>Reading a handful of these end to end, by hand, is the highest-yield hour available in the first weeks of any agent deployment. It is where you discover that it is calling an expensive tool twice because two descriptions overlap, that it silently gave up on a step and continued, or that a third of its steps are spent re-reading context it already had. None of that appears in an aggregate.\u003C\u002Fp>\n\u003Ch2>Count what arrived, not what it chose to attempt\u003C\u002Fh2>\n\u003Cp>The most common inflation in an agent metric is the denominator. A success rate computed over the tasks the system decided to take on excludes everything it declined, abandoned, or never recognised as a task — which is precisely the hard end of the distribution. Compute the rate over everything that came in, and report the attempted rate separately if you want it. The gap between the two is itself a useful figure: it is the share of the problem the agent currently does not touch.\u003C\u002Fp>\n\u003Cp>Decide explicitly what partial completion counts as. A run that did most of the work and stopped cleanly is a different outcome from one that finished with a wrong result, and both differ from one that stopped halfway having already changed something in a live system. Folding all three into failure hides the distinction that matters when you are deciding what to fix; counting the first as success flatters the agent.\u003C\u002Fp>\n\u003Ch2>Stop letting the agent mark its own work\u003C\u002Fh2>\n\u003Cp>Asking a model whether it completed the task is asking it to grade its own homework in the style it wrote it, and the answer is usually yes. Self-reported success is the least reliable signal in the system and the one most dashboards are built on.\u003C\u002Fp>\n\u003Cp>Replace it wherever the task allows with a check something else performs: a test that passes, a schema that validates, a record that now exists with the expected values, a total that reconciles against a second source. Where no such check exists, use human review on a random sample rather than on the cases somebody happened to open — a sample chosen by curiosity measures curiosity. A second model as a judge is a reasonable intermediate step, provided you have checked its agreement with human scores on a sample first and re-check it when either model changes.\u003C\u002Fp>\n\u003Ch2>Track the work that did not disappear\u003C\u002Fh2>\n\u003Cp>Every automation percentage has an honest counterpart: the share of runs a person had to step into. Count the explicit escalations, but also the quiet corrections made afterwards by whoever received the output, the cases customers reopened, the runs abandoned and redone by hand, and the approvals that were genuinely read. Counting only what the agent escalated itself measures its self-awareness, not its performance.\u003C\u002Fp>\n\u003Cp>Segment this, because the average hides the shape. Interventions concentrate — in one request type, one customer segment, one integration that returns awkward data, one hour of the day when an upstream system is slow. A high rate driven by a narrow band is an engineering problem you can fix; the same rate spread evenly is a scope problem, and the answer is to narrow what the agent attempts rather than to keep tuning it.\u003C\u002Fp>\n\u003Cp>Read it against cost, too. Intervention is where the promised labour saving actually goes, so an agent with an unremarkable rate but expensive interventions — senior people, slow context rebuilding, an annoyed customer — can be worse value than one that intervenes more often but cheaply.\u003C\u002Fp>\n\u003Ch2>Assume the failures will be silent\u003C\u002Fh2>\n\u003Cp>An agent rarely fails by crashing. It returns something fluent and on time that happens to be wrong, or it runs to completion having quietly done nothing: a retrieval that came back empty, a tool that returned an empty list rather than an error, a filter that stopped filtering. Uptime, latency and error rate all look excellent throughout, which is why conventional monitoring is so reassuring about AI features and so uninformative.\u003C\u002Fp>\n\u003Cp>Detection has to assert on outcomes rather than on execution. Expected volumes and their absence. Distributions that should be stable — steps per run, tools per run, cost per task — where a sudden move is a signal even before anyone knows what it means. A small fixed set of known inputs re-run on a schedule and checked. And reconciliation against something that would have to be wrong in the same way to agree with you.\u003C\u002Fp>\n\u003Cp>The organisational version of this test is the discovery path. If failures are found by customers, by a colleague who happened to look, or in a quarterly review, they are silent regardless of how green the dashboards are, and the interval between a failure starting and somebody noticing is the number worth driving down.\u003C\u002Fp>\n\u003Ch2>Re-measure whenever anything moves\u003C\u002Fh2>\n\u003Cp>An agent's numbers are a trend, not a fact about next week. They move when the model changes, when a prompt or a tool description is edited, when an upstream system starts returning different data, and when the mix of incoming requests shifts — and that last one causes more unexplained drops than the other three together, because nothing in the system changed at all.\u003C\u002Fp>\n\u003Cp>Keep recorded runs and replay them against fixed tool responses after each change, so you can see which previously working paths now diverge. And report any rate next to the distribution of tasks it was measured on. A success rate without that context is not a measurement, it is a slogan.\u003C\u002Fp>","Measuring a multi-step agent in production: judge the path, fix the denominator, stop self-scoring, track interventions, and expect the failures to be silent.",[10,11,12,13,14],"ai agent evaluation","task success rate","intervention rate","agent monitoring","agent trajectory","how-to","2026-09-05T03:30:02+00:00",[18,23,27,31,36,41],{"slug":19,"title":20,"excerpt":21,"updated_at":22},"ai-tool-pricing-models-seat-vs-usage-vs-credits","AI Tool Pricing Models: Seat-Based vs Usage-Based vs Credits","The three common ways AI tools charge — per seat, per usage, and by credits — and how to reason about which one will actually be cheaper for the way your team works.","2026-08-05T14:32:26+00:00",{"slug":24,"title":25,"excerpt":26,"updated_at":22},"how-ai-image-generators-differ-diffusion-vs-the-rest","How AI Image Generators Differ: Diffusion vs the Rest, in Plain Terms","A non-technical explanation of how AI image generators work, why the diffusion approach became dominant, and what practical differences to expect between tools.",{"slug":28,"title":29,"excerpt":30,"updated_at":22},"how-to-automate-your-workflow-without-code","How to Automate Your Workflow Without Code","A practical sequence for building automations that survive: picking the right process, mapping it before touching a tool, and handling the failure cases that break most first attempts.",{"slug":32,"title":33,"excerpt":34,"updated_at":35},"how-to-build-a-chatbot-without-coding","How to Build a Chatbot Without Coding","A practical route to a working chatbot using no-code tools: deciding scope, connecting your own content, handling the questions it cannot answer, and knowing what it will cost.","2026-08-05T14:32:27+00:00",{"slug":37,"title":38,"excerpt":39,"updated_at":40},"how-to-build-an-evaluation-set-that-does-not-go-stale","How to Build an Evaluation Set That Does Not Go Stale","Most evaluation sets are assembled once, from whatever was easy to collect, and then quietly stop representing the work. How to build one from failures you already have, keep it fed from production, protect it from the system it measures, and know when it has expired.","2026-09-09T03:30:02+00:00",{"slug":42,"title":43,"excerpt":44,"updated_at":45},"how-to-change-a-prompt-without-breaking-production","How to Change a Prompt Without Breaking Production","Prompts get edited in a text box and shipped in seconds, which is why they break things quietly: no compiler, no stack trace, no obvious moment of failure. Give them the release discipline code gets.","2026-08-24T03:30:02+00:00"]