Cost Per Conversation

Cost per conversation is the total inference and infrastructure cost of serving one complete exchange between a user and an assistant, from the first message to the point the interaction ends. It is a unit metric rather than a total, and its value is that it can be compared with something meaningful: the cost of a human handling the same interaction, or the revenue the interaction supports. Calculating it honestly means counting everything the conversation caused, not just the visible replies. A single user turn can trigger several model calls — a query rewrite, a retrieval, a generation, a safety check, a summarisation of history — and an agentic assistant may add tool calls and planning steps on top. Conversation history is resent with each turn, so cost grows superlinearly with length: the tenth turn is far more expensive than the first, which is why average cost per conversation is dominated by the long tail of unusually long sessions. The metric behaves differently from total spend in a useful way. Totals rise with adoption, which is good news, and therefore hide regressions. Cost per conversation isolates efficiency from growth: it moves when a prompt gains a paragraph, when retrieval starts returning more chunks, when a router sends more traffic to an expensive model, or when a change causes users to need more turns to get an answer. Each of those is invisible in the monthly invoice until it is large. Interpreting it requires a second metric beside it. A cheap conversation that fails to help is not a saving, because the user goes on to open a ticket, and the true cost is the sum of both. Pairing cost per conversation with resolution rate, escalation rate or user satisfaction is what makes it a decision tool rather than a target to be gamed; optimising it alone reliably produces a cheaper assistant that deflects less. The distribution matters more than the mean. Most conversations cluster near a modest cost while a small proportion consume many times that, usually because of long histories, large retrieved contexts or agent loops that fail to terminate. Watching a high percentile alongside the average finds the pathological cases that averages hide, and those cases are usually where both the cost and the quality problems live. Practically, instrument at the conversation level from the start: a conversation identifier attached to every model call, with token counts, model version, tenant and feature. Without that, cost per conversation can only be estimated by dividing a monthly invoice by a session count, which is a number too coarse to act on.