Inference Cost

Inference cost is the recurring expense of running a model to serve requests, as distinct from the one-off cost of building or fine-tuning one. It is the cost that never goes away: it grows with adoption, with the size of the context you send, with how many steps a workflow takes, and with every retry. For most teams shipping AI features, inference is the dominant ongoing cost and the one that determines whether a feature is viable at scale. The mechanics are worth understanding precisely because they are unintuitive. Hosted models generally charge separately for input and output tokens, and output is typically the more expensive of the two. A long system prompt sent with every request is a fixed tax on every call. Retrieved context is often the largest input component and the easiest to inflate accidentally. Reasoning-heavy models can consume a large number of tokens producing an answer whose visible length is short. Multi-step systems multiply all of this. An agent that plans, calls three tools, reads results and revises has made several model calls for one user action, each carrying the accumulated conversation. Cost per user action, not cost per call, is therefore the number to track, and the ratio between them is one of the more revealing metrics an AI system can expose. The levers divide into three groups. Send less: trim system prompts, retrieve fewer and better chunks, summarise history rather than replaying it, and cache the stable prefix of a prompt where the provider supports it. Use less: route simple requests to a smaller model, reserve the expensive one for cases that need it, and constrain output length where the task allows. Do less: deduplicate identical requests, cache answers to repeated questions, and avoid retry loops that silently double spend. Self-hosting changes the shape of the cost rather than removing it. A rented accelerator is paid for whether or not it is busy, so the economics depend on sustained volume and on how efficiently the serving stack batches work. Below a certain steady load, a metered endpoint is usually cheaper, and the crossover point moves with provider pricing, model efficiency and how spiky your traffic is. The discipline that keeps this manageable is measuring cost per unit of value from the beginning: per conversation, per document processed, per ticket resolved. Absolute spend rises with success and therefore hides regressions, while a unit figure exposes the prompt change that added context or the router change that quietly sent more traffic to the expensive path.