How to Cut the Running Cost of an AI Feature

Measure per unit of work before you change anything

Absolute spend rises with adoption, which is good news, so it hides efficiency problems. The number to watch is cost per unit of work — per conversation, per document, per resolved ticket — because that isolates how expensive each interaction is from how many of them there are.

Getting it requires instrumenting the application, not reading the invoice. Every model call should record input and output token counts, the model and version, the feature it served, the tenant and the workflow step. Without those fields, cost is a single line nobody can break down, and every optimisation is a guess.

Find where the tokens actually are

Before optimising, look at the composition of a typical request. In most systems the largest input component is retrieved context, followed by the system prompt and then conversation history. Output is smaller in volume but usually more expensive per token, and reasoning-heavy models can spend a great deal producing a short visible answer.

Look at the distribution too, not just the average. A small share of requests with enormous contexts or long agent loops often accounts for a disproportionate part of the bill, and fixing those pathological cases is cheaper and safer than trimming everything.

Send less: the retrieval and prompt pass

Returning fewer, better-ranked passages usually beats returning many, and it frequently improves accuracy as well, because models attend imperfectly across long inputs and marginal documents dilute the ones that matter. Reranking a small candidate set is often the cheapest quality improvement available.

Then audit the system prompt line by line — every word is paid for on every call — and stop replaying conversation history verbatim once it grows, summarising older turns while keeping recent ones intact. Few-shot examples earn their tokens only until the pattern is established.

Choose smaller where the task allows

Not every request needs the largest model. Routing simple, well-specified work to a smaller model and reserving the expensive one for cases that need it is usually the single biggest lever available, and a cascade that escalates only when a cheap answer fails a check keeps quality while paying premium rates for a minority of traffic.

This only works if you can tell when the cheap answer was inadequate. Structured tasks give you that cheaply — does it parse, does it match the schema, does the citation exist — while open-ended generation is much harder to route safely.

Repeat less: caching and deduplication

Where a provider supports caching a stable prompt prefix, ordering the prompt so the unchanging part comes first converts a repeated cost into a much smaller one. Beyond that, collapse concurrent identical requests into one execution, cache answers to questions that genuinely recur, and scope any cache by tenant and user so a shared answer never crosses a permission boundary.

Stop the accidental spend

Retries after a timeout set below the real latency of the operation quietly double cost. Agent loops with no step limit can run until something stops them. Failed generations that are retried without a backoff turn a provider incident into a large invoice. Each of these is a bug rather than an optimisation opportunity, and each is usually worth more than a week of prompt trimming.

Protect quality while you do it

Run every change against a fixed evaluation set before it ships, and watch the quality metric alongside cost in production. Cost reduction pursued alone will eventually find the trim that saves money and breaks the feature — a shorter prompt that drops an instruction, a smaller model that fails on the awkward cases, a cache that serves a stale answer.

Finally, set the guardrails that stop the next surprise: spend caps, per-tenant limits, budget alerts on cost per unit rather than on the total, and a kill switch that falls back to something the product can survive.