Token Efficiency

Token efficiency is the practice of achieving a given quality of output using fewer tokens of input and output. It is one of the few optimisations in an AI system that improves three things at once: cost falls because providers charge per token, latency falls because there is less to process and generate, and accuracy often rises because a shorter, better-organised prompt is easier for a model to follow. The last point surprises people who assume more context is safer. Models attend imperfectly across long inputs, and material in the middle of a large context is more likely to be underweighted than material at the edges. Padding a prompt with marginally relevant documents therefore does not merely cost more; it can actively dilute the instructions and evidence that matter, which is why aggressive retrieval settings often make answers worse as well as more expensive. The biggest wins are usually structural rather than stylistic. A system prompt sent with every request should be audited hard, because every word in it is paid for on every call. Retrieved context is typically the largest and most inflatable component, and returning fewer, better-ranked passages usually beats returning many. Conversation history replayed in full grows without bound, and summarising older turns while keeping recent ones verbatim is standard practice. Few-shot examples earn their tokens only until the pattern is established; beyond that they are cost with no effect. Output deserves the same scrutiny, and often more, since output tokens are usually the more expensive kind. Asking for a structured result rather than prose, setting a sensible maximum length, telling the model not to restate the question, and avoiding requests for explanation that nobody reads all cut spend directly. Where a model is asked to reason before answering, the reasoning is real work and worth paying for — but only where the task needs it. Caching is the other half of the discipline. Where a provider supports caching a stable prompt prefix, ordering the prompt so that the unchanging part comes first turns a repeated cost into a much smaller one. Semantic caching of whole answers helps where the same questions recur, and simple deduplication of identical concurrent requests removes spend nobody intended to make. Measure it as tokens per unit of work — per conversation, per document, per resolved case — alongside the quality metric, and watch both when anything in the configuration changes. Efficiency pursued without a quality measurement beside it eventually finds the trim that saves money and quietly breaks the feature.