KV Cache

The KV cache (key-value cache) is the memory an LLM uses during generation to avoid recomputing work. As a transformer generates each new token, it needs the attention keys and values of every previous token; rather than recompute them at every step, it caches them. This is why generating the thousandth token is fast — but it also means memory grows linearly with context length, so a long conversation or a big document can consume gigabytes of GPU memory just for the cache. For SaaS builders, the KV cache explains several realities: why long contexts cost more and run slower, why "prompt caching" features (which persist the KV cache for a reused prefix) can cut latency and price dramatically on repeated system prompts, and why serving many concurrent long-context users is memory-bound. When your inference bill or latency balloons with context length, the KV cache is usually the reason — reusing cached prefixes and trimming context are the main levers you have. Mechanically, the cache holds the key and value projection vectors that each attention layer already computed for every token in the sequence, so the model never re-derives them; only the newest token's projections are calculated fresh at each step. Its footprint grows with sequence length and batch size together, which is why the cache — not the weights — is usually what limits how many concurrent requests a single GPU can hold. That constraint propagates all the way to the price list: cache pressure caps a provider's requests-per-GPU, and requests-per-GPU sets the floor under per-token pricing. It is also the technical basis for the "cached input" discounts several providers now advertise. Two misconceptions are worth clearing up. First, the KV cache does not persist between separate API calls by default — it lives in the memory of whichever inference server is handling an active or recently active context, which is exactly why cross-request prompt caching has to be an explicit provider feature rather than something that happens for free. Second, a bigger cache does not make a model smarter: it is purely an efficiency mechanism, never a capability enhancement, so "more cache" is never the answer to a quality problem. In practice your levers are structural rather than tunable — put the stable part of your prompt (system instructions, few-shot examples, a fixed reference document) at the front so a provider's prefix cache can hit it, keep the volatile part at the end, and resist appending unbounded conversation history when a rolling summary would do. Serving stacks manage the cache in pages to limit fragmentation, which is why paged attention and continuous batching are almost always discussed together.