VRAM

VRAM is the dedicated high-bandwidth memory built onto a GPU, and during training or inference it holds the model's weights, the activations produced as data flows through the network, and the KV cache accumulated over a generation. Its capacity is a hard constraint rather than a performance preference: if the working set does not fit, the model does not run on that device at all, and no amount of patience compensates. That binary quality is why VRAM tends to be the first number teams check when evaluating whether a given open-weight model can be self-hosted on hardware they own or can rent, and why cloud GPU instances are commonly priced and selected by memory tier more than by any other single spec. It is also the direct motivation behind two of the most common workarounds in the stack — quantization, which shrinks the weights' memory footprint by lowering numerical precision, and model parallelism, which splits one model across several GPUs so their combined memory can hold it. Two details routinely catch people out. VRAM requirements do not scale purely linearly with parameter count at a fixed precision: activation memory, framework overhead, and above all the KV cache — which grows with context length and batch size together — add substantially to the raw weight footprint, so a model that looks like it should just fit often does not once you serve real concurrent traffic at long context. And having enough VRAM to load a model does not guarantee acceptable speed. Memory bandwidth and compute throughput are independent properties of the card, and a GPU with generous capacity but modest bandwidth can load a large model and still generate tokens slowly, which is a common and expensive surprise when selecting hardware on capacity alone. The KV cache deserves separate emphasis because it is the part that scales with your traffic rather than with your model choice, and therefore the part that breaks capacity planning done on a quiet development machine. Weights are a fixed cost the moment you pick a model; cache is a variable cost that grows with every concurrent user and every additional token of context, which means a deployment that runs comfortably in testing can hit the memory ceiling under production concurrency at long context lengths without anything about the model having changed. Plan capacity against your worst realistic combination of context length and concurrency, not against the weights alone. If the numbers do not fit, the ordered set of levers is roughly: quantize the weights, cap or summarize context, reduce maximum concurrency per device, and only then split the model across GPUs, since parallelism adds communication overhead and operational complexity that the earlier options do not.