GPU (Graphics Processing Unit)

A GPU (Graphics Processing Unit) is a processor originally designed for rendering graphics that turned out to be exceptionally well-suited to the kind of massively parallel matrix multiplication that neural networks require — a single GPU can perform thousands of simple arithmetic operations simultaneously across its many cores, versus a CPU's handful of cores optimized for sequential, general-purpose tasks. This architectural fit is why GPUs (particularly NVIDIA's data-center chips like the A100 and H100) became the foundational hardware of the modern AI industry, both for training large models (which requires enormous parallel compute over weeks) and for inference (running trained models to serve user requests, where GPU memory bandwidth and parallel throughput determine how fast and how many requests can be served simultaneously). This matters for SaaS builders in a few direct ways. If you're calling a hosted LLM API (Claude, GPT, Gemini), GPU cost and availability is the provider's problem, but it's baked into your per-token pricing — GPU scarcity and cost is a major reason frontier-model API pricing is what it is, and why smaller/distilled/quantized models (which need less GPU memory and compute per request) are so much cheaper to call. If you're self-hosting an open-weight model (Llama, Mistral) for data-sovereignty or cost-at-scale reasons, GPU selection becomes a direct, first-order infrastructure decision: a model's parameter count and precision (quantized or not) determines how much GPU VRAM you need, which determines whether you need one consumer GPU, one data-center GPU, or a multi-GPU cluster with fast interconnects. A concrete worked example: a startup self-hosting a 13B-parameter model in FP16 needs roughly 26GB of GPU VRAM just to load the weights, plus overhead for the context window and batching — comfortably fitting on a single NVIDIA A100 (40GB or 80GB variant), whereas the same model in full precision at 70B parameters needs roughly 140GB, requiring either multiple GPUs with model-parallelism or aggressive quantization to fit on cheaper hardware. GPU availability and pricing (spot vs. on-demand cloud instances, or long-term reserved capacity) has become a genuine strategic constraint for AI-heavy startups, not just a line-item cost. For teams weighing self-hosting against API access purely on cost, the breakeven point depends heavily on volume and utilization: API pricing bakes in the provider's GPU cost plus margin but requires zero upfront investment or idle-capacity risk, while self-hosting requires either purchasing hardware (large upfront cost, ongoing depreciation) or renting cloud GPU instances (which are expensive per-hour if utilization is low, since GPUs are billed whether or not they're actively serving requests) — self-hosting typically only becomes cost-effective at sustained high volume, where the fixed GPU cost is amortized across enough requests to beat API per-token pricing, which is why most SaaS products under meaningful scale default to API access rather than managing their own GPU fleet.