[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"glossary-quantization::en":3,"gloss-cluster-quantization::en":23,"gloss-next-quantization::en":9},{"slug":4,"category":5,"name":6,"definition":7,"meta_desc":8,"faq":9,"schema_markup":9,"related":10},"quantization","core-ai","Quantization","Quantization is a model-compression technique that reduces the numerical precision used to store a model's weights and\u002For activations — for example, converting weights from 16-bit floating point (FP16) down to 8-bit integers (INT8) or even 4-bit (INT4) representations — dramatically shrinking the model's memory footprint and speeding up inference, at the cost of some accuracy loss (usually small if done carefully). This matters enormously for practical AI deployment because model size directly determines what hardware you need and how fast\u002Fcheap inference runs: a 70-billion-parameter model at full FP16 precision needs roughly 140GB of GPU memory just to load, which requires multiple expensive high-end GPUs; the same model quantized to 4-bit needs roughly 35GB, fitting on a single consumer or prosumer GPU, with only a modest quality trade-off for most tasks. For SaaS builders, quantization is the key lever that makes self-hosting open-weight models (Llama, Mistral, Qwen) economically viable, and it's also used by API providers internally to serve models faster and cheaper at scale — when a provider offers a \"fast\" or \"mini\" tier, quantization (alongside genuinely smaller model architectures) is often part of how they achieve it. A concrete worked example: a startup wants to self-host a code-completion model for on-premise\u002Fdata-sovereignty reasons. The full-precision 34B-parameter model requires ~68GB of VRAM — too large for a single A100 GPU (80GB, but tight with overhead). By applying 4-bit quantization (using a technique like GPTQ or AWQ), the same model shrinks to roughly 18GB, comfortably fitting on a single consumer-grade RTX 4090 (24GB), and inference speed improves because less data needs to move through memory — often the actual bottleneck in LLM inference, not raw compute. Quantization isn't free: aggressive quantization (below 4-bit) can measurably degrade output quality, especially on tasks requiring precise reasoning, so production deployments typically benchmark quantized model quality against the full-precision baseline before shipping. Different quantization methods trade off speed, memory, and accuracy differently, and builders self-hosting models should know the main families: post-training quantization (PTQ) methods like GPTQ and AWQ quantize an already-trained model without retraining, fast to apply but with somewhat more accuracy loss at very low bit-widths; quantization-aware training (QAT) incorporates the precision reduction during training itself, generally preserving more accuracy but requiring access to retrain the model, which isn't possible with closed models and is expensive even with open weights. For most SaaS builders self-hosting an open-weight model, an 8-bit or 4-bit PTQ quantization (via a well-supported library like llama.cpp, GGUF, or bitsandbytes) is the practical default, benchmarked against the full-precision baseline on your actual task before committing.","Quantization reduces the numeric precision of a model's weights (e.g., 16-bit to 4-bit) to shrink its size and speed up inference, with some accuracy trade-off.",null,[11,14,17,20],{"slug":12,"name":13},"distillation","Knowledge Distillation",{"slug":15,"name":16},"gpu","GPU (Graphics Processing Unit)",{"slug":18,"name":19},"inference","Inference",{"slug":21,"name":22},"parameter","Parameter",[24,28,32,36,39,42,45,48,51,54,57,60],{"slug":25,"category":5,"name":26,"updated_at":27},"agentic","Agentic AI","2026-08-24T02:46:36+00:00",{"slug":29,"category":5,"name":30,"updated_at":31},"alignment-tax","Alignment Tax","2026-08-24T02:46:37+00:00",{"slug":33,"category":5,"name":34,"updated_at":35},"artificial-intelligence","Artificial Intelligence (AI)","2026-08-24T02:46:38+00:00",{"slug":37,"category":5,"name":38,"updated_at":27},"attention","Attention",{"slug":40,"category":5,"name":41,"updated_at":35},"beam-search","Beam Search",{"slug":43,"category":5,"name":44,"updated_at":31},"benchmark-contamination","Benchmark Contamination",{"slug":46,"category":5,"name":47,"updated_at":31},"catastrophic-forgetting","Catastrophic Forgetting",{"slug":49,"category":5,"name":50,"updated_at":35},"computer-vision","Computer Vision",{"slug":52,"category":5,"name":53,"updated_at":31},"constitutional-ai","Constitutional AI",{"slug":55,"category":5,"name":56,"updated_at":27},"context-window","Context Window",{"slug":58,"category":5,"name":59,"updated_at":35},"deep-learning","Deep Learning",{"slug":61,"category":5,"name":62,"updated_at":27},"diffusion-model","Diffusion Model"]