Quantization reduces the numerical precision of model weights and/or activations — typically from FP32 or BF16/FP16 down to INT8 or INT4 — to lower memory usage and speed up inference, at the cost of some potential quality loss.

Benefits: lower memory usage, faster inference, cheaper deployment, and the ability to run larger models on smaller hardware — which is why quantization is especially useful for edge deployment, local models, and cost-sensitive production systems. The trade-off is possible quality loss from lower numerical precision, and some tasks degrade more than others under quantization (precision-sensitive tasks tend to be more affected than general conversational tasks). There are different approaches: post-training quantization (quantize after training is done, simplest but can lose more quality), quantization-aware training (the model is trained or fine-tuned with quantization effects simulated, usually preserves more quality), weight-only quantization (only weights are compressed, activations stay higher precision), and activation quantization (activations are compressed too, for maximum savings).