GLOSSARY
Quantization
Compressing a model's weights from 16-bit numbers to 8, 4 or fewer bits — making big models run on smaller hardware with a modest quality cost.
Neural networks store knowledge as numbers — usually 16-bit floating point values. Quantization rounds those numbers to smaller representations: 8-bit integers, 4-bit, even 1.58-bit in research. A 70B-parameter model that needs 140 GB of memory at full precision fits in ~40 GB at 4-bit, making it runnable on a single high-end GPU.
The quality tradeoff is smaller than intuition suggests for many tasks, which is why local-AI tools ship quantized model files by default. When you see file names like Q4_K_M or Q8_0 in a model registry, those are quantization formats.