What is LLM Quantization
Shrinking billion-parameter models without throwing away what they know
TL;DR | An Analogy
Quantization is like reprinting a novel in a smaller font: the story stays the same, the book fits in your pocket. It converts a model's high-precision weights into lower-precision numbers so the model runs faster and cheaper.
The idea
A trained LLM stores each weight as a 32- or 16-bit floating-point number. Quantization maps those weights to a coarser number format — most commonly 8-bit integers (INT8) or 4-bit integers (INT4) — slashing memory by 2–8×. Because GPU/CPU memory bandwidth is the dominant bottleneck at inference time, smaller weights also mean faster token generation, often with only marginal quality loss.
Where it shows up
System-design interviews — Interviewers ask how you'd serve a 70B-parameter model on two A100s instead of eight. Quantization is the first lever to reach for.
On-call / cost incidents — If GPU memory is exhausted and latency is spiking, a quantized model can halve VRAM usage and buy time while you scale hardware.
Real systems — llama.cpp runs GGUF-quantized models (Q4_K_M, Q5_K_S, etc.) on consumer laptops. vLLM, TensorRT-LLM, and Ollama all ship quantization as a first-class serving option. Hugging Face bitsandbytes lets you load a model in 4-bit with two extra Python arguments.
Read the detailed breakdown›
The precision problem
A standard float32 weight uses 32 bits to represent a number with ~7 decimal digits of precision. For a 7B-parameter model that's ~28 GB — before the KV cache or activations touch memory. float16/bfloat16 halves that to ~14 GB, which is already standard practice. Quantization pushes further.
How weights get remapped
The core idea is affine quantization: for a tensor of weights W, compute a scale s and zero-point z, then store Q = round(W / s) + z as an integer. To use the weight, dequantize on the fly: W ≈ s × (Q − z). The scale and zero-point are small metadata tensors kept in float.
Per-tensor vs per-channel — Using one (s, z) pair for an entire weight matrix (per-tensor) is cheapest but least accurate. Per-channel quantization assigns one pair per output channel, preserving more of the weight distribution and significantly improving quality.
Post-training quantization (PTQ) vs quantization-aware training (QAT)
- PTQ runs after training with no gradient updates. GPTQ and AWQ are the dominant PTQ algorithms for LLMs. GPTQ uses a layer-wise quantization approach based on second-order (Hessian-weighted) optimization to minimize reconstruction error. AWQ identifies salient weights based on activation magnitudes and applies scaling to better preserve them within the quantized range.
- QAT simulates quantization during fine-tuning so the model learns to tolerate the precision loss. Higher quality, much higher compute cost.
Activation quantization
Quanting weights alone is weight-only quantization — activations are still computed in float16. Full INT8 inference (W8A8) also quantizes activations, enabling true integer matrix-multiply on hardware (Tensor Cores, CPU VNNI), which is faster but harder to calibrate because activations vary per input.
The memory-bandwidth-compute triangle
At inference, especially with batch size 1 (streaming one user's tokens), the GPU is memory-bandwidth-bound: it spends most time loading weights, not doing arithmetic. Going from 16-bit to 4-bit weights means 4× less data to move per forward pass → roughly 4× more tokens per second, even if compute stays the same. (Important Note: In practice this yields meaningful but sub-linear throughput gains (often 1.5–3×) due to dequantization overhead and kernel efficiency)
Quality trade-offs
Perplexity is the standard proxy metric. INT8 weight-only quantization of a large model (≥13B) is nearly lossless (perplexity increase < 1%). INT4 introduces a modest regression; models below ~7B parameters feel the pain more because each weight carries more responsibility.