A Visual Guide to Quantization
Source: https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization Author: Maarten Grootendorst Date: 2024-08-10
Summary
Excellent visual explainer on quantization — the technique of reducing model weight precision from float16/32 to int8/4. Key reference for understanding how models are made smaller and faster.
Key Claims
- Quantization basics: reduce the bit-width of model weights (FP32 → FP16 → INT8 → INT4). Each halving approximately halves memory.
- Quality vs. compression tradeoff: INT8 loses ~1-2% quality; INT4 loses 3-5%; below INT4 quality degrades rapidly.
- Post-training quantization (PTQ): quantize after training. Simple but suboptimal.
- Quantization-aware training (QAT): train with quantization noise, then quantize. Better quality at same compression.
- Why it matters: a 70B model in FP16 needs 140GB VRAM; in INT4, it needs ~40GB — fits on 2 A100s instead of requiring 4.
Concepts
- Scaling & Compute — quantization democratizes large model access
- RL Infrastructure — quantized models for RL rollout generation