Large Transformer Model Inference Optimization
Source: https://lilianweng.github.io/posts/2023-01-10-inference-optimization/ Author: Lilian Weng Date: 2023
Summary
Comprehensive survey of inference optimization techniques for large transformer models. Covers quantization, knowledge distillation, pruning, KV cache optimization, speculative decoding, and hardware-level techniques.
Key Claims
- Four main optimization categories: model compression, parallelism, memory optimization, hardware efficiency
- KV cache is the primary memory bottleneck for long-context inference
- Speculative decoding: small draft model proposes tokens, large model verifies in parallel
- Quantization-aware training vs post-training quantization tradeoffs
- Inference is bottlenecked by memory bandwidth, not compute (at small batch sizes)
Entities
Concepts
- Scaling & Compute — inference as the deployment-time compute challenge