LLM Inference Economics from First Principles
Source: https://piotr.io/llm-inference-economics Author: Piotr Mazurek Date: 2025-05-28
Summary
Bottom-up analysis of LLM inference economics: what drives the cost per token, how costs scale with model size and hardware, and what the economics look like for different use cases.
Key Claims
- Primary cost driver: GPU memory bandwidth, not compute. Most LLM inference is memory-bandwidth bound during decode.
- Cost breakdown: 70-80% of inference cost is model weight loading; 20-30% is attention computation.
- The “memory wall”: inference speed is limited by how fast you can move model weights from VRAM to compute cores.
- Batch efficiency: batching multiple requests shares weight loading overhead — single-request inference is 10x more expensive per token.
- Quantization economics: 4-bit quantization reduces memory by 4x at ~5% quality degradation — often the right tradeoff.
- The $0.001/token barrier: reaching this cost threshold requires 4-bit quantization + aggressive batching + optimized serving.
Concepts
- Scaling & Compute — economics determine who can afford to run large models
- RL Infrastructure — inference cost is bottleneck for RL rollout generation
Vault Notes
- Kaon GPU One Pager — Flow’s 350-GPU H100+4090 fleet; KV cache, speculative decoding, cost-per-token targets
- ML Engineer Inference JD — Flow’s inference engineering hiring; vLLM, quantization, KV cache optimization