The Next 1000x Cost Saving of LLM
Source: https://huizimao.substack.com/p/the-next-1000x-cost-saving-of-llm Author: Huizi Mao Date: 2025-11-26
Summary
Analysis of where the next orders-of-magnitude cost reduction in LLM inference will come from. Identifies speculative decoding, mixture of experts, and adaptive computation as the primary levers.
Key Claims
- The first 1000x reduction: already happened — from GPT-3 pricing in 2020 to commodity inference today.
- Next levers: (1) speculative decoding (use a fast small model to predict, use the large model to verify — 2-3x speedup), (2) MoE (only activate 1/8 of parameters per token), (3) adaptive computation (easier tokens get less compute).
- The asymmetric bet: “thinking” tokens (o1-style) are expensive but only needed for hard tasks. Routing cheap tasks to cheap models gets 5-10x savings.
- Where it ends: at some point, reducing inference cost too much destroys quality. The floor is determined by the model quality needed for the task.
Concepts
- Scaling & Compute — cost reduction trajectory
- Test-Time Compute — adaptive test-time compute as cost management