Test-Time Compute
Definition
Test-time compute (TTC) refers to the computational resources spent during inference to improve model outputs, as opposed to resources spent during training. The key insight: models can “think longer” on hard problems rather than always taking the same number of compute steps per token.
Why It Helps
Three complementary explanations from Lilian Weng:
- Compute as resource: CoT dramatically increases FLOPs per answer token, allowing variable compute proportional to problem difficulty.
- System 1 → System 2: Mirrors Kahneman’s dual-process theory — fast autoregressive prediction = System 1; deliberate reasoning steps = System 2.
- Latent variable modeling: Thoughts
zare hidden variables that enrichP(y|x)— sampling multiple chains of thought is sampling from the posteriorP(z|x,y).
Main Approaches
| Approach | Mechanism | Limitation |
|---|---|---|
| Best-of-N (parallel) | Generate N outputs, pick best | Bounded by model capability |
| Self-consistency | Majority vote across CoT chains | No external verification |
| Sequential revision | Iteratively self-correct | Requires external feedback; naive self-correction often fails |
| Process reward models | Step-by-step scoring | Expensive to train |
| RL on checkable tasks | Policy gradient on verifiable rewards | Only works for problems with ground truth |
The RL Path to Better Reasoning
The biggest TTC gains came from RL training on problems with automatically verifiable answers (STEM, coding with unit tests). This produced:
- OpenAI o1, o3
- DeepSeek-R1 (policy gradient only, no supervision needed)
See RL Infrastructure for the training side.
Open Questions
- Faithfulness: Do CoT chains reflect actual computation, or are they post-hoc rationalization? Evidence suggests models often explain reasoning differently from how they compute. See How Claude Thinks.
- Scaling laws: How does test-time compute scale with problem difficulty? Larger models benefit more from thinking time.
- Continuous space thinking: Is discrete token CoT the right substrate? Alternatives: recurrent architectures, latent variable approaches.