Alex Xi's Notes

lilian-weng-why-we-think

Jul 06, 20262 min read

Why We Think — Lilian Weng

URL: https://lilianweng.github.io/posts/2025-05-01-thinking/
Author: Lilian Weng
Published: May 2025

Summary

A comprehensive technical review of why and how language models benefit from “thinking time” — test-time compute (TTC) through chain-of-thought (CoT) reasoning. Weng synthesizes research from psychology, latent variable modeling, and RL to explain the mechanisms behind thinking tokens.

Key Claims

  • System 1 vs System 2 analogy: Fast autoregressive generation = System 1 (Kahneman). CoT lets models engage in deliberate System 2 reasoning by generating intermediate steps.
  • Compute as a resource: CoT lets models perform far more FLOPs per answer token. A key property: variable compute depending on problem hardness.
  • Latent variable framing: CoT thoughts can be modeled as hidden variable z that mediates P(y|x). Methods like best-of-N are sampling from the posterior P(z|x,y).
  • RL for reasoning: The biggest gains come from training with RL on problems with checkable answers (STEM, coding). Policy gradient on these problems led to o1, o3, and DeepSeek-R1.
  • Parallel vs sequential revision: Best-of-N (parallel) is simpler but bounded by model capability. Sequential revision (self-correction) only works reliably with external feedback — naive self-correction without feedback often doesn’t improve.
  • Faithfulness problem: Models often explain their reasoning differently from how they actually compute. CoT may be post-hoc rationalization rather than the actual reasoning path.
  • Scaling laws for thinking time: Larger models benefit more from additional thinking time. There are regularities in how performance scales with test-time compute.
  • Continuous-space thinking: Alternative approaches include recurrent architectures (thinking tokens) and thinking as latent continuous variables, not just discrete token sequences.

Key Papers Referenced

  • Wei et al. 2022 — coined “chain-of-thought”
  • DeepSeek-AI 2025 — R1 tech report (policy gradient → strong performance)
  • Zelikman et al. 2022 (STaR), Wang et al. 2023 (self-consistency)
  • Kahneman 2013 — Thinking Fast and Slow

Relevance to Alex’s Work

Test-time compute and CoT are the foundation for coding agents and reasoning-heavy tasks. The RL training pathway (checkable rewards → policy gradient) is directly relevant to RL Infrastructure work. The faithfulness problem has implications for interpretability.

Backlinks

  • No backlinks found