Inferring Model Size from Knowledge (Factual Knowledge Doesn’t Compress)
Author: @bojie_li | Date: 2026-04-29
Argument that closed labs can hide model sizes but can’t hide what their models know. Factual knowledge doesn’t compress — the amount of factual recall a model demonstrates places a lower bound on its parameter count.
Key Claims
- Factual knowledge stored in model weights has an information-theoretic lower bound
- You can estimate model size from the breadth and accuracy of factual recall
- This means closed-source model sizes can be reverse-engineered from behavior
- Knowledge compression has fundamental limits (unlike reasoning which can be more parameter-efficient)
Takeaways
- Clever approach to de-anonymizing model sizes from frontier labs
- Distinction between knowledge (hard to compress) and reasoning (more compressible) is key
- Implications for understanding the real scale of GPT-5, Claude, Gemini etc.
- Connects to information theory and rate-distortion bounds