Gemma 4 Architecture Analysis
Source: https://x.com/eliebakouch Author: Elie Bakouch Date: 2025-2026
Summary
Technical breakdown of Gemma 4’s architecture. Notes per-layer embedding only on smaller layers, and describes it as a “galaxybrained architecture” — not a standard transformer. Very subtle differences between model sizes.
Key Claims
- Gemma 4 uses per-layer embedding only on the smaller models in the family
- Architecture is non-standard — explicitly “not a standard transformer”
- Each model in the family has subtle but meaningful architectural differences
- Points to Gemma3n as another non-standard predecessor
Concepts
- Scaling & Compute — architectural innovation as complement to scale