Attention-Free Foundation Model at 14B Scale
Source: https://x.com (tweet) Author: unknown (research team) Date: 2025-2026
Summary
A completely attention-free foundation model trained at the 14B scale for $4,000. Performance matches other models of similar scale including transformers and hybrid models. Authors describe this as “the end of the transformer era marching slowly closer.”
Key Claims
- Completely attention-free — no self-attention mechanism
- 14B parameter scale — not a toy model
- Training cost: $4,000 — remarkably cheap
- Performance: matches transformers and hybrid models at 14B scale
- Represents the “end of transformer era marching slowly closer”
Concepts
- Scaling & Compute — architecture alternatives to transformers