Megakernels Remain Underrated for Training
Author: @willdepue | Date: 2026-05-06
Will Depue argues megakernels (fusing multiple operations into single GPU kernel launches) are underrated for training. References @flappyairplanes hinting at powerful training megakernel work that could significantly improve throughput.
Key Claims
- Megakernels (fused multi-op kernels) are underexploited in training pipelines
- @flappyairplanes working on powerful training megakernel implementations
- Kernel fusion reduces memory bandwidth bottlenecks and launch overhead
- Potential for significant training speedups beyond what current frameworks achieve
Takeaways
- Custom kernel engineering remains a competitive edge for training infrastructure
- The gap between framework-generated kernels and hand-optimized fused kernels is still large
- Training-specific megakernels could shift the compute-efficiency frontier