Cursor Warp Decode for MoE
Cursor’s blog post on “warp decode” — a new approach to MoE inference decode on Blackwell GPUs that achieves 1.84x throughput improvement with better accuracy.
Key Claims
- Standard MoE inference organizes computation around experts (expert-centric). For small-batch decode, this creates 5 bookkeeping stages that do no actual computation
- Warp decode flips parallelism: each GPU warp (32 lanes) computes one output value, streaming across all routed experts
- Eliminates: padding, scatter, combine step, activation gather buffer, per-expert output buffer
- Result: 1.84x throughput, outputs 1.4x closer to FP32 reference (better accuracy from avoiding intermediate MXFP8 quantization)
- Two fused kernels replace entire pipeline:
moe_gate_up_3d_batchedandmoe_down_3d_batched - Sustains 3.95 TB/s on B200 (58% of measured 6.8 TB/s peak)
Technical Details
- Each warp is completely independent — no shared mutable state, no cross-warp synchronization
- 32 KB of intermediate buffer traffic eliminated per token
- Warp-level butterfly reduction via
__shfl_xor_sync— single hardware primitive, bypasses shared memory - Keeps activations in BF16 and accumulators in FP32 throughout (avoids rounding error from MXFP8)
- Scales linearly with output dimension
Takeaways
- This is specific to small-batch decode (e.g., autoregressive generation for one user). Prefill and large-batch still benefit from expert-centric approach
- “Kernels that improve both performance and accuracy are rare” — warp decode is one
- Speeds up Cursor’s Composer training pipeline (faster RL iteration)
- Tested on Qwen-3 style model on B200 GPUs
Connections
- MoE Architectures — the models this optimizes
- Real-Time RL for Cursor Composer — faster decode enables faster RL iteration
- Kernel Engineering — GPU kernel engineering as 2026 bottleneck
- Coding Agents — Cursor as a leading coding agent