Hitting 1,000 Tokens per Second on a Single RTX 5090
Source: https://blog.alpindale.net/posts/5090_decode_optimization/ Author: AlpinDale Date: 2026-02-09
Summary
Engineering post achieving 1,000 tokens/second LLM inference on a single consumer RTX 5090 GPU through custom decoding kernels.
Key Claims
- 1,000 tok/s on a single consumer GPU: achieved through speculative decoding + custom CUDA kernels.
- The RTX 5090 advantage: doubled memory bandwidth vs. 4090 (1.79 TB/s), plus better flash attention support.
- Speculative decoding: small “draft” model predicts 4-8 tokens, large model verifies in parallel — 2-3x speedup.
- Custom kernel: rewrote the draft model’s decoding kernel for the 5090’s specific SM architecture.
- Implication: at 1000 tok/s, interactive AI assistants become genuinely instant — no perceptible latency.
Concepts
- RL Infrastructure — fast inference enables faster RL rollout generation
- Scaling & Compute — edge inference optimization