Prefill-Decode Disaggregation
Separating the prefill (prompt processing) and decode (token generation) phases of LLM inference onto different hardware, because they have fundamentally different compute profiles.
Why It Matters
- Prefill is compute-intensive (matrix multiplications over the full prompt)
- Decode is memory-bandwidth-intensive (one token at a time, need to read all KV cache)
- Running both on same hardware means one phase is always underutilized
Evolution
- Single-cluster PD: Standard approach (Mooncake, vLLM, SGLang). Prefill and decode on separate nodes within same RDMA fabric.
- Cross-datacenter PD (PrfaaS): Selective offloading of long prefills to compute-dense clusters, KVCache over commodity Ethernet. Enabled by hybrid attention architectures reducing KV by ~10x.
Hardware Divergence
- Prefill-optimized: NVIDIA Rubin CPX (high compute throughput)
- Decode-optimized: Groq LPU (extreme memory bandwidth)
- These can’t easily share the same RDMA fabric → need cross-datacenter solutions
Key Constraint
The “bandwidth wall”: for dense-attention models, a single 32K request generates ~60 Gbps of KVCache traffic. Hybrid attention (Kimi Delta Attention, sliding window) reduces this by 10x, making cross-datacenter transfer feasible.
Connections
- PrfaaS — cross-datacenter disaggregation
- TurboQuant KV Cache — KV compression for cheaper transfer
- LLM Inference Economics — cost implications