Prefill-as-a-Service: KVCache Cross-Datacenter
Paper from Moonshot AI + Tsinghua University on cross-datacenter prefill-decode disaggregation for LLM serving.
Key Claims
- PD disaggregation is the standard, but KVCache transfer keeps prefill and decode tightly coupled within a single RDMA fabric
- Hybrid-attention architectures (like Kimi Linear/KDA, sliding window attention) reduce KVCache by ~10x, making cross-datacenter transfer plausible
- PrfaaS selectively offloads only long-context prefills to compute-dense clusters, transfers KVCache over commodity Ethernet to decode clusters
- Achieves 54% higher throughput than homogeneous PD and 32% higher than naive heterogeneous baselines on internal 1T-parameter hybrid model
- Key insight: reduced KVCache is necessary but not sufficient — need system-side selective offloading + bandwidth-aware scheduling
Architecture
- Length-based threshold routing: only sufficiently long requests cross datacenter
- Bandwidth-aware scheduler reacts to fluctuating link conditions
- Global KVCache manager with hybrid prefix-cache pool
- Enables independent scaling of prefill (compute-dense) and decode (bandwidth-optimized) capacity
Takeaways
- Hardware is diverging: NVIDIA Rubin CPX for prefill throughput, Groq LPU for decode bandwidth
- The “bandwidth wall” is the real constraint — for dense-attention models, a single 32K request generates ~60 Gbps of KVCache traffic
- Cross-datacenter serving becomes practical only with hybrid attention + intelligent routing
- This is the next step beyond Mooncake’s single-datacenter KVCache-as-resource approach
Connections
- Scaling & Compute — heterogeneous hardware for different inference phases
- TurboQuant KV Cache — complementary approach: compress KV for transfer
- Transformer Inference Optimization — broader survey of inference techniques
- RL Infrastructure — similar disaggregation patterns for training