turboquant-kv-cache-compression

TurboQuant: Extreme KV Cache Compression

Source: https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/ Author: Google Research Date: 2025

Summary

TurboQuant compresses KV cache to 3 bits with zero accuracy loss and no fine-tuning required. Achieves up to 8x performance improvement over standard 32-bit on H100 GPUs, 6x KV memory reduction. Combines PolarQuant (polar coordinates) with Quantized Johnson-Lindenstrauss (1-bit error correction).

Key Claims

• KV cache quantized to 3 bits — no training or fine-tuning required
• Up to 8x performance improvement over 32-bit unquantized keys on H100
• At least 6x KV memory reduction
• Zero accuracy loss — theoretical guarantees + empirical benchmark validation
• PolarQuant: converts Cartesian → polar coordinates to eliminate memory overhead
• QJL: 1-bit quantization to correct residual errors from PolarQuant
• Enables faster semantic search at scale

Concepts

• Scaling & Compute — memory efficiency as scaling enabler
• Context Engineering — KV cache as stored context