GPU Glossary
Source: https://modal.com/gpu-glossary Author: Modal Date: 2026-06-09
Summary
Modal’s comprehensive GPU glossary — covering hardware terms (SM, VRAM, NVLink, PCIe), memory hierarchy (L1/L2/HBM), and performance metrics (FLOPS, memory bandwidth, occupancy). Primary reference for understanding GPU performance discussions.
Key Claims
- Most important GPU performance concept: memory bandwidth, not FLOPS. Most LLM inference is memory-bandwidth bound.
- Key hierarchy: registers > L1 cache > L2 cache > HBM (VRAM) > CPU RAM. Each level is 10-100x slower than the previous.
- NVLink vs PCIe: NVLink (NVIDIA’s proprietary) provides 900 GB/s inter-GPU bandwidth; PCIe provides ~64 GB/s. Critical for multi-GPU training.
Concepts
- RL Infrastructure — GPU fundamentals for ML infrastructure
- Scaling & Compute — hardware fundamentals for compute scaling