Keeping 20,000 GPUs Healthy
Source: https://modal.com/blog/gpu-health Author: Modal Date: 2026-01-15
Summary
Modal’s engineering post on operating over 20,000 GPUs across major cloud providers. Describes the failure modes, monitoring systems, and operational practices for GPU fleet management at scale.
Key Claims
- GPU failure modes: (1) thermal throttling (silent degradation), (2) memory errors (ECC correctable vs. uncorrectable), (3) NVLink failures (inter-GPU communication), (4) complete GPU death.
- Most failures are subtle: GPUs don’t crash, they degrade silently — producing wrong outputs or slowing down.
- Detection: continuous health checks (GEMM benchmarks, memory tests) running on idle GPUs to catch degradation.
- Multi-cloud complexity: AWS, GCP, Azure have different failure profiles and different health APIs.
- Key insight: fleet reliability requires proactive exclusion of degraded GPUs, not just response to crashes.
- Economic insight: a training run that completes on a degraded GPU produces subtly wrong results — harder to detect than a crash.
Entities
- Modal — serverless compute platform for AI workloads
Concepts
- RL Infrastructure — GPU fleet health is prerequisite for RL at scale
- Scaling & Compute — reliability matters as much as raw compute