A Postmortem of Three Recent Issues
Source: https://www.anthropic.com/engineering/postmortem-three-issues Author: Anthropic Engineering Date: 2025-09-18
Summary
Anthropic’s public postmortem on three infrastructure incidents between August and mid-September 2025. Commendable transparency from an AI lab on reliability failures. Covers root causes, mitigation, and lessons learned.
Key Claims
- Three incidents: (1) API latency spike from KV cache pressure, (2) model serving interruption from deployment misconfiguration, (3) partial outage from upstream cloud provider.
- KV cache issue: during a traffic spike, available KV cache filled, causing queue buildup and latency. Fix: dynamic KV cache prioritization.
- Deployment misconfiguration: automated deployment pushed a config change that caused serving errors. Fix: staged rollouts with automated canary testing.
- Cloud upstream: the third incident wasn’t Anthropic’s fault — but their blast radius was too large because of insufficient multi-region routing.
- Meta-lesson: Anthropic’s transparency sets a bar other AI labs should match.
Entities
Concepts
- RL Infrastructure — infrastructure reliability principles apply to training too