xAI Post-Training RL Overhaul
Source: https://x.com (tweet) Author: xAI team Date: 2025-2026
Summary
xAI’s post-training team (dozen people) overhauled their RL recipe using two new sources of signal: (1) user preference data from real conversations, and (2) agentic reward models that grade using strong reasoning capabilities. They also scaled RL by an order of magnitude.
Key Claims
- Team of ~12 overhauled RL recipe for Grok
- Signal source 1: user preference on real conversations (implicit human feedback)
- Signal source 2: agentic reward models with strong reasoning capabilities
- Scaled RL by an order of magnitude
- Key innovation: using reasoning models as reward models (recursive)
Connection to Other Sources
Connects to RL from User Conversations (implicit feedback signal) and Process Reward Models.
Concepts
- RL Infrastructure — production RL recipe at scale
- Reward Hacking — reasoning-based reward models as anti-gaming mechanism