The Era of Real-World Human Interaction: RL from User Conversations
Source: https://arxiv.org/abs/… Author: Various (arXiv) Date: 2025-10-09
Summary
Academic paper arguing that the next phase of RL for LLMs is “RL from user conversations” — using live user interaction signals (satisfaction, follow-up questions, corrections) as reward signals, rather than human preference labels or verifiable task outcomes.
Key Claims
- Beyond RLHF: RLHF requires explicit human labeling. RL from user conversations uses implicit signals from actual usage.
- Signals available: session length (longer = more engaged), follow-up questions (confusion or curiosity?), corrections (user rewriting AI output), abandonment.
- Challenge: signals are noisy and confounded. A short session could mean the user found the answer or gave up.
- Benefit: scale. Millions of users generate billions of implicit signals — far more than any labeling operation.
- Key risk: optimizing for engagement metrics can produce sycophantic behavior (see sycophancy).
Concepts
- RL Infrastructure — implicit feedback as RL reward source
- Sycophancy Problem — engagement optimization → sycophancy
- AI Alignment — alignment from user signals vs. human-designed rewards