ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data
Authors: Hao Wang, Haocheng Yang, Licheng Pan, Lei Shen, Xiaoxi Li, Yinuo Wang, Zhichao Chen, Yuan Lu, Haoxuan Li, Zhouchen Lin
Date: 2026-04-09
Summary
Proposes unbiased reward modeling from implicit preference data (rather than explicit human labels). Addresses the bias problem in traditional RLHF preference collection.
Key Claims
- Implicit preferences (behavioral signals) can train reward models without explicit annotation
- Reduces annotation bias inherent in explicit preference labeling
- More scalable approach to reward model training
Concepts
- Reward Hacking — implicit data may reduce gaming of reward signal
- AI Alignment
See Also
- dpo-reward-model — DPO approach
- openrubrics-reward-modeling — rubric-based approach
- rl-from-user-conversations — RL from real user interactions