Language Model Personalization via Reward Factorization
Source: https://arxiv.org/pdf/2503.06358 Author: Idan Shenfeld, Felix Faltings, Pulkit Agrawal, Aldo Pacchiano Date: 2025
Summary
Framework to personalize LLM outputs by representing user-specific rewards as a linear combination of base reward functions. Requires only ~10 user responses to infer individual preferences. Achieves 67% win rate over default GPT-4o responses in human evaluation.
Key Claims
- User preferences exist in a low-dimensional space — enables efficient personalization
- Extends RLHF: instead of one reward function, learn a mixture of base reward functions weighted per user
- Only ~10 user responses needed to infer preference vector
- 67% win rate vs. default GPT-4o in human evals
- Works with both synthetic and real users
- Doesn’t require training separate models per user — inference-time adaptation
Concepts
- RL Infrastructure — reward factorization as RLHF extension
- Sycophancy — personalization can either enable genuine value or sycophancy depending on reward design