LLM Judges as Reward Models
Source: https://www.atla-ai.com/blog/llm-judges-as-reward-models Author: Atla AI Date: 2025-09-26
Summary
Technical post on using LLMs as judges for reward modeling — the approach where, instead of training a dedicated reward model, you use a strong LLM to evaluate outputs directly.
Key Claims
- LLM-as-judge approach: ask a strong LLM (e.g., GPT-4) to rate model outputs on relevant criteria. Use these ratings as reward signals for RL training.
- Advantages: no separate reward model to train; can be updated by improving judge prompt; can evaluate nuanced criteria.
- Disadvantages: expensive (each evaluation is an LLM call); can be gamed if the trained model learns to optimize for judge heuristics rather than quality.
- Key failure mode: the student model learns to produce outputs that look good to the judge, not outputs that are genuinely good.
- Mitigation: use multiple judges, adversarial prompting, diverse evaluation criteria.
Connection to Other Sources
Connects to RL Information Bandwidth — LLM judges can provide higher-bandwidth reward signals. Also reward hacking concern applies strongly.
Concepts
- RL Infrastructure — LLM-as-judge as reward modeling approach
- Reward Hacking — judge-gaming as reward hacking
- Synthetic Data — LLM judge evaluations are synthetic reward signals