ICL-RM: Latent Few-Shot Compression for Reward Models
Author: mouse (internal) | Date: 2026-05-02 → 2026-05-05 | Source: http://24.144.74.176:8765/report.html
Internal research report covering PCRM (Perceptual Compressed Reward Model) and PDRM (Perceptual Distilled Reward Model) approaches to compressing few-shot preference examples into latent vectors for reward model inference. The goal: decouple inference cost from few-shot length while preserving text-ICL accuracy.
Key Claims
- Text-ICL baseline: Qwen2.5-3B achieves 0.978 on toy task, 0.674 on business chat preference task (kaon_chengyu_pcrm_swift)
- PCRM with C3 init: 0.909 on toy (successful), but business stuck at 0.55-0.58 due to decoder NLU degradation
- PDRM distillation: 0.979 on toy (exceeds text-ICL teacher!), but business variants all stuck at 0.586
- Root cause diagnosis: Two failure modes on business data:
- C3 route: decoder loses chat/instruct capability during compression pretraining
- PDRM route: frozen Instruct decoder never trained to read OOD latents, attention mass only 0.6% on latent tokens
- Final conclusion: Must pretrain a custom dual-LM architecture where decoder retains chat ability — can’t just bolt compression onto existing models
Key Takeaways
- Toy-scale success doesn’t guarantee business-scale transfer (0.909/0.979 toy → 0.55 business)
- Flash-attention monkey-patch bugs can silently degrade models (0.91 → 0.49) without obvious errors
- LLaVA-style two-stage training (freeze backbone, train projector) applied to reward modeling
- Latent vectors do encode meaningful preference structure (1-D preference manifold with 7 atomic rule directions found in interpretability analysis)
- 20+ variants explored systematically before declaring path blocked