Post-Training 101

Source: https://magazine.sebastianraschka.com/p/post-training-101 Author: Han Fang (Tokens for Thoughts) Date: 2025-09-15

Summary

Comprehensive explainer on post-training for LLMs: the full pipeline from SFT through RLHF/DPO/GRPO to the final aligned model. One of the best single-source references for the post-training stack.

Key Claims

• Post-training stages: (1) SFT on high-quality instruction data, (2) reward model training, (3) RL alignment (PPO/GRPO/DPO), (4) evaluation and filtering.
• SFT vs RL: SFT teaches format and basic instruction following; RL teaches quality judgments and difficult behaviors.
• DPO insight: DPO is mathematically equivalent to a specific form of RL — no reward model needed because the preference itself encodes reward.
• The “alignment tax” is largely a myth at scale: well-tuned models can be both more capable AND more aligned than unaligned counterparts.
• Emerging technique: constitutional AI (Anthropic) uses LLM-generated critiques as reward signal, reducing human labeling.
• Key failure mode: over-optimization of proxy rewards (see reward hacking) — RLHF can make models very good at appearing helpful.

Entities

• Anthropic — constitutional AI reference

Concepts

• RL Infrastructure — full post-training pipeline
• Synthetic Data — constitutional AI uses synthetic critiques
• Reward Hacking — over-optimization failure mode
• AI Alignment — post-training is the primary alignment mechanism
• Sycophancy Problem — emerges from over-optimized RLHF