Understanding and Using Supervised Fine-Tuning (SFT) for Language Models
Source: https://cameronrwolfe.substack.com/p/understanding-and-using-supervised Author: Cameron R. Wolfe, Ph.D. Date: 2024-09-14
Summary
Comprehensive tutorial on SFT — what it is, how to do it, and when to use it. Companion to GRPO++ from the same author.
Key Claims
- SFT fundamentals: train a pretrained LLM to follow instructions by showing it (instruction, response) pairs with standard language modeling loss.
- Data quality dominates: 1,000 high-quality examples beat 100,000 mediocre examples. Quality filtering is the most important part of SFT data pipelines.
- Format matters: models learn the format of responses, not just the content. Inconsistent formatting in training data causes inconsistent outputs.
- When to use SFT vs RL: SFT for behavior shaping (formatting, style, safety); RL for capability improvement (reasoning, problem-solving).
Concepts
- RL Infrastructure — SFT as the baseline before RL
- Synthetic Data — SFT data quality principles