The Full Post-Training Stack
PRE-TRAINING Base Model SFT SFT Model REWARD MODEL TRAINING Human Preferences RL (PPO) Aligned Model DPO iterative rounds HUMAN ANNOTATORS Preference labels CONSTITUTIONAL AI / RLAIF AI-generated feedback at scale KL DIVERGENCE PENALTY Keeps policy close to SFT Prevents reward hacking main training flow DPO shortcut (bypasses reward model + RL) iterative feedback loops AI feedback replaces human labels KL divergence constraint