The Full Post-Training Stack
DPO bypass
PRE-TRAINING
Base Model
SFT
SFT Model
REWARD MODEL
TRAINING
Human feedback
AI feedback (RLAIF)
RL / PPO
Aligned Model
KL divergence anchors SFT weights
iterative rounds ↺