Both methods start from preference data and produce a better policy — but DPO collapses a three-stage pipeline into one fine-tuning step. Highlighted steps show where the key complexity lives.
RLHF (PPO)
1
SFT model as starting point
↓
2
Sample completions (y_w, y_l)
↓
3
Train reward model r(x,y)↓
4
RL loop: generate → score → update↓
↓
DPO
1
SFT model as starting point
↓
2
Collect preference dataset (x, y_w, y_l)
↓
3
Single fine-tuning step with DPO loss↓
No separate RM training
No RL loop
No PPO instability
RLHF tradeoffs
✓ Online exploration of completions
✓ Reward model interpretable separately
✗ Three training stages, complex infra
✗ PPO instability, many hyperparams
DPO tradeoffs
✓ Simple: standard fine-tuning
✓ Stable, no RL hyperparams
✗ Offline only — fixed dataset
✗ No explicit reward model to audit