Playground 3 — The KL Tug-of-War

Adjust β (KL penalty) and reward strength. The policy marker settles between reward maximization and the KL constraint. Too low → reward hacking. Too high → training stalls.

β (KL penalty) 0.10
Reward strength 0.70
← Reward hacking zone KL collapse zone →
🤖
High reward, high drift Low drift, minimal gain
Reward gain
KL penalty
Net objective