Figure 10 · 100 RL steps

Zero rollout/trainer mismatch for 100 RL steps

Rollout/trainer PPO KL

0 0 20 40 60 80 step 100 0.0 at every step

Training reward (dots: per step, line: 10-step mean)

−0.8 −0.4 0.0 +0.4 +0.8 0 20 40 60 80 step 100

AIME mean@8

0.4 0.5 0.6 0.7 0 20 40 60 80 step 100 0.451 0.664

Full model, rank-32 LoRA, GRPO on DAPO-17k, 32 GB300 GPUs. mean@8: AIME 2024–2026, temperature 1.0.