bf16 Rounding Silently Discards Most RL Policy Updates: A Hidden Training-Inference Mismatch and an Unbiased One-Line Fix
Organizations: Stanford University
Abstract
Reinforcement learning (RL) post-training of language models keeps fp32 master weights in the optimizer and serves rollouts from a bf16 copy produced by round-to-nearest conversion. At the learning rates in common use, most per-step updates are smaller than half a bf16 spacing and the conversion discards them: on Qwen3-1.7B only 0.4-0.5% of served weights change per step, and the policy that generates the training data lags the optimizer by 50-150 steps. Existing mismatch corrections cannot see this lag because they compare two copies downstream of the same rounding. We propose Fair Cast: export the master with an unbiased rounding rule (stochastic rounding or error feedback) and compute the training loss on the served copy. The change is one line at the export call, adds no hyperparameters and costs 22-38 ms per export of 1.7B parameters. Across 25 matched seed-pairs on Qwen3-0.6B to 4B with AdamW and Muon, Fair Cast is ahead on 18, tied on 6 and behind on 1, with a mean gain of 6.1 points; on zebra puzzles it reaches 0.57 +/- 0.16 over four runs where round-to-nearest collapses to 0.33 +/- 0.10. It also makes the saved fp32 checkpoint agree with the served model, and it exposes the implicit lag of round-to-nearest as a regularizer that can be restored explicitly when a task needs it.
Figures & tables
| Trainer forward on fp32 master | Trainer forward on served copy | |
|---|---|---|
| Round-to-nearest export | Current default; hidden lag (§4) | Same policy as the default: the bf16 forward of the master is the nearest export (0.6B: knights-and-knaves 0.889 against 0.865, GSM8K 0.705 against 0.742) |
| Unbiased export (SR or EF) | Collapse: EF 0.00 (4/4 runs); SR 0.6B GSM8K 0.13, MATH 0.002 | Fair Cast (§7.2–§7.3) |
| Task | Model | n | Nearest | Fair Cast | Δ |
|---|---|---|---|---|---|
| Zebra puzzles | 1.7B | 3 / 4 | 0.33 ± 0.10 | 0.57 ± 0.16 | +24 |
| Knights-and-knaves | 0.6B | 1 | 0.865 | 0.967 | +10.2 |
| Knights-and-knaves | 1.7B | 1 | 0.959 | 0.996 | +3.7 |
| Knights-and-knaves, Muon | 1.7B | 2 | 0.907 ± 0.016 | 0.988 ± 0.003 | +8.2 |
| Knights-and-knaves | 4B | 1 | 0.988 | 0.992 | +0.4 |
| Nim | 1.7B | 1 | 0.570 | 0.580 | +1.0 |
| Arm | bf16 | fp16 | FP8 via bf16 | FP8 via fp16 |
|---|---|---|---|---|
| HumanEval+, round-to-nearest | 58.5 | 16.5 | 60.4 | 28.1 |
| Countdown (500 steps), round-to-nearest | 21.9 | 10.2 | 20.3 | 10.2 |
| HumanEval+, Fair Cast | 64.6 | 62.2 | 61.0 | 61.0 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Objective | GRPO-style clipped policy gradient, group size 8, clip 0.2 |
| Prompts per optimizer step | 24 (192 rollouts) |
| Optimizer steps | 150 (500 for the Countdown long run; 300 with cosine decay to for the 1.7B MATH levels 3–5 pair) |
| Learning rate | constant, AdamW (β = 0.9, 0.999, weight decay 0.01); Muon at with momentum 0.95 and RMS-matched updates (0.2·√max(m, n)) on 2-D hidden weights, AdamW at for embeddings, norms and the output head, where indicated |
| Rollout budget | 2,048 tokens (6,144 for Nim); temperature 1.0 |
| Sampler / trainer | vLLM bf16 sampler and Hugging Face trainer on one H100, lag 1 |
RL Post-Training
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
See all 464 papers