BF16 is widely used to pretrain large language models because it has the same dynamic range as FP32 and converges similarly to FP32 training. However, in reinforcement learning with verifiable rewards, where learning rates are typically low, we find that standard mixed-precision BF16 training converges more slowly than FP16 training at the same learning rate. We attribute this gap to weight sticking: when FP32 shadow weights are quantized to BF16 using round-to-nearest, small updates often leave the BF16 weights used for rollout generation and training unchanged. Consequently, several optimizer steps may be needed before updates to the shadow weights are reflected in the BF16 weights. Raising the BF16 learning rate reduces weight sticking and matches FP16's convergence speed, but it also increases the KL divergence from the reference model. We instead apply stochastic rounding when quantizing the FP32 shadow weights to BF16. This approach matches FP16's convergence speed across different models and datasets while reducing the KL divergence relative to BF16 training with a larger learning rate. Unlike FP16, it keeps BF16's wider dynamic range and needs no loss scaling.
Figures & tables
Figure 1: Applying stochastic rounding when quantizing the FP32 shadow weights to BF16 (BF16+SR) closes the BF16–FP16 convergence speed gap across different models and datasets. The FP16 and BF16+SR reward trajectories closely overlap.
Figure 3: Under training-inference numerical parity, FP16 reaches a given mean reward and AIME score (mean@32) in fewer RL iterations than BF16 (DeepSeek-R1-1.5B, MATH-1460).
Optimizer Step
BF16
FP16
1
99.42%
95.35%
2
99.23%
93.86%
3
98.93%
91.51%
4
98.84%
90.79%
Table 1: Percentage of quantized weights equal to their initial checkpoint values after each optimizer step in the first RL iteration (DeepSeek-R1-1.5B, MATH-1460, η=10−6 ).
Figure 4: Effect of mantissa bit-width and LR on convergence (DeepSeek-R1-1.5B, MATH-1460). (a) A simulated BF16-like format with 10 mantissa bits tracks FP16. (b) A higher LR speeds up convergence for both formats, but FP16 stays ahead at every LR.
Figure 5: AIME 2024 and AIME 2025 scores (mean@32). BF16 initially trails FP16 and BF16+SR on downstream task scores but usually closes the gap as training progresses.
Figure 6: Mean response length during training. After an initial phase in which FP16 and BF16+SR produce shorter responses, all three configurations converge to similar lengths, except on Qwen3-30B-A3B.
Figure 7: Drift from the reference model at matched reward (DeepSeek-R1-1.5B, MATH-1460). BF16 with a higher LR closes the reward gap but incurs larger drift, while BF16+SR at the default LR stays close to FP16.