Lost in Rounding: bf16 Recurrent State Makes Hybrid Reasoning Models Think Longer
Organizations: PayPal AI
Abstract
Hybrid reasoning models interleave attention with recurrent mixers such as Mamba-2 and Gated DeltaNet, whose state is cached between decoding steps. Prior work showed that storing this state in FP16 or INT8 with round-to-nearest (RTN) makes Mamba-2 and GDN hybrids more verbose, and that stochastic rounding (SR) fixes it. We study the other 16-bit format that serving engines expose, bf16, on six open hybrid reasoning models and 590 competition-math problems. With bf16 RTN state, five of six models reason longer (geometric-mean length 1.07x to 1.27x; also under greedy decoding in the two models tested) and hit the 32K token cap more often. On Qwen3.5-9B, cap hits rise from 118 to 223 and accuracy falls from 85.6% to 70.8%. On LiveCodeBench, both models tested also reason longer and lose 12.6 and 13.4 points of pass@1. Rerunning the capped items with a 128K budget halves the accuracy gap but does not close it (93.6% vs. 86.6%): bf16 runs that never finish are twice as common, and 85% of them are exact repetition loops. On Nemotron-Nano-9B-v2, by contrast, 30 of 31 extra caps finish at 128K, so the cost there is mostly extra tokens (exploratory). Same-width bf16 SR removes the length increase in four of five affected models, restores Qwen3.5-9B to 87.6% and brings code length and pass@1 near FP32 levels, so the failure is mostly rounding bias, not missing resolution. With an integer-hash random generator, SR on Qwen3.5-9B matches FP32 on cap hits and accuracy (113 vs. 118; 85.6% for both) at the memory and decoding speed of bf16 RTN. In the Nemotron-Nano-v2 lineage, SR on 25% of heads (an intermediate-decay band) removes the effect, while random heads and the top teacher-forced-KL octiles do not; in Nemotron-3 and Qwen3.5-9B the effect is spread across heads. We recommend that evaluations of hybrid models report the state dtype, cap rate and output length alongside accuracy.
Figures & tables
| Model (mixer) | Length ratio | Capped | Accuracy (%) | McNemar | Both finished | |
| ( ) | FP32 bf16 | FP32 bf16 | FP32 / bf16 | |||
| Sampling | ||||||
| Nemotron-Nano-9B-v2 (M2) | 1.197 (7e-19) | 1 31 | 89.3 88.8 | 0.74 | 558 | 92.7 / 93.5 |
| Nemotron-Nano-12B-v2 (M2) | 1.188 (1e-18) | 0 17 | 90.0 90.5 | 0.70 | 573 | 92.1 / 93.2 |
| Nemotron-3-Nano-30B-A3B (M2) | 1.081 (3e-6) | 36 81 | 92.5 87.3 | 3e-7 | 509 | 98.4 / 98.6 |
| Qwen3.5-9B (GDN) | 1.265 (7e-16) | 118 223 | 85.6 70.8 | 2e-14 | 322 | 99.7 / 99.7 |
| Qwen3.5-9B ( ) | Nemotron-Nano-9B-v2 ( ) | |||
|---|---|---|---|---|
| FP32 | bf16 RTN | FP32 | bf16 RTN | |
| Unfinished at 128K | 44 | 93 | 0 | 1 |
| of which exact repetition loops | 42 | 79 | 0 | 1 |
| bf16-only 32K caps that finish at 128K | 108 / 150 | 30 / 31 | ||
| Accuracy among finished reruns | 97.3% | 97.7% | 37.5% | 25.8% |
| Median output length on rerun items | 19,951 | 41,949 | 16,314 | 37,401 |
| Model | [95% CI] | Length ratio | Capped | Accuracy (%) | vs. FP32 |
|---|---|---|---|---|---|
| SR/FP32 | FP32 / RTN / SR | FP32 / RTN / SR | McNemar | ||
| Nemotron-Nano-9B-v2 | 1.05 [0.88, 1.27] | 0.991 | 1 / 31 / 0 | 89.3 / 88.8 / 89.7 | 0.88 |
| Nemotron-Nano-12B-v2 | 0.98 [0.82, 1.16] | 1.004 | 0 / 17 / 0 | 90.0 / 90.5 / 90.3 | 0.86 |
| Nemotron-3-Nano-30B-A3B | 0.38 [ 0.37, 1.00] | 1.049 | 36 / 81 / 41 | 92.5 / 87.3 / 92.2 | 0.81 |
| Qwen3.5-9B | 0.87 [0.58, 1.26] | 1.031 | 118 / 223 / 111 | 85.6 / 70.8 / 87.6 | 0.24 |
| hash generator | 0.97 [0.70, 1.37] | 1.007 | 118 / 223 / 113 | 85.6 / 70.8 / 85.6 | 1.00 |
| Model | Length ratio | [95% CI] | Capped (of 350) | Pass@1 (%) |
|---|---|---|---|---|
| bf16/FP32 [95% CI] | FP32 / RTN / SR | FP32 / RTN / SR | ||
| Nemotron-Nano-9B-v2 | 1.37 [1.29, 1.46] | 0.95 [0.77, 1.15] | 36 / 162 / 30 | 58.3 / 45.7 / 59.4 |
| Qwen3.5-9B | 1.09 [1.05, 1.13] | 0.73 [0.27, 1.33] | 169 / 233 / 179 | 46.3 / 32.9 / 45.4 |
| SR applied to | Heads | [95% CI] | Capped | KL share |
| none (bf16 RTN) | 0 | 0 | 31 | – |
| all heads | 3456 | 1.05 [0.88, 1.27] | 0 | 94% † |
| ( ) | 864 | 1.03 [0.86, 1.23] | 1 | 52% ‡ |
| (top-2 KL octiles) | 864 | 0.59 [0.44, 0.76] | 9 | 86% ‡ |
| (size-matched control) | 864 | 0.05 [ 0.28, 0.14] | 34 | – |
| random (211 in ) | 864 | 0.06 [ 0.14, 0.24] | 34 | – |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Test | Criterion | Result | Verdict |
| Head localization, Nemotron-Nano-9B-v2 (§ 8 ) | |||
| SR on | , CI lower ; caps | 1.03 [0.86, 1.23]; caps 1 | pass |
| SR on | pass | ||
| SR on (size-matched) | pass | ||
| vs. (equal size) | difference CI lower | 0.55 [0.34, 0.80] | pass |
| SR on (top-2 KL octiles) | CI lower ; fail if | 0.44 [0.26, 0.64]; 0.59 (predicted 0.29) | pass |
| Hugging Face repository | Parallelism | Running cap FP32 / bf16 | / top- / top- |
|---|---|---|---|
| nvidia/NVIDIA-Nemotron-Nano-9B-v2 | DP8 | 44 / 88 | 0.6 / 0.95 / – |
| nvidia/NVIDIA-Nemotron-Nano-12B-v2 | DP8 | 38 / 76 | 0.6 / 0.95 / – |
| nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 | DP8 | 43 / 86 | 1.0 / 1.0 / – |
| Qwen/Qwen3.5-9B | DP6 / DP8 ∗ | 122 / 240 | 0.6 / 0.95 / 20 |
| Qwen/Qwen3-Next-80B-A3B-Thinking | TP4 DP2 | 223 / 436 | 0.6 / 0.95 / 20 |
| tiiuae/Falcon-H1R-7B | DP8 | 47 / 94 | 0.6 / 0.95 / – |