Low-precision pretraining is usually judged by validation loss. We ask whether matching loss also means learning rare facts as efficiently. We train transformers from scratch with emulated 4-bit matrix multiplications on synthetic biographies with controlled exposure counts, on their own and mixed into web text. NVFP4 needs 1.32× and 1.52× as many exposures as BF16 to match its recall while validation loss differs by only 0.5% and 2%, and 1.26× and 1.30× when both models are evaluated with BF16 matrix multiplications. Under our primary readout, the exposure cost is about 1.2× at matched loss. Zero-mean gradient noise reproduces the cost of quantizing only the backward pass, which barely moves the loss. In a controlled experiment at twice the NVFP4 error, noise fed only to Adam's second-moment estimate acts like a lower learning rate and its cost fades, whereas noise in the update lowers its signal-to-noise ratio and carries about two thirds of the cost through the end of training. Raising the learning rate to offset the smaller average step does not recover the cost, either for this noise or for NVFP4. Gradient statistics of BF16 checkpoints rank the costs of formats, partial quantizations and noise controls without fitting (Spearman 0.96). We suggest measuring rare-fact acquisition alongside loss when evaluating low-precision recipes.
Figures & tables
Figure 1: BF16 (black) and NVFP4 (red) training of the same 40M architecture, mean over six seeds per regime. (a, c) Validation loss over the last 70% of training. (b, d) Final recall against exposures per fact ( ± 1 standard error over seeds), with BF16 recall interpolated to the NVFP4 final validation loss in gray. Arrows mark the horizontal shift of the mean curves at recall 0.3, 0.5 and 0.7.
Figure 2: The model of Section 2 for one coordinate and one fact (illustrative shapes). (a) Zero-mean gradient error widens the distribution of the minibatch gradient and raises Adam’s normalizer from vi to vi+si2 . (b) Each exposure then adds less first-order progress, so the threshold τf is reached after mf times as many exposures (Proposition 2 ). (c) A common delay mf shifts the recall curve by log2mf at every level (Proposition 1 ).
Figure 3: Exposure multipliers against BF16 at matched tokens (gray) and matched validation loss (red), with 95% intervals and seed counts at matched tokens/loss (T/L). Values above 1 mean more exposures are needed. 40M synthetic data unless labeled otherwise. Further settings are in Table 1 .
Figure 4: Where the deficit sits (40M, mean over six seeds per regime). (a, b) Recall of NVFP4 minus BF16 per exposure bin during training (red: NVFP4 behind). (c) Final value-token loss gap per bin ( ± 1 standard error over seeds) and the share of synthetic validation value tokens in each bin (right axis).
Figure 5: Where the cost arises (40M synthetic, 95% intervals). (a) Partial quantization and forward noise at matched tokens (gray) and matched loss (red), with seed counts (T/L). (b) Zero-mean noise at 0.5, 1 and 2 times the NVFP4 error on backward GEMM outputs, i.i.d. (blue) or block-shaped (green), or on forward GEMM outputs (purple), at matched tokens. Dashed and dotted: NVFP4 backward-only and forward-only quantization. (c) Multiplier predicted by Eq. ( 1 ) against the observed token-matched multiplier for the 17 configurations of Table 6 . Open markers: bias ratio above 0.1 (Appendix D ). Dashed: equality.
Figure 6: Splitting gradient noise between Adam’s moments (40M synthetic, i.i.d. noise at twice the NVFP4 error, 8 seeds, 95% intervals). (a) Token-matched multiplier against BF16 at 10−3 , recomputed at every evaluation, for BF16 at 10−3/A (gray), noise in the second moment only (blue) and noise in both moments (red). (b) Horizontal gap per exposure bin against BF16 at 10−3/A , averaged over evaluations (Appendix E ). (c) Multipliers against BF16 at 10−3 for training at 10−3 (filled) and at the rate raised by A (open).
Figure 7: Share of the validation-loss gap (horizontal) and of the log2 exposure cost at matched tokens (vertical) of plain NVFP4 that each remedy removes, on shared seeds (six synthetic, three natural). Bars: 95% Fieller intervals for the cost share (Appendix B ). Below the diagonal, loss recovers faster than factual acquisition. Loss-matched values are in Table 4 .
Figure 8: Loss-matched exposure multiplier of NVFP4 against BF16 vs. training tokens per parameter for the 12M, 40M and 100M architectures (12.9M, 28.7M and 96.6M parameters with the synthetic vocabulary, all counted) on synthetic data, with 95% intervals over 3–6 seeds per point.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Matched tokens
Matched loss
n
Multiplier [95% CI]
n
Multiplier [95% CI]
Formats
FP8
6
1.11 [1.00, 1.22]
4
1.11 [0.96, 1.28]
NVFP4
6
1.32 [1.16, 1.50]
6
1.25 [1.13, 1.38]
MXFP4
6
1.61 [1.38, 1.88]
6
1.41 [1.22, 1.64]
Recipes
Appendix
Table 1: Default exposure multipliers averaged over recall levels 20–80%, with geometric means, 95% intervals in brackets and the number of seeds with a defined multiplier ( n ). Settings are grouped by the experimental factor changed.
Setting
Matched tokens
Matched loss
30%
50%
70%
30%
50%
70%
Formats
FP8
1.13 [1.02, 1.26]
1.11 [1.00, 1.22]
1.11 [0.98, 1.25]
1.14 [0.96, 1.35]
1.11 [0.96, 1.28]
1.10 [0.94, 1.29]
NVFP4
1.38 [1.22, 1.56]
1.34 [1.17, 1.53]
1.26 [1.10, 1.45]
1.32 [1.20, 1.46]
1.25 [1.14, 1.38]
1.17 [1.06, 1.30]
MXFP4
1.70 [1.44, 1.99]
1.62 [1.41, 1.87]
1.46 [1.24, 1.72]
1.53 [1.31, 1.77]
1.40 [1.20, 1.62]
1.25 [1.09, 1.43]
Recipes
Appendix
Table 2: Sensitivity to the recall level at which the exposure multiplier is computed. Geometric means with 95% intervals. Seed counts are as in Table 1 , except where a cell ends with its own n . –: unavailable or outside the BF16 loss range.
Setting
Matched tokens
Matched loss
Exact match
BF16 forward
Value prob.
Exact match
BF16 forward
Value prob.
Formats
FP8
1.10 [1.00, 1.20]
1.10 [1.00, 1.21]
1.14 [1.02, 1.28]
1.09 [0.95, 1.25]
1.10 [0.96, 1.26]
1.10 [1.00, 1.21]
NVFP4
1.31 [1.16, 1.48]
1.26 [1.11, 1.44]
1.32 [1.06, 1.64]
1.24 [1.13, 1.35]
1.21 [1.10, 1.34]
1.19 [0.99, 1.43]
MXFP4
1.62 [1.38, 1.89]
1.47 [1.29, 1.69]
1.71 [1.37, 2.13]
1.42 [1.22, 1.64]
1.32 [1.17, 1.50]
1.34 [1.08, 1.65]
Recipes
Appendix
Table 3: Sensitivity to the recall readout (value prob.: correct-value probability). BF16-forward values of NVFP4 for seeds 0, 1, 2 come from a rerun of the same configuration with saved checkpoints. Geometric means with 95% intervals. Seed counts are as in Table 1 , except where a cell ends with its own n . –: unavailable or outside the BF16 loss range.
Variant
Loss gap
m [95% CI]
p
Plain NVFP4
0.47%
1.25 [1.13, 1.38]
–
Simplified recipe (BF16 last 2 blocks, RHT also on data gradient)
0.27%
1.16 [1.04, 1.29]
0.17
BF16 cooldown (final 20%)
0.25%
1.18 [1.09, 1.29]
0.02
NVIDIA recipe (adapted)
0.19%
1.15 [1.07, 1.23]
0.008
NVFP4 with FP8 backward GEMMs
0.29%
1.19 [1.07, 1.33]
0.46
NVIDIA recipe (adapted) with FP8 backward GEMMs
0.19%
1.12 [0.98, 1.27]
0.008
Appendix
Table 4: Recipe variants of plain NVFP4 (40M synthetic). m : loss-matched multiplier (six seeds, five for BF16 cooldown, four for the recipe with FP8 backward). p : paired t -test of log2m against plain NVFP4. After Holm’s correction over the five tests, the two smallest p remain below 0.05.
Remedy
n
Loss share
Cost share
Difference
NVIDIA recipe (adapted)
6
60 [29, 79]
42 [26, 56]
18 [2, 29]
Simplified recipe
6
42 [ − 18, 59]
34 [ − 19, 70]
7 [ − 26, 37]
BF16 cooldown
6
46 [31, 105]
35 [23, 53]
11 [3, 29]
FP8 backward
6
39 [8, 67]
25 [ − 56, 70]
14 [ − 4, 42]
Recipe + FP8 backward
6
58 [29, 131]
46 [32, 65]
13 [ − 10, 49]
Adapted recipe, natural
3
41 [32, 51]
12 [ − 75, 45]
29 [–]
Appendix
Table 5: Shares of the validation-loss gap and of the log2 exposure cost (matched tokens) of plain NVFP4 that each remedy removes, in percent, with 95% Fieller intervals, and their difference (loss share minus cost share, in percentage points) with a 95% seed-bootstrap interval. –: unbounded interval, or too few seeds for a bootstrap.
Figure 9: Exposure multiplier at matched tokens (40M, seeds 0–2, 95% intervals) under the default top-1 readout (filled) and a top-five readout over the values of the attribute (open). Thin lines pair the two readouts of one seed.
Configuration
Predicted
Observed [95% CI]
n
Bias ratio
Block-shaped gradient noise, 0.5 ×
1.02
1.05 [0.86, 1.27]
3
0.03
FP8
1.02
1.11 [1.00, 1.22]
6
0.30
Forward noise, 0.5 ×
1.03
1.08 [0.90, 1.29]
3
0.51
i.i.d. gradient noise, 0.5 ×
1.05
1.02 [0.95, 1.09]
3
0.03
NVFP4, forward weights only
1.05
1.11 [0.94, 1.31]
3
0.81
NVFP4, backward only
1.06
1.13 [1.04, 1.21]
6
0.03
Appendix
Table 6: Configurations of Figure 5 (c): multiplier predicted by Eq. ( 1 ) (geometric mean over 96 facts and 3 checkpoints), observed multiplier at matched tokens with 95% interval and seed count, and gradient-error bias ratio (about 1/32 if zero-mean, Appendix D ).
Training large language models directly in 4-bit floating-point (FP) formats promises substantial improvements in throughput and energy efficiency. While some recipe design choices have been validated at scale, many promising approaches remain untested beyond small models and short token horizons, leaving open the question of which trends will hold. We present a systematic comparison of recent NVFP4 recipes at medium model scale, evaluating dense 8B and 30B-A3B MoE models trained for up to 1T tokens, with a focus on which ingredients are necessary for accuracy recovery. We propose a final recipe grounded in the principles behind established stable NVFP4 training at scale, incorporating state-of-the-art techniques such as unbiased gradient estimation with lower quantization error than stochastic rounding. To the best of our knowledge, this is the strongest FP4 training result demonstrated at this scale to date in loss gap to BF16. Through ablation studies, we find that: (i) each technique in the optimized recipe measurably improves loss trajectory, (ii) selective high-precision layers are necessary for recovering accuracy at scale, (iii) not all tensors in the backward pass benefit equally from de-biasing, leaving room to apply complementary error-reduction techniques to the remaining tensors.
Anjulie Agrusa, Andrei Panferov, Elizabeth Wei +7
NVIDIA · Institute of Science and Technology Austria · Red Hat AI
Recent NVFP4 pretraining methods mainly target transformer linear layers, leaving optimizer states, optimizer arithmetic and attention underexplored in 4-bit pipelines. This critical gap blocks stable full-stack 4-bit pretraining, as the three core modules exhibit unique numerical failure patterns: linear layers hit hard quantization noise limits with dimension-propagated error amplification; AdamW second moments are heavy-tailed non-negative values fragile to low-precision denominators; attention carries error-prone computation paths demanding strict forward-backward quantization consistency. We propose Full-Stack FP4, the first complete NVFP4 pretraining framework resolving all three stability bottlenecks via module-wise precision strategies. For linear projections, LoRA-SVD lightweight decomposition suppresses quantization noise and breaks the direct-quantization error ceiling, shrinking the linear-only loss gap from 1.40% to 0.61%. For optimizers, we design AdamW second-moment transformation for robust NVFP4 storage and fully support native NVFP4 Newton-Schulz iterations for the Root (Muon) optimizer. For attention, a mixed-precision scheme quantizes Q/K/V and backward dS while guarding vulnerable paths in BF16, paired with unified tensor reuse to sustain forward-backward alignment. We further analyze fast error accumulation in naive low-bit matrix multiplication and the extreme sensitivity of PV / dOV^T attention branches. All modules are plug-and-play with cumulative stability and efficiency improvements. Our 3B/64B-token pretraining validates near-BF16 performance with merely 1.47% loss gap, verifying feasible stable end-to-end NVFP4 LLM pretraining.
Siyu Ding, Mingchuan Ma, Jiabo Tong +3
Institute of Automation, Chinese Academy of Sciences · Sichuan University · Beijing Zhongguancun Academy +3
FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins. We show that this bias accumulates multiplicatively across layers and is amplified by the Random Hadamard Transform (RHT), providing a unified explanation for the training instability observed in existing E2M1-based FP4 recipes. In contrast, uniform grids (E1M2/INT4) bypass this grid-geometry error and better convert the improved bucket utilization from RHT into higher quantization quality. Based on this finding, we propose UFP4, a uniform 4-bit training recipe that applies RHT to all three training GEMMs while restricting stochastic rounding to dY alone. On Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining, UFP4 consistently achieves lower BF16-relative loss degradation than strong E2M1-based baselines, supported by scaling-law analysis and ablation studies. Our results suggest that future accelerators should support E1M2/INT4-style uniform 4-bit grids as first-class training primitives alongside E2M1.