What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes
Authors: Anjulie Agrusa, Andrei Panferov, Elizabeth Wei, Keith Wyss, Paul Gibbons, Erik Schultheis, Jinhang Choi, Frank Sun, Tijmen Blankevoort, Dan Alistarh
Organizations: NVIDIA · Institute of Science and Technology Austria · Red Hat AI
Training large language models directly in 4-bit floating-point (FP) formats promises substantial improvements in throughput and energy efficiency. While some recipe design choices have been validated at scale, many promising approaches remain untested beyond small models and short token horizons, leaving open the question of which trends will hold. We present a systematic comparison of recent NVFP4 recipes at medium model scale, evaluating dense 8B and 30B-A3B MoE models trained for up to 1T tokens, with a focus on which ingredients are necessary for accuracy recovery. We propose a final recipe grounded in the principles behind established stable NVFP4 training at scale, incorporating state-of-the-art techniques such as unbiased gradient estimation with lower quantization error than stochastic rounding. To the best of our knowledge, this is the strongest FP4 training result demonstrated at this scale to date in loss gap to BF16. Through ablation studies, we find that: (i) each technique in the optimized recipe measurably improves loss trajectory, (ii) selective high-precision layers are necessary for recovering accuracy at scale, (iii) not all tensors in the backward pass benefit equally from de-biasing, leaving room to apply complementary error-reduction techniques to the remaining tensors.
Figures & tables
Figure 1: Left: Relative train loss difference between each NVFP4 recipe and BF16 (lower is better), trained with NVIDIA ’25 NVFP4 and Mosaic NVFP4 recipes to 1T tokens on Nemotron 3 Nano. Right: Cumulative recipe updates, each adding one Mosaic NVFP4 component, trained to 400B tokens on Nemotron 3 Nano. All recipes keep the last 15% of layers in BF16. Recipe updates are described in Table 1 .
Recipe
Weight Quant
Weight Re-Quant
Tiled Random Hadamard Transforms
Random Sign Vectors
Technique to Address Bias in Gradients
Four Over Six Tensors
NVIDIA ’25 NVFP4
2D
No
Dim-16 WGRAD
Static
SR
None
Update 1
1D
Yes
Dim-16 WGRAD
Static
SR
None
Update 2
1D
Yes
Dim-128 WGRAD, DGRAD
Resampled
SR
None
Update 3
1D
Yes
Dim-128 WGRAD, DGRAD
Resampled
MS-EDEN
W.T, X.T
Mosaic NVFP4
1D
Yes
Dim-128 WGRAD, DGRAD
Resampled
MS-EDEN
W, X, W.T, X.T
Table 1: Recipe improvements over the prior state of the art at this model and token scale. All recipes keep the last 15% of layers in BF16 unless otherwise noted.
Figure 2: Stability study of the Mosaic NVFP4 recipe with the last 15% of the network kept in BF16 vs. all layers quantized on the Nemotron-H 8B model.
Figure 3: Different backward-quantization schemes on Nemotron 3 Nano, with all layers quantized (left) or the last 15% of layers kept in BF16 (right). With all layers quantized, the configuration applying MS-EDEN to gradient tensors and Four Over Six scale selection to the remaining backward tensors, X.T and W.T, yields the lowest observed loss gap. At this model size and token horizon, the backward-pass techniques show no clear separation when the last 15% of layers are kept in BF16.
Figure 4: Percent relative train (left) and validation (right) loss difference compared to BF16 (lower is better) evaluated on the Nemotron-H 8B model to 1T tokens. The Mosaic NVFP4 recipe has improved training and validation loss over most of the token horizon.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Model
No. Mamba-2 Layers
No. Attention Layers
No. FFN/MoE † Layers
No. KV Heads
Active/ Total Parameters
Nemotron 3 Nano
23
6
23
2
3.2B / 31.6B
Nemotron-H
24
4
24
8
8B
Appendix
Table 2: Model parameters for each experimental setup. Active parameter counts exclude embeddings. † Nemotron 3 Nano uses MoE layers.
Model
Batch Size
Min LR
LR
Nemotron 3 Nano
3072
1e-5
1e-3
Nemotron-H
768
8e-6
8e-4
Appendix
Table 3: Training parameters for each experimental setup.
Task
NVIDIA ’25 NVFP4 (0–100)
Mosaic NVFP4 (0–100)
Noise Reference (%)
Δ (%)
Nemotron 3 Nano (30B-A3B)
General Knowledge
MMLU, 5-shot
67.5
68.4
0.9
+1.3
MMLU-Pro, 5-shot CoT
45.0
47.1
6.3
+4.7
Math Average
79.4
79.2
4.6
-0.3
(Math 500, 4-shot; GSM8K, 8-shot CoT)
Appendix
Table 4: Downstream-task scores and measured run-to-run variability. For individual tasks, the noise reference is the relative range across three repeated runs. For aggregate rows, it is an average weighted by the NVIDIA ’25 NVFP4 task scores. Boldface denotes differences outside the corresponding noise reference.