cs.LGNov 9, 2026

What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts

Authors: Chang Lu, Mengjiao Zhang

Organizations: Independent Researcher.

Abstract

Low-precision pretraining is usually judged by validation loss. We ask whether matching loss also means learning rare facts as efficiently. We train transformers from scratch with emulated 4-bit matrix multiplications on synthetic biographies with controlled exposure counts, on their own and mixed into web text. NVFP4 needs 1.32×\times and 1.52×\times as many exposures as BF16 to match its recall while validation loss differs by only 0.5% and 2%, and 1.26×\times and 1.30×\times when both models are evaluated with BF16 matrix multiplications. Under our primary readout, the exposure cost is about 1.2×\times at matched loss. Zero-mean gradient noise reproduces the cost of quantizing only the backward pass, which barely moves the loss. In a controlled experiment at twice the NVFP4 error, noise fed only to Adam's second-moment estimate acts like a lower learning rate and its cost fades, whereas noise in the update lowers its signal-to-noise ratio and carries about two thirds of the cost through the end of training. Raising the learning rate to offset the smaller average step does not recover the cost, either for this noise or for NVFP4. Gradient statistics of BF16 checkpoints rank the costs of formats, partial quantizations and noise controls without fitting (Spearman 0.96). We suggest measuring rare-fact acquisition alongside loss when evaluating low-precision recipes.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes

    Oct 17, 2026Anjulie Agrusa, Andrei Panferov, Elizabeth Wei +7Low-Precision TrainingLanguage Model Pretraining

  2. Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention

    Jul 5, 2026Siyu Ding, Mingchuan Ma, Jiabo Tong +3Language Model PretrainingMixed-Precision Quantization