Gradient Quantization

Follow this topic Ask about this topic

Latest papers

9 papers1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes
Oct 17, 2026Anjulie Agrusa, Andrei Panferov, Elizabeth Wei +7Low-Precision Training Language Model Pretraining
NVIDIA · Institute of Science and Technology Austria · Red Hat AI
Scaling Influence Functions in LLMs through Eigenbasis-Corrected One-Bit Gradient Projection
Sep 29, 2026Jaeseung Heo, J Rosser, Dongwoo KimGradient Compression Influence Functions
POSTECH · University of Oxford
GeoFP8: Geometry-Aware FP8 Gradient Compression for Distributed LLM Training
Jul 8, 2026Jieying Wang, Zizhong Wang, Fangru Linghu +3Language Model Pretraining Communication-Efficient Distributed Training
Department of Computer Science, Rutgers University, Piscataway, NJ, USA · Department of Electrical and Computer Engineering, Rutgers University, Piscataway, NJ, USA
MGVQ: Synergizing Multi-dimensional Sensitivity-Aware and Gradient-Hessian Fusion for Vector Quantization
May 20, 2026Zhong Wang, Zukang Xu, Xing Hu +1Mixed-Precision Quantization Vision-Language Models
Bauman Moscow State Technical University · Houmo AI
AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs
May 1, 2026Wenxiang Lin, Juntao Huang, Luhan Zhang +5LLM Quantization Communication-Efficient Distributed Training
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Huawei Technologies Ltd.
Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
Apr 28, 2026Haoran Chen, Wentao WangSign-Based Optimization Stochastic Optimization Convergence
´Ecole polytechnique Palaiseau, France
Quantization Impact on the Accuracy and Communication Efficiency Trade-off in Federated Learning for Aerospace Predictive Maintenance
Apr 9, 2026Abdelkarim LoukiliGradient Quantization Federated Learning
OptINC: Optical In-Network-Computing for Scalable Distributed Learning
Mar 30, 2026Sijie Fei, Grace Li Zhang, Bing Li +1Gradient Quantization Communication-Efficient Distributed Training
Technical University of Munich, Munich, Germany · Technical University of Darmstadt, Darmstadt, Germany · Technical University of Ilmenau, Ilmenau, Germany
On the Gradient Complexity of Private Optimization with Private Oracles
Nov 17, 2025Michael Menart, Aleksandar NikolovQuery Complexity Oracle Complexity
Department of Computer Science, University of Toronto, Vector Institute · Department of Computer Science, University of Toronto

Oct 17, 2026J/K move · Enter open · S save

What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes

Anjulie Agrusa, Andrei Panferov, Elizabeth Wei, Keith Wyss+6

NVIDIA · Institute of Science and Technology Austria · Red Hat AI

Low-Precision Training Language Model Pretraining 4-Bit Quantization Gradient Quantization

Open paper AskPDF arXiv

Training large language models directly in 4-bit floating-point (FP) formats promises substantial improvements in throughput and energy efficiency. While some recipe design choices have been validated at scale, many promising approaches remain untested beyond small models and short token horizons, leaving open the question of which trends will hold. We present a systematic comparison of recent NVFP4 recipes at medium model scale, evaluating dense 8B and 30B-A3B MoE models trained for up to 1T tokens, with a focus on which ingredients are necessary for accuracy recovery. We propose a final recipe grounded in the principles behind established stable NVFP4 training at scale, incorporating state-of-the-art techniques such as unbiased gradient estimation with lower quantization error than stochastic rounding. To the best of our knowledge, this is the strongest FP4 training result demonstrated at this scale to date in loss gap to BF16. Through ablation studies, we find that: (i) each technique in the optimized recipe measurably improves loss trajectory, (ii) selective high-precision layers are necessary for recovering accuracy at scale, (iii) not all tensors in the backward pass benefit equally from de-biasing, leaving room to apply complementary error-reduction techniques to the remaining tensors.