Low-Precision Training

Latest papers

28 papers8 papers in the last four weeks, up 167% on the four weeks before. 0.1% of all new papers.

The Rise of Low Precision Floating Point
Oct 30, 2026Guy Lemieux, Michael L. OvertonLow-Precision Training
University of British Columbia, Department of Electrical and Computer Engineering, Vancouver, BC, Canada · New York University, Courant Institute School of Mathematics, Computing and Data Science, New York, NY, USA
Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding
Oct 29, 2026Zhichao Wang, Michael W. MahoneyLow-Precision Training Stochastic Rounding
ICSI and Department of Statistics University of California, Berkeley, USA · ICSI, LBNL and Department of Statistics University of California, Berkeley, USA
Efficient Quantization Training with Stochastic Rounding via Structured Subspace Randomness
Oct 27, 2026Zeyu Wang, Yiheng Duan, Tao Chen +5Stochastic Rounding Low-Precision Training
Google, Mountain View, CA, USA
Training an MoE LLM in MXFP4: How We Closed the Gap to BF16
Oct 23, 2026Han Wang, Zhitao Wang, Natalia Frumkin +4LLM Quantization Low-Precision Training
Advanced Micro Devices, Inc.
What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes
Oct 17, 2026Anjulie Agrusa, Andrei Panferov, Elizabeth Wei +7Low-Precision Training Language Model Pretraining
NVIDIA · Institute of Science and Technology Austria · Red Hat AI
Mixed Precision Scaling Laws for Quantized Pretraining
Oct 13, 2026Jinuk Kim, Rajarshi Saha, Muhammed Fatih Balin +1Mixed-Precision Quantization LLM Quantization
Seoul National University · Amazon Annapurna Labs
Memory-Efficient Integer-Only Training of Deep Neural Networks
Oct 12, 2026Hazem Hesham Yousef Shalby, Fabrizio Pittorino, Manuel RoveriLow-Precision Training Neural Network Quantization
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB), Politecnico di Milano, Milan, Italy
Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training
Oct 6, 2026Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2LLM Quantization Quantization-Aware Training
ISTA · EPFL, ISTA
Backward-State Policy Is Part of the Learning Algorithm
Sep 30, 2026Shuxiao Xie, Shuyang Xie, Dezhi Ran +2Mixed-Precision Quantization Backpropagation
Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC), China
Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs
Sep 29, 2026Haozhan Tang, Hao Kang, Han Cai +2Low-Precision Training LLM Training
Carnegie Mellon University · NVIDIA · NVIDIA & Massachusetts Institute of Technology
Linger and Lose: Knowledge Collapse in Low-Bit Language Models
Sep 26, 2026Prashanna Mani Paudel, Shivanand Venkanna SheshappanavarLow-Precision Training LLM Quantization
Geometric Intelligence Research Lab, Dept. of Electrical Engineering and Computer Science, University of Wyoming, USA.
Format-Aware Fusion for Fast FP4 Pretraining
Sep 4, 2026Robert HuLow-Precision Training LLM Quantization
GRAPHCORE RESEARCH
Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090
Aug 27, 2026Kairong Luo, Jiarui Cui, Yaorui Yin +8Language Model Pretraining Language Model Scaling Laws
Tsinghua University · Pengcheng Laboratory
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning
Aug 24, 2026Md Romyull IslamFine-Tuning Quantization-Aware LoRA
Kennesaw State University
One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse
Aug 3, 2026Shuxiao Xie, Shuyang Xie, Yuan Cao +3Softmax Attention Transformer Attention
Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC), China · Fudan University Institute of Systems for Advanced Computing, China · Harbin Institute of Technology, China +1
M+Adam: Low-Precision Training via Additive-Multiplicative Optimization
Jul 12, 2026Xiaoyuan Liang, Sebastian Loeschcke, Mads Toftrup +1Quantization-Aware Training Low-Precision Training
California Institute of Technology · University of Copenhagen · Aarhus University
The Silent Freeze: Predicting When Low-Precision Training Stops Learning
Jul 9, 2026Zekai ShangStochastic Rounding Low-Precision Training
University of Illinois at Urbana-Champaign, Champaign, IL, USA
ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Training
Jun 14, 2026Janghwan Lee, Sihwa Lee, Jinseok Kim +4Activation Quantization Quantization-Aware Training
1Hanyang University, 2Rebellions Inc., Republic of Korea.
GNMR: Runtime Stability Control for Low-Precision Large Language Model Training
May 30, 2026Boao Kong, Weichen Jia, Engao Zhang +6LLM Training Low-Precision Training
Peking University · Huawei Technologies Ltd.
Mapping the Schedule x Bit-Width Boundary in Sub-100M Quantisation-Aware Training
May 25, 2026Christian Brandt ThomassenLLM Quantization Learning Rate Scheduling
Dwarf A/S, Copenhagen, Denmark
Normalized Architectures are Natively 4-Bit
May 7, 2026Maxim Fishman, Brian Chmiel, Ron Banner +2LLM Quantization Neural Representation Geometry
∧NVIDIA · ◦Department of Electrical and Computer Engineering - Technion, Haifa, Israel
HiFloat4 Format for Language Model Pre-training on Ascend NPUs
Apr 9, 2026Mehran Taghian, Yunke Peng, Xing Huang +22Low-Precision Training LLM Quantization
Huawei
AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation
Apr 2, 2026Seonggon Kim, Alireza Khodamoradi, Pranathi Vasireddy +2Walsh-Hadamard Transform Low-Precision Training
Advanced Micro Devices, Inc. · POSTECH
Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets
Mar 17, 2026Kristi Topollai, Anna ChoromanskaLow-Precision Training Low-Bit Quantization
New York University
True 4-Bit Quantized Convolutional Neural Network Training on CPU: Achieving Full-Precision Parity
Mar 14, 2026Shivnath TatheLow-Precision Training 4-Bit Quantization
Independent Researcher, Pune, India
Scalable Training of Mixture-of-Experts Models with Megatron Core
Mar 8, 2026Zijie Yan, Hongxiao Bai, Xin Yao +42Low-Precision Training Mixture-of-Experts Models
NVIDIA
Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs
Mar 3, 2026Wuyue Zhang, Chongdong Huang, Chunbo You +3Low-Precision Training LLM Training
Zhejiang Lab, Hangzhou, China
SageBwd: A Trainable Low-bit Attention
Mar 2, 2026Jintao Zhang, Marco Chen, Haoxu Wang +5Attention Quantization Low-Precision Training
Tsinghua University, UC Berkeley

Oct 30, 2026J/K move · Enter open · S save

The Rise of Low Precision Floating Point

Guy Lemieux, Michael L. Overton

University of British Columbia, Department of Electrical and Computer Engineering, Vancouver, BC, Canada · New York University, Courant Institute School of Mathematics, Computing and Data Science, New York, NY, USA

Low-Precision Training

Open paper AskPDF arXiv

Floating point computing has dominated scientific computations since the 1950s because of its versatile ability to approximate real numbers with both sufficiently high precision and sufficiently great range. Since 1985, two specific formats have become standard across the industry, one with 32 bits which is adequate for many purposes and one with 64 bits which allows numbers with both much higher precision and much greater range for applications where these are needed. However, during the past decade, deep learning has come to dominate computation, and narrower floating point formats with lower precision and reduced range have steadily become more common. In this article we discuss this phenomenon, why it is happening, how various 16-bit, 8-bit and even 4-bit floating point formats have been developed, and efforts to standardize these formats.