Stochastic Rounding

Follow this topic Ask about this topic

Latest papers

8 papers2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization
Oct 8, 2026Hanyang Li, Shao Tang, Daniel Thomas Braithwaite +7Stochastic Rounding 4-Bit Quantization
University of California, Berkeley · Nubank
Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2
Oct 1, 2026Yohan Chatelain, Pablo de Oliveira CastroTransformer Inference Stochastic Rounding
Krembil Centre for Neuroinformatics, CAMH, Toronto, Canada · Universite Paris-Saclay, UVSQ, LI-PaRAD, Versailles, France
UE5M3 FP4 Block Scaling for Stable Language Model Pretraining
Sep 2, 2026Robert Hu, Carlo Luschi, Paul BalancaStochastic Rounding Language Model Pretraining
GRAPHCORE RESEARCH
The Silent Freeze: Predicting When Low-Precision Training Stops Learning
Jul 9, 2026Zekai ShangStochastic Rounding Low-Precision Training
University of Illinois at Urbana-Champaign, Champaign, IL, USA
Novel Aspects of IEEE SA P3109 Arithmetic Formats for Machine Learning
Jun 1, 2026Andrew Fitzgibbon, Christoph M. Wintersteiger, Jeffrey SarnoffStochastic Rounding
Graphcore United Kingdom · Imandra, Inc. United Kingdom · IEEE USA
Stochastic Rounding Increases Small Singular Values
May 29, 2026Linkai Ma, Tingzhou Yu, Petros DrineasSpectral Regularization Stochastic Rounding
Department of Mathematics University of Alberta
Fuzzy PyTorch: Rapid Numerical Variability Evaluation for Deep Learning Models
May 25, 2026Inés Gonzalez-Pepe, Hiba Akhaddar, Tristan Glatard +1Stochastic Rounding Uncertainty Quantification
Department of Computer Science and Software Engineering Concordia University Montreal, Canada · Krembil Centre for Neuroinformatics Centre for Addiction and Mental Health Toronto, Canada
The AetherFloat Family: Block-Scale-Free Quad-Radix Floating-Point Architectures for AI Accelerators
Feb 26, 2026Keita MorisakiStochastic Rounding AI Accelerator Inference
Independent Hardware/Software Co-Design Researcher Shinagawa City, Tokyo, Japan

Oct 8, 2026J/K move · Enter open · S save

Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization

Hanyang Li, Shao Tang, Daniel Thomas Braithwaite, Gregory Dexter+6

University of California, Berkeley · Nubank

Stochastic Rounding 4-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates. We redesign 4-bit optimizer-state quantization for AdamW from the perspective of rounding space: the coordinate in which a quantizer chooses between adjacent reconstruction levels. For the second moment, a local analysis of the quantization cell adjacent to zero shows that small mean state error need not imply small mean preconditioner error at the next step. A one-dimensional quadratic construction further shows qualitatively different optimization dynamics under state-space and preconditioner-space rounding. These results motivate Zero-Inclusive Preconditioner-space Stochastic Rounding (ZIP-SR), which retains zero in the second-moment codebook and computes stochastic-rounding probabilities in preconditioner space. As a complementary route, Zero-Excluding EDEN calibration (ZE-EDEN) uses a zero-excluding second-moment codebook and rescales the quantized second-moment block to mitigate the preconditioner distortion caused by the positive quantization floor. Both configurations use 4-bit NormalFloat (NF4) for the first moment, with targeted stochastic rounding of the LM-head first moment during the final 10% of training. Across GPT- and Llama-style pretraining experiments ranging from 130M to 2.7B parameters, both methods reduce TorchAO 4-bit AdamW's mean validation-loss gap to 32-bit AdamW at every evaluated model size, with the largest reported gap reduction reaching 70%. In full-parameter supervised fine-tuning, both recipes achieve lower validation loss than TorchAO while remaining close to 32-bit AdamW on downstream tasks.