Neural Network Quantization

Follow this topic Ask about this topic

Momentum

36 papers in the last four weeks, up 125% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 270

All topics

Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization
Oct 8, 2026Hanyang Li, Shao Tang, Daniel Thomas Braithwaite +74-Bit Quantization Neural Network Quantization
University of California, Berkeley · Nubank
Read What Matters: Query-Adaptive Quantization for KV Caches
Oct 8, 2026Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian +1KV-Cache Quantization KV-Cache Compression
Amazon · University of California, Berkeley
TR-PTQ: High-Accuracy Integer-Only Transformer Post Training Quantization via Taylor Region Reformulation
Oct 7, 2026Eliyahu Levy, Adam Teman, Yoni PugachovPost-Training Quantization Neural Network Quantization
Faculty of Engineering, Bar-Ilan University, Ramat Gan, Israel
Layerwise Error Attribution for Fast and Robust Mixed-Precision Post-Training Quantization
Oct 7, 2026Samy Houache, Yann Traonmilin, Jean-François AujolPost-Training Quantization Neural Network Quantization
IMB, UB · UB, IMB
Quantize by Drift: Label-Free Mixed-Precision Post-Training Quantization for Text Embedders
Oct 6, 2026Hyojung Han, Jongmin Kim, Seung-Hun JeonPost-Training Quantization Mixed-Precision Quantization
ThakiCloud
CurveTQ: Rotation-Free Trellis Quantization of LLM Weights via Curvature-Weighted Search
Oct 6, 2026Guanhua Ding, Zi Wang, Ruichao Li +1LLM Quantization Neural Network Quantization
Black Sesame Technologies
Few Bits, One Law: Toward W2A4KV2
Oct 6, 2026Kai Yi, Tarek Elgamal, Sruthikesh Surineni +3LLM Quantization Low-Bit Quantization
Meta AI
ApexQuant: Data-Free Elastic Quantization by Residual Re-Isotropization
Oct 6, 2026Aksel Fristrup, Sumit Pandey, Ankit KariryaaLLM Quantization Rotation-Based Quantization
Henselian (henselian.com) · Department of Computer Science University of Copenhagen Denmark · Towards Deep Learning
Activation Denoising: A Robustness View on Parallel vs Sequential LLM Quantization
Oct 5, 2026Yan Scholten, Rachel Lawrence, James Hensman +3LLM Quantization Post-Training Quantization
Dept. of Computer Science & Munich Data Science Institute, Technical University of Munich · Microsoft Research Cambridge, UK
AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation
Oct 5, 2026Hanzhi Zhang, Qiao Zhang, Qinglei Cao +4Mixed-Precision Quantization LLM Quantization
LLaVi Lab, Computer Science & Engineering, University of North Texas
SoloQ: Calibration-Free Quantization for Diffusion Language Models
Oct 5, 2026Donghyun Lee, Arkapravo Ghosh, Varun Manjunath +5Diffusion Model Quantization KV-Cache Quantization
University of Southern California · Yale University · Korea University
CentriQ: Calibration-Free Quantization of Diffusion Transformers via Exact Mean Centering
Oct 5, 2026Nataša Jovanović, Mathieu Salzmann, Saqib JavedDiffusion Transformer Diffusion Model Quantization
Tenstorrent · EPFL
Shared Stopping Decisions Change Answers in HQQ Cache Quantization
Oct 5, 2026Seunghui Jwa, Minsu Oh, Chanjun Park +1LLM Inference KV-Cache Quantization
Department of Artificial Intelligence, Jeju National University · School of Software, Soongsil University
Differentiable Bit-Widths: Co-optimizing Pruning and Quantization via SVD for Ultra-Efficient LLM Compression
Oct 5, 2026Hankyul Kang, Jongbin RyuLLM Quantization LLM Pruning
Ajou University
Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization
Oct 4, 2026Hanzhang Wang, Tianqi Shen, Zonglin Liu +3LLM Quantization Post-Training Quantization
City University of Hong Kong · The University of Hong Kong
The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization
Oct 1, 2026Chao Li, Shigeng Wang, Anbang YaoLLM Quantization Post-Training Quantization
Intel Labs China
XOR-Trellis: Ultra-Low-Complexity Dequantization and Curvature-Aware Hadamard-Free LLM Quantization
Sep 30, 2026Xiaofan Que, Nir Elkayam, Spandan Pyakurel +2LLM Quantization LLM Inference Acceleration
Arm, Inc. · Rochester Institute of Technology
A Tilted Bowl Is Not a Slippery Slope: Compressing Looped Models
Sep 30, 2026Steven Kolawole, Pearse Jim, Opegbemi M. Busoye +2Efficient Neural Network Inference Recurrent Neural Networks
Carnegie Mellon University · ML Collective
Low-Discrepancy Dither for Quantized Recurrent State Caches
Sep 30, 2026Snigdha Chandan KhilarLLM Quantization Neural Network Quantization
Independent Researcher
SteerQuant: Steering Quantization Error with Action-Guided Scaling in World-Action Models
Sep 30, 2026Yunhan Wang, Haodong Wang, Zhiming Liu +74-Bit Quantization Efficient Inference for World Action Models
Tongji · HKUST · HIT +1
JARQ: Joint Alternating Refinement for Quantization
Sep 29, 2026Xinyu Wang, Sicheng Lyu, Xiao-Wen ChangLLM Quantization Weight-Only Quantization
McGill University
ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights
Sep 29, 2026Jonathan Mei, Sang Hyub Kim, Oliver Knitter +2LLM Quantization Weight-Only Quantization
Applications R&D IonQ College Park, MD 20740, USA
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10LLM Quantization LLM Inference Acceleration
UC Berkeley · University of Washington · MIT +2
QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching
Sep 29, 2026Zunhai Su, Yuxuan Sun, Jianchao Tan +5LLM Inference Acceleration Attention Quantization
The University of Hong Kong · Meituan LongCat Team · South China University of Technology +1
TORQUE: Optimizing What (not) to Quantize Before and After Rotation
Sep 28, 2026Ran Ben Basat, Michael Mitzenmacher, Shay VargaftikRotation-Based Quantization Vector Quantization
University College London · Harvard University · VMware Research by Broadcom
IMC-CLINIC: Coupled Loss-Informed Newton Iterations for Clipping in Analog In-Memory Computing
Sep 28, 2026Yung-Chin Chen, Chia-Yu Chen, Naveen VermaLLM Quantization Compute-in-Memory
Princeton University, NJ, USA · EnCharge AI, CA, USA
From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers
Sep 28, 2026Nafiseh HosseinpourFardi, Negar Alihadi, Mahmoudreza Babaei +2Mixed-Precision Quantization LLM Quantization
Sharif University of Technology, Tehran, Iran · GISMA University of Applied Sciences, Potsdam, Germany · BRAINS, Brandenburg Research Center for Applied Intelligent Systems, Potsdam, Germany +3
QuantForge: Discovering Residual Decompositions for MXFP4 Post-Training Quantization
Sep 28, 2026Qiulin Shang, Zhoutong Wu, Jie Hu +1LLM Quantization Large Language Model-Guided Optimization
Peking University
GLF-Q: Global-Local Feature-based Quantization for Vision Transformers
Sep 28, 2026Peilin Sun, Guang Liang, Jin Tong +1Post-Training Quantization Low-Bit Quantization
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · Zhongguancun Academy, Beijing 100094, China
QuantaSpike: Short-Window Spike-Driven Quantization for Large Language Models
Sep 28, 2026Bang Hu, Guowei Zhu, Changze Lv +4LLM Quantization Spiking Neural Networks
School of Computer Science, Fudan University, Shanghai, China

Oct 8, 2026, cs.LGJ/K move · Enter open · S save

Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization

Hanyang Li, Shao Tang, Daniel Thomas Braithwaite, Gregory Dexter+6

University of California, Berkeley · Nubank

4-Bit Quantization Neural Network Quantization

Open paper AskPDF arXiv

Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates. We redesign 4-bit optimizer-state quantization for AdamW from the perspective of \emph{rounding space}: the coordinate in which a quantizer chooses between adjacent reconstruction levels. For the second moment, a local analysis of the quantization cell adjacent to zero shows that small mean state error need not imply small mean preconditioner error at the next step. A one-dimensional quadratic construction further shows qualitatively different optimization dynamics under state-space and preconditioner-space rounding. These results motivate Zero-Inclusive Preconditioner-space Stochastic Rounding (\textbf{ZIP-SR}), which retains zero in the second-moment codebook and computes stochastic-rounding probabilities in preconditioner space. As a complementary route, Zero-Excluding EDEN calibration (\textbf{ZE-EDEN}) uses a zero-excluding second-moment codebook and rescales the quantized second-moment block to mitigate the preconditioner distortion caused by the positive quantization floor. Both configurations use 4-bit NormalFloat (NF4) for the first moment, with targeted stochastic rounding of the LM-head first moment during the final 10% of training. Across GPT- and Llama-style pretraining experiments ranging from \textbf{130M} to \textbf{2.7B} parameters, both methods reduce TorchAO 4-bit AdamW's mean validation-loss gap to 32-bit AdamW at every evaluated model size, with the largest reported gap reduction reaching \textbf{70%}. In full-parameter supervised fine-tuning, both recipes achieve lower validation loss than TorchAO while remaining close to 32-bit AdamW on downstream tasks.

Momentum

Related topics

Latest papers 270