Optimizer State Compression

Latest papers

9 papers2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

ResQAdam: Residual-Preserving Adaptive Optimization for Memory-Efficient Quantization Aware Training
Nov 6, 2026Liqun Ma, Shengkun Tang, Liangyu Wang +7Optimizer State Compression LLM Quantization
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · King Abdullah University of Science and Technology (KAUST) · Advanced Micro Devices, Inc. (AMD) +1
Guarded Compression of Adam Optimizer State with Shared-Gradient Bounds
Oct 31, 2026Annan Zhang, Shengjie Song, Wenshuai Yao +2Optimizer State Compression Quantization
Peking University · Beihang University
AdamPM: Memory-Efficient LLM Optimization via Moment Pooling
Oct 17, 2026Xiaowei Ji, Lishan Xu, Jianing An +3Optimizer State Compression Efficient Language Model Training
SKLCCSE, School of Artificial Intelligence, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China
Clean: Second-order LLM Training at Linear Memory Cost via Nystr{ö}m Sketching
Oct 3, 2026Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi +3Nyström Method Optimizer State Compression
Mila & DIRO, Université de Montréal · Critical ML, University of Waterloo · Independent Researcher +2
TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning
Oct 1, 2026Jichao Jiang, Cristian McGee, El Houcine Bergou +2Memory-Efficient Fine-Tuning LLM Fine-Tuning
University of Central Florida · Mohammed VI Polytechnic University
Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
Jul 21, 2026Nuemaan MalikDeep Learning Optimization Mixture-of-Experts Language Models
Independent Researcher
Gefen: Optimized Stochastic Optimizer
Jun 11, 2026Nadav Benedek, Tomer Koren, Ohad FriedDeep Learning Optimization Memory-Efficient Optimization
Reichman University · Tel Aviv University, Google Research
Budget-aware Auto Optimizer Configurator
May 6, 2026Kang Liu, Wei Peng, Jianchen HuDeep Learning Optimization Memory-Efficient Optimization
School of Future Technology Xi’an Jiaotong University Xi’an, China · School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China
Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation
Feb 27, 2026Zhengbo Wang, Jian Liang, Ran He +2Optimizer State Compression Memory-Efficient Training
University of Science and Technology of China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +1

Nov 6, 2026J/K move · Enter open · S save

ResQAdam: Residual-Preserving Adaptive Optimization for Memory-Efficient Quantization Aware Training

Liqun Ma, Shengkun Tang, Liangyu Wang, Lin Zhao+6

Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · King Abdullah University of Science and Technology (KAUST) · Advanced Micro Devices, Inc. (AMD)+1

Optimizer State Compression LLM Quantization Quantization-Aware Training Memory-Efficient Optimization Low-Bit Quantization

Open paper AskPDF arXiv

Quantization-aware training (QAT) is the most reliable way to obtain accurate low-bit large language models (LLMs), but training with AdamW keeps two dense FP32 states for every weight matrix. We find that low-rank memory-efficient optimizers such as GaLore and LDAdam degrade substantially in QAT. A considerable part of the QAT momentum lies outside its top-r singular subspace, and projection-based methods discard this part at every step. We further observe that this residual remains useful when only its sign is kept. Based on these observations, we propose ResQAdam. It keeps a single BF16 momentum, applies Adam only to the r x r coordinates of the principal subspace, and adds a normalized sign update computed from the residual. We evaluate ResQAdam with two from-scratch QAT methods and three QAT fine-tuning pipelines, with weights quantized down to 2 bits. It reduces optimizer-state memory by 71% to 74% relative to AdamW, stays close to AdamW in most settings, and lowers training throughput by less than 2%.