Memory-Efficient Training

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 36

All topics

Budget-aware Auto Optimizer Configurator
May 6, 2026Kang Liu, Wei Peng, Jianchen HuDeep Learning Optimization Memory-Efficient Optimization
School of Future Technology Xi’an Jiaotong University Xi’an, China · School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China
Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm
May 2, 2026Wen-Da Wei, Han-Bin Fang, Yang-Di Liu +3LLM Compression Gradient Compression
Nanjing University, Nanjing, China · Tsinghua University, Beijing, China · Huazhong University of Science and Technology, Wuhan, China +1
AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs
May 1, 2026Wenxiang Lin, Juntao Huang, Luhan Zhang +5LLM Quantization Communication-Efficient Distributed Training
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Huawei Technologies Ltd.
Gaussians on a Diet: High-Quality Memory-Bounded 3D Gaussian Splatting Training
Apr 21, 2026Yangming Zhang, Jian Xu, Chaojian Li +73D Gaussian Splatting 3D Gaussian Pruning
Dept. of Computer Science University of Texas at Arlington · School of Computer Science Georgia Institute of Technology · School of Computing University of Georgia +2
GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
Apr 17, 2026Jixuan Leng, Si Si, Hsiang-Fu Yu +2LLM Alignment Direct Preference Optimization
Carnegie Mellon University · Google DeepMind · Google
HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation
Mar 16, 2026Qiyuan Chen, Xian Wu, Yi Wang +1Communication-Efficient Distributed Training Zeroth-Order Optimization
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China.

May 6, 2026, cs.AIJ/K move · Enter open · S save

Budget-aware Auto Optimizer Configurator

Kang Liu, Wei Peng, Jianchen Hu

School of Future Technology Xi’an Jiaotong University Xi’an, China · School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China

Deep Learning Optimization Memory-Efficient Optimization Efficient Language Model Training Memory-Efficient Training

Open paper AskPDF arXiv

Optimizer states occupy massive GPU memory in large-scale model training. However, gradients in different network blocks exhibit distinct behaviors, such as varying directional stability and scale anisotropy, implying that expensive optimizer states are not universally necessary and using a global optimizer is often memory-inefficient. We propose the Budget-Aware Optimizer Configurator (BAOC) to reduce memory cost by assigning suitable optimizer configurations to individual blocks under given budgets. Specifically, BAOC samples gradient streams to derive statistical metrics that quantify the potential performance risk of applying cheaper configurations (e.g., low precision or removing momentum). It then solves a constrained allocation problem to minimize total risk under memory and time budgets, selecting a budget-feasible configuration for each block. Experiments across vision, language, and diffusion workloads demonstrate that BAOC maintains training quality while significantly reducing the memory usage of optimizer states. The code is available at https://anonymous.4open.science/r/BAOC-45C6.

Momentum

Related topics

Latest papers 36

Budget-aware Auto Optimizer Configurator

Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm

AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs

Gaussians on a Diet: High-Quality Memory-Bounded 3D Gaussian Splatting Training

GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization

HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation