Memory-Efficient Training

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 34

All topics

Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training
Oct 6, 2026Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2LLM Quantization Quantization-Aware Training
ISTA · EPFL, ISTA
Cut Binary Cross Entropy: Efficient Large-Vocabulary Loss and Gradient Kernels for Sequential Recommendation
Oct 4, 2026Yaoyiran Li, Haowen Ning, Mohamed HammadSequential Recommendation Recommender Systems
Google Cloud London, United Kingdom · Google Cloud Mountain View, USA
Clean: Second-order LLM Training at Linear Memory Cost via Nyström Sketching
Oct 3, 2026Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi Shikai Fang +2Second-Order Optimization Memory-Efficient Training
Mila & DIRO, Université de Montréal · Critical ML, University of Waterloo · Independent Researcher +1
Scale-Split Neural Operator for Memory- and Data-Efficient 3D Turbulence Prediction
Sep 30, 2026Shaoxiang Qin, Yucheng Zhao, Zongyi Li +2PDE Surrogate Modeling Neural Operators
University of Tennessee, Knoxville · McGill University · New York University +1
Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation
Sep 28, 2026Yongliang Miao, Shuang Liu, Yanguang Liu +2Language Model Distillation On-Policy Distillation
The Chinese University of Hong Kong, Shenzhen · Carnegie Mellon University · New Jersey Institute of Technology +1
HiLoRe: What to Store, Compress, or Recompute for Efficient GRPO Training
Sep 27, 2026Xinrui Chen, Mengyang Li, Ou Wu +1Group Relative Policy Optimization Policy Learning
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China · Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, China · University of Southern Queensland
Low-Rank Friction for Memory-Efficient Transformer Pretraining
Sep 24, 2026Rajit Rajpal, Benedict LeimkuhlerDeep Learning Optimization Low-Rank Compression
School of Mathematics University of Edinburgh Edinburgh, UK EH9 3FD
MENO: Memory-Efficient Neural Operator
Sep 23, 2026Shengyang Xu, Weijun Zhang, Jun Hu +1PDE Surrogate Modeling PDE Operator Learning
School of Mathematical Sciences, Peking University, Beijing 100871, China · Chongqing Research Institute of Big Data, Peking University, Chongqing 401329, China · National Engineering Laboratory for Big Data Analysis and Applications, Peking University, Beijing 100871, China
CacheDyG: Decoupling Temporal Propagation for Efficient Dynamic Graph Learning
Sep 22, 2026PinHeng Zong, Ye YuanTemporal GNNs Temporal Link Prediction
College of Computer and Information Science, Southwest University, Chongqing, China
Layer-wise Curriculum Learning for Efficient LLM Compression
Sep 16, 2026Donggeon Lee, Dooyeon Na, Seungmin Oh +1LLM Compression Curriculum Learning
Ajou University, South Korea
LeanGRPO: Eliminating Redundant Recomputation in Diffusion RL
Sep 3, 2026Sijie Wang, Zhiqiang Tan, Xinrui Yang +1RL for Diffusion Models RL Post-Training
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen
LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training
Aug 12, 2026Xiaojun Wu, Cehao Yang, Honghao Liu +5Efficient Language Model Training LLM Training
1IDEA Research · 2The Hong Kong University of Science and Technology (Guangzhou) · 3DataArcTech Ltd.
Batch Size or Negatives? A Selection Rule for Memory-Constrained Recommender Training
Aug 11, 2026Artyom Sabitov, Daniil Volkov, Alexey ZaytsevMemory-Efficient Optimization Recommender Systems
Applied AI Institute, Moscow, Russia Risk department
Gradient Under Microscope: Benchmarking Resource Utilization of Memory-Efficient Gradient Computation Methods
Aug 9, 2026Sarthak Mahapatra, Zihan Zhou, Khatoon Khedri +2Efficient Language Model Training Adaptive Gradient Methods
Memory-Efficient Activation Checkpointing with Sliding Window and Hirschberg's Algorithm for 0/1 Knapsack Solving in PyTorch
Aug 9, 2026Jędrzej MaczanDynamic Programming Memory-Efficient Optimization
Cohere Labs Community Poland
ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling
Aug 8, 2026Wentao Dai, Xuanran Li, Yuxiang Zhang +2On-Device Training Memory-Efficient Fine-Tuning
Department of Computer Science and Engineering, Southern University of Science and Technology, China · Department of Mathematics, Southern University of Science and Technology, China · School of Computing, Montclair State University, New Jersey, USA
MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training
Aug 5, 2026Masato FujitakeNeural Network Optimization Memory-Efficient Optimization
FA Research, Fast Accounting Co., Ltd.
Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
Jul 21, 2026Nuemaan MalikDeep Learning Optimization Mixture-of-Experts Language Models
Independent Researcher
Backpropagation-Free Trunk Training via the Split Forward Gradients
Jul 18, 2026Tian Qin, Wei-Min HuangBackpropagation Gradient Descent
Math department Lehigh University Bethlehem, PA 18015, USA
MegaSlide-DiT: Memory-Centric Adaptation and Deformable Local Attention for Efficient Video Diffusion
Jul 17, 2026Jiacheng Liu, Jason LiuVideo Diffusion Models Diffusion Transformer
Trendinsight Lab / UC San Diego
Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
Jul 2, 2026Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao +3Communication-Efficient Distributed Training High-Performance Computing
Salesforce AI Research
FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training
Jun 22, 2026Dikshant Kukreja, Kritarth Prasad, Avinash Anand +6Automatic Differentiation Backpropagation
Puch AI · Singapore Institute of Technology · Nanyang Technological University +2
Gefen: Optimized Stochastic Optimizer
Jun 11, 2026Nadav Benedek, Tomer Koren, Ohad FriedDeep Learning Optimization Memory-Efficient Optimization
Reichman University · Tel Aviv University, Google Research
Unifying Data, Memory, and Compute Efficiency in LLM training: A Survey
Jun 9, 2026Vanessa Schmidt, Huy Hoang Nguyen, Cédric Jung +2Efficient Language Model Training LLM Training
Chair of Information Theory and Data Analytics (INDA), RWTH Aachen University, 52074 Aachen, Germany · AIT Austrian Institute of Technology GmbH, 1210 Vienna, Austria · Automation and Control Institute, Technische Universität Wien (TUW), 1040 Vienna, Austria
GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning
Jun 1, 2026Liyan Tan, Yequan Zhao, Yifan Yang +3Fine-Tuning Memory-Efficient Fine-Tuning
University of California, Santa Barbara
Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
May 30, 2026Qiao Xiao, Boqian Wu, Patrik Okanovic +6Language Model Pretraining Efficient Language Model Training
Eindhoven University of Technology · University of Luxembourg · University of Twente +1
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
May 25, 2026Zili Zhang, Chengxu Yang, Shenglong Zhang +8Deep Learning Optimization Multimodal Large Language Models
1Peking University · †Work done during internships at Xiaohongshu, Inc. · 2Independent Researcher +2
ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning
May 20, 2026Yongkang Liu, Zijing Wang, Mengjie Zhao +7Fine-Tuning Memory-Efficient Fine-Tuning
Northeastern University, China · Shanghai Jiao Tong University, China · CIS, LMU Munich, Germany +2
Beyond Square Roots: Explicit Memory-Efficient Factorization for Multi-Epoch Private Learning
May 18, 2026Nikita P. Kalinin, Aki Rehn, Joel Daniel Andersson +2Differentially Private Stochastic Gradient Descent Differential Privacy
Institute of Science and Technology Austria · University of Helsinki
PowerStep: Memory-Efficient Adaptive Optimization via $\ell_p$ -Norm Steepest Descent
May 11, 2026Yao Lu, Dengdong Fan, Shixun Zhang +1Gradient Descent Memory-Efficient Optimization
Pengcheng Laboratory · Peking University
Budget-aware Auto Optimizer Configurator
May 6, 2026Kang Liu, Wei Peng, Jianchen HuDeep Learning Optimization Memory-Efficient Optimization
School of Future Technology Xi’an Jiaotong University Xi’an, China · School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China
Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm
May 2, 2026Wen-Da Wei, Han-Bin Fang, Yang-Di Liu +3LLM Compression Gradient Compression
Nanjing University, Nanjing, China · Tsinghua University, Beijing, China · Huazhong University of Science and Technology, Wuhan, China +1
AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs
May 1, 2026Wenxiang Lin, Juntao Huang, Luhan Zhang +5LLM Quantization Communication-Efficient Distributed Training
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Huawei Technologies Ltd.
Gaussians on a Diet: High-Quality Memory-Bounded 3D Gaussian Splatting Training
Apr 21, 2026Yangming Zhang, Jian Xu, Chaojian Li +73D Gaussian Splatting 3D Gaussian Pruning
Dept. of Computer Science University of Texas at Arlington · School of Computer Science Georgia Institute of Technology · School of Computing University of Georgia +2
GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
Apr 17, 2026Jixuan Leng, Si Si, Hsiang-Fu Yu +2LLM Alignment Direct Preference Optimization
Carnegie Mellon University · Google DeepMind · Google
HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation
Mar 16, 2026Qiyuan Chen, Xian Wu, Yi Wang +1Communication-Efficient Distributed Training Zeroth-Order Optimization
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China.

Oct 6, 2026, cs.LGJ/K move · Enter open · S save

Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

Alexandra Volkova, Matin Ansaripour, Erik Schultheis, Christoph H. Lampert+1

ISTA · EPFL, ISTA

LLM Quantization Quantization-Aware Training Mixed-Precision Quantization Memory-Efficient Training

Open paper AskPDF arXiv

Quantization-aware training (QAT) leverages lower-precision arithmetic to reduce the cost of LLM deployment, but aggressive quantization degrades final model performance. A common remedy is mixed-precision training, in which high precision is assigned to some of the layers to maintain performance while keeping the cost constrained. This approach then requires precision assignments for model layers during training. We provide a new approach, called Q-PACE, consisting of a second-order sensitivity model that predicts the loss increase as a sum of quantization noise MSE weighted by per-layer curvature coefficients. During training, we periodically re-compute these coefficients using perturbations across layers, and re-assign precision. Pretraining and supervised fine-tuning experiments on LLMs of up to 4B parameters show that Q-PACE consistently improves over existing mixed-precision training recipes, and achieves comparable loss at substantially lower total memory budgets. We further find that quantization sensitivity is highly predictable by depth and layer type, and its stability during training allows for infrequent, cheap recalibration.

Momentum

Related topics

Latest papers 34