Memory-Efficient Training

Follow this topic Ask about this topic

Memory-efficient training studies how to reduce the memory used by model weights, activations, gradients, and optimizer states so larger models or longer sequences can be trained on limited hardware. Common approaches include quantizing or factorizing optimizer states and activations, sparse updates, activation checkpointing and recomputation, and offloading data between GPU, host memory, and storage. Systems work combines these methods with parallelism, fused kernels, and memory-aware scheduling to improve throughput as well as fit. The main tension is that memory savings can add computation, communication, or approximation error, so the best trade-off depends on the model, hardware, and training workload.

Latest papers

90 papers25 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

Decoupled Prune-and-Reconstruct: Training on a Compact Core with Double-Layer Sparsification
Oct 25, 2026Siqi Gu, Gaozhen Ma, Liheng Huang +4Graph Neural Networks Memory-Efficient Training
School of Physics and Electronic Engineering, Nanyang Normal University, Wolong District, Nanyang, 473061, Henan, China · School of Integrated Circuits, Huazhong University of Science and Technology, 1037 Luoyu Road, Hongshan District, Wuhan, 430074, Hubei, China · School of Architecture and Urban Planning, Huazhong University of Science and Technology, 1037 Luoyu Road, Hongshan District, Wuhan, 430074, Hubei, China +1
Scalable Unrolled Chains for MRI Reconstruction
Oct 24, 2026Emiljo Mëhillaj, Ender KonukogluAccelerated MRI Reconstruction Memory-Efficient Training
Computer Vision Lab, ETH Zurich, Zurich, Switzerland · LOOP Zurich - Medical Research Center, Zurich, Switzerland
Stochastic Pruning of the DPO Computation Graph: 1.4x Faster Training and Lower Peak Memory
Oct 21, 2026Yubing Wu, Wei Chen, Junmei Yang +2Direct Preference Optimization Memory-Efficient Training
South China University of Technology · Shenzhen Technology University
Flash RNN-T: Memory-Efficient Joint Network and Transducer Loss Computation
Oct 16, 2026Mahmoud AshrafMemory-Efficient Training Streaming ASR
Tarteel AI
Local proxies for memory-efficient training
Oct 16, 2026Wu S. Zihan, Brian A. Yuan, Guillaume BellecLocal Learning Memory-Efficient Training
Machine Learning Research TU Wien Vienna, Austria
Scalable Levenberg-Marquardt Training Framework for Deep Learning
Oct 16, 2026Xianghao Wang, Johan Barthélemy, John TaylorNeural Network Optimization Second-Order Optimization
School of Computing Australian National University Canberra, Australia · NVIDIA CA, United States
BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks
Oct 15, 2026Max Zhu, Bohang Sun, Pietro Lio +1Activation Compression Memory Compression
University of Cambridge · University of Oxford
LCT: Losslessly Compressed Tensors with GPU friendly execution for Training
Oct 15, 2026Max Zhu, Bohang Sun, Pietro Lio +1Memory Compression Memory-Efficient Training
University of Cambridge · University of Oxford
Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
Oct 14, 2026Sifan Yang, Hongyi Tao, Dingzhi Yu +5Subspace Optimization Memory-Efficient Training
State Key Laboratory of Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Alibaba Token Hub, Alibaba Group +1
Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining
Oct 12, 2026Alexander Miasnikov, Ruslan Kabirov, Gleb Molodtsov +7Muon Optimizer LLM Training
BRAIn Lab · HSE University · MBZUAI
GpFlash: Maximizing Speed and Minimizing Memory for Gaussian Process Training and Prediction
Oct 12, 2026Andre MarchildonGaussian Processes Gaussian Process Regression
Applied Scientist at Amazon Prime Air, Seattle, USA
Memory-Efficient Integer-Only Training of Deep Neural Networks
Oct 12, 2026Hazem Hesham Yousef Shalby, Fabrizio Pittorino, Manuel RoveriLow-Precision Training Neural Network Quantization
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB), Politecnico di Milano, Milan, Italy
BARTER: Compact Dynamic Sparse Training with Standard Dense Kernels
Oct 12, 2026Yueyuan Sui, Payal Mohapatra, Qi Zhu +1Dynamic Sparse Training Neural Network Pruning
Northwestern University, USA · Arm Inc., USA
When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerialization
Oct 9, 2026Mahesh Reddy PagadalaMemory-Efficient Training
Aston University, Birmingham, UK
Structure-Aware Numerical Stabilization for Half-Precision Proximal Policy Optimization
Oct 9, 2026Linxuan Wu, Liang ChenRL Stability Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences Beijing, China
Phase-HDC: Replacing Optimizer History with Gradient Thresholds in Discrete Phase Learning
Oct 7, 2026Ahmed NebliMemory-Efficient Training Sign-Based Optimization
Mathalyse Research
Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training
Oct 6, 2026Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2LLM Quantization Quantization-Aware Training
ISTA · EPFL, ISTA
Memory-Efficient Expert Routing for Distributed MoE Training
Oct 5, 2026Arnab Kanti Tarafder, Jaume Guasch-Martí, Gokcen Kestor +1Expert Routing Mixture-of-Experts Models
William & Mary · Barcelona Supercomputing Center
Cut Binary Cross Entropy: Efficient Large-Vocabulary Loss and Gradient Kernels for Sequential Recommendation
Oct 4, 2026Yaoyiran Li, Haowen Ning, Mohamed HammadSequential Recommendation Recommender Systems
Google Cloud London, United Kingdom · Google Cloud Mountain View, USA
Clean: Second-order LLM Training at Linear Memory Cost via Nystr{ö}m Sketching
Oct 3, 2026Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi +3Nyström Method Second-Order Optimization
Mila & DIRO, Université de Montréal · Critical ML, University of Waterloo · Independent Researcher +2
Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPU
Oct 2, 2026Jafar Badour, Maurice van Keulen, Elena MocanuDynamic Sparse Training Sparse Neural Networks
University of Twente
Scale-Split Neural Operator for Memory- and Data-Efficient 3D Turbulence Prediction
Sep 30, 2026Shaoxiang Qin, Yucheng Zhao, Zongyi Li +2PDE Surrogate Modeling Neural Operators
University of Tennessee, Knoxville · McGill University · New York University +1
Probe-Space Preconditioning for Fast and Stable Zero-Order Training
Sep 29, 2026Francois Chaubard, Mykel J. Kochenderfer, Chris RéZeroth-Order Optimization Memory-Efficient Fine-Tuning
Stanford University, Stanford, CA, USA.
Backpropagated Output Momentum: Relocating Optimizer History from Parameters to Task Space
Sep 29, 2026Yuchen Li, Zongqi Fan, Nguyen H. Tran +1Deep Learning Optimization Neural Network Optimization
University of Sydney
Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation
Sep 28, 2026Yongliang Miao, Shuang Liu, Yanguang Liu +2Language Model Distillation On-Policy Distillation
The Chinese University of Hong Kong, Shenzhen · Carnegie Mellon University · New Jersey Institute of Technology +1
HiLoRe: What to Store, Compress, or Recompute for Efficient GRPO Training
Sep 27, 2026Xinrui Chen, Mengyang Li, Ou Wu +1Group Relative Policy Optimization Activation Compression
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China · Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, China · University of Southern Queensland
LocalProp: Neuro-Localized Memory-Efficient Backpropagation
Sep 26, 2026Diana-Nicoleta Grigore, Iuliana Georgescu, Radu Tudor IonescuLocal Learning Backpropagation
Department of Computer Science University of Bucharest, Romania
Deterministic Regime Switching and Feasibility Inversion in Dynamic Tensor Rematerialization
Sep 25, 2026Mahesh Reddy PagadalaMemory-Efficient Training
Aston University
Low-Rank Friction for Memory-Efficient Transformer Pretraining
Sep 24, 2026Rajit Rajpal, Benedict LeimkuhlerDeep Learning Optimization Low-Rank Compression
School of Mathematics University of Edinburgh Edinburgh, UK EH9 3FD
MENO: Memory-Efficient Neural Operator
Sep 23, 2026Shengyang Xu, Weijun Zhang, Jun Hu +1PDE Surrogate Modeling PDE Operator Learning
School of Mathematical Sciences, Peking University, Beijing 100871, China · Chongqing Research Institute of Big Data, Peking University, Chongqing 401329, China · National Engineering Laboratory for Big Data Analysis and Applications, Peking University, Beijing 100871, China

Oct 25, 2026J/K move · Enter open · S save

Decoupled Prune-and-Reconstruct: Training on a Compact Core with Double-Layer Sparsification

Siqi Gu, Gaozhen Ma, Liheng Huang, Weiqi He+3

School of Physics and Electronic Engineering, Nanyang Normal University, Wolong District, Nanyang, 473061, Henan, China · School of Integrated Circuits, Huazhong University of Science and Technology, 1037 Luoyu Road, Hongshan District, Wuhan, 430074, Hubei, China · School of Architecture and Urban Planning, Huazhong University of Science and Technology, 1037 Luoyu Road, Hongshan District, Wuhan, 430074, Hubei, China+1

Graph Neural Networks Memory-Efficient Training Graph Sparsification Neural Network Pruning Graph Reconstruction

Open paper AskPDF arXiv

On million-node graphs, the memory wall arrives before the compute wall: a GNN that never fits in VRAM cannot deliver the all-node inference its accuracy promises. The standard remedy, pruning, has always been an accuracy tax - paid once and never refunded. DPR refuses this framing. Its central insight is that the mask that shrinks training can also scaffold recovery: one compact node set bounds both computations of graph learning, serving as the encoder's training graph and, at inference, as the anchor from which a sparse structure-aware decoder rebuilds every pruned node (double-layer sparsification). The order is inverted - prune for efficiency, then reconstruct for accuracy - and the tax is repaid: the deployed 30%-budget model trains in 1,450 MiB of peak memory on one 6 GB GPU and recovers 96.9% of its dense-reconstruction reference, beating a same-GPU rerun of the edge-sparsification baseline DSpar by +1.36 pp at 50.1% lower peak training memory; against the same-protocol full-batch no-prune encoder the deployed point pays a measured 2.64 pp accuracy tax for a 2.8x smaller training footprint. Pruning without reconstruction leaves the encoder at 0.378 within the same 40-epoch window (Fig. 4), and all-node serving runs 46x faster than sampling-based GraphSAGE, giving up approx. 5.4 pp of test accuracy. The framework scales to a 2.45M-node production graph at 6,285 MiB of peak training memory; extended variants (long-range context, external text features, a CAN-bus application) are reported as extensions and do not compete with the deployed method, and the 41.8%-budget M4 point, scored on a coarsened super-node graph with its own split, is reported separately rather than compared with the original-node baselines. DPR thus turns aggressive node pruning from an accuracy tax into a safe training accelerator for VRAM-limited all-node inference.