Overview

Memory-efficient training studies how to reduce the memory used by model weights, activations, gradients, and optimizer states so larger models or longer sequences can be trained on limited hardware. Common approaches include quantizing or factorizing optimizer states and activations, sparse updates, activation checkpointing and recomputation, and offloading data between GPU, host memory, and storage. Systems work combines these methods with parallelism, fused kernels, and memory-aware scheduling to improve throughput as well as fit. The main tension is that memory savings can add computation, communication, or approximation error, so the best trade-off depends on the model, hardware, and training workload.

Latest papers

CardsList

100 papers13 papers in the last four weeks, down 54% on the four weeks before. 0.1% of all new papers.

  1. Supercharging Olmo-core for Efficient and Scalable MoE Training

    Nov 13, 2026Tianhua Tao, Akshita Bhagia, Dirk Groeneveld +8Mixture-of-Experts ModelsCommunication-Efficient Mixture-of-Experts

  2. OOM: Offload-Optimized Full-Parameter MoE Training via Memory-Execution Co-Design

    Nov 8, 2026Ruijia Yang, Shiyuan Lin, Chengxi Liao +1Memory-Efficient TrainingMixture-of-Experts Models

  3. Sensitivity-Gated Accumulator Caching for On-Device Learning of Low-Bit Hand Networks

    Nov 7, 2026Soroor Ghandali, Kanika Mamgain, Manjot Kaur +2On-Device TrainingLow-Bit Quantization

  4. Scaling Laws for SAE Training Data

    Nov 5, 2026Nikita Koriagin, Nikita Balagansky, Daniil GavrilovSparse AutoencodersMemory-Efficient Training

  5. Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training

    Nov 2, 2026Avni Mittal, Avinash Anand, Ashutosh Kumar +7Mixture-of-Experts ModelsMemory-Efficient Training

  6. Architecture-Aware Mixed Precision for Replica-Exchange Training of Geometry-Aware Neural Operators with Uncertainty Quantification

    Nov 2, 2026Ujin Jeon, Michael Penwarden, Wenqian Chen +1Mixed-Precision TrainingGeometry-Aware Neural Operators

  7. State-Aware Activation Recomputation with Replay Contracts and Selective Preservation

    Oct 31, 2026Annan Zhang, Shengjie Song, Wenshuai Yao +2Memory-Efficient Training

  8. Decoupled Prune-and-Reconstruct: Training on a Compact Core with Double-Layer Sparsification

    Oct 25, 2026Siqi Gu, Gaozhen Ma, Liheng Huang +4Graph Neural NetworksMemory-Efficient Training

  9. Scalable Unrolled Chains for MRI Reconstruction

    Oct 24, 2026Emiljo Mëhillaj, Ender KonukogluUnrolled Neural NetworksAccelerated MRI Reconstruction

  10. Stochastic Pruning of the DPO Computation Graph: 1.4x Faster Training and Lower Peak Memory

    Oct 21, 2026Yubing Wu, Wei Chen, Junmei Yang +2Direct Preference OptimizationMemory-Efficient Training

  11. Local proxies for memory-efficient training

    Oct 16, 2026Wu S. Zihan, Brian A. Yuan, Guillaume BellecLocal LearningMemory-Efficient Training

  12. Scalable Levenberg-Marquardt Training Framework for Deep Learning

    Oct 16, 2026Xianghao Wang, Johan Barthélemy, John TaylorNeural Network OptimizationSecond-Order Optimization

  13. BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks

    Oct 15, 2026Max Zhu, Bohang Sun, Pietro Lio +1Activation CompressionMemory Compression

  14. LCT: Losslessly Compressed Tensors with GPU friendly execution for Training

    Oct 15, 2026Max Zhu, Bohang Sun, Pietro Lio +1Memory CompressionMemory-Efficient Training

  15. Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon

    Oct 14, 2026Sifan Yang, Hongyi Tao, Dingzhi Yu +5Subspace OptimizationMemory-Efficient Training

  16. Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining

    Oct 12, 2026Alexander Miasnikov, Ruslan Kabirov, Gleb Molodtsov +7Muon OptimizerLLM Training

  17. Memory-Efficient Integer-Only Training of Deep Neural Networks

    Oct 12, 2026Hazem Hesham Yousef Shalby, Fabrizio Pittorino, Manuel RoveriLow-Precision TrainingNeural Network Quantization

  18. BARTER: Compact Dynamic Sparse Training with Standard Dense Kernels

    Oct 12, 2026Yueyuan Sui, Payal Mohapatra, Qi Zhu +1Dynamic Sparse TrainingNeural Network Pruning

  19. Structure-Aware Numerical Stabilization for Half-Precision Proximal Policy Optimization

    Oct 9, 2026Linxuan Wu, Liang ChenRL StabilityReinforcement Learning

  20. Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

    Oct 6, 2026Alexandra Volkova, Matin Ansaripour, Erik Schultheis +2LLM QuantizationQuantization-Aware Training

  21. Memory-Efficient Expert Routing for Distributed MoE Training

    Oct 5, 2026Arnab Kanti Tarafder, Jaume Guasch-Martí, Gokcen Kestor +1Expert RoutingMixture-of-Experts Models

  22. Cut Binary Cross Entropy: Efficient Large-Vocabulary Loss and Gradient Kernels for Sequential Recommendation

    Oct 4, 2026Yaoyiran Li, Haowen Ning, Mohamed HammadSequential RecommendationRecommender Systems

  23. Clean: Second-order LLM Training at Linear Memory Cost via Nystr{ö}m Sketching

    Oct 3, 2026Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi +3Nyström MethodOptimizer State Compression