Overview

Memory-efficient training studies how to reduce the memory used by model weights, activations, gradients, and optimizer states so larger models or longer sequences can be trained on limited hardware. Common approaches include quantizing or factorizing optimizer states and activations, sparse updates, activation checkpointing and recomputation, and offloading data between GPU, host memory, and storage. Systems work combines these methods with parallelism, fused kernels, and memory-aware scheduling to improve throughput as well as fit. The main tension is that memory savings can add computation, communication, or approximation error, so the best trade-off depends on the model, hardware, and training workload.

Latest papers

CardsList

100 papers13 papers in the last four weeks, down 54% on the four weeks before. 0.1% of all new papers.

  1. POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformation

    Mar 5, 2026Zeju Qiu, Lixin Liu, Adrian Weller +2LLM TrainingMemory-Efficient Training

  2. Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation

    Feb 27, 2026Zhengbo Wang, Jian Liang, Ran He +2Optimizer State CompressionMemory-Efficient Training

  3. FastODT: A tree-based framework for efficient continual learning

    Feb 27, 2026Daniel Bretsko, Piotr Walas, Devashish Khulbe +3Online Continual LearningDecision Tree Learning

  4. FlashOptim: Optimizers for Memory-Efficient Training

    Feb 26, 2026Jose Javier Gonzalez Ortiz, Abhay Gupta, Christopher Rinard +1Memory-Efficient TrainingMixed-Precision Quantization

  5. PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Training

    Feb 26, 2026Yanyi Li, Yimu Zhang, Cong FangLLM CompressionMemory-Efficient Training

  6. Rethinking Efficiency in Neural Combinatorial Optimization: Batched Preference Optimization with Mamba

    Feb 24, 2026Zhenxing Xu, Zeyuan Ma, Weidong Bao +4Neural Combinatorial OptimizationMemory-Efficient Training

  7. Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation

    Feb 16, 2026Killian Bakong, Laurent Massoulié, Edouard Oyallon +1BackpropagationDeep Learning Optimization

  8. ZenFlow: Enabling Stall-Free Offloading for LLM Training

    May 18, 2025Tingfeng Lan, Yusen Wu, Bin Ma +7LLM TrainingLLM Fine-Tuning