Memory-Efficient Fine-Tuning

Overview

This field studies how to adapt pretrained models when memory for weights, activations, gradients, and optimizer states is limited, including on consumer GPUs and edge devices. Common approaches include LoRA and quantized adapters, selective or sparse updates, activation checkpointing and offloading, and zeroth-order optimization that avoids storing backpropagated gradients. Work also targets full fine-tuning and long-context training through streamed parameter updates, compressed states, and tiered memory systems. A central tension is that fewer trainable parameters do not always reduce activation memory, and memory savings can trade off against accuracy, speed, energy use, or implementation complexity.

Latest papers

CardsList

69 papers7 papers in the last four weeks, down 59% on the four weeks before. 0.1% of all new papers.

  1. Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment

    Nov 16, 2026Weiyun Huang, Alessio GalatoloPreference Learning for LLMsDirect Preference Optimization

  2. SMELL: Sparsity-aware Memory Efficient Federated Fine-tuning for Long-context LLMs

    Nov 14, 2026Yin Xu, Yongbo Yang, He Sun +2Federated Fine-TuningMemory-Efficient Fine-Tuning

  3. Direction or Magnitude? Learning Scalars over Task-Calibrated Directions

    Nov 13, 2026Ziqing Wen, Xiaohui Song, Jiahuan Wang +5LLM Fine-TuningParameter-Efficient Fine-Tuning

  4. Wanda-Guided Sparse Perturbations for Efficient Zeroth-Order Optimization

    Nov 12, 2026Ziqing Wen, Xiaohui Song, Jiahuan Wang +4Zeroth-Order OptimizationLLM Fine-Tuning

  5. Zeroth-Order Riemannian Optimization on Fixed-Rank Manifolds for LLM Fine-Tuning

    Nov 7, 2026Xinjue Wang, Xiuheng Wang, Yejun Zhang +2Zeroth-Order OptimizationRiemannian Optimization

  6. ENGINE: Endogenous Variational MultiScale Optimization for Zeroth-Order LLM Fine-Tuning

    Nov 2, 2026Zhuoli Ouyang, Changxi Chi, Siyuan Li +1Zeroth-Order OptimizationLLM Fine-Tuning

  7. DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation

    Oct 26, 2026Paul Raj, Simon Caldwell, Daniel Whitmore +1Mixture-of-Experts Fine-TuningMemory-Efficient Fine-Tuning

  8. PACT: NVFP4 Data-Flow LoRA Fine-Tuning on Consumer GPUs

    Oct 18, 2026Lei Gao, Hossein Entezari Zarch, Chaoyi Jiang +1Memory-Efficient Fine-TuningParameter-Efficient Fine-Tuning

  9. AdamPM: Memory-Efficient LLM Optimization via Moment Pooling

    Oct 17, 2026Xiaowei Ji, Lishan Xu, Jianing An +3Optimizer State CompressionEfficient Language Model Training

  10. Local proxies for memory-efficient training

    Oct 16, 2026Wu S. Zihan, Brian A. Yuan, Guillaume BellecLocal LearningMemory-Efficient Training

  11. LCT: Losslessly Compressed Tensors with GPU friendly execution for Training

    Oct 15, 2026Max Zhu, Bohang Sun, Pietro Lio +1Memory CompressionMemory-Efficient Training

  12. StructZO: Structured Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning

    Oct 12, 2026Xuanlin Yang, Jia Li, Zhouchen LinZeroth-Order OptimizationMemory-Efficient Fine-Tuning

  13. Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces

    Oct 10, 2026Chuang Tong, Ling ChenLLM Fine-TuningZeroth-Order Optimization

  14. Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

    Oct 9, 2026Xi Zhang, Boyu Diao, Zesen Chen +3Continual Learning for VLMsMultimodal Instruction Tuning

  15. LadderEdit: Edit-Level Residual Compression for Memory-Efficient Lifelong Editing of LLMs

    Oct 8, 2026Xiaobing Yu, Peijie Qiu, Jin Yang +5Language Model EditingMemory-Efficient Fine-Tuning

  16. MemFLoRA: Memory-Floor LoRA for CNN Adaptation at the Edge

    Oct 6, 2026Mehmet Emre Akbulut, Johannes Geier, Ulf SchlichtmannOn-Device TrainingMemory-Efficient Fine-Tuning

  17. RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

    Oct 5, 2026Muhammad Azeem Lodhi, Chao Zhou, Rebekka BurkholzMemory-Efficient Fine-TuningLLM Fine-Tuning

  18. DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategies

    Oct 4, 2026Zhishen Sun, Hongzhan Wang, Sizhe Dang +2LLM Fine-TuningEvolution Strategies

  19. TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning

    Oct 1, 2026Jichao Jiang, Cristian McGee, El Houcine Bergou +2Memory-Efficient Fine-TuningLLM Fine-Tuning

  20. Probe-Space Preconditioning for Fast and Stable Zero-Order Training

    Sep 29, 2026Francois Chaubard, Mykel J. Kochenderfer, Chris RéZeroth-Order OptimizationMemory-Efficient Fine-Tuning

  21. AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning

    Sep 28, 2026Yue Xie, Zhi Zheng, Yunpeng Ba +5Subspace OptimizationMemory-Efficient Fine-Tuning

  22. TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge

    Sep 27, 2026Houssem Sifaou, Prabodh Katti, Bipin Rajendran +1Memory-Efficient Fine-TuningZeroth-Order Optimization

  23. MpSub: A Momentum pp-Dimensional Subspace Trust-Region Method for Derivative-Free Fine-Tuning of Large Language Models

    Sep 7, 2026Yuyang Wang, Haoyu Yao, Pengcheng XieMemory-Efficient Fine-TuningZeroth-Order Optimization

  24. Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation

    Sep 1, 2026Wentao Ye, Zhanming Shen, Zhiqing Xiao +3Memory-Efficient Fine-TuningLLM Fine-Tuning

  25. SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces

    Aug 16, 2026Ziming Yu, Shuyao Xiao, Xingyu Zhao +6Memory-Efficient Fine-TuningZeroth-Order Optimization

  26. SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuning

    Aug 15, 2026Mohammad Aref Jafari-Raddani, Morteza Mohajjel KafshdoozParameter SharingParameter-Efficient Fine-Tuning

  27. TERRA: A Hierarchical Parallel Training and Memory Orchestration Framework for High-Resolution AI-based Earth Modeling

    Aug 15, 2026Ruohan Wu, Ziqi Zhu, Yang Zhao +6Distributed TrainingMemory-Efficient Fine-Tuning