Memory-Efficient Fine-Tuning

Ask about this topic

Overview

This field studies how to adapt pretrained models when memory for weights, activations, gradients, and optimizer states is limited, including on consumer GPUs and edge devices. Common approaches include LoRA and quantized adapters, selective or sparse updates, activation checkpointing and offloading, and zeroth-order optimization that avoids storing backpropagated gradients. Work also targets full fine-tuning and long-context training through streamed parameter updates, compressed states, and tiered memory systems. A central tension is that fewer trainable parameters do not always reduce activation memory, and memory savings can trade off against accuracy, speed, energy use, or implementation complexity.

Latest papers

CardsList

61 papers16 papers in the last four weeks, up 433% on the four weeks before. 0.1% of all new papers.

  1. DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation

    Oct 26, 2026Paul Raj, Simon Caldwell, Daniel Whitmore +1Mixture-of-Experts Fine-TuningMemory-Efficient Fine-Tuning

    Texas A&M University-Commerce · Michigan State University

  2. PACT: NVFP4 Data-Flow LoRA Fine-Tuning on Consumer GPUs

    Oct 18, 2026Lei Gao, Hossein Entezari Zarch, Chaoyi Jiang +1Memory-Efficient Fine-TuningParameter-Efficient Fine-Tuning

    University of Southern California

  3. AdamPM: Memory-Efficient LLM Optimization via Moment Pooling

    Oct 17, 2026Xiaowei Ji, Lishan Xu, Jianing An +3Efficient Language Model TrainingMemory-Efficient Fine-Tuning

    SKLCCSE, School of Artificial Intelligence, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

  4. Local proxies for memory-efficient training

    Oct 16, 2026Wu S. Zihan, Brian A. Yuan, Guillaume BellecLocal LearningMemory-Efficient Training

    Machine Learning Research TU Wien Vienna, Austria

  5. LCT: Losslessly Compressed Tensors with GPU friendly execution for Training

    Oct 15, 2026Max Zhu, Bohang Sun, Pietro Lio +1Memory CompressionMemory-Efficient Training

    University of Cambridge · University of Oxford

  6. StructZO: Structured Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning

    Oct 12, 2026Xuanlin Yang, Jia Li, Zhouchen LinZeroth-Order OptimizationMemory-Efficient Fine-Tuning

    Peking University · Beijing Normal University

  7. Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces

    Oct 10, 2026Chuang Tong, Ling ChenLLM Fine-TuningZeroth-Order Optimization

    State Key Laboratory of Blockchain and Data Security, Zhejiang University · College of Computer Science and Technology, Zhejiang University

  8. Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

    Oct 9, 2026Xi Zhang, Boyu Diao, Zesen Chen +3Continual Learning for VLMsMultimodal Instruction Tuning

    Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · ShanghaiTech University

  9. LadderEdit: Edit-Level Residual Compression for Memory-Efficient Lifelong Editing of LLMs

    Oct 8, 2026Xiaobing Yu, Peijie Qiu, Jin Yang +5Language Model EditingMemory-Efficient Fine-Tuning

    Yale University · Washington University in St. Louis · Icahn School of Medicine at Mount Sinai +1

  10. MemFLoRA: Memory-Floor LoRA for CNN Adaptation at the Edge

    Oct 6, 2026Mehmet Emre Akbulut, Johannes Geier, Ulf SchlichtmannOn-Device TrainingMemory-Efficient Fine-Tuning

    Technical University of Munich Munich, Germany

  11. LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches

    Oct 5, 2026Shaokun Zhang, Yifan Zhang, Jian Hu +5Memory-Efficient Fine-TuningRL for Language Model Reasoning

    NVIDIA

  12. RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

    Oct 5, 2026Muhammad Azeem Lodhi, Chao Zhou, Rebekka BurkholzMemory-Efficient Fine-TuningLLM Fine-Tuning

    Saarland University, Saarbrücken, Germany · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

  13. DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategies

    Oct 4, 2026Zhishen Sun, Hongzhan Wang, Sizhe Dang +2LLM Fine-TuningEvolution Strategies

    Xi’an Jiaotong University · SGIT AI Lab

  14. TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning

    Oct 1, 2026Jichao Jiang, Cristian McGee, El Houcine Bergou +2Memory-Efficient Fine-TuningLLM Fine-Tuning

    University of Central Florida · Mohammed VI Polytechnic University

  15. FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

    Oct 1, 2026Junkang LiuFederated Learning AggregationMemory-Efficient Fine-Tuning

    Tianjin University

  16. Probe-Space Preconditioning for Fast and Stable Zero-Order Training

    Sep 29, 2026Francois Chaubard, Mykel J. Kochenderfer, Chris RéZeroth-Order OptimizationMemory-Efficient Fine-Tuning

    Stanford University, Stanford, CA, USA.

  17. AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning

    Sep 28, 2026Yue Xie, Zhi Zheng, Yunpeng Ba +5Subspace OptimizationMemory-Efficient Fine-Tuning

    Southern University of Science and Technology · National University of Singapore · Huawei Technologies Ltd. +1

  18. TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge

    Sep 27, 2026Houssem Sifaou, Prabodh Katti, Bipin Rajendran +1Memory-Efficient Fine-TuningZeroth-Order Optimization

    Institute for Intelligent Networked Systems, Northeastern University London, UK

  19. MpSub: A Momentum pp-Dimensional Subspace Trust-Region Method for Derivative-Free Fine-Tuning of Large Language Models

    Sep 7, 2026Yuyang Wang, Haoyu Yao, Pengcheng XieMemory-Efficient Fine-TuningZeroth-Order Optimization

    Xi’an Jiaotong University, Xi’an, China · Lawrence Berkeley National Laboratory, Berkeley, CA, USA

  20. Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation

    Sep 1, 2026Wentao Ye, Zhanming Shen, Zhiqing Xiao +3Memory-Efficient Fine-TuningLLM Fine-Tuning

    Zhejiang University · Hunyuan, Tencent

  21. SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces

    Aug 16, 2026Ziming Yu, Shuyao Xiao, Xingyu Zhao +6Memory-Efficient Fine-TuningZeroth-Order Optimization

    Beijing Normal University · Singapore Management University · Xiaomi Inc. +2

  22. SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuning

    Aug 15, 2026Mohammad Aref Jafari-Raddani, Morteza Mohajjel KafshdoozParameter SharingParameter-Efficient Fine-Tuning

    Department of Electrical and Computer Engineering, Qom University of Technology, Qom, Iran.

  23. TERRA: A Hierarchical Parallel Training and Memory Orchestration Framework for High-Resolution AI-based Earth Modeling

    Aug 15, 2026Ruohan Wu, Ziqi Zhu, Yang Zhao +6Distributed TrainingMemory-Efficient Fine-Tuning

    School of Computer Science and Technology, University of Science and Technology of China, Hefei 230026, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei 230026, China · Department of Ocean Big Data and Artificial Intelligence, Laoshan Laboratory, Qingdao, China +1

  24. Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptation

    Aug 12, 2026Junming Zhang, Shuyu Yin, Peilin Liu +2On-Device TrainingTest-Time Adaptation

    Shanghai Jiao Tong University, China

  25. ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling

    Aug 8, 2026Wentao Dai, Xuanran Li, Yuxiang Zhang +2On-Device TrainingMemory-Efficient Fine-Tuning

    Department of Computer Science and Engineering, Southern University of Science and Technology, China · Department of Mathematics, Southern University of Science and Technology, China · School of Computing, Montclair State University, New Jersey, USA

  26. Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs

    Aug 5, 2026Kuanysh Akhmetzhanov, Jurn-Gyu ParkFine-TuningOn-Device Language Model Inference

    Department of Computer Science, Nazarbayev University, Astana 010000, Kazakhstan

  27. Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving

    Aug 4, 2026Xiang Li, Pengcheng Wang, Huazheng Wang +1LLM Inference EfficiencyLLM Serving

    1Purdue University · 2Oregon State University

  28. Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models

    Aug 4, 2026Lele Zheng, Weifeng Kong, Xinyi Zhang +3Memory-Efficient Fine-TuningPrivacy-Preserving Language Models

    School of Computer Science and Technology, Xidian University

  29. How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model

    Jul 28, 2026Mahendra Singh Rathor, Anagheem AzzamMemory-Efficient Fine-TuningSmall Language Models

    Independent Researcher · Jaipur, India · Hamburg, Germany

  30. \k{appa}-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating

    Jul 24, 2026Jianghui Wang, Silong Yong, Francesco Orabona +3Fine-TuningMemory-Efficient Fine-Tuning

    King Abdullah University of Science and Technology · Carnegie Mellon University