Sparse Fine-Tuning

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 14

All topics

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning
Oct 5, 2026Muhammad Azeem Lodhi, Chao Zhou, Rebekka BurkholzMemory-Efficient Fine-Tuning LLM Fine-Tuning
Saarland University, Saarbrücken, Germany · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge
Sep 27, 2026Houssem Sifaou, Prabodh Katti, Bipin Rajendran +1Memory-Efficient Fine-Tuning Zeroth-Order Optimization
Institute for Intelligent Networked Systems, Northeastern University London, UK
From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
Sep 22, 2026Zhentao Tan, Chang Liu, Yao Liu +2Mixture-of-Experts Language Models Sparse Fine-Tuning
Alibaba Group · Southeast University
CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework
Sep 15, 2026Yunxiang Fu, Meng Lou, Zicheng Liao +1Class-Incremental Learning Continual Learning
School of Computing and Data Science, The University of Hong Kong · Hong Kong Generative AI Research and Development Center
Learning how to Forget: Fine-tuning for Long-Context Sparse Attention
Aug 20, 2026Matthias Seeger, Zeyu Zhang, Vihang Patil +2Fine-Tuning KV Caching
Amazon Web Services · University of Amsterdam · Amazon +1
Personalized Federated Sparse Adaptation of Time-Series Foundation Models
Aug 5, 2026Priyanka Nihalchandani, Naman Srivastava, Varun Ojha +1Mixture of Experts Time Series Forecasting
Robert Bosch Centre for Cyber-Physical Systems, Indian Institute of Science, Bengaluru, India · School of Computing, Newcastle University, Newcastle upon Tyne, United Kingdom
Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
Jul 10, 2026Ivan Ilin, Philip Zmushko, Peter RichtárikMemory-Efficient Fine-Tuning Low-Rank Adaptation
KAUST
Robust and Interpretable Adaptation of Equivariant Materials Foundation Models via Sparsity-promoting Fine-tuning
Jun 17, 2026Youngwoo Cho, Seunghoon Yi, Wooil Yang +6Machine Learning Interatomic Potentials Equivariant Neural Networks
KAIST · Seoul National Univ. · KIAS +2
Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation
Jun 11, 2026Guo Yu, Wenlin Liu, Yulan Hu +3Sparse Fine-Tuning On-Policy Distillation
School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University · Amap, Alibaba Group
Not How Many, But Which: Parameter Placement in Low-Rank Adaptation
May 12, 2026Arijit Sehanobish, Charles LoveringAdapter Tuning Low-Rank Adaptation
Kensho Technologies
S2FT: Parameter-Efficient Fine-Tuning in Sparse Spectrum Domain
May 9, 2026Baoquan Zhang, Zhehao Yu, Lisai Zhang +5Fine-Tuning Sparse Fine-Tuning
Harbin Institute of Technology, Shenzhen · Bilibili. Inc · Shenzhen University +1
Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
May 4, 2026Prakhar Gupta, Garv Shah, Satyam Goyal +1Memory-Augmented Language Models LLM Fine-Tuning
University of Michigan
JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models
Apr 17, 2026Alexandra Dragomir, Ioana Pintilie, Antonio Barbalau +6Continual Learning for LLMs Low-Rank Adaptation
Bitdefender · ETH Zurich · University of Bucharest
LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning
Apr 16, 2026Bowen Ping, Zijun Chen, Tingfeng Hui +4Reinforcement Learning RL for Language Model Reasoning
1Peking University · 2Shanghai Jiao Tong University · 3Beijing University of Posts and Telecommunications

Oct 5, 2026, cs.LGJ/K move · Enter open · S save

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

Muhammad Azeem Lodhi, Chao Zhou, Rebekka Burkholz

Saarland University, Saarbrücken, Germany · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Memory-Efficient Fine-Tuning LLM Fine-Tuning Sparse Fine-Tuning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting foundation models by focusing training on a small parameter subset. Complementary to this idea, we introduce RoSA (Rotational Sparse Adaptation), which narrows adaptation to a subset of layers at a time. RoSA freezes lower layers close to the input throughout training and rotates a trainable block over later layers, progressively increasing the number of frozen layers close to the input. This design reduces optimizer-state memory, shortens backpropagation, and even forward propagation if activations at the last frozen layer are cached. Because RoSA is orthogonal to the choice of trainable parameterization, it can be combined with PEFT methods or sparse optimizers within each active block. Experiments across multiple LLM architectures and tasks show that RoSA reduces peak memory while maintaining strong fine-tuning performance.

Momentum

Related topics

Latest papers 14