Sparse Fine-Tuning

Ask about this topic

Overview

Sparse fine-tuning studies how to adapt pretrained models by changing only a subset of their weights or adding small, selectively updated components. Methods choose updates using gradient, activation, Fisher-information, or routing signals, or use sparse masks, memory modules, low-rank adapters, and zeroth-order optimization; applications include language models, vision, scientific prediction, and continual learning. Evaluations span language and reasoning benchmarks, long-context tasks, continual-learning settings, and domain-specific prediction, with many approaches aiming to approach dense fine-tuning using fewer trainable parameters or less memory. A central tension is how to select parameters reliably across training objectives and tasks while preserving general capabilities, since sparse updates can reduce forgetting but may also miss useful learning signals or require careful optimization.

Latest papers

CardsList

28 papers5 papers in the last four weeks, up 25% on the four weeks before. 0.0% of all new papers.

  1. Sparse Fine-Tuning for Parameter-Efficient Adversarial Training

    Oct 29, 2026Zhaoxin Wang, Zihang Ding, Handing WangAdversarial TrainingAdversarial Robustness

    School of Artificial Intelligence, Xidian University, Xi’an, China

  2. Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery

    Oct 23, 2026Marek Masiak, Andrzej Szablewski, Nicola Cancedda +1Sparse Fine-TuningLLM Interpretability

    Department of Computer Science, University of Cambridge · Neuralk-AI

  3. LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning

    Oct 22, 2026Keqi Deng, Ruchao Fan, Jinyu LiSparse Fine-TuningLLM Fine-Tuning

    Microsoft, USA

  4. Ranking and Partitioning of Attention Heads for Sparse and Continual Reinforcement Learning Fine-tuning

    Oct 15, 2026Ruizhe Shi, Zaiwen Yang, Qiubo Wang +2Continual Learning for LLMsSparse Fine-Tuning

    University of Washington · Tsinghua University · University of Hong Kong +1

  5. RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

    Oct 5, 2026Muhammad Azeem Lodhi, Chao Zhou, Rebekka BurkholzMemory-Efficient Fine-TuningLLM Fine-Tuning

    Saarland University, Saarbrücken, Germany · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

  6. Understanding LLM Parameter Update Sparsity through the Lens of Fisher

    Sep 28, 2026Yufan Zhang, Sagnik Mukherjee, Hao PengNeural Network Training DynamicsSparse Fine-Tuning

    University of Illinois Urbana-Champaign

  7. TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge

    Sep 27, 2026Houssem Sifaou, Prabodh Katti, Bipin Rajendran +1Memory-Efficient Fine-TuningZeroth-Order Optimization

    Institute for Intelligent Networked Systems, Northeastern University London, UK

  8. Awaken, Then Scale: Tiny Adaptation for MRI Reconstruction

    Sep 25, 2026Mohammed Wattad, Tamir Shor, Alexander M. BronsteinDomain AdaptationParameter-Efficient Fine-Tuning

    Faculty of Computer Science, Technion-Israel Institute of Technology, Haifa, Israel

  9. From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

    Sep 22, 2026Zhentao Tan, Chang Liu, Yao Liu +2Mixture-of-Experts Language ModelsSparse Fine-Tuning

    Alibaba Group · Southeast University

  10. CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework

    Sep 15, 2026Yunxiang Fu, Meng Lou, Zicheng Liao +1Class-Incremental LearningContinual Learning

    School of Computing and Data Science, The University of Hong Kong · Hong Kong Generative AI Research and Development Center

  11. Surgical Alignment in Knowledge Graph Training for Clinical Diagnosis with Large Language Models

    Aug 27, 2026Saksham Khatwani, He Cheng, Majid Afshar +2LLM Reasoning with GraphsLLM Alignment

    University of Colorado Anschutz Medical Campus · University of Wisconsin Madison · Loyola University

  12. Fine-Tuning of Transformer models with Frames

    Aug 26, 2026Harshavardhan Adepu, Li Zhang, Sanjiv Kumar +1Parameter-Efficient Fine-TuningLLM Fine-Tuning

    University of Wisconsin-Madison · Google DeepMind

  13. Thermo-FL: Thermal-Aware Robust Federated Fine-Tuning of Large Language Models for Edge AI

    Aug 21, 2026Shiva Shrestha, Kazi Shaharair Sharif, Zongxing Xie +3Federated Fine-TuningFederated Learning

    Department of Computer Science, Kennesaw State University Marietta, GA 30060, USA · School of Data Science and Analytics, Kennesaw State University Marietta, GA 30060, USA · Department of Information Technology, Kennesaw State University Marietta, GA 30060, USA

  14. Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

    Aug 20, 2026Matthias Seeger, Zeyu Zhang, Vihang Patil +2Fine-TuningKV Caching

    Amazon Web Services · University of Amsterdam · Amazon +1

  15. Personalized Federated Sparse Adaptation of Time-Series Foundation Models

    Aug 5, 2026Priyanka Nihalchandani, Naman Srivastava, Varun Ojha +1Mixture of ExpertsTime Series Forecasting

    Robert Bosch Centre for Cyber-Physical Systems, Indian Institute of Science, Bengaluru, India · School of Computing, Newcastle University, Newcastle upon Tyne, United Kingdom

  16. Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning

    Jul 10, 2026Ivan Ilin, Philip Zmushko, Peter RichtárikMemory-Efficient Fine-TuningLow-Rank Adaptation

    KAUST

  17. Robust and Interpretable Adaptation of Equivariant Materials Foundation Models via Sparsity-promoting Fine-tuning

    Jun 17, 2026Youngwoo Cho, Seunghoon Yi, Wooil Yang +6Machine Learning Interatomic PotentialsEquivariant Neural Networks

    KAIST · Seoul National Univ. · KIAS +2

  18. Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation

    Jun 11, 2026Guo Yu, Wenlin Liu, Yulan Hu +3Sparse Fine-TuningOn-Policy Distillation

    School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University · Amap, Alibaba Group

  19. Not How Many, But Which: Parameter Placement in Low-Rank Adaptation

    May 12, 2026Arijit Sehanobish, Charles LoveringAdapter TuningLow-Rank Adaptation

    Kensho Technologies

  20. S2FT: Parameter-Efficient Fine-Tuning in Sparse Spectrum Domain

    May 9, 2026Baoquan Zhang, Zhehao Yu, Lisai Zhang +5Fine-TuningSparse Fine-Tuning

    Harbin Institute of Technology, Shenzhen · Bilibili. Inc · Shenzhen University +1

  21. Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning

    May 4, 2026Prakhar Gupta, Garv Shah, Satyam Goyal +1Memory-Augmented Language ModelsLLM Fine-Tuning

    University of Michigan

  22. JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models

    Apr 17, 2026Alexandra Dragomir, Ioana Pintilie, Antonio Barbalau +6Continual Learning for LLMsLow-Rank Adaptation

    Bitdefender · ETH Zurich · University of Bucharest

  23. LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning

    Apr 16, 2026Bowen Ping, Zijun Chen, Tingfeng Hui +4Reinforcement LearningRL for Language Model Reasoning

    1Peking University · 2Shanghai Jiao Tong University · 3Beijing University of Posts and Telecommunications

  24. Improving Sparse Memory Finetuning

    Apr 6, 2026Satyam Goyal, Anirudh Kanchi, Garv Shah +1Continual Learning for LLMsSparse Fine-Tuning

    University of Michigan, Ann Arbor

  25. MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning

    Mar 25, 2026Andrea ManzoniMixture-of-Experts Fine-TuningFine-Tuning

  26. CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning

    Mar 23, 2026Shuo Wang, Ziyu Chen, Ming TangZeroth-Order OptimizationMemory-Efficient Fine-Tuning

    Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. · Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.

  27. Pruned Adaptation Modules: A Simple yet Strong Baseline for Continual Foundation Models

    Mar 22, 2026Elif Ceren Gok Yildirim, Murat Onur Yildirim, Joaquin VanschorenStorage-Efficient Continual LearningClass-Incremental Learning

    AMOR/e Lab, Eindhoven University of Technology

  28. GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selection

    Mar 10, 2026Kai Yao, Zhenghan Song, Kaixin Wu +5Sparse Fine-TuningSelective Fine-Tuning

    1Ant Group · 2Cornell University · University of Liverpool +2