Overview

Sparse fine-tuning studies how to adapt pretrained models by changing only a subset of their weights or adding small, selectively updated components. Methods choose updates using gradient, activation, Fisher-information, or routing signals, or use sparse masks, memory modules, low-rank adapters, and zeroth-order optimization; applications include language models, vision, scientific prediction, and continual learning. Evaluations span language and reasoning benchmarks, long-context tasks, continual-learning settings, and domain-specific prediction, with many approaches aiming to approach dense fine-tuning using fewer trainable parameters or less memory. A central tension is how to select parameters reliably across training objectives and tasks while preserving general capabilities, since sparse updates can reduce forgetting but may also miss useful learning signals or require careful optimization.

Latest papers

CardsList

33 papers8 papers in the last four weeks, up 33% on the four weeks before. 0.1% of all new papers.

  1. SMELL: Sparsity-aware Memory Efficient Federated Fine-tuning for Long-context LLMs

    Nov 14, 2026Yin Xu, Yongbo Yang, He Sun +2Federated Fine-TuningMemory-Efficient Fine-Tuning

  2. The Surprising Effectiveness of Deleting Weights in LLM Post-Training

    Nov 7, 2026Jack Lu, Zhenbang Yang, Mike Lasby +3LLM Post-TrainingParameter-Efficient Fine-Tuning

  3. RASP: Read-All, Sparse-Plasticity Continual Learning in a Stationary-Size LoRA

    Nov 2, 2026Wen-Hsuan Liu, Amara Okonkwo, Tom Saliencro +2Continual LoRAStorage-Efficient Continual Learning

  4. Prune Gradually, Adapt Sparsely: Efficient Pruning of Large Language Models at Extreme Sparsity

    Oct 30, 2026Jean Lucien Randrianantenaina, Hung Nguyen, Nasib Ullah +4LLM PruningNeural Network Pruning

  5. Breaking Information Islands in Sparse Tuning via Small-World Connectivity

    Oct 30, 2026Xianchao Guan, Zijun Xiong, Yifeng Wang +5Sparse Fine-TuningFine-Tuning Language Models

  6. Sparse Fine-Tuning for Parameter-Efficient Adversarial Training

    Oct 29, 2026Zhaoxin Wang, Zihang Ding, Handing WangAdversarial TrainingAdversarial Robustness

  7. Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery

    Oct 23, 2026Marek Masiak, Andrzej Szablewski, Nicola Cancedda +1Sparse Fine-TuningLLM Interpretability

  8. Ranking and Partitioning of Attention Heads for Sparse and Continual Reinforcement Learning Fine-tuning

    Oct 15, 2026Ruizhe Shi, Zaiwen Yang, Qiubo Wang +2Continual Learning for LLMsSparse Fine-Tuning

  9. RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

    Oct 5, 2026Muhammad Azeem Lodhi, Chao Zhou, Rebekka BurkholzMemory-Efficient Fine-TuningLLM Fine-Tuning

  10. Understanding LLM Parameter Update Sparsity through the Lens of Fisher

    Sep 28, 2026Yufan Zhang, Sagnik Mukherjee, Hao PengNeural Network Training DynamicsSparse Fine-Tuning

  11. TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge

    Sep 27, 2026Houssem Sifaou, Prabodh Katti, Bipin Rajendran +1Memory-Efficient Fine-TuningZeroth-Order Optimization

  12. Awaken, Then Scale: Tiny Adaptation for MRI Reconstruction

    Sep 25, 2026Mohammed Wattad, Tamir Shor, Alexander M. BronsteinDomain AdaptationParameter-Efficient Fine-Tuning

  13. CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework

    Sep 15, 2026Yunxiang Fu, Meng Lou, Zicheng Liao +1Class-Incremental LearningContinual Learning

  14. Surgical Alignment in Knowledge Graph Training for Clinical Diagnosis with Large Language Models

    Aug 27, 2026Saksham Khatwani, He Cheng, Majid Afshar +2LLM Reasoning with GraphsLLM Alignment

  15. Fine-Tuning of Transformer models with Frames

    Aug 26, 2026Harshavardhan Adepu, Li Zhang, Sanjiv Kumar +1Parameter-Efficient Fine-TuningLLM Fine-Tuning

  16. Thermo-FL: Thermal-Aware Robust Federated Fine-Tuning of Large Language Models for Edge AI

    Aug 21, 2026Shiva Shrestha, Kazi Shaharair Sharif, Zongxing Xie +3Federated Fine-TuningFederated Learning

  17. Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

    Aug 20, 2026Matthias Seeger, Zeyu Zhang, Vihang Patil +2Fine-TuningKV Caching

  18. Personalized Federated Sparse Adaptation of Time-Series Foundation Models

    Aug 5, 2026Priyanka Nihalchandani, Naman Srivastava, Varun Ojha +1Mixture of ExpertsTime Series Forecasting

  19. Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation

    Jun 11, 2026Guo Yu, Wenlin Liu, Yulan Hu +3Sparse Fine-TuningOn-Policy Distillation

  20. Not How Many, But Which: Parameter Placement in Low-Rank Adaptation

    May 12, 2026Arijit Sehanobish, Charles LoveringAdapter TuningLow-Rank Adaptation

  21. S2FT: Parameter-Efficient Fine-Tuning in Sparse Spectrum Domain

    May 9, 2026Baoquan Zhang, Zhehao Yu, Lisai Zhang +5Fine-TuningSparse Fine-Tuning

  22. JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models

    Apr 17, 2026Alexandra Dragomir, Ioana Pintilie, Antonio Barbalau +6Parameter-Efficient Fine-TuningLow-Rank Adaptation

  23. LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning

    Apr 16, 2026Bowen Ping, Zijun Chen, Tingfeng Hui +4Reinforcement LearningRL for Language Model Reasoning

  24. Improving Sparse Memory Finetuning

    Apr 6, 2026Satyam Goyal, Anirudh Kanchi, Garv Shah +1Continual Learning for LLMsSparse Fine-Tuning