Overview

Sparse fine-tuning studies how to adapt pretrained models by changing only a subset of their weights or adding small, selectively updated components. Methods choose updates using gradient, activation, Fisher-information, or routing signals, or use sparse masks, memory modules, low-rank adapters, and zeroth-order optimization; applications include language models, vision, scientific prediction, and continual learning. Evaluations span language and reasoning benchmarks, long-context tasks, continual-learning settings, and domain-specific prediction, with many approaches aiming to approach dense fine-tuning using fewer trainable parameters or less memory. A central tension is how to select parameters reliably across training objectives and tasks while preserving general capabilities, since sparse updates can reduce forgetting but may also miss useful learning signals or require careful optimization.

Latest papers

CardsList

33 papers8 papers in the last four weeks, up 33% on the four weeks before. 0.1% of all new papers.

  1. CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning

    Mar 23, 2026Shuo Wang, Ziyu Chen, Ming TangZeroth-Order OptimizationMemory-Efficient Fine-Tuning

  2. Pruned Adaptation Modules: A Simple yet Strong Baseline for Continual Foundation Models

    Mar 22, 2026Elif Ceren Gok Yildirim, Murat Onur Yildirim, Joaquin VanschorenStorage-Efficient Continual LearningClass-Incremental Learning

  3. GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selection

    Mar 10, 2026Kai Yao, Zhenghan Song, Kaixin Wu +5Sparse Fine-TuningSelective Fine-Tuning