Organizations: Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, China · Tsinghua University, China · Nanjing University of Science and Technology, China
Sparse tuning is widely used to adapt large language models due to its parameter efficiency. However, its effectiveness depends not only on the number of trainable parameters, but also on the feature-interaction topology of the sparse update's structural support. Limited or uneven connectivity in this support may isolate subsets of feature dimensions, forming information islands that restrict direct task-specific interactions through the trainable update. To address this issue, we propose Small-World Induced Fine-Tuning (SWIFT), a sparse tuning framework inspired by small-world connectivity. SWIFT combines block-diagonal sparse updates for local feature mixing with a fixed deterministic scattering permutation that routes the mixed outputs across feature groups without additional trainable parameters. We further show theoretically that certain random sparse-update models and block-diagonal updates can exhibit information islands, whereas a sufficiently scattering permutation guarantees positive block-level expansion when the trainable block-diagonal factor consists of equal-sized, structurally dense square blocks. Extensive experiments on commonsense reasoning, natural language generation, and image classification demonstrate that SWIFT outperforms the evaluated parameter-efficient fine-tuning (PEFT) baselines on average and can match or surpass full fine-tuning. Implementation is available at https://github.com/xianchaoguan/SWIFT
Figures & tables
Figure 1: Schematic feature-interaction topologies of trainable updates in full fine-tuning, SpaRTA, DiaBlo, and SWIFT. SWIFT combines local feature mixing with permutation-based routing across feature groups.
Figure 2: Overview of the SWIFT framework. SWIFT combines trainable block-diagonal updates for local feature mixing with a fixed deterministic scattering permutation that routes the mixed outputs across feature groups. The pre-trained weights remain frozen.
Method
Param. (%)
BoolQ
PIQA
SIQA
HellaS.
WinoG.
ARC-e
ARC-c
OBQA
Avg.
Full Fine-Tuning
100
73.90
86.20
79.10
93.10
85.80
88.10
78.20
84.00
83.55
Low-rank adaptation
LoRA
0.70
74.58
88.41
80.81
95.31
85.87
89.60
80.12
85.40
85.01
DoRA
0.71
76.09
88.30
80.60
95.23
86.67
90.40
80.29
88.20
85.72
PiSSA
0.70
69.70
87.20
79.50
90.30
81.00
82.90
75.50
77.80
80.49
HydraLoRA
0.53
72.20
85.00
79.80
89.60
87.60
79.40
74.00
80.40
81.00
Table 1: Performance comparison of LLaMA3-8B on eight commonsense reasoning datasets. Task scores are accuracies (%). The best and second-best scores in each column are indicated in bold and underlined , respectively, including ties.
Method
Param. (%)
GSM8K
MATH
HumanEval
MBPP
Avg.
Full Fine-Tuning
100
49.05
7.22
21.34
35.59
28.30
Low-rank adaptation
LoRA
4.75
42.30
5.50
18.29
35.34
25.36
DoRA
4.76
50.42
7.26
23.78
38.85
30.08
PiSSA
4.75
53.07
7.44
21.95
37.09
29.89
HydraLoRA
1.87
49.58
7.02
23.17
37.34
29.28
Table 2: Performance comparison of LLaMA2-7B on four natural language generation datasets. Scores are percentages. Bold and underlined entries indicate the best and second-best scores, respectively, including ties.
Method
Param. (%)
Cars
DTD
GTSRB
RESISC45
SVHN
Avg.
Full Fine-Tuning
100
60.33
73.88
98.30
93.65
96.78
84.59
Low-rank adaptation
LoRA
1.49
41.02
70.15
96.51
90.38
95.39
78.69
DoRA
1.49
40.75
71.91
97.71
90.19
95.46
79.20
PiSSA
1.49
40.41
69.62
95.84
90.58
95.84
78.46
GOAT
2.24
51.44
73.99
98.06
93.50
96.55
82.71
Table 3: Performance comparison of CLIP ViT-B/32 on five image classification datasets. Task scores are accuracies (%). Bold and underlined entries indicate the best and second-best scores, respectively.
Figure 3: Ablation study of SWIFT. (a) Update-density sensitivity on eight commonsense reasoning datasets with LLaMA3-8B. Densities of 0.20% , 0.39% , 0.78% , 1.56% , and 3.13% correspond to N=512,256,128,64,32 , respectively. (b) Comparison of permutation routing schemes on five image classification datasets with CLIP ViT-B/32. DC, AC, SS, RP, and US denote diagonal cyclic, anti-diagonal cyclic, single-row swap, random permutation, and uniform scattering, respectively.
Figure 4: End-to-end efficiency comparison on LLaMA3-8B at a matched 0.88% trainable-parameter budget on one V100-32GB GPU. (a) Total training time. (b) Total training FLOPs. (c) Peak training memory. Training durations follow the method-specific schedules described in the text. Results are averaged over three seeds; lower values are better.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Placement and seed analyses on LLaMA3-8B. (a) SWIFT and DiaBlo use identical target modules and matched parameter budgets within each placement; scores are averaged over eight commonsense reasoning datasets and three seeds. (b) SWIFT accuracy for seeds 6, 24, and 42, with the horizontal line showing their mean.
Figure 6: Training loss curves for LoRA, SpaRTA, and SWIFT when fine-tuning LLaMA3-8B on Commonsense-170K for commonsense reasoning.
Sparse fine-tuning adapts pretrained models by updating selected parameters, reducing training costs and limiting interference with existing capabilities.
Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt. We propose Super, a sparse parameter-efficient fine-tuning (PEFT) method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score [Sun et al., 2023] computed from a calibration pass. We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule. In single-seed Math17K arithmetic experiments on Llama-3.2-1B and Meta-Llama-3-8B, the best Super/Supra variants achieve the highest average accuracy among the tested schedule-selected adapter configurations. We also include a PaFi-style magnitude-only support as a closest training-free sparse baseline and find that low-score supports under both magnitude and Wanda-style orderings can be effective. These results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.
Large Language Models (LLMs) are typically static after training, yet real-world applications require continual adaptation to new knowledge without degrading existing capabilities. Standard approaches to updating models, like full finetuning or parameter-efficient methods (e.g., LoRA), face a fundamental trade-off: catastrophic forgetting. They modify shared dense representations, causing interference across tasks. Sparse Memory Finetuning (SMF) offers a promising alternative by localizing updates to a small subset of parameters in explicit memory layers. In this work, we present an open-source pipeline to retrofit existing pretrained models (Qwen-2.5-0.5B) with sparse memory modules, enabling effective continual learning on consumer hardware. We extend prior work by introducing a theoretically grounded slot-selection mechanism based on Kullback-Leibler (KL) divergence, which prioritizes memory updates for informationally "surprising" tokens relative to a background distribution. Our experiments demonstrate that our retrofitted models can acquire new factual knowledge with minimal forgetting of held-out capabilities, validating the sparse update hypothesis in a practical setting.
Full fine-tuning remains a strong way to adapt pretrained LLMs, but it updates all weights and can be expensive. LoRA reduces the number of trainable parameters, but it does not directly answer which pretrained components should be trained and which can be frozen during adaptation. We introduce TopoTuner, a topology-guided fine-tuning framework for selective freezing of attention projection matrices. \method treats each projection matrix as a row cloud and uses Wasserstein distances between persistence diagrams to measure how its topology changes during fine-tuning. TopoTuner learns a reusable freezing profile from a source dataset and transfers it to efficiently fine-tune models on out-of-domain datasets, evaluating whether task-specific topological drift generalizes across question answering and sentiment analysis tasks. Across LLaMA-3.1-8B, Mistral-7B-v0.3, and Qwen3-8B-Base, TopoTuner is competitive with full fine-tuning while training only 1-2% of the model parameters, and outperforms LoRA in 7 out of 9 model-dataset settings, which can change up to 39.57% of the projection parameters. Along with minimized updates, TopoTuner reduces training time by 20.4% relative to full fine-tuning and 5.5% relative to LoRA on average. TopoTuner opens a new direction for reusable freezing profiles, where fine-tuning behavior learned on one dataset can be shared across multiple tasks.
Department of Computer Science University of Central Florida Orlando, USA · Department of Computer Science University of Manitoba Canada · Department of Finance University of Central Florida Orlando, USA