Organizations: School of Computer Science and Engineering, Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China · Kunming University of Science and Technology, Kunming, China
Merging multiple task-specific fine-tuned checkpoints into a single multi-task model provides an attractive alternative to joint multi-task training. To mitigate cross-task interference, a common strategy is to sparsify task vectors before aggregation. However, many widely used methods rely on a uniform global sparsity ratio, overlooking differences in pruning sensitivity across tasks and parameter tensors. Consequently, such uniform sparsification may remove useful parameter updates from sensitive tensors while retaining redundant updates in others. To address this limitation, we propose TADrop, a plug-and-play sparsifier that independently determines tensor-specific keep ratios from their magnitude distributions using a scale-invariant quantile ratio. TADrop operates in a data-free manner by default, while offering optional lightweight module-level calibration when a small validation set is available. Experiments across vision, language, and multimodal benchmarks, including LLM and LoRA merging, demonstrate consistent improvements across multiple host merging methods, including a 4.99-point gain over EMR-Merging on RoBERTa GLUE. In many-task ViT-B/16 merging, the improvement increases with the number of tasks, reaching 2.26 points when merging 30 tasks. Further analyses show that TADrop preserves more task-associated neurons under matched overall sparsity. Together, these findings highlight the value of distribution-guided sparsification at the tensor level for improving model merging.
Figures & tables
Figure 1: Left : Average accuracy over eight ViT-B/32 tasks after EMR-Merging with different sparsification strategies and sparsity ratios. Right : Average accuracy of different merging methods with and without TADrop.
Figure 2: Task vectors are heterogeneous across sparsity levels, tasks, and layers/modules, motivating tensor-wise adaptive sparsification. (a) illustrates how sparsification strategy and sparsity ratio affect merged-model performance; (b) shows that different tasks exhibit different sensitivity to task vector sparsification (Vertical markers denote 5-point accuracy drop thresholds); (c) visualizes the heterogeneity of task vector magnitude distributions across layers/modules and tasks.
Figure 3: Overview of TADrop. Given task vectors {τk}k=1K , TADrop independently sparsifies each parameter tensor τkn before host merging. It (1) estimates a tensor-wise keep ratio rkn from the magnitude distribution via a quantile ratio, (2) optionally calibrates the ratio by functional module group using a small calibration set, with αg=1 in the data-free setting, (3) retains the largest-magnitude r~kn fraction, and (4) rescales the masked tensor to approximately preserve ∥τkn∥2 . The resulting sparsified task vectors {τ^k}k=1K are passed to the original host merger F without modifying its merging rule.
Methods
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg.
Individual
79.2 / 84.9
77.7 / 92.4
96.1 / 97.4
99.7 / 99.7
97.5 / 98.1
98.7 / 99.2
99.7 / 99.8
79.4 / 84.1
91.0 / 94.4
Traditional MTL
73.9 / 80.8
74.4 / 90.6
93.9 / 96.3
98.2 / 96.3
95.8 / 97.6
98.9 / 99.1
99.5 / 99.6
77.9 / 84.4
88.9 / 93.5
Weight Averaging
65.3 / 72.1
63.4 / 81.6
71.4 / 82.6
71.7 / 91.9
64.2 / 78.2
52.8 / 70.7
87.5 / 97.1
50.1 / 62.8
65.8 / 79.6
Fisher Merging
68.6 / 69.2
69.2 / 88.6
70.7 / 87.5
66.4 / 93.5
72.9 / 80.6
51.1 / 74.8
87.9 / 93.3
59.9 / 70.0
68.3 / 82.2
RegMean
65.3 / 73.3
63.5 / 81.8
75.6 / 86.1
78.6 / 97.0
78.1 / 88.0
67.4 / 84.2
93.7 / 98.5
52.0 / 60.8
71.8 / 83.7
PCB-Merging
66.7 / 76.8
65.5 / 86.2
78.5 / 89.4
79.3 / 96.5
86.4 / 88.3
77.1 / 91
98.2 / 98.6
59.1 / 73.6
76.3 / 87.5
Table 1: Multi-task performance when merging ViT models on eight tasks (a / b: a = ViT-B/32, b = ViT-L/14). Bold numbers indicate the best score within each backbone block, and underlined numbers indicate the second-best score.
Methods
COCO-Re
COCO-Captioning
ImageNet
NLVR2
VQAv2
Avg.
Acc
BLEU
CIDEr ×100
METEOR
ROUGE
Acc
Acc
Acc
Score
Individual
84.6
39.4
134.0
31.1
60.1
85.4
77.7
84.4
74.6
Weight Averaging
18.9
3.1
0.1
11.5
15.9
67.7
28.0
62.9
26.0
Task Arithmetic ( Ilharco et al., 2023 )
31.8
3.3
0.0
11.8
17.6
70.8
38.1
69.3
30.3
TIES-Merging ( Yadav et al., 2023 )
39.3
2.9
0.1
10.8
16.7
69.8
32.1
67.2
29.9
EMR-Merging ( Huang et al., 2024 )
79.5
28.9
106.0
27.2
53.4
77.4
74.8
72.1
64.9
Table 2: Performance of merging multimodal BEiT3 models on five vision-language tasks.
Method
Avg.
Individual
85.56
Weight Averaging
51.34
Fisher Merging ( Matena and Raffel, 2022 )
45.57
RegMean ( Jin et al., 2023 )
70.01
Task Arithmetic ( Ilharco et al., 2023 )
66.65
+ ImPart ( Yang et al., 2025a )
65.72
Table 3: Average multi-task performance when merging RoBERTa-base models fine-tuned on eight GLUE text classification tasks.
Model
GSM8K
ARC-C
MMLU-CS
Avg.
Qwen2.5-32B-Ins
77.94
74.49
83.00
78.48
SAND-Math
89.39
73.72
83.00
82.04
OpenThinker
60.65
72.27
86.00
72.97
SWE-Swiss
78.70
71.25
84.00
77.98
Task Arithmetic
82.11
73.29
85.00
80.13
+ TADrop(w/o)
84.69
73.98
83.00
80.55
Table 4: Performance of merging three models fine-tuned from Qwen2.5-32B-Instruct on three benchmarks.
Method
Avg.
Task Arithmetic ( Ilharco et al., 2023 )
61.35
+ TADrop(w/o)
63.32
TIES-Merging ( Yadav et al., 2023 )
63.62
+ TADrop(w/o)
64.19
KnOTS-TIES ( Stoica et al., 2025 )
68.30
+ TADrop(w/o)
69.16
Table 5: Average performance of LoRA model merging on the eight vision classification tasks using CLIP ViT-B/32 with rank-16 adapters.
Figure 4: Scalability of TADrop in 30-task model merging on ViT-B/16. Left : Absolute average accuracy as the number of merged tasks grows from 8 to 30. Right : Performance gain over the EMR-Merging baseline.
Figure 5: Average performance versus task-vector retention on ViT-L/14 and RoBERTa. Colors distinguish host merging methods, while markers indicate different sparsification strategies.
Backbone
Method
Baseline
w/ Top- k
w/ TADrop
ViT-L/14
TA
100.00 / 84.50
31.54 / 84.34
31.54 / 86.70
TIES
20.00 / 84.40
30.00 / 84.64
30.00 / 86.80
RoBERTa
TA
100.00 / 66.65
20.65 / 68.45
20.65 / 69.78
TIES
19.86 / 64.04
18.67 / 65.58
18.67 / 69.64
Table 6: Comparison of uniform Top- k and TADrop under matched retention budgets. Each entry reports retention (%) / average performance. Baseline denotes the original host method.
Figure 6: Retention of task-labeled neurons under matched overall sparsity. TADrop consistently outperforms uniform-ratio magnitude-based sparsification.
Figure 7: Ablation on quantile choices (Qa,Qb) on ViT-B (8 tasks). Bars show the change in average accuracy relative to each host method’s baseline.
Method
EMR-Merging
w/o R
(1−p) R
ℓ2 R
Avg.
88.7
86.8
71.1
90.8
Table 7: Ablation results of rescaling strategies on the 8 ViT tasks. R denotes the rescaling operation.
Method
Sparsification Signal
Sparsity Allocation / Selection
Auxiliary Data
TADrop (Ours)
Tensor magnitude quantiles
Tensor-wise adaptive keep ratios with magnitude-based selection
Table 8: Conceptual comparison of representative task-vector sparsification and selection methods in terms of their decision signals, parameter retention strategies, and auxiliary data requirements.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Chat
Math
Code
Method
MMLU
HellaSwag
TruthfulQA
GSM8K
HumanEval
MBPP
Individual
0.4793
0.75
0.31
0.45
0.23
0.2
Ties-Merging ( Yadav et al., 2023 )
0.4707
0.74
0.27
0.23
0.29
0.29
+ TADrop(w/o)
0.4768
0.75
0.27
0.38
0.29
0.27
+ TADrop
0.4858
0.76
0.28
0.41
0.29
0.26
EMR-Merging ( Huang et al., 2024 )
0.4753
0.72
0.3
0.47
0.27
0.24
Appendix
Table 10: Results on chat, math, and code benchmarks for Llama-2-7B. Bold numbers indicate the best score within each backbone block.
Method
CoLA
SST-2
MRPC
STS-B
QQP
MNLI
QNLI
RTE
Avg.
Individual
60.18
94.04
89.22
90.63
91.41
87.20
92.71
79.06
85.56
Weight Averaging
13.96
64.11
69.36
31.84
75.36
42.19
58.70
55.23
51.34
Fisher Merging ( Matena and Raffel, 2022 )
6.11
53.56
65.21
22.80
58.51
34.80
67.64
55.96
45.57
RegMean ( Jin et al., 2023 )
36.67
90.60
75.74
62.68
83.55
70.02
82.35
58.48
70.01
Task Arithmetic ( Ilharco et al., 2023 )
18.78
85.89
79.90
74.03
83.78
59.08
69.67
62.09
66.65
+ ImPart ( Yang et al., 2025a )
21.29
85.67
79.17
72.40
82.24
54.43
69.21
61.37
65.72
Appendix
Table 11: per-task performance when merging RoBERTa-base models fine-tuned on eight GLUE text classification tasks. Bold numbers indicate the best score within each backbone block, and underlined numbers indicate the second-best score.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg.
Task Arithmetic
92.66
81.17
69.27
36.55
46.15
42.23
51.23
71.53
61.35
+ TADrop(w/o)
94.19
81.51
71.00
47.47
44.10
43.27
52.05
72.99
63.32
TIES-Merging
94.60
82.30
69.47
49.91
45.20
36.52
58.33
72.63
63.62
+ TADrop(w/o)
94.52
82.35
69.76
51.33
45.09
38.47
58.42
73.63
64.19
KnOTS-TIES
92.93
82.87
71.07
49.87
54.55
49.24
71.88
73.99
68.30
+ TADrop(w/o)
93.53
81.36
71.43
51.74
56.44
50.07
74.43
74.27
69.16
Appendix
Table 12: Detailed LoRA merging results on eight vision classification tasks using CLIP ViT-B/32 with rank-16 adapters. Bold numbers indicate the best performance within each merging method group.
Model merging integrates task-specific fine-tuned models into a single multi-task model, but often suffers from parameter interference caused by conflicting task-vector updates. Existing methods typically mitigate conflicts by pruning task vectors based on weight magnitude or random heuristics, treating Transformers as unstructured ``bags of parameters'' and overlooking their inherent modularity. In this paper, we propose Contribution-Aware Structured Sparsity (CASS), a unified framework that reduces parameter interference by identifying and preserving task-specific components. At the core of CASS is a contribution-aware structured mask that identifies task-relevant attention heads and FFN neurons. We instantiate this mask in two settings: CASS-Merging, the primary post-hoc setting where masks serve as a plug-and-play denoising filter for existing merging operators, and CASS-Tuning, an extension for scenarios with fine-tuning access where masks constrain gradients to reduce structural overlap between task vectors. Our analysis shows that task-relevant components are sparse and partially disjoint, supporting structured component-level filtering as an effective way to reduce merging interference. Extensive experiments across vision (ViT, 20 tasks) and language (RoBERTa, 8 tasks; Qwen2.5, 4 tasks) benchmarks demonstrate that CASS improves a range of representative merging baselines.
Yan Li, Guiping Cao, Meng Xu +5
Pengcheng Laboratory, Shenzhen, China · City University of Hong Kong, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China +2
Model merging has emerged as a promising paradigm for enabling multi-task capabilities without additional training. However, traditional basic merging methods often experience performance degradation due to parameter conflicts, even when applied to similar tasks. While recent personalized merging frameworks successfully preserve task-specific information to maintain performance, they typically incur storage overhead. In this paper, we propose Decomposition, Thresholding, and Scaling (DTS), an approximation-based personalized merging framework that pushes task-specific storage efficiency. DTS first applies singular value decomposition to the task-specific information and retains only a small subset of singular values and vectors. It then introduces a novel thresholding strategy that partitions singular vector elements into groups and assigns a scaling factor to each group. To enable generalization to unseen tasks, we further extend DTS with a variant that fuses task-specific information in a data-free manner based on the semantic similarity of task characteristics. Extensive experiments demonstrate that DTS consistently outperforms state-of-the-art baselines while requiring only 1% extra storage per task. Furthermore, experiments on unseen tasks show that the DTS variant achieves significantly better generalization performance. Our code is available at https://github.com/krumpguo/DTS.
Kuangpu Guo, Aijing Yu, Jian Liang +4
University of Science and Technology of China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +3
Model merging aims to combine multiple fine-tuned models derived from a common pretrained model into a single multi-task model without additional joint training. Recent spectral merging methods improve over simple weight averaging by exploiting low-rank structures of task-specific updates, but they commonly assign the same rank capacity to every task. This uniform allocation ignores that task vectors can have heterogeneous spectral complexity, causing the shared merging space to be used suboptimally. In this paper, we propose Spectral Energy-proportional Rank Allocation (SERA), a simple task-adaptive strategy that allocates ranks according to the singular-value energy structure of each task vector. By assigning richer spectral capacity to complex or isolated tasks and fewer directions to compact tasks, SERA extends SVD-based model merging from uniform-capacity merging to task-dependent capacity allocation. Experiments under standard vision model merging protocols show that SERA improves multi-task merging performance while preserving the same total rank budget as existing spectral merging methods. Further analysis demonstrates that task-level spectral concentration is closely related to the per-task effect of adaptive rank allocation, providing insight into when and why SERA is effective.