One Size Does Not Fit All: Distribution-Guided Tensor-Wise Adaptive Sparsification for Model Merging

Yingfeng Luo; Dingyang Lin; Junxin Wang; Kaiyan Chang; Ziqiang Xu; Weiqiao Shan; Yinqiao Li; Tong Zheng

cs.CLAug 8, 2025

One Size Does Not Fit All: Distribution-Guided Tensor-Wise Adaptive Sparsification for Model Merging

Authors: Yingfeng Luo, Dingyang Lin, Junxin Wang, Kaiyan Chang, Ziqiang Xu, Weiqiao Shan, Yinqiao Li, Tong Zheng,

Organizations: School of Computer Science and Engineering, Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China · Kunming University of Science and Technology, Kunming, China

Abstract

Merging multiple task-specific fine-tuned checkpoints into a single multi-task model provides an attractive alternative to joint multi-task training. To mitigate cross-task interference, a common strategy is to sparsify task vectors before aggregation. However, many widely used methods rely on a uniform global sparsity ratio, overlooking differences in pruning sensitivity across tasks and parameter tensors. Consequently, such uniform sparsification may remove useful parameter updates from sensitive tensors while retaining redundant updates in others. To address this limitation, we propose TADrop, a plug-and-play sparsifier that independently determines tensor-specific keep ratios from their magnitude distributions using a scale-invariant quantile ratio. TADrop operates in a data-free manner by default, while offering optional lightweight module-level calibration when a small validation set is available. Experiments across vision, language, and multimodal benchmarks, including LLM and LoRA merging, demonstrate consistent improvements across multiple host merging methods, including a 4.99-point gain over EMR-Merging on RoBERTa GLUE. In many-task ViT-B/16 merging, the improvement increases with the number of tasks, reaching 2.26 points when merging 30 tasks. Further analyses show that TADrop preserves more task-associated neurons under matched overall sparsity. Together, these findings highlight the value of distribution-guided sparsification at the tensor level for improving model merging.

Figures & tables

Figure 1: Left : Average accuracy over eight ViT-B/32 tasks after EMR-Merging with different sparsification strategies and sparsity ratios. Right : Average accuracy of different merging methods with and without TADrop.

Figure 2: Task vectors are heterogeneous across sparsity levels, tasks, and layers/modules, motivating tensor-wise adaptive sparsification. (a) illustrates how sparsification strategy and sparsity ratio affect merged-model performance; (b) shows that different tasks exhibit different sensitivity to task vector sparsification (Vertical markers denote 5-point accuracy drop thresholds); (c) visualizes the heterogeneity of task vector magnitude distributions across layers/modules and tasks.

Figure 3: Overview of TADrop. Given task vectors

\{\tau_{k}\}_{k=1}^{K}

, TADrop independently sparsifies each parameter tensor

\tau_{k}^{n}

before host merging. It (1) estimates a tensor-wise keep ratio

r_{k}^{n}

from the magnitude distribution via a quantile ratio, (2) optionally calibrates the ratio by functional module group using a small calibration set, with

\alpha_{g}=1

in the data-free setting, (3) retains the largest-magnitude

\tilde{r}_{k}^{n}

fraction, and (4) rescales the masked tensor to approximately preserve

\lVert\tau_{k}^{n}\rVert_{2}

. The resulting sparsified task vectors

\{\hat{\tau}_{k}\}_{k=1}^{K}

are passed to the original host merger

\mathcal{F}

without modifying its merging rule.

Methods	SUN397	Cars	RESISC45	EuroSAT	SVHN	GTSRB	MNIST	DTD	Avg.
Individual	79.2 / 84.9	77.7 / 92.4	96.1 / 97.4	99.7 / 99.7	97.5 / 98.1	98.7 / 99.2	99.7 / 99.8	79.4 / 84.1	91.0 / 94.4
Traditional MTL	73.9 / 80.8	74.4 / 90.6	93.9 / 96.3	98.2 / 96.3	95.8 / 97.6	98.9 / 99.1	99.5 / 99.6	77.9 / 84.4	88.9 / 93.5
Weight Averaging	65.3 / 72.1	63.4 / 81.6	71.4 / 82.6	71.7 / 91.9	64.2 / 78.2	52.8 / 70.7	87.5 / 97.1	50.1 / 62.8	65.8 / 79.6
Fisher Merging	68.6 / 69.2	69.2 / 88.6	70.7 / 87.5	66.4 / 93.5	72.9 / 80.6	51.1 / 74.8	87.9 / 93.3	59.9 / 70.0	68.3 / 82.2
RegMean	65.3 / 73.3	63.5 / 81.8	75.6 / 86.1	78.6 / 97.0	78.1 / 88.0	67.4 / 84.2	93.7 / 98.5	52.0 / 60.8	71.8 / 83.7
PCB-Merging	66.7 / 76.8	65.5 / 86.2	78.5 / 89.4	79.3 / 96.5	86.4 / 88.3	77.1 / 91	98.2 / 98.6	59.1 / 73.6	76.3 / 87.5

Table 1: Multi-task performance when merging ViT models on eight tasks (a / b: a = ViT-B/32, b = ViT-L/14). Bold numbers indicate the best score within each backbone block, and underlined numbers indicate the second-best score.

Methods	COCO-Re	COCO-Captioning				ImageNet	NLVR2	VQAv2	Avg.
Methods	Acc	BLEU	CIDEr $\times 100$	METEOR	ROUGE	Acc	Acc	Acc	Score
Individual	84.6	39.4	134.0	31.1	60.1	85.4	77.7	84.4	74.6
Weight Averaging	18.9	3.1	0.1	11.5	15.9	67.7	28.0	62.9	26.0
Task Arithmetic ( Ilharco et al., 2023 )	31.8	3.3	0.0	11.8	17.6	70.8	38.1	69.3	30.3
TIES-Merging ( Yadav et al., 2023 )	39.3	2.9	0.1	10.8	16.7	69.8	32.1	67.2	29.9
EMR-Merging ( Huang et al., 2024 )	79.5	28.9	106.0	27.2	53.4	77.4	74.8	72.1	64.9

Table 2: Performance of merging multimodal BEiT3 models on five vision-language tasks.

Method	Avg.
Individual	85.56
Weight Averaging	51.34
Fisher Merging ( Matena and Raffel, 2022 )	45.57
RegMean ( Jin et al., 2023 )	70.01
Task Arithmetic ( Ilharco et al., 2023 )	66.65
+ ImPart ( Yang et al., 2025a )	65.72

Table 3: Average multi-task performance when merging RoBERTa-base models fine-tuned on eight GLUE text classification tasks.

Model	GSM8K	ARC-C	MMLU-CS	Avg.
Qwen2.5-32B-Ins	77.94	74.49	83.00	78.48
SAND-Math	89.39	73.72	83.00	82.04
OpenThinker	60.65	72.27	86.00	72.97
SWE-Swiss	78.70	71.25	84.00	77.98
Task Arithmetic	82.11	73.29	85.00	80.13
+ TADrop(w/o)	84.69	73.98	83.00	80.55

Table 4: Performance of merging three models fine-tuned from Qwen2.5-32B-Instruct on three benchmarks.

Method	Avg.
Task Arithmetic ( Ilharco et al., 2023 )	61.35
+ TADrop(w/o)	63.32
TIES-Merging ( Yadav et al., 2023 )	63.62
+ TADrop(w/o)	64.19
KnOTS-TIES ( Stoica et al., 2025 )	68.30
+ TADrop(w/o)	69.16

Table 5: Average performance of LoRA model merging on the eight vision classification tasks using CLIP ViT-B/32 with rank-16 adapters.

Panel of Figure 4 — Figure 4: Scalability of TADrop in 30-task model merging on ViT-B/16. Left : Absolute average accuracy as the number of merged tasks grows from 8 to 30. Right : Performance gain over the EMR-Merging baseline.

Figure 5: Average performance versus task-vector retention on ViT-L/14 and RoBERTa. Colors distinguish host merging methods, while markers indicate different sparsification strategies.

Backbone	Method	Baseline	w/ Top- $k$	w/ TADrop
ViT-L/14	TA	100.00 / 84.50	31.54 / 84.34	31.54 / 86.70
ViT-L/14	TIES	20.00 / 84.40	30.00 / 84.64	30.00 / 86.80
RoBERTa	TA	100.00 / 66.65	20.65 / 68.45	20.65 / 69.78
RoBERTa	TIES	19.86 / 64.04	18.67 / 65.58	18.67 / 69.64

Table 6: Comparison of uniform Top-

k

and TADrop under matched retention budgets. Each entry reports retention (%) / average performance. Baseline denotes the original host method.

Figure 6: Retention of task-labeled neurons under matched overall sparsity. TADrop consistently outperforms uniform-ratio magnitude-based sparsification.

Figure 7: Ablation on quantile choices

(Q_{a},Q_{b})

on ViT-B (8 tasks). Bars show the change in average accuracy relative to each host method’s baseline.

Method	EMR-Merging	w/o R	$(1-p)$ R	$\ell_{2}$ R
Avg.	88.7	86.8	71.1	90.8

Table 7: Ablation results of rescaling strategies on the 8 ViT tasks. R denotes the rescaling operation.

Method	Sparsification Signal	Sparsity Allocation / Selection	Auxiliary Data
TADrop (Ours)	Tensor magnitude quantiles	Tensor-wise adaptive keep ratios with magnitude-based selection	Optional
TIES-Merging ( Yadav et al., 2023 )	Update magnitudes	Uniform keep ratio with magnitude-based selection	None
DARE ( Yu et al., 2024 )	Random sampling	Uniform random drop probability	None
DELLA-Merging ( Deep et al., 2024 )	Magnitude ranking	Magnitude-dependent parameter dropping probabilities	None
TALL-masks ( Wang et al., 2024 )	Task-specific update contrasts	Task-specific element-wise masks	Validation for tuning
PCB-Merging ( Du et al., 2024 )	Intra-task importance and inter-task similarity	Element-wise pruning and rescaling	None

Table 8: Conceptual comparison of representative task-vector sparsification and selection methods in terms of their decision signals, parameter retention strategies, and auxiliary data requirements.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	Chat			Math	Code
Method	MMLU	HellaSwag	TruthfulQA	GSM8K	HumanEval	MBPP
Individual	0.4793	0.75	0.31	0.45	0.23	0.2
Ties-Merging ( Yadav et al., 2023 )	0.4707	0.74	0.27	0.23	0.29	0.29
+ TADrop(w/o)	0.4768	0.75	0.27	0.38	0.29	0.27
+ TADrop	0.4858	0.76	0.28	0.41	0.29	0.26
EMR-Merging ( Huang et al., 2024 )	0.4753	0.72	0.3	0.47	0.27	0.24

Appendix

Table 10: Results on chat, math, and code benchmarks for Llama-2-7B. Bold numbers indicate the best score within each backbone block.

Method	CoLA	SST-2	MRPC	STS-B	QQP	MNLI	QNLI	RTE	Avg.
Individual	60.18	94.04	89.22	90.63	91.41	87.20	92.71	79.06	85.56
Weight Averaging	13.96	64.11	69.36	31.84	75.36	42.19	58.70	55.23	51.34
Fisher Merging ( Matena and Raffel, 2022 )	6.11	53.56	65.21	22.80	58.51	34.80	67.64	55.96	45.57
RegMean ( Jin et al., 2023 )	36.67	90.60	75.74	62.68	83.55	70.02	82.35	58.48	70.01
Task Arithmetic ( Ilharco et al., 2023 )	18.78	85.89	79.90	74.03	83.78	59.08	69.67	62.09	66.65
+ ImPart ( Yang et al., 2025a )	21.29	85.67	79.17	72.40	82.24	54.43	69.21	61.37	65.72

Appendix

Table 11: per-task performance when merging RoBERTa-base models fine-tuned on eight GLUE text classification tasks. Bold numbers indicate the best score within each backbone block, and underlined numbers indicate the second-best score.

Method	SUN397	Cars	RESISC45	EuroSAT	SVHN	GTSRB	MNIST	DTD	Avg.
Task Arithmetic	92.66	81.17	69.27	36.55	46.15	42.23	51.23	71.53	61.35
+ TADrop(w/o)	94.19	81.51	71.00	47.47	44.10	43.27	52.05	72.99	63.32
TIES-Merging	94.60	82.30	69.47	49.91	45.20	36.52	58.33	72.63	63.62
+ TADrop(w/o)	94.52	82.35	69.76	51.33	45.09	38.47	58.42	73.63	64.19
KnOTS-TIES	92.93	82.87	71.07	49.87	54.55	49.24	71.88	73.99	68.30
+ TADrop(w/o)	93.53	81.36	71.43	51.74	56.44	50.07	74.43	74.27	69.16

Appendix

Table 12: Detailed LoRA merging results on eight vision classification tasks using CLIP ViT-B/32 with rank-16 adapters. Bold numbers indicate the best performance within each merging method group.

Model Merging

Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.

See all 192 papers

CASS: Contribution-Aware Structured Sparsity for Model Merging
Sep 28, 2026Yan Li, Guiping Cao, Meng Xu +5Multi-Task Learning Structured Sparsity
Pengcheng Laboratory, Shenzhen, China · City University of Hong Kong, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China +2
Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging
Dec 1, 2025Kuangpu Guo, Aijing Yu, Jian Liang +4Model Merging
University of Science and Technology of China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +3
Not All Task Vectors Need Equal Rank: Energy-Proportional Allocation for Model Merging
Sep 21, 2026Hyunjoong Cho, Jinhyeok JangMulti-Task Learning Task Vector Merging
LG CNS · ETRI · UST

Sep 28, 2026J/K move · Enter open · S save

CASS: Contribution-Aware Structured Sparsity for Model Merging

Yan Li, Guiping Cao, Meng Xu, Tao Jiang+4

Pengcheng Laboratory, Shenzhen, China · City University of Hong Kong, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China+2

Multi-Task Learning Structured Sparsity Task Vector Merging Attention Head Pruning Model Merging

Open paper AskPDF arXiv

Model merging integrates task-specific fine-tuned models into a single multi-task model, but often suffers from parameter interference caused by conflicting task-vector updates. Existing methods typically mitigate conflicts by pruning task vectors based on weight magnitude or random heuristics, treating Transformers as unstructured ``bags of parameters'' and overlooking their inherent modularity. In this paper, we propose Contribution-Aware Structured Sparsity (CASS), a unified framework that reduces parameter interference by identifying and preserving task-specific components. At the core of CASS is a contribution-aware structured mask that identifies task-relevant attention heads and FFN neurons. We instantiate this mask in two settings: CASS-Merging, the primary post-hoc setting where masks serve as a plug-and-play denoising filter for existing merging operators, and CASS-Tuning, an extension for scenarios with fine-tuning access where masks constrain gradients to reduce structural overlap between task vectors. Our analysis shows that task-relevant components are sparse and partially disjoint, supporting structured component-level filtering as an effective way to reduce merging interference. Extensive experiments across vision (ViT, 20 tasks) and language (RoBERTa, 8 tasks; Qwen2.5, 4 tasks) benchmarks demonstrate that CASS improves a range of representative merging baselines.

Abstract

Model Merging

Explore similar work

CASS: Contribution-Aware Structured Sparsity for Model Merging

Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging

Not All Task Vectors Need Equal Rank: Energy-Proportional Allocation for Model Merging