CASS: Contribution-Aware Structured Sparsity for Model Merging

Yan Li; Guiping Cao; Meng Xu; Tao Jiang; Yaguang Song; Ming Tao; Yaowei Wang; Dongmei Jiang

cs.AISep 28, 2026

CASS: Contribution-Aware Structured Sparsity for Model Merging

Authors: Yan Li, Guiping Cao, Meng Xu, Tao Jiang, Yaguang Song, Ming Tao, Yaowei Wang, Dongmei Jiang

Organizations: Pengcheng Laboratory, Shenzhen, China · City University of Hong Kong, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China · Harbin Institute of Technology, Shenzhen, China · Northwestern Polytechnical University, Xi’an, China

Abstract

Model merging integrates task-specific fine-tuned models into a single multi-task model, but often suffers from parameter interference caused by conflicting task-vector updates. Existing methods typically mitigate conflicts by pruning task vectors based on weight magnitude or random heuristics, treating Transformers as unstructured ``bags of parameters'' and overlooking their inherent modularity. In this paper, we propose \textbf{C}ontribution-\textbf{A}ware \textbf{S}tructured \textbf{S}parsity (CASS), a unified framework that reduces parameter interference by identifying and preserving task-specific components. At the core of CASS is a contribution-aware structured mask that identifies task-relevant attention heads and FFN neurons. We instantiate this mask in two settings: CASS-Merging, the primary post-hoc setting where masks serve as a plug-and-play denoising filter for existing merging operators, and CASS-Tuning, an extension for scenarios with fine-tuning access where masks constrain gradients to reduce structural overlap between task vectors. Our analysis shows that task-relevant components are sparse and partially disjoint, supporting structured component-level filtering as an effective way to reduce merging interference. Extensive experiments across vision (ViT, 20 tasks) and language (RoBERTa, 8 tasks; Qwen2.5, 4 tasks) benchmarks demonstrate that CASS improves a range of representative merging baselines.

Figures & tables

Figure 1 : Sparsification-based model merging methods.

Panel of Figure 2 — Figure 2 : Unified summation form of Transformer.

Panel of Figure 3 — Figure 3 : (Left) CDF of neuron contribution scores (Coding task). (Right) Jaccard similarity of top-20% neurons across tasks (Layer 14).

Figure 4 : The proposed C ontribution- A ware S tructured S parsity (CASS) framework.

Method	ViT-B/32		ViT-B/16
Method	Base	w/ CASS-M	Base	w/ CASS-M
TA	61.37	65.29 (+3.92)	65.02	69.99 (+4.97)
TIES	63.88	65.52 (+1.64)	66.54	69.34 (+2.80)
DARE	61.40	65.24 (+3.84)	65.03	69.90 (+4.87)
PCB	63.72	65.75 (+2.03)	67.74	69.78 (+2.04)
TSV	73.67	74.91 (+1.24)	78.37	78.98 (+0.61)

Table 1 : Results on Vision Tasks. Average accuracy (%) on 20 tasks.

Method	RoBERTa		Qwen2.5-0.5B		Qwen2.5-1.5B
Method	Base	w/ CASS-M	Base	w/ CASS-M	Base	w/ CASS-M
Base Model	47.00	–	0.720	–	0.768	–
Fine-Tuned	85.91	–	1.000	–	1.000	–
Post-Mask	79.33	–	0.910	–	0.975	–
TA	66.36	70.12 (+3.76)	0.743	0.776 (+.034)	0.789	0.837 (+.048)
TIES	70.05	71.15 (+1.10)	0.695	0.773 (+.078)	0.803	0.829 (+.026)

Table 2 : Results on RoBERTa and Qwen2.5. Average performance on 8 tasks for RoBERTa and average normalized performance on 4 tasks for Qwen2.5.

Method	Qwen2.5-0.5B		Qwen2.5-1.5B
Method	Base	w/ CASS-T	Base	w/ CASS-T
Base Model	0.720	–	0.768	–
Fine-Tuned	1.000	–	1.000	–
Masked FT	0.954	–	0.924	–
TA	0.743	0.811 (+.069)	0.789	0.820 (+.031)
TIES	0.695	0.816 (+.121)	0.803	0.867 (+.064)

Table 3 : Results of CASS-Tuning on Qwen2.5. Average normalized performance on 4 tasks.

Table 4 : OOD Generalization on ViT and Qwen.

Figure 5 : Probe sample size sensitivity.

Figure 6 : Comparison between TA, random structured masks, and CASS masks. Random masks use the same sparsity budget as CASS.

Figure 7 : Component-wise ablation study. “w/ MHA” and “w/ FFN” denote applying CASS masks only to attention heads or FFN neurons, respectively.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

\tau_{t}=\theta_{t}-\theta_{base}.

Appendix

Algorithm 1 CASS-Merging

\mu_{h,\mathrm{global}}=\frac{1}{K}\sum_{t=1}^{K}\mathcal{A}_{h,t}.

Appendix

Algorithm 2 CASS-Tuning

Panel of Figure 8 — Figure 8 : CDF of FFN neuron contribution scores across tasks. The CDF curves exhibit a sharp rise near zero across all four task domains, indicating that only a small subset of FFN neurons contributes prominently to each task. This suggests that task-relevant FFN activity is highly concentrated rather than uniformly distributed over all neurons.

Panel of Figure 9 — Figure 9 : Layer-wise Jaccard similarity of active FFN neurons across all 28 layers. Each heatmap shows the pairwise overlap between task-specific active neuron sets in one Transformer layer of Qwen2.5-1.5B-Instruct. The heatmaps are ordered from Layer 0 to Layer 27 in row-major order, from left to right and top to bottom. For each task, we select the top-20% FFN neurons according to their contribution scores and compute the Jaccard similarity between the selected sets. Darker off-diagonal entries indicate lower cross-task overlap. The consistently low off-diagonal values across many layers suggest that task-relevant FFN activity tends to be concentrated in partially separated neuron subsets, supporting the use of neuron-level structured masks in CASS.

Panel of Figure 10 — Figure 10 : Attention head specialization across tasks. Task-selectivity scores reveal that only a subset of heads is strongly specialized for each task, and the selected top heads exhibit limited overlap across different task domains.

Figure 11 : Layer-wise distribution of top-20% task-selective heads. Different tasks exhibit different depth preferences. Coding and Mathematics heads are more concentrated in deeper layers, while Safety heads appear more frequently in relatively shallower layers. Instruction-following heads are distributed more broadly across layers.

Capability	Fine-tuning Dataset	Evaluation Dataset	Metric
Math	DART-Math [ 47 ]	minerva_math500 [ 19 ]	math_verify accuracy
Instruction	TULU-3 persona IF [ 25 ]	ifeval [ 63 ]	prompt-level strict accuracy
Coding	Magicoder [ 52 ]	mbppplus [ 2 ]	pass@1
Safety	WildGuardMix [ 15 ]	wildguardtest [ 15 ]	Micro Harm Rate ( $\downarrow$ )

Appendix

Table 5 : Qwen2.5 tasks, datasets, and metrics. We evaluate four decoder-only capabilities covering mathematical reasoning, instruction following, code generation, and safety.

Hyperparameter	Value
Fine-tuning Method	LoRA
LoRA Rank ( $r$ )	32
LoRA Alpha	64
LoRA Dropout	0.1
Learning Rate	$2\times 10^{-4}$
LR Scheduler	Cosine

Appendix

Table 6 : Fine-tuning hyperparameters for Qwen2.5.

Method	Qwen2.5-0.5B-Instruct					Qwen2.5-1.5B-Instruct
Method	Code	Safe	Math	Instr.	Avg.	Code	Safe	Math	Instr.	Avg.
Base Model	0.676	0.708	0.765	0.731	0.720	0.687	0.775	0.855	0.754	0.768
Fine-Tuned	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
Post-Mask	0.964	0.836	0.961	0.878	0.910	0.953	1.015	1.051	0.882	0.975
TA	0.801	0.687	0.765	0.718	0.743	0.735	0.778	0.942	0.703	0.789
w/ CASS-M	0.820	0.684	0.814	0.788	0.776	0.808	0.777	1.065	0.698	0.837

Appendix

Table 7 : Detailed CASS-Merging results on Qwen2.5. Per-capability normalized performance on Qwen2.5-0.5B-Instruct and Qwen2.5-1.5B-Instruct. CASS-M is applied as a post-hoc structured task-vector filter before each merge operator.

Method	Qwen2.5-0.5B-Instruct					Qwen2.5-1.5B-Instruct
Method	Code	Safe	Math	Instr.	Avg.	Code	Safe	Math	Instr.	Avg.
Base Model	0.676	0.708	0.765	0.731	0.720	0.687	0.775	0.855	0.754	0.768
Fine-Tuned	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
Masked FT	0.966	0.988	0.902	0.962	0.954	0.885	1.008	0.978	0.826	0.924
TA	0.801	0.687	0.765	0.718	0.743	0.735	0.778	0.942	0.703	0.789
w/ CASS-T	0.889	0.705	0.863	0.788	0.811	0.798	0.772	1.007	0.703	0.820

Appendix

Table 8 : Detailed CASS-Tuning results on Qwen2.5. Per-capability normalized performance on Qwen2.5-0.5B-Instruct and Qwen2.5-1.5B-Instruct. CASS-T constrains task-specific fine-tuning with structured masks before merging.

Explore similar work

CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation
Aug 13, 2026Yuchen Liu, Zongzhen Yang, Binhang Qi +2Continual Model Merging Multi--Task Learning
State Key Laboratory of Complex & Critical Software Environment (CCSE), Beihang University, Beijing, China · Hangzhou Innovation Institute of Beihang University, Hangzhou, China · National University of Singapore, Singapore
Model Merging: Foundations and Algorithms
May 2, 2026Donato CrisostomiContinual Model Merging Multi--Task Learning
SAPIENZA UNIVERSITY OF ROME · GLADIA research lab · Department of Computer Science +1
Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging
Dec 1, 2025Kuangpu Guo, Aijing Yu, Jian Liang +4Continual Model Merging Multi--Task Learning
University of Science and Technology of China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +3

Aug 13, 2026, cs.AIJ/K move · Enter open · S save

CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation

Yuchen Liu, Zongzhen Yang, Binhang Qi, Hailong Sun+1

State Key Laboratory of Complex & Critical Software Environment (CCSE), Beihang University, Beijing, China · Hangzhou Innovation Institute of Beihang University, Hangzhou, China · National University of Singapore, Singapore

Continual Model Merging Multi--Task Learning Sparsity

Open paper AskPDF arXiv

Model merging has recently attracted significant attention as a promising paradigm for constructing unified multi-task models without requiring additional retraining. However, parameter conflicts and knowledge interference across tasks often degrade merged-model performance. Prior work introduced Conflict-Aware and Balanced Sparsification (CABS), which reduces parameter interference through structured pruning and sequential masking. However, CABS relies on grid search to determine scaling coefficients, resulting in exponential time complexity, while its optimization objective can be dominated by high-performance tasks, leading to suboptimal overall performance. To address these limitations, we extend CABS and propose CABS+. Specifically, Adaptive Weight Allocation (AWA) optimizes merging coefficients via a gradient-free search scheme to reduce time complexity, while an asymmetric fitness function promotes more comprehensive performance gains across tasks. Moreover, we conduct a systematic empirical study of key factors influencing model merging performance and propose Relative Synergy Score (RSS) to quantify model mergeability and guide model selection. We compare CABS+ with state-of-the-art model merging methods, including CABS, AdaMerging, and WUDIMerging, across 27 datasets and 5 models covering large language, small-scale language, and vision models. Extensive experiments verify the effectiveness and efficiency of CABS+. Compared with AdaMerging and WUDIMerging, CABS+ improves overall performance by 16.97% and 12.93%, respectively, exhibits stronger stability and robustness across varying task numbers and model architectures, uses less than 25% of the GPU memory required by AdaMerging, and achieves nearly a 4x speedup in merging time over WUDIMerging.

Abstract

Explore similar work

CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation

Model Merging: Foundations and Algorithms

Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging