An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse

Yuan Cao; Dezhi Ran; Yuzhe Guo; Mengzhou Wu; Simin Chen; Linyi Li; Wei Yang; Tao Xie

cs.AIMar 10, 2026

An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse

Authors: Yuan Cao, Dezhi Ran, Yuzhe Guo, Mengzhou Wu, Simin Chen, Linyi Li, Wei Yang, Tao Xie

Organizations: Key Lab of HCST (PKU), MOE; SCS, Peking University, Beijing, China · Computer Science Department at Columbia University, New York, USA · School of Computing Science, Simon Fraser University, Burnaby, BC, Canada · Department of Computer Science, University of Texas at Dallas, Richardson, USA

Abstract

Model merging unifies independently fine-tuned LLMs from the same base, enabling reuse and integration of parallel development efforts without retraining. However, in practice we observe that merging does not always succeed: certain combinations of task-specialist models suffer from catastrophic performance degradation after merging. We refer to this failure mode as merging collapse. Intuitively, collapse arises when the learned representations or parameter adjustments for different tasks are fundamentally incompatible, so that merging forces destructive interference rather than synergy. In this paper, we identify and characterize the phenomenon of task-level merging collapse, where certain task combinations consistently trigger huge performance degradation across all merging methods. Through extensive experiments and statistical analysis, we demonstrate that representational incompatibility between tasks is strongly correlated with merging collapse, while parameter-space conflict metrics show minimal correlation, challenging conventional wisdom in model merging literature. We provide a theoretical explanation on this phenomenon through rate-distortion theory with a dimension-dependent bound, establishing fundamental limits on task mergeability regardless of methodology.

Figures & tables

Model	Merging Techniques
Model	LA	TA	TIES	DARE	SLERP
Llama3.2-3B	17.3 $\pm$ 7.5	17.6 $\pm$ 7.8	21.8 $\pm$ 16.9	57.0 $\pm$ 27.5	17.5 $\pm$ 8.0
Llama3.1-8B	22.0 $\pm$ 22.2	22.0 $\pm$ 22.2	31.7 $\pm$ 19.9	66.1 $\pm$ 27.9	22.2 $\pm$ 23.0
Qwen2.5-3B	13.7 $\pm$ 15.1	13.6 $\pm$ 15.2	13.3 $\pm$ 13.0	27.1 $\pm$ 16.8	13.1 $\pm$ 14.7
Qwen2.5-7B	17.4 $\pm$ 23.3	17.0 $\pm$ 22.2	26.1 $\pm$ 24.3	62.6 $\pm$ 24.6	17.6 $\pm$ 22.6
Qwen2.5-14B	19.0 $\pm$ 26.3	19.1 $\pm$ 26.9	18.6 $\pm$ 25.7	31.9 $\pm$ 27.8	18.9 $\pm$ 26.3

Table 1: Merging Losses on GLUE tasks across Different Merging Techniques and Models.

Model	Merging	NLP Tasks
		COLA		MNLI		MRPC		QNLI		QQP		RTE		SST-2		WNLI
		FT	$M(\Delta)$	FT	$M(\Delta)$	FT	$M(\Delta)$	FT	$M(\Delta)$	FT	$M(\Delta)$	FT	$M(\Delta)$	FT	$M(\Delta)$	FT	$M(\Delta)$
3B	LA	82.7	81.0 ( -2.1 )	88.4	86.2 ( -2.4 )	89.5	75.5 ( -15.6 )	92.6	85.0 ( -8.2 )	84.2	62.6 ( -25.7 )	90.3	85.2 ( -5.6 )	95.4	93.7 ( -1.8 )	73.2	38.0 ( -48.1 )
	TA	82.7	81.2 ( -1.9 )	88.4	86.2 ( -2.5 )	89.5	75.5 ( -15.6 )	92.6	85.1 ( -8.1 )	84.2	62.6 ( -25.7 )	90.3	85.6 ( -5.2 )	95.4	93.6 ( -1.9 )	73.2	38.0 ( -48.1 )
	TIES	82.7	79.2 ( -4.3 )	88.4	87.8 ( -0.7 )	89.5	83.6 ( -6.6 )	92.6	67.8 ( -26.9 )	84.2	63.2 ( -24.9 )	90.3	87.7 ( -2.8 )	95.4	91.9 ( -3.7 )	73.2	46.5 ( -36.5 )
	DARE	82.7	70.4 ( -14.9 )	88.4	35.4 ( -60.0 )	89.5	82.6 ( -7.7 )	92.6	50.6 ( -45.4 )	84.2	63.2 ( -24.9 )	90.3	80.9 ( -10.4 )	95.4	73.7 ( -22.7 )	73.2	50.7 ( -30.8 )

Table 2: Effectiveness of different merging methods on NLP Tasks with Qwen2.5 Models.

Task Setting	Merging Tech.	Task
Glue	$0.575$	$2.357\times 10^{-36}$
Lots-of-LoRAs	$0.987$	$1.699\times 10^{-7}$

Table 3: P-values of One-way ANOVA F-tests ( Maxwell et al., 2017 ) for Task-level and Merging-technique-level Effects.

p<0.05

indicates statistical significance.

Conflict Metrics	Merging Techniques
Conflict Metrics	TIES	LA	DARE	SLERP	TA
Parameter Magnitude Change Ratio	0.192	0.098	0.834	0.085	0.186
Parameter Sign Change Ratio	0.379	0.460	0.882	0.408	0.659
Conflicting Parameter Magnitude Change Ratio	0.170	0.170	0.979	0.156	0.235
Average Cosine Similarity	0.262	0.459	0.635	0.444	0.296
Hidden State Distance Similarity	0.006	0.001	0.145	0.001	0.006

Table 4: P-values of Pearson correlation coefficient ( Benesty et al., 2009 ) on how conflict metrics are correlated to merging loss with different merging techniques when merging two model checkpoints.

p<0.05

indicates statistical significance.

	(a)	(a1)	(a2)
Worst Loss	-70.8	-17.8	-19.1
Worst MDS	5.05	2.52	2.89

Table 6: Comparison among three task groups.

Language Model Merging

Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.

See all 106 papers

Predicting and Repairing Merge Collapse in Large Language Models
Oct 2, 2026Jungseob Lee, Seungyoon Lee, Sugyeong Eo +3Language Model Merging Task Vector Merging
Korea University · Yonsei University Mirae Campus · Sookmyung Women’s University +1
Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions
Mar 10, 2026Mingyang Song, Mao ZhengLanguage Model Merging Model Merging
Large Language Model Department, Tencent, China
CAMFT: Conflict-Aware Mergeable Fine-Tuning for Large Language Models
Sep 6, 2026Jingang Zhou, Haiyang Guo, Yuan Ma +2LLM Fine-Tuning Language Model Merging
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences

Oct 2, 2026J/K move · Enter open · S save

Predicting and Repairing Merge Collapse in Large Language Models

Jungseob Lee, Seungyoon Lee, Sugyeong Eo, Hyeonseok Moon+2

Korea University · Yonsei University Mirae Campus · Sookmyung Women’s University+1

Language Model Merging Task Vector Merging Task Vectors

Open paper AskPDF arXiv

Large language models fine-tuned from a shared base can be merged by averaging their task vectors, but some merges collapse far below the base model, and common merge operators give no warning before evaluation. We show that one statistic of the specialists' task vectors both predicts this collapse and calibrates its repair. The power that averaging removes equals the variance of the task vectors across specialists, our measure of interference. Under a working noise model, the disturbance that a merge injects grows with the merge coefficient and with interference, yielding a pre-merge score. In our experiments on twenty-two merge configurations from four model families, only destructive merges exceed a threshold on this score. We find that statistics of sign conflict between specialists, a common target of existing merge operators, are anti-predictive. We then predicted the outcomes of fourteen merges before evaluating them, and twelve predictions were correct, including the destructive outcome of a specialist pair pushed past the threshold by continued pretraining. To address this collapse, we introduce PRISM, an operator that averages the task vectors first and then soft-thresholds each layer at a level set by the layer's interference. Without data or tuning, PRISM keeps all five destructive merges above the threshold within evaluation noise of the base model, where plain averaging falls at least 14.4 points below it or collapses entirely. We apply PRISM only above the threshold and keep the plain average for merges below it, which include all fifteen harmless ones. Code is available at https://github.com/js-lee-AI/PRISM.

Abstract

Language Model Merging

Explore similar work

Predicting and Repairing Merge Collapse in Large Language Models

Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions

CAMFT: Conflict-Aware Mergeable Fine-Tuning for Large Language Models