Predicting and Repairing Merge Collapse in Large Language Models

Jungseob Lee; Seungyoon Lee; Sugyeong Eo; Hyeonseok Moon; Jaehyung Seo; Heuiseok Lim

cs.LGOct 2, 2026

Predicting and Repairing Merge Collapse in Large Language Models

Authors: Jungseob Lee, Seungyoon Lee, Sugyeong Eo, Hyeonseok Moon, Jaehyung Seo, Heuiseok Lim

Organizations: Korea University · Yonsei University Mirae Campus · Sookmyung Women’s University · Konkuk University

Abstract

Large language models fine-tuned from a shared base can be merged by averaging their task vectors, but some merges collapse far below the base model, and common merge operators give no warning before evaluation. We show that one statistic of the specialists' task vectors both predicts this collapse and calibrates its repair. The power that averaging removes equals the variance of the task vectors across specialists, our measure of interference. Under a working noise model, the disturbance that a merge injects grows with the merge coefficient and with interference, yielding a pre-merge score. In our experiments on twenty-two merge configurations from four model families, only destructive merges exceed a threshold on this score. We find that statistics of sign conflict between specialists, a common target of existing merge operators, are anti-predictive. We then predicted the outcomes of fourteen merges before evaluating them, and twelve predictions were correct, including the destructive outcome of a specialist pair pushed past the threshold by continued pretraining. To address this collapse, we introduce PRISM, an operator that averages the task vectors first and then soft-thresholds each layer at a level set by the layer's interference. Without data or tuning, PRISM keeps all five destructive merges above the threshold within evaluation noise of the base model, where plain averaging falls at least 14.4 points below it or collapses entirely. We apply PRISM only above the threshold and keep the plain average for merges below it, which include all fifteen harmless ones. Code is available at https://github.com/js-lee-AI/PRISM.

Figures & tables

Figure 1: Merge then prune. Top: task vectors are pruned before aggregation. Bottom: Prism averages first with norm scaling and then soft-thresholds the aggregate at the layer’s interference scale. Full algorithm in Appendix A .

	Six-task evaluation							Generation
Method	GSM8K	ARC	MMLU	TQA	HellaS.	WinoG.	Avg.	GSM8K	MATH	HumanE.	MBPP
Math specialist	84.4	49.2	60.2	47.9	58.8	68.4	61.5	83.8	53.6	34.1	56.2
Coder specialist	81.8	49.0	69.5	49.8	66.4	72.0	64.8	82.4	41.4	59.8	68.4
Task Arithmetic	22.0	47.0	55.7	50.0	55.4	66.6	49.5	23.0	0 5.8	13.4	27.4
TIES	67.2	53.4	64.3	50.2	60.6	71.0	61.1	65.8	24.6	17.7	43.0
DELLA	65.2	51.6	64.2	50.3	59.8	70.4	60.3	64.0	22.0	17.7	42.4

Table 1: Qwen2.5-7B Math

+

Coder base checkpoints. Generation: GSM8K 5-shot, MATH-500 4-shot, harness HumanEval/MBPP pass@1 with completion prompts. LEWIS uses activation-based six-task/HumanEval/MBPP results and weight-sparsity GSM8K/MATH generation. Bold: best merged; underline: second best; † : calibration for activation-based LEWIS.

	Six-task evaluation							Generation
Method	GSM8K	ARC	MMLU	TQA	HellaS.	WinoG.	Avg.	GSM8K	MATH	HumanE.	MBPP
Task Arithmetic	0 4.6	32.4	35.7	49.4	43.4	57.0	37.1	0 4.0	0 1.8	0 0.0	0 0.0
TIES	18.8	37.6	41.2	46.1	50.6	61.2	42.6	19.0	0 3.8	11.6	16.2
LEWIS †	49.4	40.2	57.8	46.4	55.4	64.8	52.3	35.6	13.6	28.0	38.6
Prism (ours)	59.6	42.6	61.8	45.5	59.0	66.4	55.8	59.0	26.6	35.4	43.0

Table 2: Qwen2.5-1.5B Math

+

Coder instruction-tuned checkpoints. Columns, marks and LEWIS variants follow Table 1 .

	Norm	Coords	Mean		PPL	Coding (EvalPlus)
Method	$\\|\delta\\|/\\|\bar{\delta}\\|$	%	$c\!=\!0.5$	$c\!=\!1$	$c\!=\!1$	HumanEval	HumanEval+	MBPP	MBPP+
Base model, unmerged	0	0.0	68.0	68.0	8.6	62.8	57.9	69.8	59.5
Task Arithmetic	1.00	99.7	49.5	28.8	$4.6\!\times\!10^{5}$	0.0	0.0	0.0	0.0
TIES	0.80	34.8	61.1	30.4	$1.1\!\times\!10^{4}$	0.0	0.0	0.0	0.0
DELLA	0.82	34.9	60.3	30.9	$1.6\!\times\!10^{4}$	0.0	0.0	0.0	0.0
DARE (drop $0.9$ ) §	2.85	19.0	30.0	29.8	$7.2\!\times\!10^{6}$	0.0	0.0	0.0	0.0

Table 3: Qwen2.5-7B Math

+

Coder. Coding: vendor-native EvalPlus pass@1 at

c\!=\!0.5

where applicable, not comparable with Table 1 . LEWIS: activation-based Mean/PPL, weight-sparsity fallback for Norm/Coords/Coding. Bold: above the unmerged base. § : rescaled; † : calibration for activation-based LEWIS; SLERP and Model Stock: no coefficient.

Figure 4: (a) Densities of the interference score by outcome. (b–d) Densities of

\rho

K

and

c/\sqrt{K}

overlap. (e) At the lower learning rate, the score grows roughly as the square root of specialist training tokens. Large markers are evaluated merges.

Setting	$K$	$c$	$\rho$	$c\sqrt{\overline{I}/K}$ $(\times 10^{-3})$	Base	TA	Prism
naive averaging destroys the model
Qwen2.5-1.5B ja $+$ med, constructed ⋆	2	0.50	0.342	5.19	56.1	35.8	55.9
Qwen2.5-1.5B Math $+$ Coder	2	0.50	0.246	4.94	56.3	37.1	55.8
Qwen2.5-7B Math $+$ Coder	2	0.50	0.293	4.25	68.0	49.5	67.8
Qwen2.5-7B R1-Distill $+$ Coder ⋆	2	0.50	0.291	4.24	68.1	49.7	67.7
Qwen2.5-7B Math-PRM $+$ Coder ⋆	2	0.50	0.285	4.17	68.1	53.7	67.6

Table 4: The interference score exceeds the threshold of

1.9\!\times\!10^{-3}

only for destructive merges. Means are six-task accuracies, and self-SFT specialists are fine-tuned by us from the base. ⋆ Outcome predicted before evaluation. g Gated, so Prism equals Task Arithmetic (TA).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Mechanism on Qwen2.5-7B Math+Coder ( $K\!=\!2$ , instruction-tuned specialists, $\kappa\!=\!0.5$ ). (a) Layer-wise retention, bars shaded by coherence. (b) Retention against coherence, colored by depth. (c) Distance from the base model across settings ordered by interference.

Variant	Mean	PPL $\downarrow$
Global- $\lambda$ (single $\sigma$ )	68.1	9.0
SA-calibrated $\sigma$ (sign agreement)	67.8	9.0
Row-wise $\sigma$ (empirical Bayes)	67.8	9.1
Prism (layer-wise $\mathrm{CR}$ , default)	67.8	9.0

Appendix

Table 5: Calibration-granularity variants on Qwen2.5-7B Math+Coder (

K\!=\!2

c\!=\!0.5

). The variants use interference or sign agreement to calibrate a soft threshold.

Ablation	Variant	PPL $\downarrow$	GSM	ARC	MMLU	TQA	HS	WG	Mean
Threshold scale $\kappa$	$\kappa\!=\!0.25$ (keep $22.6\%$ )	11.6	81.2	56.6	72.0	53.9	66.6	74.2	67.4
	$\kappa\!=\!0.5$ (keep $2.3\%$ )	9.5	79.0	53.2	74.1	54.2	67.8	75.4	67.3
	$\kappa\!\in\![1,3]$ (floor, default)	9.0	81.4	51.4	74.3	55.2	67.4	77.0	67.8
Scope	MLP $+$ Attention	9.0	83.0	51.2	74.2	55.5	67.6	76.0	67.9
Scope	MLP-only (default)	9.0	81.4	51.4	74.3	55.2	67.4	77.0	67.8

Appendix

Table 6: Threshold-scale and scope ablations of Prism on Qwen2.5-7B

K\!=\!2

. floor : the

0.1\%

keep-rate floor binds in every layer. Bold = best in each block.

Instruction-tuned specialists
Method	GSM8K	ARC	MMLU	TQA	HS	WG	Mean
Task Arith.	$47.8\pm 2.2$	$48.4\pm 2.2$	$59.5\pm 0.4$	$53.0\pm 2.0$	$57.8\pm 2.2$	$69.8\pm 2.1$	$56.0\pm 0.8$
TIES	$69.4\pm 2.1$	$51.4\pm 2.2$	$65.4\pm 0.4$	$52.8\pm 2.0$	$62.6\pm 2.2$	$70.0\pm 2.1$	$61.9\pm 0.8$
DELLA	$70.4\pm 2.0$	$52.2\pm 2.2$	$65.0\pm 0.4$	$51.7\pm 2.0$	$62.2\pm 2.2$	$70.6\pm 2.0$	$62.0\pm 0.8$
LEWIS (wt.-sp.)	$77.4\pm 1.9$	$54.6\pm 2.2$	$70.6\pm 0.4$	$50.7\pm 2.0$	$66.2\pm 2.1$	$72.0\pm 2.0$	$65.2\pm 0.8$
Prism	$81.2\pm 1.7$	$51.4\pm 2.2$	$74.3\pm 0.4$	$55.0\pm 1.9$	$67.8\pm 2.1$	$75.8\pm 1.9$	$67.6\pm 0.7$

Appendix

Table 7: Benchmark scores with standard errors for Qwen2.5-7B Math

+

Coder merges of instruction-tuned and base specialists; the base-specialist merges are those of Table 1 . The final column propagates the six benchmark errors into the standard error of the mean.

		Harness							Generation
Setting	Variant	GSM8K	ARC	MMLU	TruthQA	HellaSwag	WinoGr.	Mean	GSM8K	MATH	HumanEval	MBPP
Qwen 7B $K\!=\!2$	Denoise	81.4	51.4	74.3	55.2	67.4	77.0	67.8	82.4	30.2	51.2	62.8
Qwen 7B $K\!=\!2$	Prune	73.8	53.6	70.2	50.8	65.2	71.0	64.1	73.8	15.6	31.1	39.6
Qwen 1.5B $K\!=\!2$	Denoise	59.6	42.6	61.8	45.5	59.0	66.4	55.8	59.0	26.6	35.4	43.0
Qwen 1.5B $K\!=\!2$	Prune	35.4	40.0	53.4	46.2	53.8	64.8	48.9	35.6	13.8	21.3	37.4
Qwen 7B $K\!=\!3$	Denoise	83.4	51.6	74.3	55.6	66.8	76.8	68.1	83.6	37.2	57.3	64.2

Appendix

Table 8: Prism -Denoise against Prism -Prune across settings (%). Bold marks the better variant, with ties sharing the mark.

Model	$c\sqrt{\overline{I}/K}$ $(\times 10^{-3})$	Base	TA
Qwen2.5-3B	0.076	60.7	61.5
Qwen2.5-14B	0.067	71.9	72.9

Appendix

Table 9: Instruct

+

Coder-Instruct merges of Qwen2.5 at 3B and 14B (

K\!=\!2

c\!=\!0.5

, six-task mean, %). Both score far below the threshold of Table 4 , so the recipe returns the plain average (TA).

Setting	Method	Mean (%)
Qwen $K\!=\!2$	LEWIS (act.)	56.1
	LEWIS (wt.-sp.)	64.9
	Base Model	68.0
Llama $K\!=\!1$	LEWIS (act.)	60.6
	LEWIS (wt.-sp.)	61.4
	Base Model	60.0

Appendix

Table 10: Activation-based and weight-sparsity LEWIS on three merges (six-task mean, %). Bold marks the better variant in each merge.

Scoring Method	Mean (%)	PPL $\downarrow$
Prism -Denoise (interference $\sigma$ )	67.8	9.0
LEWIS-CTP (activation $\sigma$ + denoise)	66.5	11.9
LEWIS-Act (activation + prune)	56.1	24.2

Appendix

Table 11: Scoring ablation on Qwen2.5-7B Math

+

Coder base specialists (

K\!=\!2

, six-task mean).

Method	Data used	Mean (%)
Base model, unmerged	none	68.1
AdaMerging (tuned, $c\!=\!0.1$ )	128 C4 texts	67.9
AdaMerging ( $c\!=\!0.5$ )	128 C4 texts	51.6
Greedy Soup	50 WikiText-2 texts	68.1
Fisher-weighted averaging	100 WikiText-2 texts	47.5
Prism ( $c\!=\!0.5$ )	none	67.8

Appendix

Table 12: Data-dependent baselines on Qwen2.5-7B Math+Coder (

K\!=\!2

, six-task mean). Prism uses no data.

Method	Hyperparameters	Data used
Task Arithmetic	$c\in\{0.1,0.3,0.5,0.7,1.0\}$	none
TIES	$c$ (as above); trim $k\in\{0.1,0.2\}$	none
DELLA	$c$ ; $\epsilon\!=\!0.3$ ; trim $k\!=\!0.2$	none
LEWIS	$c$ ; keep rate $\in\{0.05,0.1\}$	128 WikiText-2 texts
Fisher	$c$ ; diagonal Fisher	100 WikiText-2 texts
Greedy Soup	inclusion by perplexity	50 WikiText-2 texts

Appendix

Table 13: Baseline settings and sweeps across all experiments. The LEWIS data entry applies to the activation-based variant.

		Harness	Generation
$c$	Method	Mean	GSM8K	MATH-500	HumanEval	MBPP
$0.3$	Task Arith.	63.4	69.2	29.0	33.5	48.8
	TIES	66.1	78.4	30.2	42.1	53.8
	DELLA	65.8	78.2	27.6	42.7	53.4
	LEWIS	67.6	80.8	22.6	41.5	55.0
	Prism -Denoise	68.0	83.0	32.6	51.2	64.8

Appendix

Table 14: Scaling-coefficient sweep on Qwen2.5-7B Math

+

Coder: six-task harness mean and generation accuracy (%). LEWIS: activation-based harness mean, weight-sparsity generation. Bold = best merged in each block.

		Mean		GSM8K (gen.)		PPL $\downarrow$
Rank	$\mathrm{CR}$	Prism	TA	Prism	TA	Prism	TA
$r\!=\!8$	0.529	67.7	65.1	81.8	78.2	10.1	14.2
$r\!=\!16$	0.524	67.9	65.2	83.8	78.4	9.9	14.5
$r\!=\!32$	0.520	68.4	64.8	84.2	77.4	9.8	14.9
$r\!=\!64$	0.519	68.0	64.2	83.4	73.6	9.5	16.0
$r\!=\!128$	0.521	68.3	63.2	83.6	72.6	9.2	17.3

Appendix

Table 15: Low-rank SVD simulation on Qwen2.5-7B Math+Coder (

K\!=\!2

) across projection ranks: six-task harness mean, GSM8K generation accuracy and perplexity for Prism -Denoise and Task Arithmetic (TA), with the merge’s coherence ratio.

Method	MBPP	HumanEval
Base Model	63.4	57.3
Math Specialist	0.0	48.8
Coder Specialist	70.0	81.7
Task Arith.	34.8	11.6
TIES	45.1	18.3
DELLA	45.7	18.9

Appendix

Table 16: Coding results with instruction-tuned Qwen2.5-7B Math+Coder specialists (

K\!=\!2

), pass@1 (%). Gray italic marks specialist reference scores. LEWIS uses its weight-sparsity variant. Bold best merged, underline second best; † needs calibration data.

Model	HumanEval	HumanEval+	MBPP	MBPP+
Base model, unmerged	62.8	57.9	69.8	59.5
Prism -Denoise	65.9	59.1	71.2	61.1
Math Specialist	66.5	63.4	60.3	52.9
Coder Specialist	64.6	60.4	72.0	60.6

Appendix

Table 17: EvalPlus pass@1 (%) of the unmerged base, Prism and the two specialists on Qwen2.5-7B Math+Coder (

K\!=\!2

, base specialists). Table 3 reports the baselines under the same recipe.

Statistic	AUC	Margin	Result
$c\sqrt{\overline{I}/K}$ (used)	0.98	$0.83\times$	predictive
$c\sqrt{\overline{P}/K}$ (total power)	1.00	$21.2\times$	separates
$c\sqrt{\overline{S}/K}$ (signal power)	1.00	$23.3\times$	separates
$1-\overline{\mathrm{CR}}$ (mean incoherence)	0.08	$<\!0.01\times$	anti-predictive
$\rho$ (cancellation ratio)	0.25	$0.02\times$	anti-predictive

Appendix

Table 18: Rival pre-merge statistics on all configurations of Table 4 . AUC treats larger scores as more destructive. Margin is the smallest destructive value divided by the largest harmless one.

Method	Mean (%)
Base Model	60.2
Task Arithmetic	62.0
DELLA	61.4
TIES	61.2
LEWIS	60.3
Prism -Denoise (no gate)	59.6

Appendix

Table 19: Llama-3.1-8B

K\!=\!3

merge of OpenMath2, Hermes-3 and Llama-Guard with unscaled task vectors (six-task mean, %). The data-free

\rho

-gate detects the dispersion-dominated regime at

\rho\!=\!0.021

and returns Task Arithmetic. Bold = best.

Method	$K\!=\!2$	$K\!=\!3$	$K\!=\!4$	$K\!=\!5$	$K\!=\!6$	$K\!=\!7$	$K\!=\!8$
Base Model (ref.)	68.1	68.1	68.1	68.1	68.1	68.1	68.1
Task Arithmetic	68.7	68.5	68.0	68.1	68.1	68.1	68.0
TIES	68.4	68.2	68.2	68.1	67.9	68.1	68.2
Prism -Denoise ( $+\,\rho$ -gate)	67.9	68.1	68.1	68.1	68.0	68.0	68.1
$\rho$ (cancellation ratio)	0.34	0.42	0.46	0.55	0.60	0.65	0.70

Appendix

Table 20: Matched-recipe self-SFT scaling on Qwen2.5-7B with

c\!=\!1/K

and unscaled task vectors, merging

K\!=\!2

8

specialists trained from one base under an identical recipe.

Language Model Merging

Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.

See all 106 papers

Mar 10, 2026

An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse

Model merging unifies independently fine-tuned LLMs from the same base, enabling reuse and integration of parallel development efforts without retraining. However, in practice we observe that merging does not always succeed: certain combinations of task-specialist models suffer from catastrophic performance degradation after merging. We refer to this failure mode as merging collapse. Intuitively, collapse arises when the learned representations or parameter adjustments for different tasks are fundamentally incompatible, so that merging forces destructive interference rather than synergy. In this paper, we identify and characterize the phenomenon of task-level merging collapse, where certain task combinations consistently trigger huge performance degradation across all merging methods. Through extensive experiments and statistical analysis, we demonstrate that representational incompatibility between tasks is strongly correlated with merging collapse, while parameter-space conflict metrics show minimal correlation, challenging conventional wisdom in model merging literature. We provide a theoretical explanation on this phenomenon through rate-distortion theory with a dimension-dependent bound, establishing fundamental limits on task mergeability regardless of methodology.

Yuan Cao, Dezhi Ran, Yuzhe Guo +5

Key Lab of HCST (PKU), MOE; SCS, Peking University, Beijing, China · Computer Science Department at Columbia University, New York, USA · School of Computing Science, Simon Fraser University, Burnaby, BC, Canada +1

Language Model Merging Model Merging

Ask PDF

Mar 10, 2026

Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions

Model merging combines the parameters of multiple neural networks into a single model without additional training. As fine-tuned large language models (LLMs) proliferate, merging offers a computationally efficient alternative to ensembles and full retraining, enabling practitioners to compose specialized capabilities at minimal cost. This survey examines model merging in the LLM era through the FUSE taxonomy, organized along Foundations, Unification Strategies, Scenarios, and Ecosystem. We first establish the theoretical underpinnings of merging, including loss landscape geometry and mode connectivity, then systematically review the algorithmic space spanning weight averaging, task vector arithmetic, sparsification-enhanced methods, mixture-of-experts architectures, and evolutionary optimization. We further examine downstream applications across multi-task learning, safety alignment, domain specialization, and federated learning, and survey the supporting ecosystem of tools and evaluation benchmarks. Finally, we identify key open challenges and future directions, aiming to equip researchers and practitioners with a structured foundation for advancing model merging.

Mingyang Song, Mao Zheng

Large Language Model Department, Tencent, China

Language Model Merging Model Merging

Ask PDF

Jul 13, 2026

Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training. Standard practice merges experts at their optimal validation loss. We challenge this convention by systematically studying how training duration of domain experts affects the quality of the merged model. We fine-tune experts on five domains (Math, Code, Instruction Following, Multilingual, and Safety) across three model sizes (Qwen 3.5 0.8B, 2B, and 4B), saving checkpoints from 25% to 500% of the optimal training steps and evaluating five merging methods at each duration. Our findings reveal a striking method-dependent pattern: simple averaging degrades sharply with overfitting, while sparsification-based methods achieve their best performance well past the validation optimum. We formalize this through bias-variance decomposition analysis, drawing a parallel to random forests where averaging benefits from high-variance individual learners. These results suggest that training duration and merging method should be chosen jointly rather than independently.

Nikita Kozodoi, Zainab Afolabi, Jack Butler

1Amazon Web Services

LLM Fine-Tuning Language Model Merging Model Merging

Ask PDF

Abstract

Language Model Merging

Explore similar work

An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse

Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions

Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs