Mitigating Model Merging Conflicts via Knowledge-Aware Task Vector Purification

Bang An; Yibo Yang; Philip Torr; Bernard Ghanem

cs.AIOct 16, 2025

Mitigating Model Merging Conflicts via Knowledge-Aware Task Vector Purification

Authors: Bang An, Yibo Yang, Philip Torr, Bernard Ghanem

Organizations: King Abdullah University of Science and Technology (KAUST) · University of Oxford

Abstract

Model merging aims to integrate task-specific abilities from individually fine-tuned models into a single model without extra training. In recent model merging methods, task vector has become a fundamental building block, as it can encapsulate the residual information from finetuning. However, the merged model often suffers from notable performance degradation due to the conflicts caused by task-irrelevant redundancy in task vectors. Existing efforts in overcoming redundancy by randomly dropping elements in the parameter space involves randomness and lacks knowledge awareness. To address these challenges, in this study, we propose Purifying TAsk Vectors (PAVE) in knowledge-aware subspace. Concretely, we sample some training examples from each task, and feed them into their corresponding fine-tuned models to acquire the covariance matrices before linear layers. We then perform a context-oriented singular value decomposition, which accentuates the weight components most relevant to the target knowledge. As a result, we can split fine-tuned model weights into task-relevant and redundant components in the knowledge-aware subspace, and purify the task vector by pruning the redundant components. To induce fair pruning efforts across models, we further introduce a spectral rank allocation strategy by optimizing a normalized activated pruning error. The task vector purification by our method as a plug-and-play scheme is applicable across various task vector-based merging methods to improve their performance. In experiments, we demonstrate the effectiveness of PAVE across a diverse set of merging methods, tasks, and model architectures.

Figures & tables

Figure 1: An illustration of our method and a comparison between PAVE and prior task vectors. DARE addresses redundancy by randomly dropping elements in the parameter space without knowledge-awareness. PAVE refines task vectors by identifying the target knowledge-relevant components within knowledge-aware subspace and removing the redundant components.

Figure 2: An overall illustration of our proposed method, PAVE, which serves as a plug-and-play method for task-vector based merging methods. It begins by identifying task-aware components through a decomposition approach, and then eliminates noise and less effective components via rank-based pruning. A rank allocation strategy is designed to optimize the preserved rank ratios across individual models. Overall, rather than employing the plain task vector

W_{k}-W_{\text{B}}

, PAVE performs model merging based on purified task vectors

\text{SVD}_{r_{k}}(W_{k}C_{k})C^{-1}_{k}-W_{\text{B}},\ 1\leq k\leq K

Panel of Figure 3 — Figure 3: Model rank pruning with different decomposition methods. The results show that the context-oriented decomposition (CO-SVD) is more effective in aligning the subspace with the target knowledge. The pruned ranks, 96, 192, 384 and 512, correspond to removing 1/8, 1/4, 1/2 and 2/3 of the full rank, respectively.

Methods	CoLA	SST-2	MRPC	STSB	QQP	MNLI	QNLI	RTE	Average
Individual	60.18	94.04	89.22	90.63	91.41	87.20	92.71	79.06	85.55
RegMean	36.67	90.6	75.74	62.68	83.55	70.02	82.35	58.48	70.01
DOGE-Merging	28.64	88.53	81.13	78.46	82.34	59.53	77.27	67.87	70.46
ISO-Merging	28.47	86.93	80.64	67.63	84.74	65.23	78.75	73.29	70.71
EMR-Merging	39.96	93.35	86.27	82.77	89.72	85.45	89.57	74.37	80.18
EMR-Merging W/ DARE	40.58	93.23	86.52	82.86	89.93	85.98	89.68	75.09	80.48 +0.30

Table 1: Results of merging fine-tuned RoBERTa models on eight datasets from GLUE benchmark.

Methods	CoLA	SST-2	MRPC	STSB	QQP	MNLI	QNLI	RTE	Average
Individual	59.10	95.06	89.21	91.31	91.50	88.52	93.30	69.31	84.66
RegMean	22.59	93.00	72.55	27.99	81.89	73.77	73.44	81.80	65.87
DOGE-Merging	2.80	75.60	69.11	53.39	64.05	44.27	54.02	51.34	51.82
ISO-Merging	7.42	60.92	67.35	44.91	63.12	65.93	70.01	45.31	53.12
Twin-Merging	46.13	93.10	73.24	77.57	86.23	72.68	89.86	61.73	75.06
EMR-Merging	47.05	94.72	71.32	87.57	88.36	87.38	92.29	64.26	79.11

Table 2: Results of merging fine-tuned DeBERTa models on eight datasets from GLUE benchmark.

Methods	Mathematical Solving		Code Generating
Methods	GSM8K	MATH	Human Eval	MBPP
Individual Math	54.9	10.7	-	-
Individual Code	-	-	39.6	30.1
Average Merging	32.2	5.0	29.2	29.0
Average Merging W/ DARE	31.9 -0.3	5.0 +0.0	29.8 +0.6	28.6 -0.4
Average Merging W/ PAVE	32.4 +0.2	5.0 +0.0	31.1 +1.9	29.0 +0.0

Table 3: Results of merging fine-tuned LLaMA-2-7B models on Math and Code tasks. Model merging on generative tasks is particularly challenging, as it requires preserving the full reasoning ability across the entire generated response rather than making an instance-level prediction, and only a limited number of existing studies have explored this setting.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	CoLA	SST-2	MRPC	STSB	QQP	MNLI	QNLI	RTE	Average
Individual	59.10	95.06	89.21	91.31	91.50	88.52	93.30	69.31	84.66
EMR-Merging	47.05	94.72	71.32	87.57	88.36	87.38	92.29	64.26	79.11
EMR-Merging W/ PAVE plain	59.88	94.61	82.60	88.83	90.40	87.35	92.44	58.48	81.82
EMR-Merging W/ PAVE	61.32	94.50	84.31	89.54	90.34	87.50	93.01	60.65	82.64
Task Arithmetic	3.03	75.00	68.38	52.72	63.44	43.46	51.16	50.90	51.01
Task Arithmetic W/ PAVE plain	2.56	74.73	68.38	54.83	64.03	42.84	60.19	51.16	52.34

Appendix

Table 4: Results of merging fine-tuned DeBERTa models on eight datasets from GLUE benchmark.

Methods	CoLA	SST-2	MRPC	STSB	QQP	MNLI	QNLI	RTE	Average
Individual	59.10	95.06	89.21	91.31	91.50	88.52	93.30	69.31	84.66
EMR-Merging	47.05	94.72	71.32	87.57	88.36	87.38	92.29	64.26	79.11
EMR-Merging W/ SVD	62.20	93.58	83.82	89.40	89.45	76.72	92.53	58.48	80.77
EMR-Merging W/ ASVD	62.33	95.07	84.56	89.73	89.98	87.44	92.86	61.01	82.87
EMR-Merging W/ SVD-LLM	61.82	94.72	84.07	89.55	90.10	87.44	92.90	61.37	82.74
EMR-Merging W/ PAVE	61.32	94.50	84.31	89.54	90.34	87.50	93.01	60.65	82.64

Appendix

Table 5: Results of merging fine-tuned DeBERTa models on eight datasets from GLUE benchmark.

Panel of Figure 4 — Figure 4: Performance and runtime analysis of PAVE across different sample sizes. Top: Mean performance (solid line) and standard deviation (shaded region) over 3 random seeds of EMR-Merging with PAVE, showing consistent improvements as more samples are used. A detailed ablation has been included in Table 6 . Bottom: PAVE runtime grows with number of samples.

Methods	SUN397	Cars	RESISC45	EuroSAT	SVHN	GTSRB	MNIST	DTD	Average
EMR-Merging W/ PAVE N=4096	76.3 $\pm<0.1$	74.3 $\pm<0.1$	93.8 $\pm<0.1$	99.3 $\pm<0.1$	96.7 $\pm<0.1$	97.6 $\pm<0.1$	99.6 $\pm<0.1$	76.1 $\pm<0.1$	89.2 $\pm<0.1$
EMR-Merging W/ PAVE N=2048	76.3 $\pm<0.1$	71.5 $\pm<0.1$	94.2 $\pm<0.1$	99.2 $\pm<0.1$	96.7 $\pm<0.1$	97.6 $\pm<0.1$	99.6 $\pm<0.1$	75.0 $\pm<0.1$	88.8 $\pm<0.1$
EMR-Merging W/ PAVE N=1024	76.5 $\pm<0.1$	72.5 $\pm<0.1$	94.2 $\pm<0.1$	99.1 $\pm<0.1$	96.7 $\pm<0.1$	97.7 $\pm<0.1$	99.6 $\pm<0.1$	74.8 $\pm<0.1$	88.9 $\pm<0.1$
EMR-Merging W/ PAVE N=512	76.7 $\pm<0.1$	74.7 $\pm<0.1$	94.0 $\pm<0.1$	99.3 $\pm<0.1$	96.7 $\pm<0.1$	97.8 $\pm<0.1$	99.5 $\pm<0.1$	74.4 $\pm<0.1$	89.1 $\pm<0.1$
EMR-Merging W/ PAVE N=256	76.2 $\pm<0.1$	74.1 $\pm<0.1$	93.5 $\pm<0.1$	99.1 $\pm<0.1$	96.3 $\pm<0.1$	97.1 $\pm<0.1$	99.2 $\pm<0.1$	76.3 $\pm<0.1$	89.0 $\pm<0.1$
EMR-Merging W/ PAVE N=128	75.8 $\pm<0.1$	74.1 $\pm<0.1$	93.4 $\pm=0.1$	98.9 $\pm<0.1$	96.4 $\pm<0.1$	97.0 $\pm<0.1$	99.5 $\pm<0.1$	74.6 $\pm=0.1$	88.7 $\pm=0.1$

Appendix

Table 6: Detailed ablation on Figure 4 .

Methods	Average Performance	Time (min)
WUDI	68.04	2.1
WUDI W/ PAVE	71.29 +3.25	4.4 +2.3

Appendix

Table 7: Comparison of computational cost and average performance. Adding PAVE to WUDI improves the average performance while introducing an acceptable increase in computation time.

Methods	Mathematical Solving		Code Generating
Methods	GSM8K	MATH	Human Eval	MBPP
Individual Math	54.9	10.7	-	-
Individual Code	-	-	39.6	30.1
Average Merging	32.2	5.0	29.2	29.0
Average Merging W/ DARE	31.9 -0.3	5.0 +0.0	29.8 +0.6	28.6 -0.4
Average Merging w/ PAVE	32.4 +0.2	5.0 +0.0	31.1 +1.9	29.0 +0.0

Appendix

Table 8: Results of merging fine-tuned LLaMA-2-7B models on Math and Code tasks.

Methods	SUN397	Cars	RESISC45	EuroSAT	SVHN	GTSRB	MNIST	DTD	Average
Individual ViT-B/32	75.3	77.7	96.1	99.7	97.5	98.7	99.7	79.4	90.5
EMR-Merging	75.2	72.8	93.5	99.5	96.9	98.1	99.6	74.4	88.7
EMR-Merging W/ PAVE	76.3	74.3	93.8	99.3	96.7	97.6	99.6	76.1	89.2 +0.5
Individual ViT-B/16	79.0	87.0	96.4	99.1	97.7	99.0	99.7	82.3	92.5
EMR-Merging	78.5	82.6	95.4	99.1	97.6	98.8	99.6	78.3	91.2
EMR-Merging W/ PAVE	80.2	82.8	96.1	99.2	97.6	98.7	99.6	78.9	91.7 +0.5

Appendix

Table 9: Results of merging fine-tuned ViT models on 8 image classification benchmarks.

Panel of Table 10 — Table 10: Normalized assigned pruning ranks across layers

Task	CoLA	SST-2	MRPC	STSB	QQP	MNLI	QNLI	RTE
Assigned Rank / Uniform Rank	0.96	1.21	0.84	1.17	1.13	1.09	0.87	0.72

Appendix

Table 11: Average normalized assigned pruning ranks for different downstream tasks.

WizardMath-7B-V1.0 on GSM8K	SVD	CO-SVD
Pruning rank = 16	48.59	51.32
Pruning rank = 64	47.08	50.94
Pruning rank = 128	44.50	50.49
Pruning rank = 256	36.16	49.35

Appendix

Table 12: Comparison of SVD and CO-SVD under different pruning ranks for WizardMath-7B-V1.0 on GSM8K.

Model Merging

Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.

See all 192 papers

PACT: Preserving Anchored Cores in Task-vectors for Model Merging
Jun 17, 2026Ningyuan Shi, Zhipeng Zhou, Hao Wang +2Multi-Task Learning Task Vector Merging
Shanghai Jiao Tong University · Nanyang Technological University · The Hong Kong University of Science and Technology (Guangzhou)
Model Merging in the Essential Subspace
Feb 23, 2026Longhua Li, Lei Qi, Qi Tian +1Task Interference Model Merging
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shanghai, China
GRAM: Group-wise Rank-Aware Model Merging via Subspace Alignment
Oct 21, 2026Di Wu, Shuaidong PanLanguage Model Merging Model Merging
JPMorgan Chase & Co.

Jun 17, 2026J/K move · Enter open · S save

PACT: Preserving Anchored Cores in Task-vectors for Model Merging

Ningyuan Shi, Zhipeng Zhou, Hao Wang, Chunyan Miao+1

Shanghai Jiao Tong University · Nanyang Technological University · The Hong Kong University of Science and Technology (Guangzhou)

Multi-Task Learning Task Vector Merging Model Merging Latent Space Alignment

Open paper AskPDF arXiv

Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model. Most existing model merging approaches follow the Task Arithmetic paradigm, which decomposes fine-tuned weights into pre-trained parameters and task vectors, and performs merging exclusively in the task-vector space. The effectiveness of this paradigm implicitly relies on the assumption that task-specific knowledge is encoded solely within task vectors. We argue that this assumption generally does not hold due to the intrinsic task preferences of pre-trained models. Specifically, we identify Load-Bearing Wall (LBW) dimensions, namely some task-critical knowledge that remains embedded in the pre-trained weights rather than being fully transferred into task vectors. We characterize LBW dimensions from both scalar-weight and subspace perspectives, thereby covering the major paradigms of existing model merging methods. Our analysis reveals that, by ignoring LBW dimensions, task-vector-based approaches fail to fully resolve task conflicts and may inadvertently damage task-specific knowledge encoded in the pre-trained model, leading to degradation. To address this issue, we propose PACT, which preserves the anchored task-specific cores (i.e., LBW dimensions) within task vectors by aligning their orthogonal complements with the subspace of the pre-trained weights. These aligned subspace components are then removed from the task vectors before applying existing model merging algorithms. Furthermore, we develop an efficient variant based on randomized SVD to improve scalability. PACT can be seamlessly integrated with existing methods. Extensive experiments across multiple benchmarks demonstrate that PACT consistently enhances mainstream model merging approaches and establishes new state-of-the-art performance.

Abstract

Model Merging

Explore similar work

PACT: Preserving Anchored Cores in Task-vectors for Model Merging

Model Merging in the Essential Subspace

GRAM: Group-wise Rank-Aware Model Merging via Subspace Alignment