Separating Common and Unique Directions for Model Merging

Shiting Wang; Yingjie Zou; Ke Li

cs.LGNov 3, 2026

Separating Common and Unique Directions for Model Merging

Authors: Shiting Wang, Yingjie Zou, Ke Li

Organizations: Earlham Institute, United Kingdom · Living Systems Institute, University of Exeter, United Kingdom · Department of Computer Science, University of Exeter, United Kingdom · Department of Engineering, University of Exeter, United Kingdom ·

Abstract

Model merging combines multiple fine-tuned models without additional training, and arithmetic methods have emerged as a simple yet effective way to merge models without data. However, existing arithmetic methods treat task vectors in parameter space and apply a single element-wise rule to directions that can differ in mergeability. In a shared directional basis, we find a consistent two-regime structure: a low-dispersion common bulk shared across tasks and a high-dispersion task-dominated tail that carries disproportionate energy. These findings motivate Common--Unique Subspace Decomposition (CUSD), a data-free pre-merging method that separates task vectors into common and unique directional components, merges the common components with a downstream arithmetic operator, and retains unique residuals separately. Experiments across $11$ base models, four model families, and four arithmetic-based merging operators show that CUSD improves 42 of 44 model--operator combinations and achieves up to 22.5% improvement over the corresponding baselines. Our code is publicly available at https://github.com/stwang666/CUSD

Figures & tables

Figure 1: Direction-level heterogeneity in task vectors. Motivating examples from representative matrix layers of Llama-3.2-3B ( mlp.gate_proj ) and ViT-B/16 ( mlp.c_proj ). (a, c) Distribution of cross-task dispersion scores over shared directions. (b, d) Share of total directional energy in each dispersion-ranked decile.

Figure 2: CUSD pipeline overview. Task vectors

A_{i}

are projected into an HO-GSVD-derived shared right basis

V

and directional energies

\Sigma

. A std-based dispersion score with a KDE-based threshold partitions directions into common (

\mathcal{C}

) and unique (

\mathcal{U}

). CUSD merges common components with

\mathcal{F}

and retains unique residuals through

\beta\sum_{i=1}^{K}R_{i}

Method	with CUSD	ViT-B/32	ViT-B/16	ViT-L/14	RoBERTa-B	RoBERTa-L	Llama-3B	Llama-8B	Llama-3B-Inst	Llama-8B-Inst	Gemma-2b	Gemma-9b
Fine-tuned	—	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Task Arithmetic	$\times$	71.8	76.8	87.6	53.5	58.3	69.6	93.7	73.5	82.5	94.2	92.0
Task Arithmetic	$\checkmark$	85.0 (+13.2)	88.9 (+12.1)	93.0 (+5.4)	68.7 (+15.1)	70.0 (+11.7)	69.6 (0.0)	94.1 (+0.4)	75.8 (+2.3)	86.5 (+4.0)	95.7 (+1.5)	94.8 (+2.8)
TIES-Merging	$\times$	74.0	78.5	88.5	51.1	47.3	69.2	93.7	72.7	82.6	94.1	91.1
TIES-Merging	$\checkmark$	87.0 (+13.0)	90.3 (+11.8)	94.8 (+6.3)	69.3 (+18.2)	69.8 (+22.5)	69.7 (+0.5)	94.1 (+0.4)	75.5 (+2.8)	86.2 (+3.6)	97.3 (+3.2)	95.3 (+4.2)
DARE	$\times$	71.7	76.6	87.5	54.0	58.8	68.9	94.1	73.4	83.1	94.1	91.2

Table 1: Overall performance across all models. We report average normalized scores.

Table 4

Method	Llama-3.2-3B				Gemma-2-2b
Method	w/o CUSD	CUSD ( $s_{\mathrm{abs}}$ )	CUSD ( $s_{\mathrm{rel}}$ )	$s_{\mathrm{rel}}$ - $s_{\mathrm{abs}}$	w/o CUSD	CUSD ( $s_{\mathrm{abs}}$ )	CUSD ( $s_{\mathrm{rel}}$ )	$s_{\mathrm{rel}}$ - $s_{\mathrm{abs}}$
Task Arithmetic	69.6	70.0 (+0.4)	69.6 (+0.0)	-0.4	94.2	96.9 (+2.7)	95.7 (+1.5)	-1.2
TIES-Merging	69.2	69.5 (+0.3)	69.7 (+0.5)	0.2	94.1	95.9 (+1.8)	97.3 (+3.2)	1.4
DARE	68.9	69.6 (+0.7)	70.3 (+1.4)	0.7	94.1	95.8 (+1.7)	96.2 (+2.1)	0.4
Consensus	67.4	70.0 (+2.6)	70.6 (+3.2)	0.6	88.8	91.7 (+2.9)	93.1 (+4.3)	1.4

Table 4: Absolute vs. relative dispersion scoring on decoder LLMs. We report normalized average performance; green values denote gains over the corresponding plain merge.

Model	Random	Reversed	Top-energy (matched)	Top-energy (0.2%)	Top-energy (0.5%)	Top-energy (1.0%)	CUSD
ViT-B/32	$59.20\pm 0.08$	59.57	73.78	74.75	73.68	70.97	76.48
ViT-B/16	$65.64\pm 1.17$	65.09	81.07	80.84	80.34	78.97	81.74

Table 5: Direction-selection ablation for CUSD+TA. Average accuracy (%) is reported; Random shows mean

\pm

standard deviation over three seeds.

Table 8

Figure 7: Per-task directional singular values for representative ViT, Llama, and Gemma layers. Each panel shows the HO-GSVD singular values

\sigma_{ik}

of task vectors along the top shared directions.

Panel of Figure 8 — Figure 8: Distribution of dispersion scores under absolute and relative scoring. Mean normalization changes the dispersion distribution, exposing scale-suppressed task imbalance in the evaluated LLM settings while amplifying low-energy relative fluctuations in the evaluated encoder settings.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Direction-level heterogeneity in task vectors. Directional energy structure across all evaluated model families.

Method	with CUSD	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Avg.
Fine-tuned	—	92.5	84.2	91.0	99.2	99.7	99.7	83.6	98.1	93.5
Task Arithmetic	$\times$	80.5	64.6	75.6	86.7	99.0	86.1	72.9	89.7	81.9
Task Arithmetic	$\checkmark$	87.2 (+6.7)	71.9 (+7.2)	87.7 (+12.0)	92.6 (+6.0)	98.7 (-0.3)	94.5 (+8.4)	76.1 (+3.2)	87.2 (-2.5)	87.0 (+5.1)
TIES-Merging	$\times$	81.1	65.3	77.8	87.5	99.1	87.2	73.3	90.5	82.7
TIES-Merging	$\checkmark$	87.7 (+6.6)	74.5 (+9.2)	88.4 (+10.7)	95.0 (+7.5)	99.0 (-0.1)	95.7 (+8.5)	77.2 (+3.9)	91.2 (+0.8)	88.6 (+5.9)
DARE	$\times$	80.4	64.0	75.6	86.7	99.0	86.1	72.8	89.7	81.8

Appendix

Table 8: Multi-task performance of merged ViT-L/14 models on eight tasks.

Method	with CUSD	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Avg.
Fine-tuned	—	86.5	82.1	90.2	99.2	99.8	99.7	80.6	97.9	92.0
Task Arithmetic	$\times$	59.0	51.4	64.9	76.3	98.5	71.3	54.3	89.2	70.6
Task Arithmetic	$\checkmark$	75.6 (+16.5)	62.1 (+10.7)	85.7 (+20.8)	87.1 (+10.9)	97.9 (-0.5)	89.2 (+17.9)	71.6 (+17.3)	84.7 (-4.5)	81.7 (+11.1)
TIES-Merging	$\times$	60.7	52.3	68.5	77.1	98.8	73.4	56.6	90.2	72.2
TIES-Merging	$\checkmark$	74.7 (+14.0)	65.2 (+12.9)	86.8 (+18.3)	90.3 (+13.3)	98.4 (-0.4)	90.2 (+16.8)	70.7 (+14.1)	88.4 (-1.9)	83.1 (+10.9)
DARE	$\times$	59.2	51.3	64.8	75.6	98.4	71.3	54.2	89.2	70.5

Appendix

Table 9: Multi-task performance of merged ViT-B/16 models on eight tasks.

Method	with CUSD	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Avg.
Fine-tuned	—	78.0	79.4	88.9	98.7	99.7	99.7	78.3	97.5	90.0
Task Arithmetic	$\times$	46.4	48.5	64.2	71.4	97.2	64.1	46.2	79.3	64.7
Task Arithmetic	$\checkmark$	65.1 (+18.8)	60.4 (+12.0)	81.9 (+17.7)	79.8 (+8.5)	96.5 (-0.7)	83.4 (+19.3)	65.5 (+19.4)	79.3 (0.0)	76.5 (+11.9)
TIES-Merging	$\times$	48.8	49.7	66.9	72.8	97.7	66.7	49.0	81.3	66.6
TIES-Merging	$\checkmark$	65.6 (+16.8)	63.7 (+14.0)	83.8 (+16.9)	83.7 (+10.9)	96.8 (-0.9)	87.1 (+20.4)	66.1 (+17.1)	79.8 (-1.5)	78.3 (+11.7)
DARE	$\times$	45.9	48.6	64.2	71.4	97.2	63.7	45.9	79.1	64.5

Appendix

Table 10: Multi-task performance of merged ViT-B/32 models on eight tasks.

Method	with CUSD	CoLA	MNLI	MRPC	QNLI	QQP	RTE	SST-2	STS-B	Avg.
Fine-tuned	—	55.1	87.5	88.0	92.4	89.4	67.5	93.8	88.3	82.8
Task Arithmetic	$\times$	0.0	64.6	79.5	75.5	71.3	49.1	71.1	-56.8	44.3
Task Arithmetic	$\checkmark$	6.6 (+6.6)	63.6 (-1.0)	78.2 (-1.3)	74.1 (-1.4)	37.6 (-33.7)	50.9 (+1.8)	87.7 (+16.6)	55.9 (+112.6)	56.8 (+12.5)
TIES-Merging	$\times$	0.0	64.9	79.0	73.2	76.4	46.2	65.9	-67.2	42.3
TIES-Merging	$\checkmark$	6.6 (+6.6)	62.1 (-2.8)	77.9 (-1.1)	63.6 (-9.6)	36.6 (-39.8)	53.4 (+7.2)	90.3 (+24.3)	68.4 (+135.6)	57.4 (+15.1)
DARE	$\times$	4.6	64.6	79.4	75.6	71.7	48.7	71.3	-58.6	44.7

Appendix

Table 11: Multi-task performance of merged RoBERTa-Base models on the 8-task GLUE benchmark.

Method	with CUSD	CoLA	MNLI	MRPC	QNLI	QQP	RTE	SST-2	STS-B	Avg.
Fine-tuned	—	59.1	90.5	89.0	94.5	90.5	70.0	96.6	91.7	85.2
Task Arithmetic	$\times$	0.2	75.7	48.6	69.5	82.1	58.8	85.1	-22.4	49.7
Task Arithmetic	$\checkmark$	9.1 (+8.8)	72.2 (-3.4)	27.9 (-20.8)	83.5 (+14.0)	59.5 (-22.7)	70.8 (+11.9)	92.7 (+7.6)	62.0 (+84.3)	59.7 (+10.0)
TIES-Merging	$\times$	-1.4	71.0	56.3	53.3	84.6	54.5	72.1	-67.9	40.3
TIES-Merging	$\checkmark$	5.7 (+7.1)	74.5 (+3.5)	29.1 (-27.2)	85.3 (+32.0)	58.6 (-26.0)	70.0 (+15.5)	92.6 (+20.4)	60.3 (+128.2)	59.5 (+19.2)
DARE	$\times$	2.4	75.6	48.8	70.2	82.1	59.2	85.7	-23.2	50.1

Appendix

Table 12: Multi-task performance of merged RoBERTa-Large models on the 8-task GLUE benchmark.

Method	with CUSD	Humaneval+	MBPP+	GSM8k	M-MMLU	M-ARC	M-HellaSwag	Avg.
Fine-tuned	—	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Task Arithmetic	$\times$	45.7	59.4	39.2	87.0	93.4	93.0	69.6
Task Arithmetic	$\checkmark$	45.7 (0.0)	52.8 (-6.6)	45.8 (+6.6)	86.0 (-1.0)	94.0 (+0.6)	93.2 (+0.2)	69.6 (0.0)
TIES-Merging	$\times$	46.1	58.5	37.9	87.2	93.1	92.6	69.2
TIES-Merging	$\checkmark$	45.2 (-0.9)	53.3 (-5.2)	47.1 (+9.2)	85.8 (-1.4)	94.0 (+0.9)	93.0 (+0.4)	69.7 (+0.5)
DARE	$\times$	47.3	53.8	38.8	87.2	93.3	93.0	68.9

Appendix

Table 13: Performance of merged Llama-3.2-3B models. The task-specific checkpoints are fine-tuned for coding, math, and multilingual tasks; normalized scores and their average are reported.

Method	with CUSD	Humaneval+	MBPP+	GSM8k	M-MMLU	M-ARC	M-HellaSwag	Avg.
Fine-tuned	—	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Task Arithmetic	$\times$	82.5	100.0	93.7	95.5	89.7	100.8	93.7
Task Arithmetic	$\checkmark$	82.6 (+0.1)	102.4 (+2.4)	91.3 (-2.4)	96.1 (+0.6)	90.9 (+1.2)	101.3 (+0.5)	94.1 (+0.4)
TIES-Merging	$\times$	82.6	100.8	94.2	95.2	88.8	100.4	93.7
TIES-Merging	$\checkmark$	83.0 (+0.4)	104.0 (+3.2)	88.2 (-6.0)	97.1 (+1.9)	91.2 (+2.4)	101.2 (+0.8)	94.1 (+0.4)
DARE	$\times$	84.6	101.2	92.7	95.1	90.3	100.7	94.1

Appendix

Table 14: Performance of merged Llama-3.1-8B models. The task-specific checkpoints are fine-tuned for coding, math, and multilingual tasks; normalized scores and their average are reported.

Method	with CUSD	Humaneval+	MBPP+	GSM8k	M-MMLU	M-ARC	M-HellaSwag	Avg.
Fine-tuned	—	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Task Arithmetic	$\times$	55.5	46.5	64.2	85.5	96.2	93.3	73.5
Task Arithmetic	$\checkmark$	61.3 (+5.8)	54.0 (+7.5)	62.4 (-1.8)	86.4 (+0.9)	97.2 (+1.0)	93.6 (+0.3)	75.8 (+2.3)
TIES-Merging	$\times$	53.8	44.3	63.9	85.1	95.9	93.4	72.7
TIES-Merging	$\checkmark$	58.0 (+4.2)	57.5 (+13.2)	61.1 (-2.8)	86.4 (+1.3)	96.4 (+0.5)	93.6 (+0.2)	75.5 (+2.8)
DARE	$\times$	51.7	50.0	64.6	85.0	95.9	93.3	73.4

Appendix

Table 15: Performance of merged Llama-3.2-3B-Instruct models. The task-specific checkpoints are fine-tuned for coding, math, and multilingual tasks; normalized scores and their average are reported.

Method	with CUSD	Humaneval+	MBPP+	GSM8k	M-MMLU	M-ARC	M-HellaSwag	Avg.
Fine-tuned	—	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Task Arithmetic	$\times$	55.5	87.9	87.6	84.2	87.3	92.4	82.5
Task Arithmetic	$\checkmark$	70.7 (+15.2)	91.4 (+3.5)	83.8 (-3.8)	88.3 (+4.1)	90.7 (+3.4)	94.0 (+1.6)	86.5 (+4.0)
TIES-Merging	$\times$	56.9	87.9	87.8	84.5	86.7	92.1	82.6
TIES-Merging	$\checkmark$	68.8 (+11.9)	92.6 (+4.7)	85.2 (-2.6)	87.3 (+2.8)	89.6 (+2.9)	93.8 (+1.7)	86.2 (+3.6)
DARE	$\times$	59.7	87.5	87.9	83.8	87.3	92.3	83.1

Appendix

Table 16: Performance of merged Llama-3.1-8B-Instruct models. The task-specific checkpoints are fine-tuned for coding, math, and multilingual tasks; normalized scores and their average are reported.

Method	with CUSD	Humaneval+	MBPP+	GSM8k	M-MMLU	M-ARC	M-HellaSwag	Avg.
Fine-tuned	—	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Task Arithmetic	$\times$	80.6	94.9	75.0	106.8	104.0	104.0	94.2
Task Arithmetic	$\checkmark$	83.7 (+3.1)	98.0 (+3.1)	76.4 (+1.4)	107.4 (+0.6)	104.5 (+0.5)	104.1 (+0.1)	95.7 (+1.5)
TIES-Merging	$\times$	80.4	90.3	79.5	107.1	103.3	103.8	94.1
TIES-Merging	$\checkmark$	90.8 (+10.4)	94.4 (+4.1)	83.2 (+3.7)	107.6 (+0.5)	104.0 (+0.7)	103.9 (+0.1)	97.3 (+3.2)
DARE	$\times$	80.6	91.8	76.7	107.6	104.0	104.2	94.1

Appendix

Table 17: Performance of merged Gemma-2-2b models. The task-specific checkpoints are fine-tuned for coding, math, and multilingual tasks; normalized scores and their average are reported.

Method	with CUSD	Humaneval+	MBPP+	GSM8k	M-MMLU	M-ARC	M-HellaSwag	Avg.
Fine-tuned	—	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Task Arithmetic	$\times$	73.5	102.8	105.0	96.3	83.4	90.7	92.0
Task Arithmetic	$\checkmark$	72.4 (-1.1)	106.8 (+4.0)	101.3 (-3.7)	99.5 (+3.2)	90.7 (+7.3)	98.1 (+7.4)	94.8 (+2.8)
TIES-Merging	$\times$	71.5	102.0	105.0	96.2	81.8	90.3	91.1
TIES-Merging	$\checkmark$	71.5 (0.0)	109.2 (+7.2)	99.6 (-5.4)	100.3 (+4.1)	91.5 (+9.7)	99.6 (+9.3)	95.3 (+4.2)
DARE	$\times$	72.8	99.6	105.5	96.0	82.9	90.3	91.2

Appendix

Table 18: Performance of merged Gemma-2-9b models. The task-specific checkpoints are fine-tuned for coding, math, and multilingual tasks;normalized scores and their average are reported.

Figure 11: ViT-B/16 directional singular-value heatmaps. Each panel shows

\log_{10}\sigma_{i,k}

for one layer component, with directions sorted by energy.

Figure 12: Llama-3.2-3B directional singular-value heatmaps. Each panel shows

\log_{10}\sigma_{i,k}

for one layer component, with directions sorted by energy.

Figure 13: Per-method sensitivity to

\beta

across downstream arithmetic-based merging operators.

$\lambda\backslash\beta$	0.0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0	Plain TA
0.1	57.43	61.74	63.33	62.22	59.15	53.95	47.22	39.87	33.62	28.21	23.91	61.74
0.2	66.62	69.07	69.36	67.35	62.75	56.56	49.11	41.37	34.97	29.31	24.91	69.36
0.3	72.49	73.87	72.94	70.05	64.59	57.76	49.88	42.29	35.91	30.36	25.85	70.05
0.4	76.48 †	76.50	74.65	70.48	64.65	57.43	49.84	42.52	36.31	31.18	26.94	64.65 †
0.5	77.64	77.02	74.15	69.70	63.45	56.27	49.03	42.36	36.58	31.94	27.80	56.27
0.6	77.33	75.82	72.47	67.53	61.25	54.54	47.97	41.74	36.49	32.35	28.43	47.97

Appendix

Table 19: Joint sensitivity to the Task Arithmetic scale

\lambda

and unique-residual coefficient

\beta

on ViT-B/32. We report average accuracy (%) from a post-hoc sweep on the test sets. The final column reports plain TA at the same

\lambda

; bold entries outperform this baseline. † denotes the hyperparameter configuration used in the original submission.

Panel of Figure 14 — Figure 14: Per-block interference analysis for Task Arithmetic on ViT models.

Panel of Figure 15 — Figure 15: Per-block interference analysis for TIES-Merging on ViT models.

Panel of Figure 16 — Figure 16: Per-block interference analysis for Consensus Merging on ViT models.

Data-Free Model Merging

See all 8 papers

Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion
Sep 19, 2026Aditya Pola, Vineeth N. BalasubramanianModel Merging
IIT Hyderabad · Microsoft Research, India
A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic
Oct 6, 2026Sin-Han Yang, Shih-Cheng Huang, Chieh-Yen Lin +3Task Vector Merging Multi-Task Optimization
Appier AI Research · National Taiwan University
Model Merging: Foundations and Algorithms
May 2, 2026Donato CrisostomiMulti-Task Learning Task Vector Merging
SAPIENZA UNIVERSITY OF ROME · GLADIA research lab · Department of Computer Science +1

Sep 19, 2026J/K move · Enter open · S save

Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion

Aditya Pola, Vineeth N. Balasubramanian

IIT Hyderabad · Microsoft Research, India

Model Merging

Open paper AskPDF arXiv

Model merging consolidates fine-tuned experts into one multi-task model without retraining. All existing data-free methods approach this problem entirely in weight space. Restricted to arithmetic on parameters, these methods never observe how each expert behaves, a signal that only emerges through forward evaluation. Accessing this behavioral signal requires inputs to evaluate on, which the data-free setting prohibits. We propose Merge++, a post-hoc method that addresses this by inverting the expert checkpoints to synthesize task-representative images, then distilling expert knowledge into the merged model using those images. Merge++ requires no additional data beyond the checkpoints themselves. It applies universally across merging algorithms and operates as a complementary refinement stage independent of the underlying weight-space method. The method consistently improves merging algorithms ranging from simple task arithmetic to state-of-the-art spectral methods, with average gains of +2 to +8 points and up to +25.9 on individual configurations.

Abstract

Data-Free Model Merging

Explore similar work

Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion

A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic

Model Merging: Foundations and Algorithms