CORAM: Coherent Orthogonal Rotation for Model Merging

Xinyi Sui; Ziran Liu; Nam Ling; Wei Wang; Wei Jiang

cs.LGAug 18, 2026

CORAM: Coherent Orthogonal Rotation for Model Merging

Authors: Xinyi Sui, Ziran Liu, Nam Ling, Wei Wang, Wei Jiang

Organizations: Santa Clara University · Shanghai Institute for Mathematics and Interdisciplinary Sciences / Fudan University · Futurewei Technologies, Inc.

Abstract

Merging finetuned models combines specialized capabilities without joint training or access to the original data. Most methods operate by linear arithmetic in Euclidean weight space, which cannot carry the geometry of the update. Orthogonal Model Merging (OrthoMerge) uses a single orthogonal transform for each weight matrix, but such a transform cannot change singular values. We propose CORAM, which partitions each target matrix into row slices, represents every expert slice by its singular value decomposition in the corresponding base-model SVD frame, and merges the task-specific factors on their corresponding manifolds. Because manifold averaging contracts the merged update, CORAM applies an amplification coefficient $λ=κ\hat{c}$ . The scale c_hat is estimated from the expert and merged update norms and is approximately $\sqrt{N}$ for $N$ experts with comparable update magnitudes. The restoration strength kappa is selected from the dispersion of expert updates without evaluating candidate merged models. This rule remains within 0.72 points of the best swept value on all evaluated suites. CORAM also includes spread slicing to distribute highly updated rows across slices and a residual pathway for non-target layers. Across four suites covering three model families, 3B to 9B scales, and language and vision-language experts, CORAM improves over OrthoMerge by 0.25 to 1.35 points and matches or exceeds the strongest weight-space baselines.

Figures & tables

Figure 1: Overview of CORAM. Each expert slice is factored in the base frame into a rotation, a spectral shift $\mathit{dS}_{i}=S_{i}S_{0}^{-1}-I$ , and a right factor. Each factor is averaged in its own space. The coefficient $\lambda$ is given by Eq. ( 3 ).

	In-domain					In-domain Avg		Out-of-domain
Method	MATH500	HEval $+$	SciQA	CSQA	SIQA	$\kappa$ -rule	best $\lambda$	M-ARC	AGIEval	OOD Avg
Llama-3.1-8B	17.80	21.52	71.40	70.60	48.11	45.89		33.02	30.27	31.65
Task-specific FT	19.00	38.54	91.82	82.47	56.81	57.73		–	–	–
Linear	23.40	32.26	83.59	76.82	51.84	53.58		34.41	32.27	33.34
Task Arithmetic	24.80	37.20	86.29	79.52	53.33	56.23		34.97	32.68	33.83
TIES	21.40	40.43	82.87	78.30	54.04	55.41		34.40	32.53	33.47

Table 1: Top–T1 : merging five orthogonally finetuned Llama-3.1-8B experts. Bottom–T2 : merging five fully finetuned Llama-3.2-3B experts from MergeBench. Higher is better. Per column, the best result among merging methods is in bold (base-model and task-specific-FT reference rows excluded). CORAM-C denotes the conflict-aware variants (masking neurons where experts disagree).

+

SS adds spread slicing (a task-informed row permutation) and

+

RP the residual pathway (conventional merge of the non-target layers). CORAM rows evaluate the search-free

\kappa

-rule checkpoint (grid

\lambda{=}2.60

for both suites). “best

\lambda

” is the sweep optimum, for reference (see Setup).

	In-domain						In-domain Avg		Out-of-domain
Method	MMSI	EmbSp.	MMMU Med	PathVQA	OCRB.	CharXiv	$\kappa$ -rule	best $\lambda$	IFEval	MMB.	OOD Avg
Qwen2.5-VL-7B-It.	27.80	69.97	53.10	66.30	84.70	67.20	61.51		63.03	83.93	73.48
Task-specific FT	32.60	70.58	55.17	66.81	85.00	72.50	63.78		–	–	–
Linear	29.20	71.29	55.17	68.47	84.80	67.30	62.71		58.23	84.19	71.21
Task Arithmetic	29.10	71.07	55.86	68.38	84.60	66.10	62.52		59.33	84.19	71.76
TIES	32.10	71.54	57.93	68.44	82.80	69.40	63.70		54.53	84.02	69.27

Table 2: T3 : merging three Qwen2.5-VL-7B-Instruct vision–language experts. Layout and bolding as in Table 1 .

\kappa

-rule grid point

\lambda{=}2.00

. CORAM-C denotes the conflict-aware variants (masking neurons where experts disagree).

+

RP adds the residual pathway (conventional merge of the non-target layers). The OrthoMerge out-of-domain entries are obtained from our zero-shot re-evaluation.

	In-domain					In-domain Avg		Out-of-domain
Method	Instr.	Math	Coding	Multi.	Safety	$\kappa$ -rule	best $\lambda$	MMLU †	AGIEval	OOD Avg
Gemma-2-9B	14.23	69.83	43.37	54.63	34.39	43.29		70.41	38.01	54.21
Task-specific FT	65.06	79.76	58.51	55.91	76.00	67.05		–	–	–
Linear	27.17	81.05	51.52	53.79	59.31	54.57		67.88	37.75	52.81
Task Arithmetic	27.54	82.34	51.65	51.31	54.26	53.42		64.44	35.70	50.07
TIES	24.03	82.94	46.60	45.64	52.84	50.41		60.60	34.92	47.76

Table 3: T4 : merging five fully finetuned Gemma-2-9B experts from MergeBench with uneven update magnitudes (

D{\approx}16

). The rule selects

\kappa{=}0.5

, corresponding to

\lambda{=}1.10

. Layout and bolding follow Table 1 . CORAM-C masks neurons where experts disagree.

+

SS adds a task-informed row permutation, and

+

RP merges the non-target layers.

Configuration	T1	T2	T3	T4
CORAM	57.72	42.75	63.95	54.49
$+$ RP	n/a	42.31	64.38	54.56
$+$ SS	57.84	42.93	63.96	54.15
$+$ SS $+$ RP	n/a	43.30	64.27	54.71
CORAM-C	57.60	42.96	64.02	54.32
$+$ RP	n/a	43.32	64.20	54.37

Table 4: Best in-domain average reached at the

\kappa

-rule point (best configuration per method and suite). RP is inapplicable on T1 (no non-target residual).

Appendix figures & tables45 assets

Supplementary material from the paper’s appendix.

Appendix

	Rotation log (all slices)			Cond. (contiguous)			Cond. (spread)
Suite	Mean err	Max err	$\theta{>}3$	Median	p95	Max	p95	Max
T1	$8.8{\times}10^{-16}$	$1.9{\times}10^{-15}$	0	1.3	2.3	706	2.8	38
T2	$8.8{\times}10^{-16}$	$2.2{\times}10^{-15}$	0	1.3	2.6	822	3.1	28
T3	$8.8{\times}10^{-16}$	$1.9{\times}10^{-15}$	0	1.3	5.2	635	5.3	282
T4	$1.4{\times}10^{-15}$	$5.1{\times}10^{-10}$	64	1.3	2.0	43	2.5	18

Appendix

Table 1: Numerical audit of the geometric operators at

h=8

, over every slice of every expert (0.48M–1.16M rotations per suite). Rotation-logarithm reconstruction errors are at machine precision; the near-

\pi

fallback fires on 64 of 1.16M rotations on T4 only and is handled by the exact eigendecomposition path. No slice has

\sigma_{\min}<10^{-8}

. Variance-spread slicing leaves the median condition number unchanged but collapses the tail (maximum

706\!\to\!38

822\!\to\!28

635\!\to\!282

43\!\to\!18

), confirming its conditioning motivation.

Figure 1: Spread slicing. Rows are scored by their relative update magnitudes, ranked by the selected priority, and assigned to slices so that highly updated rows are distributed evenly. The inverse permutation restores the original row order after merging.

Branch	Avg	$\\|\Delta\\|/\\|\Delta_{\mathrm{TA}}\\|$	$\cos(\Delta,\Delta_{\mathrm{TA}})$
$dS$ only	27.23	0.006	0.024
$Q$ only	26.97	0.768	0.001
$V$ only	33.98	0.788	0.253
$Q{+}dS$	27.30	0.768	0.001
$dS{+}V$	34.24	0.788	0.253
$Q{+}V$ ( lr )	34.69	0.207	0.966

Appendix

Table 2: Branch diagnosis on T2 (Llama-3.2-3B,

h{=}16

\lambda{=}1

). Each row merges a subset of the per-slice factors and keeps the rest at their base values. Only the branches containing both rotations are direction-correct, and their magnitude contracts to

0.21\,\|\Delta_{\mathrm{TA}}\|

. The sweep optima in Table 7 are at

h{=}8

Base	$N$	$D$	$c_{\mathrm{RMS}}$	$\sqrt{N}$	Rel. diff.
Llama-3.1-8B	5	3.5	2.09	2.24	6.7%
Llama-3.2-3B	5	2.6	2.03	2.24	9.4%
Qwen2.5-VL-7B	3	1.3	1.72	1.73	0.6%
Gemma-2-2B	5	1.67	2.09	2.24	6.6%
Gemma-2-9B	5	16.0	2.19	2.24	2.3%

Appendix

Table 3: Measured contraction scale. The first four bases form the low-dispersion case. The Gemma-2-9B experts have substantially more uneven update magnitudes.

Figure 2: Swept restoration strength $\kappa_{\mathrm{opt}}=\lambda^{*}/\sqrt{N}$ against the dispersion statistic $D$ . Dashed lines mark the two shared constants.

Name	Description
full	Merges the rotation, relative spectral shift, and right factor. No conflict masking, spread slicing, or residual pathway is applied.
lr	Merges the rotation and right factor while keeping the spectral factor at its base value $S_{0}$ .
nca	Merges the non-conflicting columns geometrically and discards the conflicting columns.
nca+ave	Merges the non-conflicting columns geometrically and adds the average of the conflicting columns as a Euclidean residual.
ca	Merges the conflicting columns geometrically and discards the non-conflicting columns.
ca+ave	Merges the conflicting columns geometrically and adds the average of the non-conflicting columns as a Euclidean residual.

Appendix

Table 4: Configuration names used in the supplementary tables. The suffixes ta , ties , and tsvm denote the merger used for the non-target residual pathway.

Suite	Main-paper row	Internal line	$h$	SS priority	RP merger	Conflict variant	Rule $\lambda$	Best $\lambda$
T1	CORAM	full	8	None	None	None	2.60	2.40 ( $h{=}16$ )
T1	CORAM-C	nca+ave	8	None	None	nca+ave	2.60	2.25
T1	CORAM-C+SS	SS(ca+ave,energy)	8	energy	None	ca+ave	2.60	2.30
T2	CORAM+SS+RP	SS(lr,mean).ties	8	mean	TIES	None	2.60	2.60
T2	CORAM-C+RP	nca.ties	8	None	TIES	nca	2.60	2.60
T2	CORAM-C+SS+RP	SS(ca+ave,variance).ties	8	variance	TIES	ca+ave	2.60	2.60

Appendix

Table 5: Mapping between the main-paper rows and the internal configuration names (Table 4 ), with the

\kappa

-rule and swept-best coefficients. All checkpoints use

h=8

, except the T1 CORAM best-

\lambda

entry (

h=16

Line	$\lambda^{*}$	Avg	Line	$\lambda^{*}$	Avg
full	2.50	57.94	SS(full,mean)	2.40	58.08
lr	2.20	57.80	SS(full,variance)	2.30	57.99
nca	2.50	57.73	SS(full,energy)	2.40	57.97
nca+ave	2.25	57.74	SS(ca+ave,energy)	2.30	58.21
ca	2.10	47.45	full ( $h=16$ )	2.40	57.98
ca+ave	2.50	57.87	lr ( $h=16$ )	2.40	57.97

Appendix

Table 6: T1 per-line sweep optima. Strict conflict masking without an averaged residual gives a substantially lower result on T1.

Line	$\lambda^{*}$	Avg	Line	$\lambda^{*}$	Avg
full	2.80	42.76	nca.ta	2.60	43.00
lr	2.75	42.44	nca.ties	2.60	43.32
nca	2.50	42.96	nca.tsvm	2.80	43.06
nca+ave	2.60	42.33	nca+ave.ties	2.55	43.18
ca+ave	2.60	42.56	ca+ave.ties	2.55	42.96
full.ties	2.60	43.03	SS(lr,mean)	2.80	42.99

Appendix

Table 7: Selected T2 per-line sweep optima at

h=8

Line	$\lambda^{*}$	Avg	Line	$\lambda^{*}$	Avg
full	1.60	63.25	nca.ta	1.75	63.75
lr	1.60	63.43	nca+ave.ta	1.70	63.60
nca	1.60	63.07	ca+ave.ta	1.70	63.40
nca+ave	1.75	63.42	lr.ta	2.00	63.64
ca+ave	1.75	63.41	SS(nca,energy)	1.60	63.72
full.ta	1.80	63.61	SS(nca+ave,variance)	1.75	63.73

Appendix

Table 8: Selected T3 per-line sweep optima at

h=8

. All entries are five-task in-domain averages (CharXiv excluded).

Line	$\lambda^{*}$	Avg	Line	$\lambda^{*}$	Avg
full	1.20	54.56	nca.ties	1.30	53.86
lr	1.00	54.74	nca+ave.ties	1.20	54.72
nca	1.30	53.89	ca+ave.ties	1.00	54.67
nca+ave	1.00	54.78	lr.ties	1.25	54.84
ca+ave	1.10	54.32	full.ties	1.20	54.67
lr.ta	1.10	54.48	SS(lr,mean).ta	1.25	54.99

Appendix

Table 9: Selected T4 per-line sweep optima at

h=8

. The optima lie between

\lambda=1.00

and

1.30

Figure 3: In-domain average versus $\lambda$ at $h=8$ . Blue: the representative line ( full on T1/T2, lr on T3, lr.ta on T4); orange: the strongest branch of the same suite (SS(ca $+$ ave,energy), SS(ca $+$ ave,variance).ties, lr.ta , SS(lr,mean).ta). T3 curves are five-task averages, hence no OrthoMerge reference line. Dashed vertical lines mark the $\kappa$ -rule coefficient; stars mark the swept optimum of the representative line; dotted horizontal lines mark the strongest OrthoMerge variant. The T4 panel is truncated at $\lambda\leq 1.7$ for readability; beyond this point the curve declines monotonically to 43.5 at $\lambda=2.9$ (full grid in Table 10 ).

Suite	Line	Rule $\lambda$	Rule score	$\lambda^{*}$	Best score	Gap
T1	full	2.60	57.72	2.50	57.94	0.22
	SS(full,mean)	2.60	57.49	2.40	58.08	0.59
	lr	2.60	57.48	2.20	57.80	0.32
	SS(full,variance)	2.60	57.47	2.30	57.99	0.52
	nca	2.60	57.54	2.50	57.73	0.19
	SS(full,energy)	2.60	57.72	2.40	57.97	0.25

Appendix

Table 10: Rule-selected versus swept-optimal scores for every retained line at

h=8

(plus the two retained

h=16

lines on T1). T3 entries are five-task averages. Lines whose swept optimum coincides with the rule coefficient have zero gap by construction.

Suite	Line	$n_{\lambda}$	$\lambda$ range	$\lambda^{*}$	Best	Rule
T1	ca	9	2.00–2.75	2.10	47.45	47.09
	ca $+$ ave	12	1.00–3.25	2.50	57.87	57.60
	full	7	2.25–2.80	2.50	57.94	57.72
	lr	7	2.25–2.80	2.25	57.80	57.48
	nca	9	2.00–2.75	2.50	57.73	57.54
	nca $+$ ave	12	1.00–3.25	2.25	57.74	57.10

Appendix

Table 11: Layer 1: conflict variants under contiguous slicing.

Line	$n_{\lambda}$	$\lambda$ range	$\lambda^{*}$	Best	Rule
SS(ca $+$ ave,energy)	9	2.00–2.75	2.30	58.21	57.49
SS(ca $+$ ave,margin)	9	2.00–2.75	2.30	57.80	57.62
SS(ca $+$ ave,mean)	9	2.00–2.75	2.60	57.98	57.98
SS(ca $+$ ave,owner)	9	2.00–2.75	2.20	57.98	57.76
SS(ca $+$ ave,variance)	9	2.00–2.75	2.25	58.08	57.36
SS(full,energy)	5	2.30–2.70	2.40	57.97	57.72

Appendix

Table 12: Layer 2 (T1): spread-only lines.

Line	$n_{\lambda}$	$\lambda$ range	$\lambda^{*}$	Best	Rule
SS(ca $+$ ave,energy)	5	2.60–2.90	2.80	42.83	42.18
SS(ca $+$ ave,margin)	5	2.60–2.90	2.70	42.75	42.44
SS(ca $+$ ave,mean)	5	2.60–2.90	2.70	42.77	42.49
SS(ca $+$ ave,owner)	5	2.60–2.90	2.75	42.78	42.51
SS(ca $+$ ave,variance)	5	2.60–2.90	2.75	42.80	41.93
SS(ca,energy)	4	2.60–2.80	2.75	26.71	26.42

Appendix

Table 13: Layer 2 (T2): spread-only lines.

Line	$n_{\lambda}$	$\lambda$ range	$\lambda^{*}$	Best	Rule
SS(ca $+$ ave,energy)	7	1.50–2.00	1.80	63.52	63.45
SS(ca $+$ ave,margin)	5	1.50–2.00	1.70	63.20	62.98
SS(ca $+$ ave,mean)	5	1.50–2.00	1.60	63.29	63.19
SS(ca $+$ ave,owner)	5	1.50–2.00	1.50	63.16	63.05
SS(ca $+$ ave,variance)	7	1.50–2.00	1.75	63.52	63.27
SS(full,energy)	7	1.50–2.00	1.75	63.31	63.30

Appendix

Table 14: Layer 2 (T3): spread-only lines.

Line	$n_{\lambda}$	$\lambda$ range	$\lambda^{*}$	Best	Rule
SS(ca $+$ ave,energy)	5	1.00–1.40	1.00	53.72	53.68
SS(ca $+$ ave,margin)	6	1.00–1.40	1.10	53.28	53.28
SS(ca $+$ ave,mean)	6	1.00–1.40	1.00	53.72	53.38
SS(ca $+$ ave,owner)	5	1.10–1.40	1.10	53.67	53.67
SS(ca $+$ ave,variance)	6	1.00–1.40	1.00	53.51	53.49
SS(full,energy)	5	1.00–1.30	1.00	53.29	53.17

Appendix

Table 15: Layer 2 (T4): spread-only lines.

Suite	Line	$n_{\lambda}$	$\lambda$ range	$\lambda^{*}$	Best	Rule
T2	ca $+$ ave.ta	12	2.10–3.00	2.55	42.77	42.61
	ca $+$ ave.ties	12	2.10–3.00	2.55	42.96	42.96
	ca $+$ ave.tsvm	12	2.10–3.00	2.65	42.83	42.65
	full.ta	8	2.20–2.90	2.60	42.72	42.72
	full.ties	8	2.20–2.90	2.60	43.03	43.03
	full.tsvm	8	2.20–2.90	2.60	42.79	42.79

Appendix

Table 16: Layer 3: residual-pathway lines without spread slicing.

Line	$n_{\lambda}$	$\lambda$ range	$\lambda^{*}$	Best	Rule
SS(ca $+$ ave,energy).ties	4	2.60–2.80	2.60	43.34	43.34
SS(ca $+$ ave,margin).ties	4	2.60–2.80	2.60	43.22	43.22
SS(ca $+$ ave,mean).ties	4	2.60–2.80	2.60	42.72	42.72
SS(ca $+$ ave,owner).ties	4	2.60–2.80	2.60	42.77	42.77
SS(ca $+$ ave,variance).ties	4	2.60–2.80	2.60	43.48	43.48
SS(full,energy).ties	4	2.60–2.80	2.60	43.07	43.07

Appendix

Table 17: Layer 4 (T2): combined spread

+

residual lines.

Line	$n_{\lambda}$	$\lambda$ range	$\lambda^{*}$	Best	Rule
SS(ca $+$ ave,energy).ta	6	1.50–2.00	1.70	63.42	63.13
SS(ca $+$ ave,margin).ta	6	1.50–2.00	1.70	63.66	63.43
SS(ca $+$ ave,mean).ta	6	1.50–2.00	1.70	63.64	63.12
SS(ca $+$ ave,owner).ta	6	1.50–2.00	1.70	63.53	63.16
SS(ca $+$ ave,variance).ta	6	1.50–2.00	1.70	63.59	63.29
SS(nca $+$ ave,energy).ta	6	1.50–2.00	1.80	63.56	63.30

Appendix

Table 18: Layer 4 (T3): combined spread

+

residual lines.

Line	$n_{\lambda}$	$\lambda$ range	$\lambda^{*}$	Best	Rule
SS(ca $+$ ave,energy).ta	6	1.00–1.40	1.00	54.49	54.11
SS(ca $+$ ave,margin).ta	6	1.00–1.40	1.25	54.51	53.96
SS(ca $+$ ave,mean).ta	6	1.00–1.40	1.10	53.85	53.85
SS(ca $+$ ave,owner).ta	4	1.00–1.40	1.00	53.87	53.81
SS(ca $+$ ave,variance).ta	6	1.00–1.40	1.00	54.46	54.36
SS(full,energy).ta	6	1.00–1.40	1.00	54.41	53.90

Appendix

Table 19: Layer 4 (T4): combined spread

+

residual lines.

Suite	Rule-selected line	Rule $\lambda$	Rule Avg	Swept-best line	Best $\lambda$	Best Avg	Best OM	Best WS
T1	SS(ca $+$ ave,mean)	2.60	57.98	SS(ca $+$ ave,energy)	2.30	58.21	57.13	56.23
T2	SS(ca $+$ ave,variance).ties	2.60	43.48	SS(ca $+$ ave,variance).ties	2.60	43.48	42.13	43.45
T3	lr.ta	2.00	64.38	lr.ta	2.00	64.38	63.56	63.70
T4	SS(lr,variance).ta	1.10	54.71	SS(lr,mean).ta	1.25	54.99	54.74	54.57

Appendix

Table 20: Best in-domain averages at the

\kappa

-rule point and at the swept optimum, with the internal line (Table 4 ) realizing each value. The rule-selected column reports the best configuration evaluated directly at the

\kappa

-rule point. It may correspond to a different internal line than the swept best within the same component family (T1 and T4 here), and may differ from the main-table line, which is selected by its swept optimum (Table 5 ). Best OM is the strongest OrthoMerge variant and Best WS the strongest weight-space baseline for each suite, from the main tables. The rule-selected CORAM exceeds the strongest OrthoMerge variant on T1–T3; against the strongest weight-space baseline it leads on T1 and T3 and is comparable on T2 (43.48 vs. 43.45, within evaluation noise; Section H). On T4 it trails OrthoMerge by

0.03

at the rule point and exceeds it after the sweep.

$\lambda^{*}$	1.00	1.10	1.20	1.25	1.30
Number of lines	16	10	9	4	6

Appendix

Table 21: Distribution of the swept-optimal coefficient across the retained T4 lines.

$D_{0}$	Assignments	Per-suite gaps (T1–T4)	Max gap
4	unchanged	0.23 / 0.00 / 0.00 / 0.28	0.28
6	unchanged	0.23 / 0.00 / 0.00 / 0.28	0.28
8	unchanged	0.23 / 0.00 / 0.00 / 0.28	0.28
10	unchanged	0.23 / 0.00 / 0.00 / 0.28	0.28
12	unchanged	0.23 / 0.00 / 0.00 / 0.28	0.28

Appendix

Table 22: Threshold sensitivity. Gaps are the best-configuration rule-to-optimum differences of Table 20 . Any

D_{0}\in(3.5,16)

yields identical behavior. We fix no unique threshold; intermediate values of

D

are not represented in the current expert suites (see Limitations).

Held-out	$\hat{\kappa}$	$\lambda_{\mathrm{pred}}$	Score	Best	Gap
T1 ( full )	1.107	2.48 $\to$ 2.50	57.94	57.94	0.00
T2 ( full )	1.077	2.41 $\to$ 2.50	42.47	42.76	0.29
T3 ( lr )	1.103	1.91 $\to$ 1.90	63.07	63.43	0.36

Appendix

Table 23: Leave-one-base-out validation on the low-dispersion bases at

h=8

. T3 entries are five-task averages. Predicted coefficients are rounded to the nearest point of the original evaluation grids; for T2 this is

\lambda{=}2.50

(a later direct

\lambda{=}2.40

evaluation on newer hardware is not comparable to the original sweep and is not used). Only one high-dispersion base exists, so no leave-one-out test is possible for

\kappa=0.5

, which remains a single-base calibration (see Limitations).

Family	Statistic	Prediction	Observation
Weight norm	$\\|\Delta\\|$ -matching	$\lambda\approx 1$	Below the swept optima
Energy ratio	$E_{\mathrm{task}}/E_{\mathrm{geo}}$	$2.08/2.07/1.75$	Correct ordering; under-scaled
Spectral	$\sqrt{c}$	$1.08/2.73/2.55$	Matches T2 only
Gram	Cross-term statistic	$1.92/2.13/1.72$	Incorrect direction
Fisher energy	Function-space energy	$1.87/1.81$	Under-scaled
KL path	Calibration divergence	$\lambda\leq 1.5$	Below useful range

Appendix

Table 24: Alternative closed-form choices considered for the amplification scale.

Configuration	T1	T2	T3	T4
CORAM	57.72	42.75	63.95	54.49
+RP	n/a	42.31	64.38	54.56
+SS	57.84	42.93	63.96	54.15
+SS+RP	n/a	43.30	64.27	54.71
CORAM-C	57.60	42.96	64.02	54.32
+RP	n/a	43.32	64.20	54.37

Appendix

Table 25: Best in-domain average within each component branch at the

\kappa

-rule point. These are branch-level summaries; the matched ablations are reported below. Branch-level bests search over all spread priorities and conflict variants evaluated at the rule point, including combinations not retained in the per-line tables of Section D; the main-table rows instead report, for each branch, the line with the best swept optimum, evaluated at the rule coefficient. The two conventions can differ: on T1 the CORAM-C

+

SS branch best at the rule point is SS(ca

+

ave,mean) at 57.98, while the main-table CORAM-C

+

SS row reports SS(ca

+

ave,energy) at 57.49, whose swept optimum (58.21) is the strongest in the branch.

Suite	Matched setting	Plain	Conflict-aware	$\Delta$
T1	$h{=}8$ , base, $\lambda{=}2.60$	57.72	57.10	$-0.62$
T2	$h{=}8$ , base, $\lambda{=}2.60$	42.75	42.33	$-0.42$
T3	$h{=}8$ , base, $\lambda{=}2.00$	63.85	64.02	$+0.17$
T4	$h{=}8$ , base, $\lambda{=}1.10$	54.33	54.05	$-0.28$

Appendix

Table 26: Matched conflict-aware ablation: plain full vs. conflict-aware nca+ave ,

\kappa

-rule

\lambda

, no SS or RP. T3 averages include CharXiv.

Suite or branch	Contiguous	Spread-only	$\Delta$
T1	57.87	58.21	$+0.34$
T2	42.96	43.37	$+0.41$
T3	64.04	64.14	$+0.10$
T4, lr	54.74	54.16	$-0.58$
T4, full	54.56	53.84	$-0.72$
T4, nca+ave	54.78	53.92	$-0.86$

Appendix

Table 27: Best contiguous and spread-only lines within the same branch at

h=8

, each at its own swept optimum. T3 averages include CharXiv. The matched fixed-coefficient comparison is in Table 28 .

Suite	Branch	$h$	$\lambda$	Priority	Contiguous	Spread	$\Delta$
T1	full	8	2.60	energy	57.72	57.72	$0.00$
T2	full	8	2.60	energy	42.75	42.57	$-0.18$
T3	full	8	2.00	energy	63.85	63.87	$+0.02$
T4	full	8	1.10	energy	54.33	53.17	$-1.16$

Appendix

Table 28: Matched spread-slicing ablation: full branch, energy priority,

\kappa

-rule

\lambda

. T3 averages include CharXiv. The effect is negligible on T1–T3 and negative on T4.

Suite	Geometry	$h$	$\lambda$	Residual merger	Without RP	With RP	$\Delta$
T2	full	8	2.60	TIES	42.75	43.03	$+0.28$
T3	full	8	2.00	Task arithmetic	63.85	64.22	$+0.37$
T4	full	8	1.10	TIES	54.33	54.29	$-0.04$

Appendix

Table 29: Matched residual-pathway ablation: full branch,

\kappa

-rule

\lambda

. T3 averages include CharXiv.

Suite	Base branch	Neither	SS only	RP only	SS+RP
T2	CORAM	42.75	42.93	42.31	43.30
T2	CORAM-C	42.96	42.99	43.32	43.48
T3	CORAM	63.95	63.96	64.38	64.27
T3	CORAM-C	64.02	64.14	64.20	64.33
T4	CORAM	54.49	54.15	54.56	54.71
T4	CORAM-C	54.32	53.77	54.37	54.41

Appendix

Table 30: Branch-level interaction at the

\kappa

-rule point, with the best priority and merger per suite and branch. Matched comparisons: Tables 28 and 29 .

	Contiguous	Spread	Spread $+$ RP
Mixed-script, all (%)	3.5	17.6	12.9
Mixed-script, XSTest (%)	12.0	69.1	49.6
XSTest safe-prompt acc.	0.904	0.452	0.648
XSTest adversarial acc.	0.645	0.645	0.635
Mean length (chars)	1304	1283	1285
Empty (%)	0.0	0.0	0.0

Appendix

Table 31: Generation statistics over the 1,819 safety responses of the matched T4 comparison (owner-balance/ full ,

\lambda{=}1.20

). Spread-only merging induces mixed-script drift on open-ended prompts rather than truncation or refusal by silence; the residual pathway partially repairs it.

Method	T1	T2	T3	T4
Fixed plain CORAM ( $\kappa$ -rule)	57.72	42.75	63.95	54.49
Fixed CORAM-C $+$ SS $+$ RP ∗	57.62	43.48	63.23	54.43
Best per-suite CORAM ( $\kappa$ -rule)	57.98	43.48	64.38	54.71
OrthoMerge (best variant)	57.13	42.13	63.56	54.74
Best weight-space baseline	56.23	43.45	63.70	54.57

Appendix

Table 32: A single configuration across all suites. The fixed plain configuration trails the per-suite best by

0.22

–

0.73

points but exceeds OrthoMerge on T1–T3 and remains within

0.25

on T4. ∗ The second row freezes the strongest T2 main-table configuration (ca

+

ave conflict masking, variance spread priority, TIES residual, rule

\lambda

) and applies it to every suite; on T1 the residual pathway is not applicable and is omitted. It transfers to T1 (57.62, above OrthoMerge) and keeps its T2 result, but trails on T3 and T4, where its components were selected for other suites. The plain configuration in the first row is the more uniform suite-agnostic choice.

Table 41

Suite	Configuration	$\lambda$	$h=8$	$h=16$	$h=32$	$h=64$
T1	full	2.60	57.72	57.65	57.46	57.39
T2	full	2.60	42.75	42.40	42.57	42.29
T3	lr.ta	2.00	63.64	63.12	63.50	Collapse a
T4	lr.ta	1.10	54.48	54.23	54.25	53.44

Appendix

Table 35: Matched comparison across slice heights at the

\kappa

-rule coefficient. All non-

h

settings are fixed within each row; T3 entries are five-task averages. The

h{=}8

cells repeat the retained-line values; the T2–T4

h{=}16

cells and the T1–T3

h{=}32/64

cells were evaluated on the newer hardware, while the T4

h{=}32/64

cells use the original hardware with the main-table safety protocol. a At

h{=}64

the T3 merge collapses into degenerate unbounded generation (MMSI-Bench accuracy

0.0

); the evaluation was aborted. The best value in every row is at

h{=}8

Line	Tested $h=16$ value(s)	Best $h=16$	$h=8$ optimum
T2 nca ( $\lambda$ 2.50/2.60)	42.30 / 42.35	42.35	42.96
T2 nca.ties ( $\lambda$ 2.60)	42.80	42.80	43.32
T3 lr (grid)	62.96 / 63.54	63.54	63.43
T3 nca ( $\lambda$ 1.50–1.70)	63.07–63.18	63.18	63.07
T3 nca.ta ( $\lambda$ 1.60–2.00)	63.35–63.47	63.47	63.75
T3 nca+ave ( $\lambda$ 1.65–2.00)	63.07–63.31	63.31	63.42

Appendix

Table 36: Representative

h=16

comparisons. The strongest suite-level result remains at

h=8

. T3 rows are five-task averages (CharXiv excluded) on both sides, matching Table 8 ; CharXiv was never evaluated at

h=16

. ‡ Safety re-scored under the vLLM protocol; the affected points remain below the

h=8

optimum under either scoring. This table is not a coefficient-matched comparison; the matched rule-coefficient results are reported in Table 35 .

Figure 4: In-domain average as a function of slice height at the $\kappa$ -rule coefficient (one panel per suite; the T3 $h{=}64$ point is the degenerate collapse noted in Table 35 ).

Figure 5: In-domain versus OOD averages for all mergers in the main tables. The CORAM point is the strongest $\kappa$ -rule main-table row of each suite.

Suite	Task	Published setting	Our setting	Published score	Reproduced score	Difference
T1	M-ARC	Unspecified	Zero-shot	44.75	35.15	$-9.60$
T1	AGIEval	Unspecified	Zero-shot	38.84	33.13	$-5.71$
T2	MMLU †	Unspecified	Zero-shot	55.57	57.42	$+1.85$
T2	AGIEval	Unspecified	Zero-shot	33.79	26.87	$-6.92$
T3	IFEval	Unspecified	Zero-shot	54.90	54.53	$-0.37$
T3	MMBench	Unspecified	Zero-shot	83.59	83.68	$+0.09$

Appendix

Table 37: Published and reproduced OOD scores for the strongest OrthoMerge variant per suite (OFT for T1, TSV-M+C for T2, TIES+C for T3). Differences are reproduced minus published.

Variant	T2	T3	T4
TA+G	35.50	62.59	51.97
TIES+G	39.67	63.12	46.51
TSV-M+G	40.05	62.61	50.21
TA+C	34.12	61.95	54.74
TIES+C	41.46	63.56	46.25
TSV-M+C	42.13	62.69	52.34

Appendix

Table 38: All six OrthoMerge variants under our harness. G/C denote plain/conflict-aware OrthoMerge geometry; TA, TIES, and TSV-M denote the residual merger. The T3 entry for TIES+C is scored under the main-table judge protocol (Table 39 ); the remaining T3 entries are from an earlier judge pass and are shown for the variant ranking only.

Suite	Published best variant	Published Avg	Reproduced best variant	Reproduced Avg	Difference
T1	Released OFT checkpoint	46.25	Released OFT checkpoint	57.13	$+10.88$
T2	TSV-M+C	42.07	TSV-M+C	42.13	$+0.06$
T3	TIES+C	64.04	TIES+C	63.56	$-0.48$
T4	–	–	TA+C	54.74	–

Appendix

Table 39: Summary of the OrthoMerge reproduction. The T1 gap follows from the ScienceQA extraction difference.

Suite	Method	Orig.	s18	s38	s68	s98	Mean	Std.
T1	CORAM	57.72	57.66	57.45	57.55	57.69	57.61	0.10
T1	Task arithmetic	56.23	55.96	55.81	55.89	55.71	55.92	0.18
T2	CORAM-C $+$ SS $+$ RP	43.48	43.32	43.33	43.35	43.27	43.35	0.08
T2	DARE-TIES	43.45	43.27	43.30	43.39	43.37	43.36	0.06
T3	CORAM $+$ RP	64.38	64.37	64.38	64.37	64.33	64.37	0.02
T3	TIES	63.70	64.08	64.13	64.10	64.12	64.03	0.16

Appendix

Table 40: Multi-seed in-domain averages for the repeated CORAM checkpoints and the closest baselines. The T2 and T4 differences lie within the estimated evaluation noise; the table supports comparable performance rather than a significant advantage in these two closest comparisons.

Stage	T1: 8B	T2: 3B	T3: 7B	T4: 9B	Frequency
Slice-SVD cache, per checkpoint	27 GB	11 GB	25 GB	32 GB	Once per suite
Slice-SVD cache, suite total	162 GB	66 GB	100 GB	192 GB	Once per suite
Geometry merge	0.5–0.8 GPU hours				Once per configuration
$\lambda$ -assembly	1–3 CPU minutes				Per coefficient
Full-suite evaluation	1.7 GPU hours	1.7 GPU hours	1.4 GPU hours	4 GPU hours	Per coefficient
Sweep over $\lambda$	10–12 full-suite evaluations				Per configuration

Appendix

Table 41: Measured storage and wall-clock cost across the four suites. Cache sizes cover the base model and all experts at

h=8

. The T3 evaluation comprises the five VL tasks (0.8 GPU hours) and one CharXiv pass (about 0.6 hours). A sweep is dominated by evaluation cost; the measured sweep totals are 17–20 GPU hours on T2 and 40–48 GPU hours on T4. For reference, the per-tensor baseline merges are far cheaper: linear merging takes 31 s (3B) and 1 min 37 s (9B), and TIES takes 3 min 9 s and 10 min 48 s.

Language Model Merging

Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.

See all 110 papers

Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion
Sep 19, 2026Aditya Pola, Vineeth N. BalasubramanianModel Merging
IIT Hyderabad · Microsoft Research, India
A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic
Oct 6, 2026Sin-Han Yang, Shih-Cheng Huang, Chieh-Yen Lin +3Task Vector Merging Multi-Task Optimization
Appier AI Research · National Taiwan University
SMAT: Simple and Efficient Merge-Aware Training
Sep 27, 2026Yanggan Gu, Yuanyi Wang, Zhen Li +5Language Model Merging Model Merging
The Hong Kong Polytechnic University (PolyU) · The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong +1

Sep 19, 2026J/K move · Enter open · S save

Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion

Aditya Pola, Vineeth N. Balasubramanian

IIT Hyderabad · Microsoft Research, India

Model Merging

Open paper AskPDF arXiv

Model merging consolidates fine-tuned experts into one multi-task model without retraining. All existing data-free methods approach this problem entirely in weight space. Restricted to arithmetic on parameters, these methods never observe how each expert behaves, a signal that only emerges through forward evaluation. Accessing this behavioral signal requires inputs to evaluate on, which the data-free setting prohibits. We propose Merge++, a post-hoc method that addresses this by inverting the expert checkpoints to synthesize task-representative images, then distilling expert knowledge into the merged model using those images. Merge++ requires no additional data beyond the checkpoints themselves. It applies universally across merging algorithms and operates as a complementary refinement stage independent of the underlying weight-space method. The method consistently improves merging algorithms ranging from simple task arithmetic to state-of-the-art spectral methods, with average gains of +2 to +8 points and up to +25.9 on individual configurations.

Abstract

Language Model Merging

Explore similar work

Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion

A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic

SMAT: Simple and Efficient Merge-Aware Training