Organizations: Santa Clara University · Shanghai Institute for Mathematics and Interdisciplinary Sciences / Fudan University · Futurewei Technologies, Inc.
Merging finetuned models combines specialized capabilities without joint training or access to the original data. Most methods operate by linear arithmetic in Euclidean weight space, which cannot carry the geometry of the update. Orthogonal Model Merging (OrthoMerge) uses a single orthogonal transform for each weight matrix, but such a transform cannot change singular values. We propose CORAM, which partitions each target matrix into row slices, represents every expert slice by its singular value decomposition in the corresponding base-model SVD frame, and merges the task-specific factors on their corresponding manifolds. Because manifold averaging contracts the merged update, CORAM applies an amplification coefficient λ=κc^. The scale c_hat is estimated from the expert and merged update norms and is approximately N for N experts with comparable update magnitudes. The restoration strength kappa is selected from the dispersion of expert updates without evaluating candidate merged models. This rule remains within 0.72 points of the best swept value on all evaluated suites. CORAM also includes spread slicing to distribute highly updated rows across slices and a residual pathway for non-target layers. Across four suites covering three model families, 3B to 9B scales, and language and vision-language experts, CORAM improves over OrthoMerge by 0.25 to 1.35 points and matches or exceeds the strongest weight-space baselines.
Figures & tables
Figure 1: Overview of CORAM. Each expert slice is factored in the base frame into a rotation, a spectral shift dSi=SiS0−1−I , and a right factor. Each factor is averaged in its own space. The coefficient λ is given by Eq. ( 3 ).
In-domain
In-domain Avg
Out-of-domain
Method
MATH500
HEval +
SciQA
CSQA
SIQA
κ -rule
best λ
M-ARC
AGIEval
OOD Avg
Llama-3.1-8B
17.80
21.52
71.40
70.60
48.11
45.89
33.02
30.27
31.65
Task-specific FT
19.00
38.54
91.82
82.47
56.81
57.73
–
–
–
Linear
23.40
32.26
83.59
76.82
51.84
53.58
34.41
32.27
33.34
Task Arithmetic
24.80
37.20
86.29
79.52
53.33
56.23
34.97
32.68
33.83
TIES
21.40
40.43
82.87
78.30
54.04
55.41
34.40
32.53
33.47
Table 1: Top–T1 : merging five orthogonally finetuned Llama-3.1-8B experts. Bottom–T2 : merging five fully finetuned Llama-3.2-3B experts from MergeBench. Higher is better. Per column, the best result among merging methods is in bold (base-model and task-specific-FT reference rows excluded). CORAM-C denotes the conflict-aware variants (masking neurons where experts disagree). + SS adds spread slicing (a task-informed row permutation) and + RP the residual pathway (conventional merge of the non-target layers). CORAM rows evaluate the search-free κ -rule checkpoint (grid λ=2.60 for both suites). “best λ ” is the sweep optimum, for reference (see Setup).
In-domain
In-domain Avg
Out-of-domain
Method
MMSI
EmbSp.
MMMU Med
PathVQA
OCRB.
CharXiv
κ -rule
best λ
IFEval
MMB.
OOD Avg
Qwen2.5-VL-7B-It.
27.80
69.97
53.10
66.30
84.70
67.20
61.51
63.03
83.93
73.48
Task-specific FT
32.60
70.58
55.17
66.81
85.00
72.50
63.78
–
–
–
Linear
29.20
71.29
55.17
68.47
84.80
67.30
62.71
58.23
84.19
71.21
Task Arithmetic
29.10
71.07
55.86
68.38
84.60
66.10
62.52
59.33
84.19
71.76
TIES
32.10
71.54
57.93
68.44
82.80
69.40
63.70
54.53
84.02
69.27
Table 2: T3 : merging three Qwen2.5-VL-7B-Instruct vision–language experts. Layout and bolding as in Table 1 . κ -rule grid point λ=2.00 . CORAM-C denotes the conflict-aware variants (masking neurons where experts disagree). + RP adds the residual pathway (conventional merge of the non-target layers). The OrthoMerge out-of-domain entries are obtained from our zero-shot re-evaluation.
In-domain
In-domain Avg
Out-of-domain
Method
Instr.
Math
Coding
Multi.
Safety
κ -rule
best λ
MMLU †
AGIEval
OOD Avg
Gemma-2-9B
14.23
69.83
43.37
54.63
34.39
43.29
70.41
38.01
54.21
Task-specific FT
65.06
79.76
58.51
55.91
76.00
67.05
–
–
–
Linear
27.17
81.05
51.52
53.79
59.31
54.57
67.88
37.75
52.81
Task Arithmetic
27.54
82.34
51.65
51.31
54.26
53.42
64.44
35.70
50.07
TIES
24.03
82.94
46.60
45.64
52.84
50.41
60.60
34.92
47.76
Table 3: T4 : merging five fully finetuned Gemma-2-9B experts from MergeBench with uneven update magnitudes ( D≈16 ). The rule selects κ=0.5 , corresponding to λ=1.10 . Layout and bolding follow Table 1 . CORAM-C masks neurons where experts disagree. + SS adds a task-informed row permutation, and + RP merges the non-target layers.
Configuration
T1
T2
T3
T4
CORAM
57.72
42.75
63.95
54.49
+ RP
n/a
42.31
64.38
54.56
+ SS
57.84
42.93
63.96
54.15
+ SS + RP
n/a
43.30
64.27
54.71
CORAM-C
57.60
42.96
64.02
54.32
+ RP
n/a
43.32
64.20
54.37
Table 4: Best in-domain average reached at the κ -rule point (best configuration per method and suite). RP is inapplicable on T1 (no non-target residual).
Appendix figures & tables45 assets
Supplementary material from the paper’s appendix.
Appendix
Rotation log (all slices)
Cond. (contiguous)
Cond. (spread)
Suite
Mean err
Max err
θ>3
Median
p95
Max
p95
Max
T1
8.8×10−16
1.9×10−15
0
1.3
2.3
706
2.8
38
T2
8.8×10−16
2.2×10−15
0
1.3
2.6
822
3.1
28
T3
8.8×10−16
1.9×10−15
0
1.3
5.2
635
5.3
282
T4
1.4×10−15
5.1×10−10
64
1.3
2.0
43
2.5
18
Appendix
Table 1: Numerical audit of the geometric operators at h=8 , over every slice of every expert (0.48M–1.16M rotations per suite). Rotation-logarithm reconstruction errors are at machine precision; the near- π fallback fires on 64 of 1.16M rotations on T4 only and is handled by the exact eigendecomposition path. No slice has σmin<10−8 . Variance-spread slicing leaves the median condition number unchanged but collapses the tail (maximum 706→38 , 822→28 , 635→282 , 43→18 ), confirming its conditioning motivation.
Figure 1: Spread slicing. Rows are scored by their relative update magnitudes, ranked by the selected priority, and assigned to slices so that highly updated rows are distributed evenly. The inverse permutation restores the original row order after merging.
Branch
Avg
∥Δ∥/∥ΔTA∥
cos(Δ,ΔTA)
dS only
27.23
0.006
0.024
Q only
26.97
0.768
0.001
V only
33.98
0.788
0.253
Q+dS
27.30
0.768
0.001
dS+V
34.24
0.788
0.253
Q+V ( lr )
34.69
0.207
0.966
Appendix
Table 2: Branch diagnosis on T2 (Llama-3.2-3B, h=16 , λ=1 ). Each row merges a subset of the per-slice factors and keeps the rest at their base values. Only the branches containing both rotations are direction-correct, and their magnitude contracts to 0.21∥ΔTA∥ . The sweep optima in Table 7 are at h=8 .
Base
N
D
cRMS
N
Rel. diff.
Llama-3.1-8B
5
3.5
2.09
2.24
6.7%
Llama-3.2-3B
5
2.6
2.03
2.24
9.4%
Qwen2.5-VL-7B
3
1.3
1.72
1.73
0.6%
Gemma-2-2B
5
1.67
2.09
2.24
6.6%
Gemma-2-9B
5
16.0
2.19
2.24
2.3%
Appendix
Table 3: Measured contraction scale. The first four bases form the low-dispersion case. The Gemma-2-9B experts have substantially more uneven update magnitudes.
Figure 2: Swept restoration strength κopt=λ∗/N against the dispersion statistic D . Dashed lines mark the two shared constants.
Name
Description
full
Merges the rotation, relative spectral shift, and right factor. No conflict masking, spread slicing, or residual pathway is applied.
lr
Merges the rotation and right factor while keeping the spectral factor at its base value S0 .
nca
Merges the non-conflicting columns geometrically and discards the conflicting columns.
nca+ave
Merges the non-conflicting columns geometrically and adds the average of the conflicting columns as a Euclidean residual.
ca
Merges the conflicting columns geometrically and discards the non-conflicting columns.
ca+ave
Merges the conflicting columns geometrically and adds the average of the non-conflicting columns as a Euclidean residual.
Appendix
Table 4: Configuration names used in the supplementary tables. The suffixes ta , ties , and tsvm denote the merger used for the non-target residual pathway.
Suite
Main-paper row
Internal line
h
SS priority
RP merger
Conflict variant
Rule λ
Best λ
T1
CORAM
full
8
None
None
None
2.60
2.40 ( h=16 )
T1
CORAM-C
nca+ave
8
None
None
nca+ave
2.60
2.25
T1
CORAM-C+SS
SS(ca+ave,energy)
8
energy
None
ca+ave
2.60
2.30
T2
CORAM+SS+RP
SS(lr,mean).ties
8
mean
TIES
None
2.60
2.60
T2
CORAM-C+RP
nca.ties
8
None
TIES
nca
2.60
2.60
T2
CORAM-C+SS+RP
SS(ca+ave,variance).ties
8
variance
TIES
ca+ave
2.60
2.60
Appendix
Table 5: Mapping between the main-paper rows and the internal configuration names (Table 4 ), with the κ -rule and swept-best coefficients. All checkpoints use h=8 , except the T1 CORAM best- λ entry ( h=16 ).
Line
λ∗
Avg
Line
λ∗
Avg
full
2.50
57.94
SS(full,mean)
2.40
58.08
lr
2.20
57.80
SS(full,variance)
2.30
57.99
nca
2.50
57.73
SS(full,energy)
2.40
57.97
nca+ave
2.25
57.74
SS(ca+ave,energy)
2.30
58.21
ca
2.10
47.45
full ( h=16 )
2.40
57.98
ca+ave
2.50
57.87
lr ( h=16 )
2.40
57.97
Appendix
Table 6: T1 per-line sweep optima. Strict conflict masking without an averaged residual gives a substantially lower result on T1.
Line
λ∗
Avg
Line
λ∗
Avg
full
2.80
42.76
nca.ta
2.60
43.00
lr
2.75
42.44
nca.ties
2.60
43.32
nca
2.50
42.96
nca.tsvm
2.80
43.06
nca+ave
2.60
42.33
nca+ave.ties
2.55
43.18
ca+ave
2.60
42.56
ca+ave.ties
2.55
42.96
full.ties
2.60
43.03
SS(lr,mean)
2.80
42.99
Appendix
Table 7: Selected T2 per-line sweep optima at h=8 .
Line
λ∗
Avg
Line
λ∗
Avg
full
1.60
63.25
nca.ta
1.75
63.75
lr
1.60
63.43
nca+ave.ta
1.70
63.60
nca
1.60
63.07
ca+ave.ta
1.70
63.40
nca+ave
1.75
63.42
lr.ta
2.00
63.64
ca+ave
1.75
63.41
SS(nca,energy)
1.60
63.72
full.ta
1.80
63.61
SS(nca+ave,variance)
1.75
63.73
Appendix
Table 8: Selected T3 per-line sweep optima at h=8 . All entries are five-task in-domain averages (CharXiv excluded).
Line
λ∗
Avg
Line
λ∗
Avg
full
1.20
54.56
nca.ties
1.30
53.86
lr
1.00
54.74
nca+ave.ties
1.20
54.72
nca
1.30
53.89
ca+ave.ties
1.00
54.67
nca+ave
1.00
54.78
lr.ties
1.25
54.84
ca+ave
1.10
54.32
full.ties
1.20
54.67
lr.ta
1.10
54.48
SS(lr,mean).ta
1.25
54.99
Appendix
Table 9: Selected T4 per-line sweep optima at h=8 . The optima lie between λ=1.00 and 1.30 .
Figure 3: In-domain average versus λ at h=8 . Blue: the representative line ( full on T1/T2, lr on T3, lr.ta on T4); orange: the strongest branch of the same suite (SS(ca + ave,energy), SS(ca + ave,variance).ties, lr.ta , SS(lr,mean).ta). T3 curves are five-task averages, hence no OrthoMerge reference line. Dashed vertical lines mark the κ -rule coefficient; stars mark the swept optimum of the representative line; dotted horizontal lines mark the strongest OrthoMerge variant. The T4 panel is truncated at λ≤1.7 for readability; beyond this point the curve declines monotonically to 43.5 at λ=2.9 (full grid in Table 10 ).
Suite
Line
Rule λ
Rule score
λ∗
Best score
Gap
T1
full
2.60
57.72
2.50
57.94
0.22
SS(full,mean)
2.60
57.49
2.40
58.08
0.59
lr
2.60
57.48
2.20
57.80
0.32
SS(full,variance)
2.60
57.47
2.30
57.99
0.52
nca
2.60
57.54
2.50
57.73
0.19
SS(full,energy)
2.60
57.72
2.40
57.97
0.25
Appendix
Table 10: Rule-selected versus swept-optimal scores for every retained line at h=8 (plus the two retained h=16 lines on T1). T3 entries are five-task averages. Lines whose swept optimum coincides with the rule coefficient have zero gap by construction.
Suite
Line
nλ
λ range
λ∗
Best
Rule
T1
ca
9
2.00–2.75
2.10
47.45
47.09
ca + ave
12
1.00–3.25
2.50
57.87
57.60
full
7
2.25–2.80
2.50
57.94
57.72
lr
7
2.25–2.80
2.25
57.80
57.48
nca
9
2.00–2.75
2.50
57.73
57.54
nca + ave
12
1.00–3.25
2.25
57.74
57.10
Appendix
Table 11: Layer 1: conflict variants under contiguous slicing.
Line
nλ
λ range
λ∗
Best
Rule
SS(ca + ave,energy)
9
2.00–2.75
2.30
58.21
57.49
SS(ca + ave,margin)
9
2.00–2.75
2.30
57.80
57.62
SS(ca + ave,mean)
9
2.00–2.75
2.60
57.98
57.98
SS(ca + ave,owner)
9
2.00–2.75
2.20
57.98
57.76
SS(ca + ave,variance)
9
2.00–2.75
2.25
58.08
57.36
SS(full,energy)
5
2.30–2.70
2.40
57.97
57.72
Appendix
Table 12: Layer 2 (T1): spread-only lines.
Line
nλ
λ range
λ∗
Best
Rule
SS(ca + ave,energy)
5
2.60–2.90
2.80
42.83
42.18
SS(ca + ave,margin)
5
2.60–2.90
2.70
42.75
42.44
SS(ca + ave,mean)
5
2.60–2.90
2.70
42.77
42.49
SS(ca + ave,owner)
5
2.60–2.90
2.75
42.78
42.51
SS(ca + ave,variance)
5
2.60–2.90
2.75
42.80
41.93
SS(ca,energy)
4
2.60–2.80
2.75
26.71
26.42
Appendix
Table 13: Layer 2 (T2): spread-only lines.
Line
nλ
λ range
λ∗
Best
Rule
SS(ca + ave,energy)
7
1.50–2.00
1.80
63.52
63.45
SS(ca + ave,margin)
5
1.50–2.00
1.70
63.20
62.98
SS(ca + ave,mean)
5
1.50–2.00
1.60
63.29
63.19
SS(ca + ave,owner)
5
1.50–2.00
1.50
63.16
63.05
SS(ca + ave,variance)
7
1.50–2.00
1.75
63.52
63.27
SS(full,energy)
7
1.50–2.00
1.75
63.31
63.30
Appendix
Table 14: Layer 2 (T3): spread-only lines.
Line
nλ
λ range
λ∗
Best
Rule
SS(ca + ave,energy)
5
1.00–1.40
1.00
53.72
53.68
SS(ca + ave,margin)
6
1.00–1.40
1.10
53.28
53.28
SS(ca + ave,mean)
6
1.00–1.40
1.00
53.72
53.38
SS(ca + ave,owner)
5
1.10–1.40
1.10
53.67
53.67
SS(ca + ave,variance)
6
1.00–1.40
1.00
53.51
53.49
SS(full,energy)
5
1.00–1.30
1.00
53.29
53.17
Appendix
Table 15: Layer 2 (T4): spread-only lines.
Suite
Line
nλ
λ range
λ∗
Best
Rule
T2
ca + ave.ta
12
2.10–3.00
2.55
42.77
42.61
ca + ave.ties
12
2.10–3.00
2.55
42.96
42.96
ca + ave.tsvm
12
2.10–3.00
2.65
42.83
42.65
full.ta
8
2.20–2.90
2.60
42.72
42.72
full.ties
8
2.20–2.90
2.60
43.03
43.03
full.tsvm
8
2.20–2.90
2.60
42.79
42.79
Appendix
Table 16: Layer 3: residual-pathway lines without spread slicing.
Table 20: Best in-domain averages at the κ -rule point and at the swept optimum, with the internal line (Table 4 ) realizing each value. The rule-selected column reports the best configuration evaluated directly at the κ -rule point. It may correspond to a different internal line than the swept best within the same component family (T1 and T4 here), and may differ from the main-table line, which is selected by its swept optimum (Table 5 ). Best OM is the strongest OrthoMerge variant and Best WS the strongest weight-space baseline for each suite, from the main tables. The rule-selected CORAM exceeds the strongest OrthoMerge variant on T1–T3; against the strongest weight-space baseline it leads on T1 and T3 and is comparable on T2 (43.48 vs. 43.45, within evaluation noise; Section H). On T4 it trails OrthoMerge by 0.03 at the rule point and exceeds it after the sweep.
λ∗
1.00
1.10
1.20
1.25
1.30
Number of lines
16
10
9
4
6
Appendix
Table 21: Distribution of the swept-optimal coefficient across the retained T4 lines.
D0
Assignments
Per-suite gaps (T1–T4)
Max gap
4
unchanged
0.23 / 0.00 / 0.00 / 0.28
0.28
6
unchanged
0.23 / 0.00 / 0.00 / 0.28
0.28
8
unchanged
0.23 / 0.00 / 0.00 / 0.28
0.28
10
unchanged
0.23 / 0.00 / 0.00 / 0.28
0.28
12
unchanged
0.23 / 0.00 / 0.00 / 0.28
0.28
Appendix
Table 22: Threshold sensitivity. Gaps are the best-configuration rule-to-optimum differences of Table 20 . Any D0∈(3.5,16) yields identical behavior. We fix no unique threshold; intermediate values of D are not represented in the current expert suites (see Limitations).
Held-out
κ^
λpred
Score
Best
Gap
T1 ( full )
1.107
2.48 → 2.50
57.94
57.94
0.00
T2 ( full )
1.077
2.41 → 2.50
42.47
42.76
0.29
T3 ( lr )
1.103
1.91 → 1.90
63.07
63.43
0.36
Appendix
Table 23: Leave-one-base-out validation on the low-dispersion bases at h=8 . T3 entries are five-task averages. Predicted coefficients are rounded to the nearest point of the original evaluation grids; for T2 this is λ=2.50 (a later direct λ=2.40 evaluation on newer hardware is not comparable to the original sweep and is not used). Only one high-dispersion base exists, so no leave-one-out test is possible for κ=0.5 , which remains a single-base calibration (see Limitations).
Family
Statistic
Prediction
Observation
Weight norm
∥Δ∥ -matching
λ≈1
Below the swept optima
Energy ratio
Etask/Egeo
2.08/2.07/1.75
Correct ordering; under-scaled
Spectral
c
1.08/2.73/2.55
Matches T2 only
Gram
Cross-term statistic
1.92/2.13/1.72
Incorrect direction
Fisher energy
Function-space energy
1.87/1.81
Under-scaled
KL path
Calibration divergence
λ≤1.5
Below useful range
Appendix
Table 24: Alternative closed-form choices considered for the amplification scale.
Configuration
T1
T2
T3
T4
CORAM
57.72
42.75
63.95
54.49
+RP
n/a
42.31
64.38
54.56
+SS
57.84
42.93
63.96
54.15
+SS+RP
n/a
43.30
64.27
54.71
CORAM-C
57.60
42.96
64.02
54.32
+RP
n/a
43.32
64.20
54.37
Appendix
Table 25: Best in-domain average within each component branch at the κ -rule point. These are branch-level summaries; the matched ablations are reported below. Branch-level bests search over all spread priorities and conflict variants evaluated at the rule point, including combinations not retained in the per-line tables of Section D; the main-table rows instead report, for each branch, the line with the best swept optimum, evaluated at the rule coefficient. The two conventions can differ: on T1 the CORAM-C + SS branch best at the rule point is SS(ca + ave,mean) at 57.98, while the main-table CORAM-C + SS row reports SS(ca + ave,energy) at 57.49, whose swept optimum (58.21) is the strongest in the branch.
Suite
Matched setting
Plain
Conflict-aware
Δ
T1
h=8 , base, λ=2.60
57.72
57.10
−0.62
T2
h=8 , base, λ=2.60
42.75
42.33
−0.42
T3
h=8 , base, λ=2.00
63.85
64.02
+0.17
T4
h=8 , base, λ=1.10
54.33
54.05
−0.28
Appendix
Table 26: Matched conflict-aware ablation: plain full vs. conflict-aware nca+ave , κ -rule λ , no SS or RP. T3 averages include CharXiv.
Suite or branch
Contiguous
Spread-only
Δ
T1
57.87
58.21
+0.34
T2
42.96
43.37
+0.41
T3
64.04
64.14
+0.10
T4, lr
54.74
54.16
−0.58
T4, full
54.56
53.84
−0.72
T4, nca+ave
54.78
53.92
−0.86
Appendix
Table 27: Best contiguous and spread-only lines within the same branch at h=8 , each at its own swept optimum. T3 averages include CharXiv. The matched fixed-coefficient comparison is in Table 28 .
Suite
Branch
h
λ
Priority
Contiguous
Spread
Δ
T1
full
8
2.60
energy
57.72
57.72
0.00
T2
full
8
2.60
energy
42.75
42.57
−0.18
T3
full
8
2.00
energy
63.85
63.87
+0.02
T4
full
8
1.10
energy
54.33
53.17
−1.16
Appendix
Table 28: Matched spread-slicing ablation: full branch, energy priority, κ -rule λ . T3 averages include CharXiv. The effect is negligible on T1–T3 and negative on T4.
Suite
Geometry
h
λ
Residual merger
Without RP
With RP
Δ
T2
full
8
2.60
TIES
42.75
43.03
+0.28
T3
full
8
2.00
Task arithmetic
63.85
64.22
+0.37
T4
full
8
1.10
TIES
54.33
54.29
−0.04
Appendix
Table 29: Matched residual-pathway ablation: full branch, κ -rule λ . T3 averages include CharXiv.
Suite
Base branch
Neither
SS only
RP only
SS+RP
T2
CORAM
42.75
42.93
42.31
43.30
T2
CORAM-C
42.96
42.99
43.32
43.48
T3
CORAM
63.95
63.96
64.38
64.27
T3
CORAM-C
64.02
64.14
64.20
64.33
T4
CORAM
54.49
54.15
54.56
54.71
T4
CORAM-C
54.32
53.77
54.37
54.41
Appendix
Table 30: Branch-level interaction at the κ -rule point, with the best priority and merger per suite and branch. Matched comparisons: Tables 28 and 29 .
Contiguous
Spread
Spread + RP
Mixed-script, all (%)
3.5
17.6
12.9
Mixed-script, XSTest (%)
12.0
69.1
49.6
XSTest safe-prompt acc.
0.904
0.452
0.648
XSTest adversarial acc.
0.645
0.645
0.635
Mean length (chars)
1304
1283
1285
Empty (%)
0.0
0.0
0.0
Appendix
Table 31: Generation statistics over the 1,819 safety responses of the matched T4 comparison (owner-balance/ full , λ=1.20 ). Spread-only merging induces mixed-script drift on open-ended prompts rather than truncation or refusal by silence; the residual pathway partially repairs it.
Method
T1
T2
T3
T4
Fixed plain CORAM ( κ -rule)
57.72
42.75
63.95
54.49
Fixed CORAM-C + SS + RP ∗
57.62
43.48
63.23
54.43
Best per-suite CORAM ( κ -rule)
57.98
43.48
64.38
54.71
OrthoMerge (best variant)
57.13
42.13
63.56
54.74
Best weight-space baseline
56.23
43.45
63.70
54.57
Appendix
Table 32: A single configuration across all suites. The fixed plain configuration trails the per-suite best by 0.22 – 0.73 points but exceeds OrthoMerge on T1–T3 and remains within 0.25 on T4. ∗ The second row freezes the strongest T2 main-table configuration (ca + ave conflict masking, variance spread priority, TIES residual, rule λ ) and applies it to every suite; on T1 the residual pathway is not applicable and is omitted. It transfers to T1 (57.62, above OrthoMerge) and keeps its T2 result, but trails on T3 and T4, where its components were selected for other suites. The plain configuration in the first row is the more uniform suite-agnostic choice.
Table 41
Suite
Configuration
λ
h=8
h=16
h=32
h=64
T1
full
2.60
57.72
57.65
57.46
57.39
T2
full
2.60
42.75
42.40
42.57
42.29
T3
lr.ta
2.00
63.64
63.12
63.50
Collapse a
T4
lr.ta
1.10
54.48
54.23
54.25
53.44
Appendix
Table 35: Matched comparison across slice heights at the κ -rule coefficient. All non- h settings are fixed within each row; T3 entries are five-task averages. The h=8 cells repeat the retained-line values; the T2–T4 h=16 cells and the T1–T3 h=32/64 cells were evaluated on the newer hardware, while the T4 h=32/64 cells use the original hardware with the main-table safety protocol. a At h=64 the T3 merge collapses into degenerate unbounded generation (MMSI-Bench accuracy 0.0 ); the evaluation was aborted. The best value in every row is at h=8 .
Line
Tested h=16 value(s)
Best h=16
h=8 optimum
T2 nca ( λ 2.50/2.60)
42.30 / 42.35
42.35
42.96
T2 nca.ties ( λ 2.60)
42.80
42.80
43.32
T3 lr (grid)
62.96 / 63.54
63.54
63.43
T3 nca ( λ 1.50–1.70)
63.07–63.18
63.18
63.07
T3 nca.ta ( λ 1.60–2.00)
63.35–63.47
63.47
63.75
T3 nca+ave ( λ 1.65–2.00)
63.07–63.31
63.31
63.42
Appendix
Table 36: Representative h=16 comparisons. The strongest suite-level result remains at h=8 . T3 rows are five-task averages (CharXiv excluded) on both sides, matching Table 8 ; CharXiv was never evaluated at h=16 . ‡ Safety re-scored under the vLLM protocol; the affected points remain below the h=8 optimum under either scoring. This table is not a coefficient-matched comparison; the matched rule-coefficient results are reported in Table 35 .
Figure 4: In-domain average as a function of slice height at the κ -rule coefficient (one panel per suite; the T3 h=64 point is the degenerate collapse noted in Table 35 ).
Figure 5: In-domain versus OOD averages for all mergers in the main tables. The CORAM point is the strongest κ -rule main-table row of each suite.
Suite
Task
Published setting
Our setting
Published score
Reproduced score
Difference
T1
M-ARC
Unspecified
Zero-shot
44.75
35.15
−9.60
T1
AGIEval
Unspecified
Zero-shot
38.84
33.13
−5.71
T2
MMLU †
Unspecified
Zero-shot
55.57
57.42
+1.85
T2
AGIEval
Unspecified
Zero-shot
33.79
26.87
−6.92
T3
IFEval
Unspecified
Zero-shot
54.90
54.53
−0.37
T3
MMBench
Unspecified
Zero-shot
83.59
83.68
+0.09
Appendix
Table 37: Published and reproduced OOD scores for the strongest OrthoMerge variant per suite (OFT for T1, TSV-M+C for T2, TIES+C for T3). Differences are reproduced minus published.
Variant
T2
T3
T4
TA+G
35.50
62.59
51.97
TIES+G
39.67
63.12
46.51
TSV-M+G
40.05
62.61
50.21
TA+C
34.12
61.95
54.74
TIES+C
41.46
63.56
46.25
TSV-M+C
42.13
62.69
52.34
Appendix
Table 38: All six OrthoMerge variants under our harness. G/C denote plain/conflict-aware OrthoMerge geometry; TA, TIES, and TSV-M denote the residual merger. The T3 entry for TIES+C is scored under the main-table judge protocol (Table 39 ); the remaining T3 entries are from an earlier judge pass and are shown for the variant ranking only.
Suite
Published best variant
Published Avg
Reproduced best variant
Reproduced Avg
Difference
T1
Released OFT checkpoint
46.25
Released OFT checkpoint
57.13
+10.88
T2
TSV-M+C
42.07
TSV-M+C
42.13
+0.06
T3
TIES+C
64.04
TIES+C
63.56
−0.48
T4
–
–
TA+C
54.74
–
Appendix
Table 39: Summary of the OrthoMerge reproduction. The T1 gap follows from the ScienceQA extraction difference.
Suite
Method
Orig.
s18
s38
s68
s98
Mean
Std.
T1
CORAM
57.72
57.66
57.45
57.55
57.69
57.61
0.10
T1
Task arithmetic
56.23
55.96
55.81
55.89
55.71
55.92
0.18
T2
CORAM-C + SS + RP
43.48
43.32
43.33
43.35
43.27
43.35
0.08
T2
DARE-TIES
43.45
43.27
43.30
43.39
43.37
43.36
0.06
T3
CORAM + RP
64.38
64.37
64.38
64.37
64.33
64.37
0.02
T3
TIES
63.70
64.08
64.13
64.10
64.12
64.03
0.16
Appendix
Table 40: Multi-seed in-domain averages for the repeated CORAM checkpoints and the closest baselines. The T2 and T4 differences lie within the estimated evaluation noise; the table supports comparable performance rather than a significant advantage in these two closest comparisons.
Stage
T1: 8B
T2: 3B
T3: 7B
T4: 9B
Frequency
Slice-SVD cache, per checkpoint
27 GB
11 GB
25 GB
32 GB
Once per suite
Slice-SVD cache, suite total
162 GB
66 GB
100 GB
192 GB
Once per suite
Geometry merge
0.5–0.8 GPU hours
Once per configuration
λ -assembly
1–3 CPU minutes
Per coefficient
Full-suite evaluation
1.7 GPU hours
1.7 GPU hours
1.4 GPU hours
4 GPU hours
Per coefficient
Sweep over λ
10–12 full-suite evaluations
Per configuration
Appendix
Table 41: Measured storage and wall-clock cost across the four suites. Cache sizes cover the base model and all experts at h=8 . The T3 evaluation comprises the five VL tasks (0.8 GPU hours) and one CharXiv pass (about 0.6 hours). A sweep is dominated by evaluation cost; the measured sweep totals are 17–20 GPU hours on T2 and 40–48 GPU hours on T4. For reference, the per-tensor baseline merges are far cheaper: linear merging takes 31 s (3B) and 1 min 37 s (9B), and TIES takes 3 min 9 s and 10 min 48 s.
Model merging consolidates fine-tuned experts into one multi-task model without retraining. All existing data-free methods approach this problem entirely in weight space. Restricted to arithmetic on parameters, these methods never observe how each expert behaves, a signal that only emerges through forward evaluation. Accessing this behavioral signal requires inputs to evaluate on, which the data-free setting prohibits. We propose Merge++, a post-hoc method that addresses this by inverting the expert checkpoints to synthesize task-representative images, then distilling expert knowledge into the merged model using those images. Merge++ requires no additional data beyond the checkpoints themselves. It applies universally across merging algorithms and operates as a complementary refinement stage independent of the underlying weight-space method. The method consistently improves merging algorithms ranging from simple task arithmetic to state-of-the-art spectral methods, with average gains of +2 to +8 points and up to +25.9 on individual configurations.
Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. To perform well across multiple tasks, most existing merging methods use an additional dataset to find the coefficients for the best linear combination of task-specific weight updates. However, we identify an implicit regularization in this standard practice: searching over coefficients restricts the candidate models to a subspace spanned by task-specific weight updates. In this work, we investigate whether this regularization is actually useful. Surprisingly, empirical results show that optimizing merged-model weights without this regularization significantly boosts the performance of common merging methods across multiple architectures, domains, and even in an extremely data-limited scenario where only one instance is available per class. Moreover, directly optimizing the pretrained model weights even outperforms some existing merging methods. Analysis shows that better multi-task weights exist outside the subspace and can be found using multiple methods. We study different strategies for using the additional dataset, discussing their practical use and implications for model merging. Overall, this work calls for revisiting the existing model-merging pipeline, motivating a broader exploration of the weight space and a reconsideration of the implicit regularization induced by task arithmetic.
Model merging integrates the capabilities of multiple experts without joint retraining, but standard expert training optimizes task loss alone and does not guarantee good performance after merging. Merge-aware training (MAT) aims to improve merged performance, but existing methods do not fully account for common merging operations and add training cost. We observe that, from an expert's perspective, common merging methods can be described by three operations: Scale reweights its own update, Mask removes selected coordinates, and Perturb adds updates from other experts. Based on this view, we introduce SMAT (Simple MAT), which jointly optimizes expert loss and expected loss at simulated merged parameters generated by sampling scaling coefficients, masks, and additive noise. We further introduce periodic scheduling, kernel fusion, and parameter storage switching to make SMAT efficient, with one forward and one backward pass per step. Across four language and vision-language backbones, SMAT improves the mean score across five merging methods by 1.07-2.16 points over the strongest baseline for each backbone, with less than 2% training-time overhead over standard fine-tuning.
Yanggan Gu, Yuanyi Wang, Zhen Li +5
The Hong Kong Polytechnic University (PolyU) · The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong +1