Model merging combines separately fine-tuned experts into a single model without retraining. Standard practice trains every expert at the same LoRA rank, which couples the rank choice across experts even though each domain has its own best rank. We show this coupling is a property of the merge method, not of merging. We train 36 experts on three domains (mathematics, instruction following, multilingual) at four ranks and three model scales, then merge with six methods at uniform ranks and at each domain's own best rank. Giving experts different ranks costs averaging and utility-based selection nothing, and gains them up to 1.8 pp of multi-domain accuracy, while the conflict-resolution methods are inconsistent and can lose up to 4 pp. The split has a simple cause. Merging keeps only a fixed share of the pooled spectral components, and methods that choose them by size discard many that would have reduced loss. They also tend to discard the same expert: the instruction adapter has the smallest singular values, so it is dropped almost entirely and IFEval falls by 16 to 25 pp. Choosing components by their effect on loss instead keeps all three experts, and leaves accuracy nearly flat across an 8× rank range (0.7 pp, against 6.7 pp for TIES at 4B scale). Rank can therefore be chosen per expert whenever the merge operator does not select by magnitude.
Figures & tables
Model
Mathematics
Instruction
Multilingual
Qwen3.5-0.8B
8, 16, 32
16
4
Qwen3.5-2B
32
8
4
Qwen3.5-4B
4
32
8
Table 1: Single-task optimal LoRA rank per domain and model. Optima differ across domains and shift with scale, so any uniform choice is suboptimal for at least one domain. Note that within each domain the variance across ranks is modest (typically ≤ 2 pp), so while optimum differs, the penalty for mismatched rank is small for single-task use; the larger effect comes from merging.
Regime
Ranks (m/i/l)
R
pool
kept
E@R
uniform
4/4/4
4
12.3
0.326
0.794
uniform
8/8/8
8
24.1
0.332
0.827
uniform
16/16/16
16
48.2
0.332
0.869
uniform
32/32/32
32
96.2
0.332
0.876
diverse
8/16/4
16
28.4
0.565
0.898
diverse
4/16/32
32
52.4
0.611
0.891
Table 2: Spectral structure of the merged update (0.8B, averaged over 20–64 modules, two independent samples agreeing to ∼ 2 decimals). pool is the mean numerical rank of the merged ΔW (count of singular values >10−6σmax , which is approximately ∑iri but varies slightly across modules due to near-zero components). R=maxiri is the truncation budget; kept =R/pool ; and E@R is the Frobenius energy retained by the top- R components. Uniform assignments sit at \nicefrac13 kept at every rank, while diverse assignments retain substantially more.
Regime
Ranks
kept
overlap
clipped
frac. pos.
Eσ
Eutil
uniform
4/4/4
0.333
0.747
1.01
0.795
0.797
0.694
uniform
8/8/8
0.333
0.714
2.29
0.755
0.827
0.686
uniform
16/16/16
0.333
0.669
5.30
0.720
0.866
0.665
uniform
32/32/32
0.333
0.694
9.78
0.652
0.876
0.701
diverse
8/16/4
0.571
0.744
4.09
0.843
0.896
0.783
diverse
4/16/32
0.615
0.687
10.02
0.780
0.892
0.718
Table 3: Utility versus magnitude selection at identical budget R (0.8B, base-point gradients, 96 target matrices). frac. pos. exceeds kept in every configuration: the budget must reject loss-reducing directions. clipped is the mean count per matrix that utility keeps and magnitude discards.
Scale
Method
Math
IFEval
Multi
Avg
Δ
0.8B
GGSM
40.9
55.1
50.7
48.9
+1.4
SVD
42.4
48.6
51.2
47.4
+0.3
Simple avg
44.9
49.7
50.9
48.5
+1.8
Greedy soup
45.3
39.9
48.0
44.4
−4.1
TIES
37.6
39.7
47.9
41.7
−0.5
KnOTS
37.9
34.2
47.6
39.9
−2.1
Table 4: Method comparison across scales, best diverse configuration per method, where Δ=(best diverse)−(best uniform) . GGSM, SVD and averaging hold within noise, TIES and KnOTS perform inconsistently across scales while greedy soup degrades. IFEval drives the spread.
Figure 1: Left: sensitivity to uniform rank at 4B, where utility selection spans 0.7 pp against 6.7 pp for TIES. Right: share of the retained top- R subspace per expert under magnitude truncation (SVD), where one expert is often dropped almost entirely (e.g. 16/16/16→0.93/0.01/0.06 ). GGSM selects by utility rather than magnitude, so the eviction pattern differs; we leave that analysis to future work.
Scale
Method
Math
IFEval
Multi
Avg
0.8B
GGSM (utility)
38.6
55.5
48.5
47.5
SVD (magnitude)
39.7
47.7
50.8
46.1
Simple averaging
42.7
47.1
50.3
46.7
Greedy soup
44.7
50.8
50.0
48.5
TIES
39.3
39.2
48.1
42.2
KnOTS
41.9
36.0
48.2
42.1
Table 5: Per-domain scores at uniform rank 4 at 0.8B, 2B and 4B. IFEval collapses for TIES and KnOTS at every scale.
Scale
Method
r4
r8
r16
r32
Range
0.8B
GGSM (utility)
47.5
47.5
46.9
46.3
1.2
SVD (magnitude)
46.1
46.8
47.1
45.9
1.2
Simple averaging
46.7
46.5
46.2
44.9
1.8
KnOTS
42.1
37.7
35.0
34.6
7.5
TIES
42.2
41.0
39.7
37.9
4.3
2B
GGSM (utility)
60.7
61.2
61.7
60.6
1.1
Table 6: Uniform-rank sweep across scales (same-rank merges). GGSM is the most rank-stable at every scale; KnOTS and TIES are the most sensitive.
Figure 2: Left: the fraction of pooled components carrying positive first-order utility far exceeds the fraction the budget admits, which is fixed at T1 for T tasks under any uniform assignment. Right: the number of high-utility components that magnitude selection discards grows roughly tenfold from r=4 to r=32 (left axis), tracking the decline of TIES while utility-based selection stays flat (right axis).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Scale
Ranks (m/i/l)
Math
IFEval
Multi
Avg
0.8B
8/32/4 (optimal)
37.6
39.7
47.9
41.7
4/8/16
39.3
39.4
46.2
41.6
8/16/4
39.0
38.4
46.9
41.4
4/32/8
35.6
38.4
50.1
41.4
32/8/4
39.9
35.3
48.8
41.3
2B
16/8/4 (optimal)
62.8
54.3
55.9
57.7
Appendix
Table 7: TIES on diverse ranks across scales. At every size the best diverse configuration underperforms the best uniform-rank baseline (0.8B: 42.2 at r4; 2B: 57.3 at r8; 4B: 75.1 at r8) except at 2B where the diverse-rank setup is marginally higher.
Scale
Method
Math
IFEval
Multi
Avg
0.8B
GGSM (utility)
39.0
56.7
49.8
48.5
Simple averaging
41.6
48.2
49.8
46.5
SVD (magnitude)
41.3
47.3
51.2
46.6
Greedy soup
45.3
39.9
48.0
44.4
KnOTS
37.9
34.2
47.6
39.9
TIES
39.0
38.4
46.9
41.4
Appendix
Table 8: All methods on the per-domain optimal ranks at each scale (0.8B: 8/16/4; 2B: 32/8/4; 4B: 4/32/8). GGSM leads on Avg at every scale; TIES and KnOTS trail because IFEval collapses.
Method
Ingredients
Math
IFEval
Multi
Avg
GGSM (same)
16/16/16
61.4
67.1
56.7
61.7
Greedy (same)
m4, i4
64.6
61.6
57.6
61.3
GGSM (diverse)
8/16/4
61.5
67.5
58.2
62.4
Greedy (diverse)
m32, i4, l8
64.5
61.2
55.1
60.3
Appendix
Table 9: 2B results: Greedy soup vs GGSM on same-rank and diverse-rank configurations.
Scale
Domain
Best Rank
Score
0.8B
Math (GSM8K)
r8/r16/r32
34.5
Instruction (IFEval)
r16
54.2
Multilingual
r4
48.0
2B
Math (GSM8K)
r32
56.6
Instruction (IFEval)
r8
67.3
Multilingual
r4
57.7
Appendix
Table 10: Single-task expert baselines across scales (best rank per domain and its score).
Figure 3: Per-domain scores across uniform ranks at 4B. IFEval (center) shows the starkest method divergence: GGSM and SVD maintain 75.1 to 82.3 while TIES and KnOTS collapse to 49.2 to 70.4. Math (left) and Multilingual (right) vary little across methods.
Figure 4: Left: IFEval scores at 4B, uniform rank 4. TIES and KnOTS lose 16 to 25 pp relative to the single-task expert baseline (dashed line), while GGSM slightly exceeds it. Right: TIES performance across diverse rank configurations vs best uniform (r8). Every diverse configuration underperforms uniform on aggregate, primarily due to IFEval collapse.