Model merging combines separately fine-tuned experts into a single model without retraining. Standard practice trains every expert at the same LoRA rank, which couples the rank choice across experts even though each domain has its own best rank. We show this coupling is a property of the merge method, not of merging. We train 36 experts on three domains (mathematics, instruction following, multilingual) at four ranks and three model scales, then merge with six methods at uniform ranks and at each domain's own best rank. Giving experts different ranks costs averaging and utility-based selection nothing, and gains them up to 1.8 pp of multi-domain accuracy, while the conflict-resolution methods are inconsistent and can lose up to 4 pp. The split has a simple cause. Merging keeps only a fixed share of the pooled spectral components, and methods that choose them by size discard many that would have reduced loss. They also tend to discard the same expert: the instruction adapter has the smallest singular values, so it is dropped almost entirely and IFEval falls by 16 to 25 pp. Choosing components by their effect on loss instead keeps all three experts, and leaves accuracy nearly flat across an 8× rank range (0.7 pp, against 6.7 pp for TIES at 4B scale). Rank can therefore be chosen per expert whenever the merge operator does not select by magnitude.
Figures & tables
Model
Mathematics
Instruction
Multilingual
Qwen3.5-0.8B
8, 16, 32
16
4
Qwen3.5-2B
32
8
4
Qwen3.5-4B
4
32
8
Table 1: Single-task optimal LoRA rank per domain and model. Optima differ across domains and shift with scale, so any uniform choice is suboptimal for at least one domain. Note that within each domain the variance across ranks is modest (typically ≤ 2 pp), so while optimum differs, the penalty for mismatched rank is small for single-task use; the larger effect comes from merging.
Regime
Ranks (m/i/l)
R
pool
kept
E@R
uniform
4/4/4
4
12.3
0.326
0.794
uniform
8/8/8
8
24.1
0.332
0.827
uniform
16/16/16
16
48.2
0.332
0.869
uniform
32/32/32
32
96.2
0.332
0.876
diverse
8/16/4
16
28.4
0.565
0.898
diverse
4/16/32
32
52.4
0.611
0.891
Table 2: Spectral structure of the merged update (0.8B, averaged over 20–64 modules, two independent samples agreeing to ∼ 2 decimals). pool is the mean numerical rank of the merged ΔW (count of singular values >10−6σmax , which is approximately ∑iri but varies slightly across modules due to near-zero components). R=maxiri is the truncation budget; kept =R/pool ; and E@R is the Frobenius energy retained by the top- R components. Uniform assignments sit at \nicefrac13 kept at every rank, while diverse assignments retain substantially more.
Regime
Ranks
kept
overlap
clipped
frac. pos.
Eσ
Eutil
uniform
4/4/4
0.333
0.747
1.01
0.795
0.797
0.694
uniform
8/8/8
0.333
0.714
2.29
0.755
0.827
0.686
uniform
16/16/16
0.333
0.669
5.30
0.720
0.866
0.665
uniform
32/32/32
0.333
0.694
9.78
0.652
0.876
0.701
diverse
8/16/4
0.571
0.744
4.09
0.843
0.896
0.783
diverse
4/16/32
0.615
0.687
10.02
0.780
0.892
0.718
Table 3: Utility versus magnitude selection at identical budget R (0.8B, base-point gradients, 96 target matrices). frac. pos. exceeds kept in every configuration: the budget must reject loss-reducing directions. clipped is the mean count per matrix that utility keeps and magnitude discards.
Scale
Method
Math
IFEval
Multi
Avg
Δ
0.8B
GGSM
40.9
55.1
50.7
48.9
+1.4
SVD
42.4
48.6
51.2
47.4
+0.3
Simple avg
44.9
49.7
50.9
48.5
+1.8
Greedy soup
45.3
39.9
48.0
44.4
−4.1
TIES
37.6
39.7
47.9
41.7
−0.5
KnOTS
37.9
34.2
47.6
39.9
−2.1
Table 4: Method comparison across scales, best diverse configuration per method, where Δ=(best diverse)−(best uniform) . GGSM, SVD and averaging hold within noise, TIES and KnOTS perform inconsistently across scales while greedy soup degrades. IFEval drives the spread.
Figure 1: Left: sensitivity to uniform rank at 4B, where utility selection spans 0.7 pp against 6.7 pp for TIES. Right: share of the retained top- R subspace per expert under magnitude truncation (SVD), where one expert is often dropped almost entirely (e.g. 16/16/16→0.93/0.01/0.06 ). GGSM selects by utility rather than magnitude, so the eviction pattern differs; we leave that analysis to future work.
Scale
Method
Math
IFEval
Multi
Avg
0.8B
GGSM (utility)
38.6
55.5
48.5
47.5
SVD (magnitude)
39.7
47.7
50.8
46.1
Simple averaging
42.7
47.1
50.3
46.7
Greedy soup
44.7
50.8
50.0
48.5
TIES
39.3
39.2
48.1
42.2
KnOTS
41.9
36.0
48.2
42.1
Table 5: Per-domain scores at uniform rank 4 at 0.8B, 2B and 4B. IFEval collapses for TIES and KnOTS at every scale.
Scale
Method
r4
r8
r16
r32
Range
0.8B
GGSM (utility)
47.5
47.5
46.9
46.3
1.2
SVD (magnitude)
46.1
46.8
47.1
45.9
1.2
Simple averaging
46.7
46.5
46.2
44.9
1.8
KnOTS
42.1
37.7
35.0
34.6
7.5
TIES
42.2
41.0
39.7
37.9
4.3
2B
GGSM (utility)
60.7
61.2
61.7
60.6
1.1
Table 6: Uniform-rank sweep across scales (same-rank merges). GGSM is the most rank-stable at every scale; KnOTS and TIES are the most sensitive.
Figure 2: Left: the fraction of pooled components carrying positive first-order utility far exceeds the fraction the budget admits, which is fixed at T1 for T tasks under any uniform assignment. Right: the number of high-utility components that magnitude selection discards grows roughly tenfold from r=4 to r=32 (left axis), tracking the decline of TIES while utility-based selection stays flat (right axis).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Scale
Ranks (m/i/l)
Math
IFEval
Multi
Avg
0.8B
8/32/4 (optimal)
37.6
39.7
47.9
41.7
4/8/16
39.3
39.4
46.2
41.6
8/16/4
39.0
38.4
46.9
41.4
4/32/8
35.6
38.4
50.1
41.4
32/8/4
39.9
35.3
48.8
41.3
2B
16/8/4 (optimal)
62.8
54.3
55.9
57.7
Appendix
Table 7: TIES on diverse ranks across scales. At every size the best diverse configuration underperforms the best uniform-rank baseline (0.8B: 42.2 at r4; 2B: 57.3 at r8; 4B: 75.1 at r8) except at 2B where the diverse-rank setup is marginally higher.
Scale
Method
Math
IFEval
Multi
Avg
0.8B
GGSM (utility)
39.0
56.7
49.8
48.5
Simple averaging
41.6
48.2
49.8
46.5
SVD (magnitude)
41.3
47.3
51.2
46.6
Greedy soup
45.3
39.9
48.0
44.4
KnOTS
37.9
34.2
47.6
39.9
TIES
39.0
38.4
46.9
41.4
Appendix
Table 8: All methods on the per-domain optimal ranks at each scale (0.8B: 8/16/4; 2B: 32/8/4; 4B: 4/32/8). GGSM leads on Avg at every scale; TIES and KnOTS trail because IFEval collapses.
Method
Ingredients
Math
IFEval
Multi
Avg
GGSM (same)
16/16/16
61.4
67.1
56.7
61.7
Greedy (same)
m4, i4
64.6
61.6
57.6
61.3
GGSM (diverse)
8/16/4
61.5
67.5
58.2
62.4
Greedy (diverse)
m32, i4, l8
64.5
61.2
55.1
60.3
Appendix
Table 9: 2B results: Greedy soup vs GGSM on same-rank and diverse-rank configurations.
Scale
Domain
Best Rank
Score
0.8B
Math (GSM8K)
r8/r16/r32
34.5
Instruction (IFEval)
r16
54.2
Multilingual
r4
48.0
2B
Math (GSM8K)
r32
56.6
Instruction (IFEval)
r8
67.3
Multilingual
r4
57.7
Appendix
Table 10: Single-task expert baselines across scales (best rank per domain and its score).
Figure 3: Per-domain scores across uniform ranks at 4B. IFEval (center) shows the starkest method divergence: GGSM and SVD maintain 75.1 to 82.3 while TIES and KnOTS collapse to 49.2 to 70.4. Math (left) and Multilingual (right) vary little across methods.
Figure 4: Left: IFEval scores at 4B, uniform rank 4. TIES and KnOTS lose 16 to 25 pp relative to the single-task expert baseline (dashed line), while GGSM slightly exceeds it. Right: TIES performance across diverse rank configurations vs best uniform (r8). Every diverse configuration underperforms uniform on aggregate, primarily due to IFEval collapse.
Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training. Standard practice merges experts at their optimal validation loss. We challenge this convention by systematically studying how training duration of domain experts affects the quality of the merged model. We fine-tune experts on five domains (Math, Code, Instruction Following, Multilingual, and Safety) across three model sizes (Qwen 3.5 0.8B, 2B, and 4B), saving checkpoints from 25% to 500% of the optimal training steps and evaluating five merging methods at each duration. Our findings reveal a striking method-dependent pattern: simple averaging degrades sharply with overfitting, while sparsification-based methods achieve their best performance well past the validation optimum. We formalize this through bias-variance decomposition analysis, drawing a parallel to random forests where averaging benefits from high-variance individual learners. These results suggest that training duration and merging method should be chosen jointly rather than independently.
Model merging offers a training-free way to combine multiple post-trained expert models, but merging experts obtained through reinforcement learning (RL) remains challenging. Existing spectral merging methods often assume that leading singular directions contain the main task signal, while lower-energy residual components can be compressed, selected, or attenuated to reduce interference. We find that this assumption does not hold for RL task vectors: after decomposing each task vector into a leading spectral head and a residual component, both parts can independently recover substantial behavior knowledge, while exhibiting different merging properties. The head is highly concentrated and informative but more prone to sharp cross-expert conflicts, whereas the residual component is more dispersed and provides a more stable basis for aggregation. Based on this observation, we propose ResMerge, a residual-based spectral merging framework for RL experts. ResMerge first constructs a stable residual backbone with Spherical Residual Consensus Adaptation, which estimates a reliability-weighted consensus direction on the Frobenius sphere. It then reintroduces leading-head information through a Lightweight Head Correction module gated by positive cross-expert agreement. Experiments across multiple RL expert groups and capability domains show that ResMerge better preserves expert capabilities than representative task-vector and spectral merging baselines. The implementation of ResMerge is publicly available at https://github.com/sunyd0303-cpu/ResMerge-release.
Yandu Sun, Zhiyan Hou, Haokai Ma +6
1Southeast University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences +4
Merging low-rank adapters (LoRAs) promises to eliminate the overhead of swapping task-specific weights at inference time. However, existing merging methods assume every layer needs the same rank budget. Further, some methods assume that rank budget needs to be split equally among the tasks too. We show this uniform-budget assumption is a major source of the performance gap between merged and per-task LoRAs. However, rank selection is an NP hard problem. To this end, we introduce Net Utility, a data free metric that first decomposes every task LoRA by its Singular Value Decomposition (SVD) and scores each of those singular directions by its task utility and its interference with other tasks directions. Next, we globally pool these scores to select singular directions with the highest values with a constraint on the total number of directions selected. The proposed Net Utility metric is applied on top of five different merging methods across three different merging spaces. The merging is done over two sets of tasks, vision and language tasks. Net utility based rank allocation outperforms its counterparts without that allocation. On average, over vision tasks it achieves +2.1% improvement in performance, and +2.2% improvement over the language tasks.
Avinash Amballa, Yashas Malur Saidutta, Wenbo Li +2