Merging different models into a single model is highly desirable for multi-task deployment, particularly in regulated domains like healthcare. In such settings, models are often locally fine-tuned due to strict data privacy policies, necessitating that aggregation operates exclusively on the fine-tuned weights. Existing methods apply a uniform merge across all tasks, ignoring the geometric relationships among their learned subspaces. When task updates occupy misaligned subspaces, projecting them onto a shared low-rank basis discards task-relevant directions, a failure mode named subspace interference. Through this study, we show that the per-task projection error of a merged model is governed by the principal-angle alignment between each task and the shared subspace, so that grouping geometrically compatible tasks provably reduces this error within each group. Motivated by this observation, we propose Group-wise Rank-Aware Model Merging (GRAM), which uses the principal-angle geometry of task vectors to decide which tasks should be merged together and then performs a standard SVD-based merge within each coalition; the pipeline requires only the fine-tuned weights at merge time. With only one additional model (\nK=2), GRAM significantly mitigates subspace interference and consistently outperforms prior merging methods across language understanding (GLUE with Flan-T5), 8-task CLIP vision, and medical instruction tuning (Qwen2-7B-Instruct with four English/Chinese medical LoRAs).
Figures & tables
Figure 1: How tasks are grouped on the Grassmannian governs merging quality. Insets: three partitions of the per-task low-rank subspaces visualised on the Grassmannian—all tasks merged into a single global coalition (top, low intra-coalition similarity), three groups (middle), and the two-group GRAM partition (bottom, high intra-coalition similarity). Curve: as intra-coalition similarity increases, merged-model accuracy rises (left axis, solid) and the within-group subspace projection error ∑pairs∈G(1−cos2θ) falls (right axis, dashed). Highlighted markers tie each partition to its location on the trade-off curve: GRAM places geometrically compatible tasks in the same coalition, reducing the alignment penalty that global merging incurs.
Figure 2: The GRAM pipeline. Each task vector ΔWi is decomposed via SVD to extract its top- r left- and right-singular subspaces (Ui,Vi) (left). The collection of Ui ’s is fed into a subspace clustering step that partitions the T tasks into K coalitions by chordal distance (middle). Within each coalition, the selected top- ki singular components from member tasks are concatenated and re-orthonormalized via thin SVD to yield a single rank- r merged update (right).
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
Avg.
Individual (oracle)
69.1 / 80.1
82.7 / 88.5
85.5 / 89.2
90.9 / 94.4
84.0 / 87.1
84.4 / 91.3
92.9 / 95.2
87.4 / 90.9
84.6 / 89.6
Weight Averaging
69.7 / 74.6
59.7 / 84.3
78.9 / 84.1
90.1 / 92.8
83.8 / 86.2
80.5 / 85.9
91.2 / 94.8
72.0 / 88.0
78.2 / 86.3
Task Arithmetic
69.5 / 76.9
59.0 / 85.4
78.7 / 85.3
90.1 / 93.9
83.9 / 85.8
79.4 / 88.1
91.5 / 95.2
72.9 / 87.8
78.1 / 87.3
Concrete TA
69.1 / 76.6
58.1 / 86.3
78.4 / 86.3
89.9 / 93.9
83.5 / 85.9
79.4 / 88.4
91.6 / 95.2
73.4 / 87.9
78.0 / 87.5
DOGE TA
69.1 / 78.4
71.9 / 88.1
80.9 / 86.5
90.3 / 93.8
83.5 / 86.3
79.8 / 87.7
92.5 / 95.1
71.1 / 87.7
79.9 / 88.0
TIES-Merging
68.3 / 77.1
56.3 / 85.1
79.4 / 86.3
89.8 / 93.9
83.7 / 86.0
79.4 / 87.7
91.6 / 95.1
71.2 / 88.0
77.5 / 87.4
Table 1: GLUE results on Flan-T5-Base and Flan-T5-Large (LoRA r=16 ): validation accuracy, Spearman ρ×100 for STS-B. Each cell reports Base / Large in that order. Bold : best merged result per column for each variant. Individual: per-task fine-tuned model (oracle upper bound).
Method
MNIST
SVHN
GTSRB
DTD
Cars
RESISC45
EuroSAT
SUN397
Avg.
Individual (oracle)
99.7 / 99.8
97.6 / 98.1
99.1 / 99.2
97.7 / 98.5
78.2 / 92.2
95.9 / 97.3
99.8 / 99.8
75.3 / 82.0
92.9 / 95.9
Weight Averaging
86.5 / 96.8
62.7 / 77.7
53.3 / 68.9
54.7 / 66.8
62.6 / 81.7
71.9 / 82.3
71.0 / 88.7
65.1 / 71.5
66.0 / 79.3
Task Arithmetic
96.9 / 99.0
78.1 / 86.0
68.2 / 84.9
59.6 / 72.2
56.2 / 81.9
68.1 / 86.3
76.0 / 91.9
57.3 / 72.9
70.1 / 84.4
TIES-Merging
97.0 / 99.0
80.5 / 87.6
69.3 / 83.7
66.2 / 77.5
61.1 / 83.3
73.2 / 87.7
79.4 / 93.8
62.7 / 74.0
73.7 / 85.8
AdaMerging++
98.2 / 99.0
90.1 / 90.2
73.2 / 95.7
81.3 / 93.7
69.2 / 88.6
78.2 / 91.5
93.9 / 97.2
65.7 / 74.7
81.2 / 91.3
TSV-Merge
99.3 / 99.6
91.5 / 95.1
91.8 / 96.3
83.8 / 93.4
69.5 / 89.3
84.8 / 93.2
94.1 / 98.5
65.6 / 76.8
85.0 / 92.8
Table 2: Eight-task vision results on fully fine-tuned CLIP ViT-B/32 and ViT-L/14: test accuracy (%). Each cell reports B/32 / L/14 in that order. Bold : best merged result per column for each variant. Individual: per-task fine-tuned model (oracle upper bound).
MedQA
CMExam
HealthcareMagic
cMedQA2
Method
Acc.
Acc.
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
Avg Impr.
Avg Rank.
Pretrained
51.92
74.77
1.05
25.90
3.27
12.65
9.77
40.30
19.05
20.94
–
–
Individual (oracle)
60.25
88.63
5.43
30.60
8.07
18.48
11.17
41.71
19.23
24.02
+69.62%
4.55
Weight Averaging
56.64
85.63
4.49
30.71
7.13
17.70
10.78
41.26
20.08
22.28
+55.23%
7.65
Task Arithmetic
57.97
87.69
5.00
30.33
7.72
18.11
5.55
32.70
12.02
19.15
+49.40%
7.05
TIES-Merging
58.05
87.69
4.23
30.72
7.03
17.58
11.30
42.06
20.60
22.85
+54.18%
6.10
Table 3: Medical merging on Qwen2-7B-Instruct with four LoRA experts. MedQA / CMExam: accuracy; HealthcareMagic / cMedQA2: BLEU-4 / ROUGE-1/2/L. Avg Impr. : mean relative improvement over Pretrained across 10 columns; Avg Rank. : mean per-column rank ( 1 = best). Bold / underline : best / second-best per column among merging methods. Individual: per-task fine-tuned oracle.
Model
GLUE
ViT-B/32
GRAM
83.8
88.8
− PA-Sim
80.8_{\,{\color[rgb]{0.7,0,0}-3.0}}
88.3_{\,{\color[rgb]{0.7,0,0}-0.5}}
− Coalition
81.7_{\,{\color[rgb]{0.7,0,0}-2.1}}
85.4_{\,{\color[rgb]{0.7,0,0}-3.4}}
− Adapt-Rank
82.2_{\,{\color[rgb]{0.7,0,0}-1.6}}
88.7_{\,{\color[rgb]{0.7,0,0}-0.1}}
Table 4: Effects of the proposed modules. Subscripts ( red ) report the change relative to full GRAM.
Table 5: Clustering-algorithm ablation (Flan-T5-Base / GLUE): four ways of producing the K=2 partition from the same subspace geometry, plus the random control, under identical merging and evaluation.
Figure 3: Grouping cost vs. number of tasks T on one RTX 3080 (bars: wall-clock time; lines: peak GPU memory). T=8 is the real task set; larger T uses synthetic tasks with the same layer shapes and rank.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
K
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
Avg.
Extra params
1 (weight averaging)
69.7 / 74.6
59.7 / 84.3
78.9 / 84.1
90.1 / 92.8
83.8 / 86.2
80.5 / 85.9
91.2 / 94.8
72.0 / 88.0
78.2 / 86.3
—
1 (global merge)
69.1 / 79.2
74.2 / 86.0
80.4 / 87.3
89.7 / 93.6
83.5 / 85.6
80.9 / 87.0
92.7 / 95.3
84.6 / 88.7
81.9 / 87.8
—
2 (GRAM)
69.5 / 79.3
82.8 / 89.0
84.3 / 88.7
90.2 / 93.8
84.0 / 85.7
79.8 / 88.1
93.2 / 95.4
86.7 / 89.9
83.8 / 88.7
+1.8 M / +4.7 M
3
69.5 / 79.3
82.7 / 88.5
84.3 / 88.7
90.2 / 93.8
84.0 / 85.7
79.8 / 88.1
93.2 / 95.4
87.4 / 90.9
83.9 / 88.8
+3.5 M / +9.4 M
4
69.1 / 79.9
82.7 / 88.5
84.8 / 89.5
90.2 / 93.8
83.7 / 85.8
78.0 / 88.4
93.2 / 95.2
87.4 / 90.9
83.6 / 89.0
+5.3 M / +14.2 M
5
69.1 / 80.1
82.7 / 88.5
84.8 / 89.5
90.2 / 93.8
83.7 / 85.8
78.0 / 88.4
92.9 / 95.2
87.4 / 90.9
83.6 / 89.0
+7.1 M / +18.9 M
Appendix
Table 6: Sensitivity to the number of groups K on Flan-T5-Base and Flan-T5-Large / GLUE; each cell reports Base / Large. Partitions come from the same clustering GRAM uses; every group is re-merged with the adaptive-rank merge ( r=16 ) and freshly evaluated with the Table 1 pipeline, which reproduces the published K=2 row exactly on Large (on Base, re-merged K=1 scores 81.9 vs. 81.7 for the original checkpoint—SVD tie-breaking on near-degenerate spectra). The Weight Averaging row and the Large K∈{1,8} anchors are the corresponding Table 1 rows.
Setting
Intra-sim
Inter-sim
Ratio
Avg Acc
No grouping ( K=1 , global merge)
0.097
—
—
81.7
GRAM K=2
0.143
0.035
4.06
83.8
Random K=2
0.089
0.102
0.87
82.4
GRAM K=3
0.150
0.036
4.21
83.9
Random K=3
0.119
0.089
1.34
83.1
Appendix
Table 7: GRAM vs. random grouping quality. Intra-sim and Inter-sim denote the average subspace similarity ( 1−dc2/r ) within and between groups, respectively. Ratio = Intra-sim / Inter-sim. Higher ratio indicates better separation.
K
2
3
4
5
6
7
8 (full task ID)
Group-routing acc.
100.00
100.00
100.00
100.00
99.99
99.98
99.98
Appendix
Table 8: Group-routing accuracy (%) of the Mahalanobis task mapper under the partition chain for every K , Flan-T5-Base / GLUE. K=8 requires full task identification.
Figure 4: Number of singular values required to capture ≥99% of Frobenius energy, per task (rows) and per LoRA layer (columns), on Flan-T5-Base. Decoder layers (left half) are typically low-rank ( 1 – 4 significant directions); encoder layers (right half) are higher-rank ( 10 – 16 ). The cross-task and cross-layer heterogeneity motivates the adaptive rank allocator in section 4.3 .
Figure 5: Pairwise projection error ∑sσi,s2sin2θs between task subspaces on ViT-B/32, under K=1 (TSV-Merge, left) and K=2 (GRAM, right). Reordering at K=2 by GRAM’s partition exposes a block-diagonal structure: within-group pairs incur lower error than across-group pairs, consistent with Lemma B.3 .
Method
MNIST
SVHN
GTSRB
DTD
Cars
RESISC45
EuroSAT
SUN397
Avg.
Individual (oracle)
99.8
97.8
99.0
98.3
87.5
97.0
99.8
78.8
94.8
Weight Averaging
94.4
72.6
60.6
54.0
69.9
75.2
81.3
68.6
72.1
Task Arithmetic
98.3
84.5
76.1
60.4
68.1
73.7
86.4
64.4
76.5
TIES-Merging
98.3
84.8
73.9
64.8
71.7
77.8
89.4
68.0
78.6
AdaMerging++
98.4
93.1
92.3
64.0
79.2
81.1
95.6
69.2
84.1
TSV-Merge
99.3
92.9
94.5
85.2
81.0
88.5
97.4
70.6
88.7
Appendix
Table 9: Eight-task vision benchmark results (test accuracy) using CLIP ViT-B/16 with full fine-tuning. Bold : best merged result per column; underline : second best. Individual adapters use one task-specific model per task (oracle upper bound).
Model merging aims to integrate multiple task-specific fine-tuned models derived from a shared pre-trained checkpoint into a single multi-task model without additional training. Despite extensive research, task interference remains a major obstacle that often undermines the performance of merged models. In this paper, we propose ESM (Essential Subspace Merging) , a robust framework for effective model merging. We begin by performing Principal Component Analysis (PCA) on feature shifts induced by parameter updates. The resulting principal directions span an essential subspace that dominantly influences feature representations. Each task's parameter update matrix is projected onto its respective essential subspace for low-rank decomposition before merging. This methodology mitigates inter-task interference while preserving core task-specific functionality. Furthermore, we introduce a multi-level polarized scaling strategy that amplifies parameters containing critical knowledge and suppresses redundant ones, preventing essential knowledge from being overwhelmed during fusion. Extensive experiments across multiple task sets and model scales demonstrate that our method achieves state-of-the-art performance in multi-task model merging. Code is available at https://github.com/kiddo127/ESM.
Longhua Li, Lei Qi, Qi Tian +1
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shanghai, China
Model merging provides an efficient way to construct multi-task generalist models without additional training, but its performance often degrades under severe task interference. Task interference in model merging primarily stems from superposition, where task-specific features become entangled within the parameter space. This entanglement renders conventional decomposition methods insufficient for effectively isolating useful task directions from interfering components. In this paper, we propose a sparse-representation-based merging framework that uses Sparse Autoencoders (SAEs) to project task vectors into a high-dimensional sparse feature space, enabling feature-level disentanglement before fusion. To reduce computational overhead, we further introduce a lightweight Group-Ranked Zeroth-Order Optimizer (GR-ZOO) to identify task-critical layers for selective merging. Experiments on both Qwen2.5-1.5B and Qwen2.5-7B demonstrate that our method consistently outperforms representative baselines, including Task Arithmetic, TIES-Merge, DARE, Fisher-Merge,and several recent training-free merging methods, across mathematical reasoning, code generation, instruction following, and general knowledge tasks. In a highly conflicting four-task setting on Qwen2.5-1.5B, our method further achieves a 2.78% improvement over the strongest baseline.
Yihang Zhang, Shengke Sun, Junjie Wen +1
Central South University · Nanjing University of Science and Technology · Hefei University of Technology
Model merging enables training-free multi-task learning by aggregating knowledge from models fine-tuned from a shared checkpoint, but their task updates can interfere. Using proxy data, we find that task updates induce output shifts whose energy concentrates in a few dominant directions, while many low-energy directions can accumulate across tasks and impair performance. We call the span of the dominant directions the essential subspace. This observation motivates a data-aware framework to preserve essential task knowledge while reducing interference. Essential Subspace Decomposition (ESD) identifies each task's essential subspace from activation shifts measured on proxy data and represents its update compactly while preserving functional behavior. Essential Subspace Merging (ESM) weights each task-update direction by its corresponding output-shift eigenvalue, thereby prioritizing core task knowledge; it then orthogonalizes the weighted directions across tasks to reduce interference. ESM++ extends this merged model by dynamically integrating low-rank task-specific residual experts for adaptive knowledge aggregation. Experiments across diverse task sets and model scales show that ESM and ESM++ effectively aggregate task knowledge while mitigating inter-task interference. Code is available at https://github.com/kiddo127/ESM.
Longhua Li, Lei Qi, Qi Tian +1
the School of Computer Science and Engineering, Southeast University, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China, 211189 · Huawei Inc., Shenzhen 518129, China