GRAM: Group-wise Rank-Aware Model Merging via Subspace Alignment

Di Wu; Shuaidong Pan

cs.AIOct 21, 2026

GRAM: Group-wise Rank-Aware Model Merging via Subspace Alignment

Authors: Di Wu, Shuaidong Pan

Organizations: JPMorgan Chase & Co.

Abstract

Merging different models into a single model is highly desirable for multi-task deployment, particularly in regulated domains like healthcare. In such settings, models are often locally fine-tuned due to strict data privacy policies, necessitating that aggregation operates exclusively on the fine-tuned weights. Existing methods apply a uniform merge across all tasks, ignoring the geometric relationships among their learned subspaces. When task updates occupy misaligned subspaces, projecting them onto a shared low-rank basis discards task-relevant directions, a failure mode named subspace interference. Through this study, we show that the per-task projection error of a merged model is governed by the principal-angle alignment between each task and the shared subspace, so that grouping geometrically compatible tasks provably reduces this error within each group. Motivated by this observation, we propose Group-wise Rank-Aware Model Merging (GRAM), which uses the principal-angle geometry of task vectors to decide which tasks should be merged together and then performs a standard SVD-based merge within each coalition; the pipeline requires only the fine-tuned weights at merge time. With only one additional model ( $\nK{=}2$ ), GRAM significantly mitigates subspace interference and consistently outperforms prior merging methods across language understanding (GLUE with Flan-T5), 8-task CLIP vision, and medical instruction tuning (Qwen2-7B-Instruct with four English/Chinese medical LoRAs).

Figures & tables

Figure 1: How tasks are grouped on the Grassmannian governs merging quality. Insets: three partitions of the per-task low-rank subspaces visualised on the Grassmannian—all tasks merged into a single global coalition (top, low intra-coalition similarity), three groups (middle), and the two-group GRAM partition (bottom, high intra-coalition similarity). Curve: as intra-coalition similarity increases, merged-model accuracy rises (left axis, solid) and the within-group subspace projection error $\sum_{\mathrm{pairs}\in G}(1{-}\cos^{2}\theta)$ falls (right axis, dashed). Highlighted markers tie each partition to its location on the trade-off curve: GRAM places geometrically compatible tasks in the same coalition, reducing the alignment penalty that global merging incurs.

Figure 2: The GRAM pipeline. Each task vector $\Delta W_{i}$ is decomposed via SVD to extract its top- $r$ left- and right-singular subspaces $(U_{i},V_{i})$ (left). The collection of $U_{i}$ ’s is fed into a subspace clustering step that partitions the $T$ tasks into $K$ coalitions by chordal distance (middle). Within each coalition, the selected top- $k_{i}$ singular components from member tasks are concatenated and re-orthonormalized via thin SVD to yield a single rank- $r$ merged update (right).

Method	CoLA	MNLI	MRPC	QNLI	QQP	RTE	SST-2	STS-B	Avg.
Individual (oracle)	69.1 / 80.1	82.7 / 88.5	85.5 / 89.2	90.9 / 94.4	84.0 / 87.1	84.4 / 91.3	92.9 / 95.2	87.4 / 90.9	84.6 / 89.6
Weight Averaging	69.7 / 74.6	59.7 / 84.3	78.9 / 84.1	90.1 / 92.8	83.8 / 86.2	80.5 / 85.9	91.2 / 94.8	72.0 / 88.0	78.2 / 86.3
Task Arithmetic	69.5 / 76.9	59.0 / 85.4	78.7 / 85.3	90.1 / 93.9	83.9 / 85.8	79.4 / 88.1	91.5 / 95.2	72.9 / 87.8	78.1 / 87.3
Concrete TA	69.1 / 76.6	58.1 / 86.3	78.4 / 86.3	89.9 / 93.9	83.5 / 85.9	79.4 / 88.4	91.6 / 95.2	73.4 / 87.9	78.0 / 87.5
DOGE TA	69.1 / 78.4	71.9 / 88.1	80.9 / 86.5	90.3 / 93.8	83.5 / 86.3	79.8 / 87.7	92.5 / 95.1	71.1 / 87.7	79.9 / 88.0
TIES-Merging	68.3 / 77.1	56.3 / 85.1	79.4 / 86.3	89.8 / 93.9	83.7 / 86.0	79.4 / 87.7	91.6 / 95.1	71.2 / 88.0	77.5 / 87.4

Table 1: GLUE results on Flan-T5-Base and Flan-T5-Large (LoRA

r{=}16

): validation accuracy, Spearman

\rho{\times}100

for STS-B. Each cell reports Base / Large in that order. Bold : best merged result per column for each variant. Individual: per-task fine-tuned model (oracle upper bound).

Method	MNIST	SVHN	GTSRB	DTD	Cars	RESISC45	EuroSAT	SUN397	Avg.
Individual (oracle)	99.7 / 99.8	97.6 / 98.1	99.1 / 99.2	97.7 / 98.5	78.2 / 92.2	95.9 / 97.3	99.8 / 99.8	75.3 / 82.0	92.9 / 95.9
Weight Averaging	86.5 / 96.8	62.7 / 77.7	53.3 / 68.9	54.7 / 66.8	62.6 / 81.7	71.9 / 82.3	71.0 / 88.7	65.1 / 71.5	66.0 / 79.3
Task Arithmetic	96.9 / 99.0	78.1 / 86.0	68.2 / 84.9	59.6 / 72.2	56.2 / 81.9	68.1 / 86.3	76.0 / 91.9	57.3 / 72.9	70.1 / 84.4
TIES-Merging	97.0 / 99.0	80.5 / 87.6	69.3 / 83.7	66.2 / 77.5	61.1 / 83.3	73.2 / 87.7	79.4 / 93.8	62.7 / 74.0	73.7 / 85.8
AdaMerging++	98.2 / 99.0	90.1 / 90.2	73.2 / 95.7	81.3 / 93.7	69.2 / 88.6	78.2 / 91.5	93.9 / 97.2	65.7 / 74.7	81.2 / 91.3
TSV-Merge	99.3 / 99.6	91.5 / 95.1	91.8 / 96.3	83.8 / 93.4	69.5 / 89.3	84.8 / 93.2	94.1 / 98.5	65.6 / 76.8	85.0 / 92.8

Table 2: Eight-task vision results on fully fine-tuned CLIP ViT-B/32 and ViT-L/14: test accuracy (%). Each cell reports B/32 / L/14 in that order. Bold : best merged result per column for each variant. Individual: per-task fine-tuned model (oracle upper bound).

	MedQA	CMExam	HealthcareMagic				cMedQA2
Method	Acc.	Acc.	BLEU-4	ROUGE-1	ROUGE-2	ROUGE-L	BLEU-4	ROUGE-1	ROUGE-2	ROUGE-L	Avg Impr.	Avg Rank.
Pretrained	51.92	74.77	1.05	25.90	3.27	12.65	9.77	40.30	19.05	20.94	–	–
Individual (oracle)	60.25	88.63	5.43	30.60	8.07	18.48	11.17	41.71	19.23	24.02	+69.62%	4.55
Weight Averaging	56.64	85.63	4.49	30.71	7.13	17.70	10.78	41.26	20.08	22.28	+55.23%	7.65
Task Arithmetic	57.97	87.69	5.00	30.33	7.72	18.11	5.55	32.70	12.02	19.15	+49.40%	7.05
TIES-Merging	58.05	87.69	4.23	30.72	7.03	17.58	11.30	42.06	20.60	22.85	+54.18%	6.10

Table 3: Medical merging on Qwen2-7B-Instruct with four LoRA experts. MedQA / CMExam: accuracy; HealthcareMagic / cMedQA2: BLEU-4 / ROUGE-1/2/L. Avg Impr. : mean relative improvement over Pretrained across

10

columns; Avg Rank. : mean per-column rank (

1

= best). Bold / underline : best / second-best per column among merging methods. Individual: per-task fine-tuned oracle.

Model	GLUE	ViT-B/32
GRAM	83.8	88.8
$-$ PA-Sim	$80.8_{\,{\color[rgb]{0.7,0,0}-3.0}}$	$88.3_{\,{\color[rgb]{0.7,0,0}-0.5}}$
$-$ Coalition	$81.7_{\,{\color[rgb]{0.7,0,0}-2.1}}$	$85.4_{\,{\color[rgb]{0.7,0,0}-3.4}}$
$-$ Adapt-Rank	$82.2_{\,{\color[rgb]{0.7,0,0}-1.6}}$	$88.7_{\,{\color[rgb]{0.7,0,0}-0.1}}$

Table 4: Effects of the proposed modules. Subscripts ( red ) report the change relative to full GRAM.

Partitioning of the same subspace geometry	$K{=}2$ partition	Avg.
GRAM	{CoLA, MRPC, QNLI, QQP, RTE, SST-2} $\mid$ {MNLI, STS-B}	83.8
Agglomerative (average linkage)	identical partition	83.8
Max-modularity communities (exact, all 127 bipartitions)	{CoLA, MNLI, SST-2, STS-B} $\mid$ {MRPC, QNLI, QQP, RTE}	82.9
Random equipartition	random $4/4$	82.4
No grouping ( $K{=}1$ , re-merged)	all tasks	81.9

Table 5: Clustering-algorithm ablation (Flan-T5-Base / GLUE): four ways of producing the

K{=}2

partition from the same subspace geometry, plus the random control, under identical merging and evaluation.

Figure 3: Grouping cost vs. number of tasks $T$ on one RTX 3080 (bars: wall-clock time; lines: peak GPU memory). $T{=}8$ is the real task set; larger $T$ uses synthetic tasks with the same layer shapes and rank.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

$K$	CoLA	MNLI	MRPC	QNLI	QQP	RTE	SST-2	STS-B	Avg.	Extra params
1 (weight averaging)	69.7 / 74.6	59.7 / 84.3	78.9 / 84.1	90.1 / 92.8	83.8 / 86.2	80.5 / 85.9	91.2 / 94.8	72.0 / 88.0	78.2 / 86.3	—
1 (global merge)	69.1 / 79.2	74.2 / 86.0	80.4 / 87.3	89.7 / 93.6	83.5 / 85.6	80.9 / 87.0	92.7 / 95.3	84.6 / 88.7	81.9 / 87.8	—
2 (GRAM)	69.5 / 79.3	82.8 / 89.0	84.3 / 88.7	90.2 / 93.8	84.0 / 85.7	79.8 / 88.1	93.2 / 95.4	86.7 / 89.9	83.8 / 88.7	$+1.8$ M / $+4.7$ M
3	69.5 / 79.3	82.7 / 88.5	84.3 / 88.7	90.2 / 93.8	84.0 / 85.7	79.8 / 88.1	93.2 / 95.4	87.4 / 90.9	83.9 / 88.8	$+3.5$ M / $+9.4$ M
4	69.1 / 79.9	82.7 / 88.5	84.8 / 89.5	90.2 / 93.8	83.7 / 85.8	78.0 / 88.4	93.2 / 95.2	87.4 / 90.9	83.6 / 89.0	$+5.3$ M / $+14.2$ M
5	69.1 / 80.1	82.7 / 88.5	84.8 / 89.5	90.2 / 93.8	83.7 / 85.8	78.0 / 88.4	92.9 / 95.2	87.4 / 90.9	83.6 / 89.0	$+7.1$ M / $+18.9$ M

Appendix

Table 6: Sensitivity to the number of groups

K

on Flan-T5-Base and Flan-T5-Large / GLUE; each cell reports Base / Large. Partitions come from the same clustering GRAM uses; every group is re-merged with the adaptive-rank merge (

r{=}16

) and freshly evaluated with the Table 1 pipeline, which reproduces the published

K{=}2

row exactly on Large (on Base, re-merged

K{=}1

scores 81.9 vs. 81.7 for the original checkpoint—SVD tie-breaking on near-degenerate spectra). The Weight Averaging row and the Large

K{\in}\{1,8\}

anchors are the corresponding Table 1 rows.

Setting	Intra-sim	Inter-sim	Ratio	Avg Acc
No grouping ( $K{=}1$ , global merge)	0.097	—	—	81.7
GRAM $K{=}2$	0.143	0.035	4.06	83.8
Random $K{=}2$	0.089	0.102	0.87	82.4
GRAM $K{=}3$	0.150	0.036	4.21	83.9
Random $K{=}3$	0.119	0.089	1.34	83.1

Appendix

Table 7: GRAM vs. random grouping quality. Intra-sim and Inter-sim denote the average subspace similarity (

1-d_{c}^{2}/r

) within and between groups, respectively. Ratio = Intra-sim / Inter-sim. Higher ratio indicates better separation.

$K$	2	3	4	5	6	7	8 (full task ID)
Group-routing acc.	100.00	100.00	100.00	100.00	99.99	99.98	99.98

Appendix

Table 8: Group-routing accuracy (%) of the Mahalanobis task mapper under the partition chain for every

K

, Flan-T5-Base / GLUE.

K{=}8

requires full task identification.

Figure 4: Number of singular values required to capture $\geq 99\%$ of Frobenius energy, per task (rows) and per LoRA layer (columns), on Flan-T5-Base. Decoder layers (left half) are typically low-rank ( $1$ – $4$ significant directions); encoder layers (right half) are higher-rank ( $10$ – $16$ ). The cross-task and cross-layer heterogeneity motivates the adaptive rank allocator in section 4.3 .

Figure 5: Pairwise projection error $\sum_{s}\sigma_{i,s}^{2}\sin^{2}\theta_{s}$ between task subspaces on ViT-B/32, under $K{=}1$ (TSV-Merge, left) and $K{=}2$ (GRAM, right). Reordering at $K{=}2$ by GRAM’s partition exposes a block-diagonal structure: within-group pairs incur lower error than across-group pairs, consistent with Lemma B.3 .

Method	MNIST	SVHN	GTSRB	DTD	Cars	RESISC45	EuroSAT	SUN397	Avg.
Individual (oracle)	99.8	97.8	99.0	98.3	87.5	97.0	99.8	78.8	94.8
Weight Averaging	94.4	72.6	60.6	54.0	69.9	75.2	81.3	68.6	72.1
Task Arithmetic	98.3	84.5	76.1	60.4	68.1	73.7	86.4	64.4	76.5
TIES-Merging	98.3	84.8	73.9	64.8	71.7	77.8	89.4	68.0	78.6
AdaMerging++	98.4	93.1	92.3	64.0	79.2	81.1	95.6	69.2	84.1
TSV-Merge	99.3	92.9	94.5	85.2	81.0	88.5	97.4	70.6	88.7

Appendix

Table 9: Eight-task vision benchmark results (test accuracy) using CLIP ViT-B/16 with full fine-tuning. Bold : best merged result per column; underline : second best. Individual adapters use one task-specific model per task (oracle upper bound).

Language Model Merging

Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.

See all 109 papers

Model Merging in the Essential Subspace
Feb 23, 2026Longhua Li, Lei Qi, Qi Tian +1Task Interference Model Merging
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shanghai, China
Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement
Aug 26, 2026Yihang Zhang, Shengke Sun, Junjie Wen +1Language Model Merging Model Merging
Central South University · Nanjing University of Science and Technology · Hefei University of Technology
Data-Aware Model Merging in the Essential Subspace for Multi-Task Knowledge Aggregation
Jun 17, 2026Longhua Li, Lei Qi, Qi Tian +1Multi-Task Learning Model Merging
the School of Computer Science and Engineering, Southeast University, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China, 211189 · Huawei Inc., Shenzhen 518129, China

Feb 23, 2026J/K move · Enter open · S save

Model Merging in the Essential Subspace

Longhua Li, Lei Qi, Qi Tian, Xin Geng

School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shanghai, China

Task Interference Model Merging Task Vector Merging

Open paper AskPDF arXiv

Model merging aims to integrate multiple task-specific fine-tuned models derived from a shared pre-trained checkpoint into a single multi-task model without additional training. Despite extensive research, task interference remains a major obstacle that often undermines the performance of merged models. In this paper, we propose ESM (Essential Subspace Merging) , a robust framework for effective model merging. We begin by performing Principal Component Analysis (PCA) on feature shifts induced by parameter updates. The resulting principal directions span an essential subspace that dominantly influences feature representations. Each task's parameter update matrix is projected onto its respective essential subspace for low-rank decomposition before merging. This methodology mitigates inter-task interference while preserving core task-specific functionality. Furthermore, we introduce a multi-level polarized scaling strategy that amplifies parameters containing critical knowledge and suppresses redundant ones, preventing essential knowledge from being overwhelmed during fusion. Extensive experiments across multiple task sets and model scales demonstrate that our method achieves state-of-the-art performance in multi-task model merging. Code is available at https://github.com/kiddo127/ESM.

Abstract

Language Model Merging

Explore similar work

Model Merging in the Essential Subspace

Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement

Data-Aware Model Merging in the Essential Subspace for Multi-Task Knowledge Aggregation