Organizations: School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shanghai, China
Model merging aims to integrate multiple task-specific fine-tuned models derived from a shared pre-trained checkpoint into a single multi-task model without additional training. Despite extensive research, task interference remains a major obstacle that often undermines the performance of merged models. In this paper, we propose ESM (Essential Subspace Merging) , a robust framework for effective model merging. We begin by performing Principal Component Analysis (PCA) on feature shifts induced by parameter updates. The resulting principal directions span an essential subspace that dominantly influences feature representations. Each task's parameter update matrix is projected onto its respective essential subspace for low-rank decomposition before merging. This methodology mitigates inter-task interference while preserving core task-specific functionality. Furthermore, we introduce a multi-level polarized scaling strategy that amplifies parameters containing critical knowledge and suppresses redundant ones, preventing essential knowledge from being overwhelmed during fusion. Extensive experiments across multiple task sets and model scales demonstrate that our method achieves state-of-the-art performance in multi-task model merging. Code is available at https://github.com/kiddo127/ESM.
Figures & tables
Figure 1: Essential Subspace Decomposition (ESD) versus Singular Value Decomposition (SVD). Unlike SVD, which decomposes the task matrix solely based on weights, ESD decomposes them based on feature shift distributions. When truncating components for merging, ESD’s expected truncation error is directly related to the magnitude of the discarded eigenvalues and yields higher knowledge retention.
Figure 2: Performance evaluation of layer-wise task matrix loading under different ordering strategies on a pre-trained ViT-B/32 backbone. (a) Direct loading of task matrices. (b) Loading with layer-wise norms averaged to reflect directional importance.
Figure 3: Performance evaluation on the 8-task benchmark when loading merged task matrices layer-by-layer into a pre-trained backbone under different ordering strategies.
Figure 4: (a) The proposed Polarized Scaling uses the norm of parameter updates as scaling factors to amplify essential parameters submerged by redundant ones. (b) This scaling is applied at three distinct levels: across tasks, dimensions, and layers.
Method
ViT-B/32
ViT-B/16
ViT-L/14
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
Zero-shot
48.3
57.2
56.1
55.3
61.3
59.7
64.7
68.2
65.2
Fine-tuned
92.8
90.9
91.3
94.6
92.8
93.2
95.8
94.3
94.7
Weight Averaging [ 46 ]
66.3 (72.1)
64.3 (71.1)
61.0 (67.5)
72.2 (76.6)
69.5 (74.8)
65.3 (70.4)
79.6 (83.2)
76.7 (81.1)
71.6 (75.6)
Task Arithmetic [ 18 ]
70.8 (76.5)
65.3 (72.1)
60.5 (66.8)
75.4 (79.6)
70.5 (75.9)
65.8 (70.8)
84.9 (88.7)
79.4 (84.0)
74.0 (78.1)
TIES-Merging [ 52 ]
75.1 (81.0)
68.0 (74.8)
63.4 (69.9)
79.7 (84.3)
73.2 (78.7)
68.2 (73.3)
86.9 (90.7)
79.5 (84.1)
75.7 (79.8)
Table 1: Average absolute accuracy on model merging benchmarks, with normalized average accuracy shown as subscripts in parentheses. “ Zero-shot ” (pre-trained model) and “ Fine-tuned ” (fine-tuned models) results are presented as the lower and upper bounds, respectively.
Method
Decomposition
Polarized Scaling
ViT-B/16
ViT-L/14
SVD
ESD
\ \ \beta_{\ell}\ \
st(ℓ)
cj(ℓ)
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
Baseline
✓
✗
✗
✗
✗
89.0 (93.9)
84.6 (91.0)
80.6 (86.5)
93.0 (97.0)
89.2 (94.4)
87.7 (92.5)
✓
✗
✓
✓
✓
89.6 (94.6)
85.4 (91.9)
82.1 (88.1)
93.4 (97.4)
89.6 (95.0)
88.1 (93.0)
✗
✓
✗
✗
✗
90.9 (95.9)
85.8 (92.2)
82.8 (88.7)
94.5 (98.6)
90.7 (95.9)
89.5 (94.2)
✗
✓
✓
✗
✗
91.4 (96.4)
86.6 (93.1)
83.7 (89.6)
94.6 (98.7)
90.9 (96.2)
90.0 (94.8)
ESM (Ours)
✗
✓
✓
✓
✓
91.8 (97.0)
87.4 (94.1)
84.9 (91.1)
94.8 (98.9)
91.3 (96.8)
90.4 (95.3)
Table 2: Ablation study of the proposed Essential Subspace Decomposition (ESD)-based merging and Polarized Scaling. Results are reported in terms of average absolute accuracy, with normalized average accuracy shown as subscripts in parentheses.
Model
Forward
PCA
Orthogonalization
ViT-B/32
0.03 s/task
1.31 s/task
13.74 s (once)
ViT-B/16
0.04 s/task
1.39 s/task
13.89 s (once)
ViT-L/14
0.06 s/task
5.20 s/task
70.83 s (once)
Table 3: Time cost of major operations during model merging. “ Forward ”: forward pass of 32 proxy samples to obtain ΔO . “ PCA ”: principal component analysis performed on ΔO . “ Orthogonalization ”: orthogonalization of the matrices Pcat and Acat .
Figure 5: Energy retention as a function of the fraction of principal components retained from the task matrices. Results are reported for each of the 20 tasks as well as the average.
Figure 6: CKA similarity with the fine-tuned model when retaining only 5% of principal components using SVD and ESD.
Method
Sampling
ViT-B/32
ViT-B/16
ViT-L/14
SVD
-
86.6
89.6
93.4
ESD
Random (ID)
88.4 ±0.06
91.8 ±0.04
94.8 ±0.07
ESD
Class Imbalance
88.4 ±0.10
91.8 ±0.06
94.8 ±0.04
ESD
Random (OOD)
88.3 ±0.04
91.8 ±0.04
94.8 ±0.01
Table 4: Impact of proxy dataset composition on merging performance, evaluated on the 8-task benchmark. “ Random (ID) ”: random sampling from in-distribution task data. “ Class Imbalance ”: sampling only a single class per task. “ Random (OOD) ”: random sampling from out-of-distribution ImageNet-1k [ 34 ] .
Figure 7: Ablation study on the number of proxy samples used in ESM, reporting the average accuracy on the 8-task benchmark. Each configuration was evaluated over five random runs.
Method
Scaling
ViT-B/32
8 tasks
14 tasks
20 tasks
None
-
86.7 (93.4)
81.1 (89.0)
78.1 (85.5)
Reverse
1/γ
82.9 (89.2)
76.3 (83.7)
72.6 (79.4)
Noise −−
min(γ,1)
87.4 (94.1)
83.0 (91.2)
80.5 (88.1)
Signal ++
max(γ,1)
87.7 (94.6)
82.8 (91.0)
80.2 (87.8)
PS (Ours)
γ
88.4 (95.3)
83.7 (92.0)
81.3 (88.9)
Table 5: Detailed ablation study of the proposed Polarized Scaling. The symbol γ denotes the scaling factor at three different levels: s , c , or β . The following variants are compared: (i) “ Reverse ”: taking the reciprocal of the scaling factors; (ii) “ Noise −− ”: retaining only factors <1 to suppress noisy parameters; (iii) “ Signal ++ ”: retaining only factors >1 to enhance important parameters.
Figure 8: Layer-wise scaling coefficients for merged ViT-B/16.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Illustration of task invasion: using the task matrices from one task to invade the fine-tuned model of another, performed layer-by-layer based on the norm order. Rows represent the invaded task, and columns represent the invading task.
Method
ViT-B/32
ViT-B/16
ViT-L/14
ID
OOD
ID
OOD
ID
OOD
Baseline
85.9
49.9
89.0
54.7
93.0
65.9
ESM (Ours)
88.4
51.3
91.8
55.5
94.8
66.4
Appendix
Table 6: Performance on the 8-task benchmark. “ID”: the average accuracy of the merged model across all tasks. “OOD”: the result of merging fine-tuned models from 7 tasks and testing on the remaining unseen task. Each task is alternately used as the test task, and the reported OOD score is the average across all such settings.
ViT-B/32
ViT-B/16
ViT-L/14
T =8
T =14
T =20
T =8
T =14
T =20
T =8
T =14
T =20
0.88
0.76
0.69
0.82
0.76
0.67
0.91
0.72
0.64
Appendix
Table 7: Scaling coefficient α selected using the validation set. T represents the total number of merged task-specific expert models.
Figure 10: Performance comparison between the baseline method TSV-M [ 11 ] and our proposed ESM framework.
the School of Computer Science and Engineering, Southeast University, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China, 211189 · Huawei Inc., Shenzhen 518129, China