Organizations: School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shanghai, China
Model merging aims to integrate multiple task-specific fine-tuned models derived from a shared pre-trained checkpoint into a single multi-task model without additional training. Despite extensive research, task interference remains a major obstacle that often undermines the performance of merged models. In this paper, we propose ESM (Essential Subspace Merging) , a robust framework for effective model merging. We begin by performing Principal Component Analysis (PCA) on feature shifts induced by parameter updates. The resulting principal directions span an essential subspace that dominantly influences feature representations. Each task's parameter update matrix is projected onto its respective essential subspace for low-rank decomposition before merging. This methodology mitigates inter-task interference while preserving core task-specific functionality. Furthermore, we introduce a multi-level polarized scaling strategy that amplifies parameters containing critical knowledge and suppresses redundant ones, preventing essential knowledge from being overwhelmed during fusion. Extensive experiments across multiple task sets and model scales demonstrate that our method achieves state-of-the-art performance in multi-task model merging. Code is available at https://github.com/kiddo127/ESM.
Figures & tables
Figure 1: Essential Subspace Decomposition (ESD) versus Singular Value Decomposition (SVD). Unlike SVD, which decomposes the task matrix solely based on weights, ESD decomposes them based on feature shift distributions. When truncating components for merging, ESD’s expected truncation error is directly related to the magnitude of the discarded eigenvalues and yields higher knowledge retention.
Figure 2: Performance evaluation of layer-wise task matrix loading under different ordering strategies on a pre-trained ViT-B/32 backbone. (a) Direct loading of task matrices. (b) Loading with layer-wise norms averaged to reflect directional importance.
Figure 3: Performance evaluation on the 8-task benchmark when loading merged task matrices layer-by-layer into a pre-trained backbone under different ordering strategies.
Figure 4: (a) The proposed Polarized Scaling uses the norm of parameter updates as scaling factors to amplify essential parameters submerged by redundant ones. (b) This scaling is applied at three distinct levels: across tasks, dimensions, and layers.
Method
ViT-B/32
ViT-B/16
ViT-L/14
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
Zero-shot
48.3
57.2
56.1
55.3
61.3
59.7
64.7
68.2
65.2
Fine-tuned
92.8
90.9
91.3
94.6
92.8
93.2
95.8
94.3
94.7
Weight Averaging [ 46 ]
66.3 (72.1)
64.3 (71.1)
61.0 (67.5)
72.2 (76.6)
69.5 (74.8)
65.3 (70.4)
79.6 (83.2)
76.7 (81.1)
71.6 (75.6)
Task Arithmetic [ 18 ]
70.8 (76.5)
65.3 (72.1)
60.5 (66.8)
75.4 (79.6)
70.5 (75.9)
65.8 (70.8)
84.9 (88.7)
79.4 (84.0)
74.0 (78.1)
TIES-Merging [ 52 ]
75.1 (81.0)
68.0 (74.8)
63.4 (69.9)
79.7 (84.3)
73.2 (78.7)
68.2 (73.3)
86.9 (90.7)
79.5 (84.1)
75.7 (79.8)
Table 1: Average absolute accuracy on model merging benchmarks, with normalized average accuracy shown as subscripts in parentheses. “ Zero-shot ” (pre-trained model) and “ Fine-tuned ” (fine-tuned models) results are presented as the lower and upper bounds, respectively.
Method
Decomposition
Polarized Scaling
ViT-B/16
ViT-L/14
SVD
ESD
\ \ \beta_{\ell}\ \
st(ℓ)
cj(ℓ)
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
Baseline
✓
✗
✗
✗
✗
89.0 (93.9)
84.6 (91.0)
80.6 (86.5)
93.0 (97.0)
89.2 (94.4)
87.7 (92.5)
✓
✗
✓
✓
✓
89.6 (94.6)
85.4 (91.9)
82.1 (88.1)
93.4 (97.4)
89.6 (95.0)
88.1 (93.0)
✗
✓
✗
✗
✗
90.9 (95.9)
85.8 (92.2)
82.8 (88.7)
94.5 (98.6)
90.7 (95.9)
89.5 (94.2)
✗
✓
✓
✗
✗
91.4 (96.4)
86.6 (93.1)
83.7 (89.6)
94.6 (98.7)
90.9 (96.2)
90.0 (94.8)
ESM (Ours)
✗
✓
✓
✓
✓
91.8 (97.0)
87.4 (94.1)
84.9 (91.1)
94.8 (98.9)
91.3 (96.8)
90.4 (95.3)
Table 2: Ablation study of the proposed Essential Subspace Decomposition (ESD)-based merging and Polarized Scaling. Results are reported in terms of average absolute accuracy, with normalized average accuracy shown as subscripts in parentheses.
Model
Forward
PCA
Orthogonalization
ViT-B/32
0.03 s/task
1.31 s/task
13.74 s (once)
ViT-B/16
0.04 s/task
1.39 s/task
13.89 s (once)
ViT-L/14
0.06 s/task
5.20 s/task
70.83 s (once)
Table 3: Time cost of major operations during model merging. “ Forward ”: forward pass of 32 proxy samples to obtain ΔO . “ PCA ”: principal component analysis performed on ΔO . “ Orthogonalization ”: orthogonalization of the matrices Pcat and Acat .
Figure 5: Energy retention as a function of the fraction of principal components retained from the task matrices. Results are reported for each of the 20 tasks as well as the average.
Figure 6: CKA similarity with the fine-tuned model when retaining only 5% of principal components using SVD and ESD.
Method
Sampling
ViT-B/32
ViT-B/16
ViT-L/14
SVD
-
86.6
89.6
93.4
ESD
Random (ID)
88.4 ±0.06
91.8 ±0.04
94.8 ±0.07
ESD
Class Imbalance
88.4 ±0.10
91.8 ±0.06
94.8 ±0.04
ESD
Random (OOD)
88.3 ±0.04
91.8 ±0.04
94.8 ±0.01
Table 4: Impact of proxy dataset composition on merging performance, evaluated on the 8-task benchmark. “ Random (ID) ”: random sampling from in-distribution task data. “ Class Imbalance ”: sampling only a single class per task. “ Random (OOD) ”: random sampling from out-of-distribution ImageNet-1k [ 34 ] .
Figure 7: Ablation study on the number of proxy samples used in ESM, reporting the average accuracy on the 8-task benchmark. Each configuration was evaluated over five random runs.
Method
Scaling
ViT-B/32
8 tasks
14 tasks
20 tasks
None
-
86.7 (93.4)
81.1 (89.0)
78.1 (85.5)
Reverse
1/γ
82.9 (89.2)
76.3 (83.7)
72.6 (79.4)
Noise −−
min(γ,1)
87.4 (94.1)
83.0 (91.2)
80.5 (88.1)
Signal ++
max(γ,1)
87.7 (94.6)
82.8 (91.0)
80.2 (87.8)
PS (Ours)
γ
88.4 (95.3)
83.7 (92.0)
81.3 (88.9)
Table 5: Detailed ablation study of the proposed Polarized Scaling. The symbol γ denotes the scaling factor at three different levels: s , c , or β . The following variants are compared: (i) “ Reverse ”: taking the reciprocal of the scaling factors; (ii) “ Noise −− ”: retaining only factors <1 to suppress noisy parameters; (iii) “ Signal ++ ”: retaining only factors >1 to enhance important parameters.
Figure 8: Layer-wise scaling coefficients for merged ViT-B/16.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Illustration of task invasion: using the task matrices from one task to invade the fine-tuned model of another, performed layer-by-layer based on the norm order. Rows represent the invaded task, and columns represent the invading task.
Method
ViT-B/32
ViT-B/16
ViT-L/14
ID
OOD
ID
OOD
ID
OOD
Baseline
85.9
49.9
89.0
54.7
93.0
65.9
ESM (Ours)
88.4
51.3
91.8
55.5
94.8
66.4
Appendix
Table 6: Performance on the 8-task benchmark. “ID”: the average accuracy of the merged model across all tasks. “OOD”: the result of merging fine-tuned models from 7 tasks and testing on the remaining unseen task. Each task is alternately used as the test task, and the reported OOD score is the average across all such settings.
ViT-B/32
ViT-B/16
ViT-L/14
T =8
T =14
T =20
T =8
T =14
T =20
T =8
T =14
T =20
0.88
0.76
0.69
0.82
0.76
0.67
0.91
0.72
0.64
Appendix
Table 7: Scaling coefficient α selected using the validation set. T represents the total number of merged task-specific expert models.
Figure 10: Performance comparison between the baseline method TSV-M [ 11 ] and our proposed ESM framework.
Model merging enables training-free multi-task learning by aggregating knowledge from models fine-tuned from a shared checkpoint, but their task updates can interfere. Using proxy data, we find that task updates induce output shifts whose energy concentrates in a few dominant directions, while many low-energy directions can accumulate across tasks and impair performance. We call the span of the dominant directions the essential subspace. This observation motivates a data-aware framework to preserve essential task knowledge while reducing interference. Essential Subspace Decomposition (ESD) identifies each task's essential subspace from activation shifts measured on proxy data and represents its update compactly while preserving functional behavior. Essential Subspace Merging (ESM) weights each task-update direction by its corresponding output-shift eigenvalue, thereby prioritizing core task knowledge; it then orthogonalizes the weighted directions across tasks to reduce interference. ESM++ extends this merged model by dynamically integrating low-rank task-specific residual experts for adaptive knowledge aggregation. Experiments across diverse task sets and model scales show that ESM and ESM++ effectively aggregate task knowledge while mitigating inter-task interference. Code is available at https://github.com/kiddo127/ESM.
Longhua Li, Lei Qi, Qi Tian +1
the School of Computer Science and Engineering, Southeast University, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China, 211189 · Huawei Inc., Shenzhen 518129, China
Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. To perform well across multiple tasks, most existing merging methods use an additional dataset to find the coefficients for the best linear combination of task-specific weight updates. However, we identify an implicit regularization in this standard practice: searching over coefficients restricts the candidate models to a subspace spanned by task-specific weight updates. In this work, we investigate whether this regularization is actually useful. Surprisingly, empirical results show that optimizing merged-model weights without this regularization significantly boosts the performance of common merging methods across multiple architectures, domains, and even in an extremely data-limited scenario where only one instance is available per class. Moreover, directly optimizing the pretrained model weights even outperforms some existing merging methods. Analysis shows that better multi-task weights exist outside the subspace and can be found using multiple methods. We study different strategies for using the additional dataset, discussing their practical use and implications for model merging. Overall, this work calls for revisiting the existing model-merging pipeline, motivating a broader exploration of the weight space and a reconsideration of the implicit regularization induced by task arithmetic.
Model merging has shown that multitask models can be created by directly combining the parameters of different models that are each specialized on tasks of interest. However, models trained independently on distinct tasks often exhibit interference that degrades the merged model's performance. To solve this problem, we formally define the notion of Cross-Task Interference as the drift in the representation of the merged model relative to its constituent models. Reducing cross-task interference is key to improving merging performance. To address this issue, we propose our method, Resolving Interference (RI), a light-weight adaptation framework which disentangles expert models to be functionally orthogonal to the space of other tasks, thereby reducing cross-task interference. RI does this whilst using only unlabeled auxiliary data as input (i.e., no task-data is needed), allowing it to be applied in data-scarce scenarios. RI consistently improves the performance of state-of-the-art merging methods by up to 3.8% and generalization to unseen domains by up to 2.3%. We also find RI to be robust to the source of auxiliary input while being significantly less sensitive to tuning of merging hyperparameters. Our codebase is available at: https://github.com/pramesh39/resolving_interference
Pratik Ramesh, George Stoica, Arun Iyer +2
Georgia Tech · University of Washington · Microsoft Research +3