Model merging aims to combine multiple task-specific expert models into a single model while preserving generalization across diverse tasks. However, interference among experts, especially when they are trained on different objectives, often leads to significant performance degradation. Despite recent progress, resolving this interference without data access, retraining, or architectural modification remains a fundamental challenge. This paper provides a theoretical analysis demonstrating that the input covariance of each task, which is a key factor for optimal merging, can be implicitly estimated from the parameter differences of its fine-tuned model, even in a fully data-free setting. Building on this insight, we introduce \acem, an Adaptive Covariance Estimation framework that effectively mitigates inter-task interference. Our approach features a principled, closed-form solution that contrasts with prior iterative or heuristic methods. Extensive experiments on both vision and language benchmarks demonstrate that \acem sets a new state-of-the-art among data-free methods. It consistently outperforms existing baselines; for example, \acem achieves an average absolute improvement of 4% over the previous methods across seven tasks on GPT-2. Owing to its efficient closed-form formulation, \acem delivers superior performance with a modest computational cost, providing a practical and theoretically grounded solution for model merging.
Figures & tables
Figure 1: Comparison of inter-task heterogeneity ( γ ) across architectures. ViT-B/16 shows relatively uniform scaling ( γ<0.25 ), whereas RoBERTa-Base exhibits substantially stronger variation ( γ>0.3 ) across layers. This contrast motivates the need for an adaptive scaling strategy.
Figure 2: Comparison between the preliminary closed-form solution Wˉpre and the final merged model Wˉ . Left: The singular value spectrum of Wˉpre is highly concentrated: the top 5% singular values capture more than 99% of the total energy, indicating severe spectral imbalance. In contrast, Wˉ exhibits a substantially flatter spectrum. Right: Despite this concentration, their leading singular vectors are nearly identical (cosine similarity ≈1 ), showing that Wˉpre already identifies the correct structural subspace. Spectral refinement preserves this subspace while restoring a more stable energy distribution.
Method
ViT-B/32
ViT-B/16
ViT-L/14
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
Zero-shot
48.3
57.2
56.1
55.3
61.3
59.7
64.7
68.2
65.2
Fine-tuned
92.8
90.9
91.3
94.6
92.8
93.2
95.8
94.3
94.7
Weight Averaging [ 5 ]
66.3
64.3
61.0
72.2
69.5
65.3
79.6
76.7
71.6
Task Arithmetic [ 4 ]
70.8
65.3
60.5
75.4
70.5
65.8
84.9
79.4
74.0
Consensus TA [ 25 ]
75.0
70.4
65.4
79.4
74.4
69.8
86.3
82.2
79.0
Table 1: Main results on vision benchmarks. We report the average absolute accuracy (%) across three ViT backbones (ViT-B/32, ViT-B/16, and ViT-L/14) on task sets of increasing cardinality (8, 14, and 20). “Zero-shot” and “Fine-tuned” serve as reference lower and upper bounds, respectively. The best result among merging methods in each setting is highlighted in bold .
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
Avg.
Individual
76.8
82.1
80.4
88.3
89.6
65.3
91.2
82.0
Weight Averaging [ 5 ]
55.0
55.1
51.0
57.6
76.7
44.8
52.5
56.1
Fisher Merging [ 6 ]
54.8
58.0
39.5
63.3
81.5
49.1
64.7
58.7
Iso-CTS [ 20 ]
58.4
56.1
43.4
56.0
79.7
51.3
67.0
58.82
Iso-C [ 20 ]
65.7
59.0
48.8
57.8
83.6
52.4
66.4
61.95
RegMean [ 7 ]
61.7
70.4
65.4
69.7
78.8
56.0
79.7
68.8
Table 2: Performance on the GLUE benchmark for different merging methods applied to GPT-2. All metrics are reported in their standard evaluation form. “Individual” denotes the upper bound obtained by single-task fine-tuned models.
Model
Method
CoLA
SST-2
MRPC
STS-B
QQP
QNLI
MNLI
RTE
Avg.
RoBERTa-Base
Pre-trained
0.0
53.8
85.0
4.0
37.5
53.1
37.1
71.2
41.7
Individual
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Weight Averaging [ 5 ]
0.0
59.2
85.8
47.0
45.4
63.9
48.0
71.2
52.6
Task Arithmetic [ 4 ]
8.4
88.3
89.6
32.8
82.0
85.4
75.5
80.4
67.8
Ties-Merging [ 11 ]
31.8
88.9
86.2
10.9
61.1
85.9
83.0
69.6
64.7
Task Arithmetic (w/ DARE) [ 14 ]
0.0
88.1
86.6
30.2
84.3
79.1
64.0
77.2
63.7
Table 3: Normalized performance on the GLUE benchmark for RoBERTa-Base and RoBERTa-Large. The average score is reported, and the best results in each column are shown in bold .
#
Method Configuration
RoBERTa-L
GPT-2
ViT-B/16
E1
Basic closed-form
80.05
68.72
89.91
E2
+ Adaptive normalization
88.04
71.50
89.91
E3
+ Collective structural prior
86.79
71.51
90.60
E4
+ Spectral refinement
91.68
74.09
90.60
Table 4: Ablation study of ACE-Merging components. We report the average score (%) on RoBERTa-Large (GLUE), GPT-2 (GLUE), and ViT-B/16 (8 tasks). The best configuration for each model is highlighted in bold .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Empirical distributions of ΔWt across architectures, layer types, and tasks. Each subplot shows the histogram of the entries of ΔWt for one representative layer from RoBERTa-Base, GPT-2, and ViT-B/16. The distributions are consistently zero-centered and approximately Gaussian, supporting the local zero-mean assumption used in our theoretical analysis and suggesting that the dominant task-specific information is captured by the second-order structure of the parameter updates.
Figure 8Figure 9
ViT-L/14 (8 tasks)
GPT-2 (7 tasks)
Method
Time (s) ↓
VRAM (GB) ↓
Time (s) ↓
VRAM (GB) ↓
WUDI (1000 iters)
126.68
6.22
62.71
4.71
TSV-M
88.39
1.91
25.87
0.77
Iso-CTS
118.58
11.12
42.83
10.54
ACE (ours)
4.32
2.04
4.70
1.02
Appendix
Table 6: Measured merge-time and peak GPU memory on a single A800 (80GB), reproduced from the rebuttal.
ViT-B/32
ViT-B/16
ViT-L/14
Dataset
8
14
20
8
14
20
8
14
20
MNIST
99.46
99.22
92.65
99.44
99.32
96.40
99.58
99.59
96.65
Cars
71.67
68.59
61.90
81.54
78.24
72.40
90.34
88.46
85.37
DTD
89.63
82.45
75.80
90.69
83.19
76.33
96.97
93.19
88.67
EuroSAT
95.81
88.37
83.70
97.78
96.96
94.56
99.52
99.30
98.22
GTSRB
92.44
85.49
76.43
92.94
88.20
80.74
97.34
97.42
93.49
Appendix
Table 7: ACE-Merging accuracy (%) on ViT-B/32, ViT-B/16, and ViT-L/14 across increasing numbers of tasks.
Tasks
xquad_zh
xquad_vi
gsm8k_cot
mathqa
humaneval
multilingual
27.24
42.20
–
–
–
coding
–
–
–
34.30
49.39
math
–
–
74.00
–
–
Average
8.40
32.72
73.62
34.24
46.34
Task Arithmetic
8.06
32.88
73.09
34.14
46.34
ACE (ours)
16.42
37.68
69.75
34.57
50.61
Appendix
Table 8: LLaMA-3 merging results (raw scores). Single-task fine-tuning experts and merged models evaluated on five benchmarks.
Method
xquad_zh
xquad_vi
gsm8k_cot
mathqa
humaneval
Avg.
Average
30.84
77.54
99.49
99.83
93.82
80.30
Task Arithmetic
29.59
77.91
98.77
99.53
93.82
79.92
ACE (ours)
60.28
89.29
94.26
100.79
102.47
89.42
Appendix
Table 9: Normalized performance on LLaMA-3 (%). Each value is the ratio between the merged model and the best single-task fine-tuning expert for that benchmark.
Model merging provides a way of cheaply combining individual models to produce a model that inherits each individual's capabilities. While some merging methods can approach the performance of multitask training, they are often heuristically motivated and lack theoretical justification. A principled alternative is to pose model merging as a layer-wise optimization problem that directly minimizes interference between tasks. However, this formulation requires estimating per-layer covariance matrices from data, which may not be available when performing merging. In contrast, many of the heuristically-motivated methods do not require auxiliary data, making them practically advantageous. In this work, we revisit the interference minimization framework and show that, under certain conditions, covariance matrices can be estimated directly from difference matrices, eliminating the need for data while also reducing computational costs. We validate our approach across vision and language benchmarks on models ranging from 86M parameters to 7B parameters, outperforming previous data-free state-of-the-art merging methods
Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo +2
Mila & DIRO, Universit´e de Montr´eal · University of Toronto & Vector Institute
Model merging aims to consolidate multiple task-specific models fine-tuned on different datasets into a unified architecture that performs cross-domain proficiency. Current data-free model merging methods often struggle to scale as they rely on simple parameter-level heuristics that ignore inter-layer dependencies and non-uniform distribution of expertise. This work proposes SA-Merging, which is built upon connectivity-based saliency formulations from structural pruning (e.g., SynFlow) and extends them to the data-free model merging setting. We define a saliency score over task vectors relative to a shared base model, and further introduce merge-aware modulation that incorporates agreement across experts to mitigate task interference. Based on this formulation, an iterative saliency-aware merging procedure progressively removes non-informative updates while preserving end-to-end connectivity. Furthermore, we extend SA-Merging to introduce rank-wise saliency decomposition for LoRAs without compromising their structural integrity. Extensive experiments on vision and language tasks demonstrate the effectiveness of our saliency-based approach, further reducing the gap between data-free and test-time adaptation methods.
Jungin Park, Jiyoung Lee, Kwanghoon Sohn
Yonsei University, Seoul, South Korea · Ewha Womans University, Seoul, South Korea.
Model merging has shown that multitask models can be created by directly combining the parameters of different models that are each specialized on tasks of interest. However, models trained independently on distinct tasks often exhibit interference that degrades the merged model's performance. To solve this problem, we formally define the notion of Cross-Task Interference as the drift in the representation of the merged model relative to its constituent models. Reducing cross-task interference is key to improving merging performance. To address this issue, we propose our method, Resolving Interference (RI), a light-weight adaptation framework which disentangles expert models to be functionally orthogonal to the space of other tasks, thereby reducing cross-task interference. RI does this whilst using only unlabeled auxiliary data as input (i.e., no task-data is needed), allowing it to be applied in data-scarce scenarios. RI consistently improves the performance of state-of-the-art merging methods by up to 3.8% and generalization to unseen domains by up to 2.3%. We also find RI to be robust to the source of auxiliary input while being significantly less sensitive to tuning of merging hyperparameters. Our codebase is available at: https://github.com/pramesh39/resolving_interference
Pratik Ramesh, George Stoica, Arun Iyer +2
Georgia Tech · University of Washington · Microsoft Research +3