Model merging has shown that multitask models can be created by directly combining the parameters of different models that are each specialized on tasks of interest. However, models trained independently on distinct tasks often exhibit interference that degrades the merged model's performance. To solve this problem, we formally define the notion of Cross-Task Interference as the drift in the representation of the merged model relative to its constituent models. Reducing cross-task interference is key to improving merging performance. To address this issue, we propose our method, Resolving Interference (RI), a light-weight adaptation framework which disentangles expert models to be functionally orthogonal to the space of other tasks, thereby reducing cross-task interference. RI does this whilst using only unlabeled auxiliary data as input (i.e., no task-data is needed), allowing it to be applied in data-scarce scenarios. RI consistently improves the performance of state-of-the-art merging methods by up to 3.8% and generalization to unseen domains by up to 2.3%. We also find RI to be robust to the source of auxiliary input while being significantly less sensitive to tuning of merging hyperparameters. Our codebase is available at: https://github.com/pramesh39/resolving_interference
Figures & tables
Figure 1: Resolving Interference ( RI ) is a lightweight adaptation strategy that mitigates cross-task interference, enhancing the performance of existing model-merging techniques.
Figure 2: Resolving Interference ( RI ) for the dog classifier involves passing unlabeled auxiliary images through (i) the frozen pretrained backbone f(θ0) , (ii) the frozen dog classifier f(θ0+τdog) , and (iii) a trainable copy f(θ0+τdog∗) . A twin distillation loss preserves output distribution across the dog head hdog while forcing output across all other heads (e.g., hcat ) to match the pretrained backbone, producing an adapted task vector τdog∗ . The same adaptation strategy is repeated for all expert models.
Merging Method
ViT-B/32
ViT-B/16
ViT-L/14
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
Zero-shot
48.3
57.2
56.1
55.3
61.3
59.7
64.7
68.2
65.2
Finetuned
92.8
90.9
91.3
94.6
92.8
93.2
95.8
94.3
94.7
Averaging
66.1
64.3
61.0
72.3
69.4
65.3
79.5
76.7
71.6
Averaging + RI (Ours)
65.7
64.6
61.4
71.3
69.1
64.9
78.9
75.8
71.1
TA
69.2
60.7
56.1
75.2
67.3
63.1
84.8
78.8
73.3
Table 1: Resolving Interference ( RI ) consistently improves existing merging techniques on the popular 8/14/20 task vision benchmarks. Achieving up to 10% improvement on TA and pushing SOTA merging techniques like TSV-M by up to 3.8% . Each entry reports average accuracy. In line with our focus on the task-data–free setting where tuning merging hyperparameters is not possible, we make use of the recommended defaults for all merging baselines with and without RI .
Merging Method
Clipart
Infograph
Painting
Quickdraw
Sketch
Mean(%)
S-0
S-1
S-0
S-1
S-0
S-1
S-0
S-1
S-0
S-1
Split-0 model
76.2
70.0
49.3
38.0
67.7
51.9
17.2
18.5
66.3
59.4
51.5
Split-1 model
70.1
73.9
44.7
38.4
64.7
55.6
13.3
17.9
62.0
61.1
50.2
Averaging
76.5
74.7
50.1
40.1
69.2
54.9
16.1
20.0
67.7
62.7
53.2
Averaging + RI (Ours)
79.1
76.3
51.9
46.7
70.7
56.2
18.0
19.7
69.6
64.1
55.2
TA
77.5
75.6
51.0
41.0
70.1
55.5
16.9
20.3
68.5
63.6
54.0
Table 2: Resolving Interference ( RI ) improves generalization performance of merging techniques on unseen domains by upto 2.3% , while improving by up to 3.9% compared to the task-expert models. Each dataset is split into two parts (denoted Split-0 (S-0) and Split-1 (S-1)), and the performance of original split-specific experts is reported in the top grey rows.
Figure 3: The Twin-Distillation loss (Eq. 4 ) on auxiliary data reduces sharply (left), followed by a similar decline in the cross-task interference of the merged model measured on task-specific data (middle), which leads to significant improvement when using existing merging techniques(right).
Distance Metric
TSV-M
Iso-C
Iso-CTS
Average
None (Without RI)
76.5
75.1
77.7
76.4
RI
MSE
78.2
76.7
78.4
77.7
Cross-Entropy
79.9
76.3
78.5
78.2
KL-Divergence
80.3
77.5
79.3
79.0
Table 3: Comparison of distance metrics used to reduce RI loss on the merging performance. Minimizing KL-Divergence yields the highest improvement across merging methods.
ViT-B/32 (20 tasks)
Accuracy (%)
Zero-Shot
56.1
Finetuned
91.3
Zero-Shot + Distill
63.5
Merging Method
Baseline
+ Distill Aux
+ RI (Ours)
Iso-C
75.1
75.0
77.5
Iso-CTS
77.7
76.4
79.3
Table 4: Resolving Interference ( RI ) and Merging performs (green) outperforms merging followed by Multitask Distillation using auxiliary data (blue).
Figure 4: What makes a good source of auxiliary data for RI? RI benefits from a wide range of auxiliary sources, including synthetic and real data, as long as they exhibit sufficient visual diversity. While gains are observed across all sources, datasets that are closer to the target task distribution yield the largest improvements, with task data (oracle) providing the strongest performance.
Method
Default
Tuned on Aux Data
Tuned on Task Data
Baseline
+ RI (Ours)
Baseline
+ RI (Ours)
Baseline
+ RI (Ours)
TA
56.1
64.7
60.2
63.4
61.3
64.5
TIES
58.0
67.7
61.0
66.1
63.7
68.9
KnOTS
66.7
69.3
62.3
67.8
66.6
70.5
Iso-C
75.1
77.5
67.7
74.6
75.1
77.4
Iso-CTS
77.7
79.3
65.8
70.9
77.8
79.3
Table 5: Sensitivity to hyperparameter tuning. Models adapted with RI are less sensitive to tuning on privileged task-specific validation data, while tuning based on cross-task interference using auxiliary data proves far less effective.
# Tasks
Training Steps
Time / Expert
Peak GPU Mem.
8
2500
7m 07s
4.8 GB
14
2500
8m 33s
4.8 GB
20
2500
8m 50s
4.8 GB
Table 6: Computational profile of RI measured on an NVIDIA A40 (48 GB VRAM). Each expert is adapted independently. Runtime grows sub-linearly with the number of tasks, while peak memory usage remains constant.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Merging Methods
2 Tasks
8 Tasks
14 Tasks
20 Tasks
TA
0.42
0.30
0.22
0.15
TIES
1.0
KnOTS
1.0
Iso-C
1.9
1.3
1.o
0.9
Iso-CTS
2.1
1.5
1.2
1.1
TSVM
1
Appendix
Table A1 : Scaling coefficients for different numbers of tasks.
Method
No RI
1.0 ×
0.8 ×
0.6 ×
0.4 ×
0.2 ×
Iso-C
75.1
77.4
77.4
77.5
77.6
77.8
Iso-CTS
77.7
79.1
79.0
79.0
79.1
79.1
TSV-M
76.5
80.1
80.1
80.2
80.2
80.3
Appendix
Table A1: Effect of auxiliary data size on RI performance. Data fraction 1.0 × corresponds to 320k auxiliary images used over 2500 steps with batch size 128. RI remains effective even under aggressive subsampling of auxiliary data.
Scaling Coeff.
0.0
0.025
0.05
0.075
0.1
0.125
0.15
0.175
0.2
0.225
0.25
0.275
0.3
Acc. (%)
56.1
59.3
61.4
62.9
64.0
64.6
64.7
64.3
63.4
62.1
60.4
58.6
56.6
Appendix
Table A2: Effect of the scaling coefficient on Averaging+RI in the ViT-B/32 20-task setting. Performance peaks at moderate scaling values, indicating that default averaging underutilizes the adapted task vectors produced by RI.
Figure B.1 : What makes a good source of auxiliary data for RI?
Model merging provides an efficient way to construct multi-task generalist models without additional training, but its performance often degrades under severe task interference. Task interference in model merging primarily stems from superposition, where task-specific features become entangled within the parameter space. This entanglement renders conventional decomposition methods insufficient for effectively isolating useful task directions from interfering components. In this paper, we propose a sparse-representation-based merging framework that uses Sparse Autoencoders (SAEs) to project task vectors into a high-dimensional sparse feature space, enabling feature-level disentanglement before fusion. To reduce computational overhead, we further introduce a lightweight Group-Ranked Zeroth-Order Optimizer (GR-ZOO) to identify task-critical layers for selective merging. Experiments on both Qwen2.5-1.5B and Qwen2.5-7B demonstrate that our method consistently outperforms representative baselines, including Task Arithmetic, TIES-Merge, DARE, Fisher-Merge,and several recent training-free merging methods, across mathematical reasoning, code generation, instruction following, and general knowledge tasks. In a highly conflicting four-task setting on Qwen2.5-1.5B, our method further achieves a 2.78% improvement over the strongest baseline.
Yihang Zhang, Shengke Sun, Junjie Wen +1
Central South University · Nanjing University of Science and Technology · Hefei University of Technology
Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. To perform well across multiple tasks, most existing merging methods use an additional dataset to find the coefficients for the best linear combination of task-specific weight updates. However, we identify an implicit regularization in this standard practice: searching over coefficients restricts the candidate models to a subspace spanned by task-specific weight updates. In this work, we investigate whether this regularization is actually useful. Surprisingly, empirical results show that optimizing merged-model weights without this regularization significantly boosts the performance of common merging methods across multiple architectures, domains, and even in an extremely data-limited scenario where only one instance is available per class. Moreover, directly optimizing the pretrained model weights even outperforms some existing merging methods. Analysis shows that better multi-task weights exist outside the subspace and can be found using multiple methods. We study different strategies for using the additional dataset, discussing their practical use and implications for model merging. Overall, this work calls for revisiting the existing model-merging pipeline, motivating a broader exploration of the weight space and a reconsideration of the implicit regularization induced by task arithmetic.
Multi-task model merging aims to consolidate several task-specific experts into a unified model, yet static merging consistently suffers from parameter interference. While dynamic merging models aim to bridge this gap, many works rely on the costly storage and loading of redundant expert components at inference. In this work, from the perspective of task expert, we view parameter interference as parameter perturbation introduced to each expert during merging process. We show that such parameter perturbations can be modeled as affine transformation, which can be approximated as additive offsets. Motivated by these, we propose Recover Task eXpert (ReTeX), a framework that predicts those offsets, in order to undo parameter interference and recover task-expert performance from a single merged checkpoint. To recover the appropriate expert when task identity is unknown, we introduce a router-free task identifier based on SVD subspace signatures computed offline before inference. At inference, the identifier selects the task whose subspace yields the smallest projection residual for a given input. As a result, ReTeX recovers over 95% of individual-expert performance in both vision and NLP domains, while significantly improving generalization to unseen tasks. Crucially, we also show that the parameter offset prediction leads to emergent adaptive interpolation of expert knowledge for out-of-distribution (OOD) tasks. ReTeX adaptively interpolates seen expert knowledge to handle unseen tasks. Our code is available at https://github.com/BAIKLAB/ReTeX
Jinwook Jung, Taegyu Kim, Kumju Jo +1
Department of Artificial Intelligence, 2Department of Data Science Hanyang University