Resolving Interference (RI): Disentangling Models for Improved Model Merging

Pratik Ramesh; George Stoica; Arun Iyer; Leshem Choshen; Judy Hoffman

cs.LGMar 13, 2026

Resolving Interference (RI): Disentangling Models for Improved Model Merging

Authors: Pratik Ramesh, George Stoica, Arun Iyer, Leshem Choshen, Judy Hoffman

Organizations: Georgia Tech · University of Washington · Microsoft Research · MIT · IBM Research · University of California, Irvine

Abstract

Model merging has shown that multitask models can be created by directly combining the parameters of different models that are each specialized on tasks of interest. However, models trained independently on distinct tasks often exhibit interference that degrades the merged model's performance. To solve this problem, we formally define the notion of Cross-Task Interference as the drift in the representation of the merged model relative to its constituent models. Reducing cross-task interference is key to improving merging performance. To address this issue, we propose our method, Resolving Interference (RI), a light-weight adaptation framework which disentangles expert models to be functionally orthogonal to the space of other tasks, thereby reducing cross-task interference. RI does this whilst using only unlabeled auxiliary data as input (i.e., no task-data is needed), allowing it to be applied in data-scarce scenarios. RI consistently improves the performance of state-of-the-art merging methods by up to 3.8% and generalization to unseen domains by up to 2.3%. We also find RI to be robust to the source of auxiliary input while being significantly less sensitive to tuning of merging hyperparameters. Our codebase is available at: https://github.com/pramesh39/resolving_interference

Figures & tables

Figure 1: Resolving Interference ( RI ) is a lightweight adaptation strategy that mitigates cross-task interference, enhancing the performance of existing model-merging techniques.

Figure 2: Resolving Interference ( RI ) for the dog classifier involves passing unlabeled auxiliary images through (i) the frozen pretrained backbone $f(\theta_{0})$ , (ii) the frozen dog classifier $f(\theta_{0}+\tau_{\text{dog}})$ , and (iii) a trainable copy $f(\theta_{0}+\tau_{\text{dog}}^{*})$ . A twin distillation loss preserves output distribution across the dog head $h_{\text{dog}}$ while forcing output across all other heads (e.g., $h_{\text{cat}}$ ) to match the pretrained backbone, producing an adapted task vector $\tau_{\text{dog}}^{*}$ . The same adaptation strategy is repeated for all expert models.

Merging Method	ViT-B/32			ViT-B/16			ViT-L/14
Merging Method	8 tasks	14 tasks	20 tasks	8 tasks	14 tasks	20 tasks	8 tasks	14 tasks	20 tasks
Zero-shot	48.3	57.2	56.1	55.3	61.3	59.7	64.7	68.2	65.2
Finetuned	92.8	90.9	91.3	94.6	92.8	93.2	95.8	94.3	94.7
Averaging	66.1	64.3	61.0	72.3	69.4	65.3	79.5	76.7	71.6
Averaging + RI (Ours)	65.7	64.6	61.4	71.3	69.1	64.9	78.9	75.8	71.1
TA	69.2	60.7	56.1	75.2	67.3	63.1	84.8	78.8	73.3

Table 1: Resolving Interference ( RI ) consistently improves existing merging techniques on the popular 8/14/20 task vision benchmarks. Achieving up to 10% improvement on TA and pushing SOTA merging techniques like TSV-M by up to 3.8% . Each entry reports average accuracy. In line with our focus on the task-data–free setting where tuning merging hyperparameters is not possible, we make use of the recommended defaults for all merging baselines with and without RI .

Merging Method	Clipart		Infograph		Painting		Quickdraw		Sketch		Mean(%)
Merging Method	S-0	S-1	S-0	S-1	S-0	S-1	S-0	S-1	S-0	S-1	Mean(%)
Split-0 model	76.2	70.0	49.3	38.0	67.7	51.9	17.2	18.5	66.3	59.4	51.5
Split-1 model	70.1	73.9	44.7	38.4	64.7	55.6	13.3	17.9	62.0	61.1	50.2
Averaging	76.5	74.7	50.1	40.1	69.2	54.9	16.1	20.0	67.7	62.7	53.2
Averaging + RI (Ours)	79.1	76.3	51.9	46.7	70.7	56.2	18.0	19.7	69.6	64.1	55.2
TA	77.5	75.6	51.0	41.0	70.1	55.5	16.9	20.3	68.5	63.6	54.0

Table 2: Resolving Interference ( RI ) improves generalization performance of merging techniques on unseen domains by upto 2.3% , while improving by up to 3.9% compared to the task-expert models. Each dataset is split into two parts (denoted Split-0 (S-0) and Split-1 (S-1)), and the performance of original split-specific experts is reported in the top grey rows.

Figure 3: The Twin-Distillation loss (Eq. 4 ) on auxiliary data reduces sharply (left), followed by a similar decline in the cross-task interference of the merged model measured on task-specific data (middle), which leads to significant improvement when using existing merging techniques(right).

Distance Metric		TSV-M	Iso-C	Iso-CTS	Average
None (Without RI)		76.5	75.1	77.7	76.4
RI	MSE	78.2	76.7	78.4	77.7
	Cross-Entropy	79.9	76.3	78.5	78.2
	KL-Divergence	80.3	77.5	79.3	79.0

Table 3: Comparison of distance metrics used to reduce RI loss on the merging performance. Minimizing KL-Divergence yields the highest improvement across merging methods.

ViT-B/32 (20 tasks)	Accuracy (%)
Zero-Shot	56.1
Finetuned	91.3
Zero-Shot + Distill	63.5
Merging Method	Baseline	+ Distill ${}_{\text{Aux}}$	+ RI (Ours)
Iso-C	75.1	75.0	77.5
Iso-CTS	77.7	76.4	79.3

Table 4: Resolving Interference ( RI ) and Merging performs (green) outperforms merging followed by Multitask Distillation using auxiliary data (blue).

Figure 4: What makes a good source of auxiliary data for RI? RI benefits from a wide range of auxiliary sources, including synthetic and real data, as long as they exhibit sufficient visual diversity. While gains are observed across all sources, datasets that are closer to the target task distribution yield the largest improvements, with task data (oracle) providing the strongest performance.

Method	Default		Tuned on Aux Data		Tuned on Task Data
Method	Baseline	+ RI (Ours)	Baseline	+ RI (Ours)	Baseline	+ RI (Ours)
TA	56.1	64.7	60.2	63.4	61.3	64.5
TIES	58.0	67.7	61.0	66.1	63.7	68.9
KnOTS	66.7	69.3	62.3	67.8	66.6	70.5
Iso-C	75.1	77.5	67.7	74.6	75.1	77.4
Iso-CTS	77.7	79.3	65.8	70.9	77.8	79.3

Table 5: Sensitivity to hyperparameter tuning. Models adapted with RI are less sensitive to tuning on privileged task-specific validation data, while tuning based on cross-task interference using auxiliary data proves far less effective.

# Tasks	Training Steps	Time / Expert	Peak GPU Mem.
8	2500	7m 07s	4.8 GB
14	2500	8m 33s	4.8 GB
20	2500	8m 50s	4.8 GB

Table 6: Computational profile of RI measured on an NVIDIA A40 (48 GB VRAM). Each expert is adapted independently. Runtime grows sub-linearly with the number of tasks, while peak memory usage remains constant.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Merging Methods	2 Tasks	8 Tasks	14 Tasks	20 Tasks
TA	0.42	0.30	0.22	0.15
TIES	1.0
KnOTS	1.0
Iso-C	1.9	1.3	1.o	0.9
Iso-CTS	2.1	1.5	1.2	1.1
TSVM	1

Appendix

Table A1 : Scaling coefficients for different numbers of tasks.

Method	No RI	1.0 $\times$	0.8 $\times$	0.6 $\times$	0.4 $\times$	0.2 $\times$
Iso-C	75.1	77.4	77.4	77.5	77.6	77.8
Iso-CTS	77.7	79.1	79.0	79.0	79.1	79.1
TSV-M	76.5	80.1	80.1	80.2	80.2	80.3

Appendix

Table A1: Effect of auxiliary data size on RI performance. Data fraction 1.0

\times

corresponds to 320k auxiliary images used over 2500 steps with batch size 128. RI remains effective even under aggressive subsampling of auxiliary data.

Scaling Coeff.	0.0	0.025	0.05	0.075	0.1	0.125	0.15	0.175	0.2	0.225	0.25	0.275	0.3
Acc. (%)	56.1	59.3	61.4	62.9	64.0	64.6	64.7	64.3	63.4	62.1	60.4	58.6	56.6

Appendix

Table A2: Effect of the scaling coefficient on Averaging+RI in the ViT-B/32 20-task setting. Performance peaks at moderate scaling values, indicating that default averaging underutilizes the adapted task vectors produced by RI.

Figure B.1 : What makes a good source of auxiliary data for RI?

Explore similar work

Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement
Aug 26, 2026Yihang Zhang, Shengke Sun, Junjie Wen +1Language Model Merging Model Merging
Central South University · Nanjing University of Science and Technology · Hefei University of Technology
A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic
Oct 6, 2026Sin-Han Yang, Shih-Cheng Huang, Chieh-Yen Lin +3Task Vector Merging Multi-Task Optimization
Appier AI Research · National Taiwan University
Learning to Recover Task Experts from a Multi-Task Merged Model
Jun 25, 2026Jinwook Jung, Taegyu Kim, Kumju Jo +1OOD Generalization Multi-Task Learning
Department of Artificial Intelligence, 2Department of Data Science Hanyang University

Aug 26, 2026J/K move · Enter open · S save

Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement

Yihang Zhang, Shengke Sun, Junjie Wen, Feng Zeng

Central South University · Nanjing University of Science and Technology · Hefei University of Technology

Language Model Merging Model Merging Task Vector Merging

Open paper AskPDF arXiv

Model merging provides an efficient way to construct multi-task generalist models without additional training, but its performance often degrades under severe task interference. Task interference in model merging primarily stems from superposition, where task-specific features become entangled within the parameter space. This entanglement renders conventional decomposition methods insufficient for effectively isolating useful task directions from interfering components. In this paper, we propose a sparse-representation-based merging framework that uses Sparse Autoencoders (SAEs) to project task vectors into a high-dimensional sparse feature space, enabling feature-level disentanglement before fusion. To reduce computational overhead, we further introduce a lightweight Group-Ranked Zeroth-Order Optimizer (GR-ZOO) to identify task-critical layers for selective merging. Experiments on both Qwen2.5-1.5B and Qwen2.5-7B demonstrate that our method consistently outperforms representative baselines, including Task Arithmetic, TIES-Merge, DARE, Fisher-Merge,and several recent training-free merging methods, across mathematical reasoning, code generation, instruction following, and general knowledge tasks. In a highly conflicting four-task setting on Qwen2.5-1.5B, our method further achieves a 2.78% improvement over the strongest baseline.

Abstract

Explore similar work

Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement

A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic

Learning to Recover Task Experts from a Multi-Task Merged Model