Model merging consolidates fine-tuned experts into one multi-task model without retraining. All existing data-free methods approach this problem entirely in weight space. Restricted to arithmetic on parameters, these methods never observe how each expert behaves, a signal that only emerges through forward evaluation. Accessing this behavioral signal requires inputs to evaluate on, which the data-free setting prohibits. We propose Merge++, a post-hoc method that addresses this by inverting the expert checkpoints to synthesize task-representative images, then distilling expert knowledge into the merged model using those images. Merge++ requires no additional data beyond the checkpoints themselves. It applies universally across merging algorithms and operates as a complementary refinement stage independent of the underlying weight-space method. The method consistently improves merging algorithms ranging from simple task arithmetic to state-of-the-art spectral methods, with average gains of +2 to +8 points and up to +25.9 on individual configurations.
Figures & tables
Figure 1: Overview of Merge++. Step 1: Each fine-tuned expert is inverted to synthesize task-representative images without real data. Step 2: Expert checkpoints are merged using any weight-space method. Step 3: The merged model is refined via multi-teacher distillation on the inverted images using Lfeat and Llogit , yielding the improved Merge++ model.
ViT-B/32
ViT-B/16
ViT-L/14
Method
8T
14T
20T
8T
14T
20T
8T
14T
20T
TA [ICLR’23]
base
69.0
53.1
33.6
75.7
56.6
32.8
84.1
58.4
38.1
+MERGE++
+6.6
+12.6
+21.6
+4.6
+12.1
+21.1
+1.6
+18.7
+25.9
TIES [NeurIPS’23]
base
72.8
43.2
37.4
77.1
35.4
26.5
84.5
63.2
63.7
+MERGE++
+3.5
+5.1
+5.3
+3.1
+5.5
+5.2
+0.5
+5.2
+2.5
Iso-CTS [ICML’25]
base
82.7
75.7
63.7
87.6
77.9
64.4
92.2
87.9
79.4
Table 1: Average accuracy (%) across tasks. Gains are consistent across all methods, architectures, and task scales, growing larger at higher task counts where interference is more severe.
Figure 2: Ablations and analysis. (a) Average accuracy gain from Merge++ across all baselines on the 8-task setting under three data sources. (b) Effect of synthetic dataset size on B/32 with Task Arithmetic. (c) Expert alignment gain per layer after Merge++, measured on real validation images across 8 tasks.
Model merging combines fine-tuned checkpoints into a single multi-task model without retraining. Existing methods - such as task arithmetic, model soups, TIES, and DARE - are computationally efficient and empirically successful, but rely on heuristic design choices and lack formal optimality guarantees. We show that merging can be formulated as a convex quadratic programme over residual updates, yielding weights that minimise a squared-output calibration objective using calibration inputs and fine-tuned model outputs, and subsuming existing methods as special cases. Our framework yields a closed-form diagnostic - the fraction of residual energy captured by a chosen basis - that predicts downstream merge quality using only the calibration set. Empirically, the QP matches or outperforms existing methods in the single-layer setting, and we characterise when the optimal basis provides significant gains over the cheaper diagonal QP. We extend to multi-layer merging via a sequential layer-wise algorithm and demonstrate consistent gains across language and vision benchmarks.
Bethan Evans, Benjamin Etheridge, Stephen Roberts +1
Department of Mathematics, University of Oxford, Oxford, UK · Department of Engineering Science, University of Oxford, Oxford, UK.
Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. To perform well across multiple tasks, most existing merging methods use an additional dataset to find the coefficients for the best linear combination of task-specific weight updates. However, we identify an implicit regularization in this standard practice: searching over coefficients restricts the candidate models to a subspace spanned by task-specific weight updates. In this work, we investigate whether this regularization is actually useful. Surprisingly, empirical results show that optimizing merged-model weights without this regularization significantly boosts the performance of common merging methods across multiple architectures, domains, and even in an extremely data-limited scenario where only one instance is available per class. Moreover, directly optimizing the pretrained model weights even outperforms some existing merging methods. Analysis shows that better multi-task weights exist outside the subspace and can be found using multiple methods. We study different strategies for using the additional dataset, discussing their practical use and implications for model merging. Overall, this work calls for revisiting the existing model-merging pipeline, motivating a broader exploration of the weight space and a reconsideration of the implicit regularization induced by task arithmetic.
Model merging has shown that multitask models can be created by directly combining the parameters of different models that are each specialized on tasks of interest. However, models trained independently on distinct tasks often exhibit interference that degrades the merged model's performance. To solve this problem, we formally define the notion of Cross-Task Interference as the drift in the representation of the merged model relative to its constituent models. Reducing cross-task interference is key to improving merging performance. To address this issue, we propose our method, Resolving Interference (RI), a light-weight adaptation framework which disentangles expert models to be functionally orthogonal to the space of other tasks, thereby reducing cross-task interference. RI does this whilst using only unlabeled auxiliary data as input (i.e., no task-data is needed), allowing it to be applied in data-scarce scenarios. RI consistently improves the performance of state-of-the-art merging methods by up to 3.8% and generalization to unseen domains by up to 2.3%. We also find RI to be robust to the source of auxiliary input while being significantly less sensitive to tuning of merging hyperparameters. Our codebase is available at: https://github.com/pramesh39/resolving_interference
Pratik Ramesh, George Stoica, Arun Iyer +2
Georgia Tech · University of Washington · Microsoft Research +3