Model Merging via Data-Free Covariance Estimation

Marawan Gamal Abdel Hameed; Derek Tam; Pascal Jr Tikeng Notsawo; Colin Raffel; Guillaume Rabusseau

cs.LGApr 1, 2026

Model Merging via Data-Free Covariance Estimation

Authors: Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo, Colin Raffel, Guillaume Rabusseau

Organizations: Mila & DIRO, Universit´e de Montr´eal · University of Toronto & Vector Institute

Abstract

Model merging provides a way of cheaply combining individual models to produce a model that inherits each individual's capabilities. While some merging methods can approach the performance of multitask training, they are often heuristically motivated and lack theoretical justification. A principled alternative is to pose model merging as a layer-wise optimization problem that directly minimizes interference between tasks. However, this formulation requires estimating per-layer covariance matrices from data, which may not be available when performing merging. In contrast, many of the heuristically-motivated methods do not require auxiliary data, making them practically advantageous. In this work, we revisit the interference minimization framework and show that, under certain conditions, covariance matrices can be estimated directly from difference matrices, eliminating the need for data while also reducing computational costs. We validate our approach across vision and language benchmarks on models ranging from 86M parameters to 7B parameters, outperforming previous data-free state-of-the-art merging methods

Figures & tables

Panel of Figure 1 — Figure 1: Left: RegMean requires data to compute activation covariances ${\bm{C}}_{t}$ , while ACTMat estimates them directly from the difference matrices as ${\bm{\Delta}}_{t}^{\top}{\bm{\Delta}}_{t}\approx\mathbf{C}_{t}$ . Right: On T5-Large, ACTMat nearly matches RegMean’s accuracy without any data, while substantially outperforming other data-free baselines.

Panel of Figure 2 — Figure 2: Empirical measurement of the three angular error terms in Theorem 3.1 on ViT-B/16. (a) Cross-term error ${\epsilon}^{(\mathrm{cross})}$ . (b) Correlation error ${\epsilon}^{(\mathrm{corr})}$ . (c) Drift error ${\epsilon}^{(\mathrm{drift})}$ . All three terms are relatively small across layers and tasks, indicating that ${\bm{\Delta}}_{t}^{\top}{\bm{\Delta}}_{t}$ is well-aligned with the final covariance ${\bm{C}}_{t}^{(K)}$ (angular distance is normalized).

Figure 3: Distribution of scaling-coefficient ratios

\kappa_{i}/\kappa_{j}

over all dataset pairs, for each layer in the ViT-B/16 model.

\kappa_{i}=\|{\bm{C}}_{i}\|_{F}/\|{\bm{\Delta}}_{i}^{\top}{\bm{\Delta}}_{i}\|_{F}

and measures the ratio of the norm of the covariance matrix to the norm of the ACTMat estimate of the covariance matrix.

Figure 4: Comparison between test accuracy of merging methods across multiple settings (NLP models fine-tuned on 7 tasks and vision models fine-tuned on 8 tasks). Hatched bars indicate that the method is not data-free. Stacked bars with a dotted pattern indicate the performance of the method (bottom bar) and performance of the method when combined with KnOTS ( Stoica et al., 2025 ) for LoRA fine-tuned models (improvement only for Iso-C).

Method ( $\downarrow$ )	HumanEval		HumanEval+		AIME ’24		AIME ’25		IFEval	Avg
Eval ( $\rightarrow$ )	@1	@10	@1	@10	@1	@32	@1	@32	@1	@1
Zero-shot	50.8	82.2	47.3	78.6	22.1	66.7	21.6	53.3	30.7	34.5
Expert	62.0	85.3	57.0	84.7	38.2	76.7	30.7	66.7	82.4	54.1
RegMean	59.1	85.7	54.8	81.1	30.8	73.3	27.8	60.0	74.7	49.4
Average	59.9	85.9	54.5	81.9	34.6	83.3	30.5	60.0	67.5	49.4
Iso-C	59.0	84.4	53.8	82.9	32.5	76.7	31.5	63.3	63.0	48.0

Table 1: Comparison between merging methods in the RL Zero setting, where models are fine-tuned using RLVR starting from the OLMo-3-7B base model. We bold the best and underline the second best data-free method for each dataset.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Vision (OpenCLIP ViT)	Language (T5)
Optimizer	AdamW	AdamW
Learning rate	$1\times 10^{-5}$ (cosine schedule w/ $500$ -step warm-up)	$1\times 10^{-4}$ , constant
Weight decay	$0.1$	$0$
Batch size	$128$	$256$ (base); $64$ (large)
Gradient clipping	$1.0$	none
Precision	float32	bfloat16

Appendix

Table 3: Fine-tuning configuration for the vision and language experts.

Task	OLMES task id
HumanEval	codex_humaneval::tulu
HumanEval+	codex_humanevalplus::tulu
AIME 2024	aime:zs_cot_r1::pass_at_32_2024_deepseek
AIME 2025	aime:zs_cot_r1::pass_at_32_2025_deepseek
IFEval	ifeval::tulu

Appendix

Table 4: OLMES task recipes used for evaluating the merged RL experts.

Method	ViT-B/16	ViT-B/32	ViT-L/14
RegMean ( nn.MultiheadAttention )	83.1	79.4	87.2
RegMean	87.6	83.0	90.0

Appendix

Table 5: Performance comparison between RegMean implemented with default the nn.MultiheadAttention implementation vs a custom implementation enabling the covariances of query, key and value projection matrices to be captured.

Method ( $\downarrow$ )	Data-free	NLP		Vision
Model ( $\rightarrow$ )		T5-B	T5-L	ViT-B/16	ViT-B/32	ViT-L/14
Zero-shot	-	54.0	51.3	55.5	48.2	65.2
Experts	-	76.0	82.4	94.6	90.4	94.1
Multitask	-	78.7	82.7	92.1	89.8	93.7
RegMean	✗	74.5	80.8	87.6	83.0	90.0
TA	✗	71.4	69.7	76.1	69.7	84.3

Appendix

Table 6: Comparison between merging methods across multiple settings (NLP models fine-tuned on 7 tasks and vision models fine-tuned on 8 tasks). All model parameters are fine-tuned. The ✓ symbol indicates that no data is used by the method. Results are reported on test sets.

Method ( $\downarrow$ )	Data-free	NLP		Vision
Model ( $\rightarrow$ )		T5-B	T5-L	ViT-B/16	ViT-B/32	ViT-L/14
Zero-shot	-	54.0	51.3	55.5	48.2	65.2
Experts	-	76.5	79.2	84.1	80.7	89.0
RegMean	✗	75.9	77.7	80.3	76.5	86.9
TA	✗	66.7	70.9	75.7	70.5	83.0
TA	✓	54.2	55.0	66.1	53.5	77.6

Appendix

Table 7: Comparison between merging methods across multiple settings (NLP models fine-tuned on 7 tasks and vision models fine-tuned on 8 tasks). Model parameters are fine-tuned using LoRA . The ✓ symbol indicates that no data is used by the method. Results are reported on test sets

Method	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Avg
Zero-shot	64.7	44.7	55.8	43.3	51.8	66.3	65.5	52.0	55.5
Experts	87.5	98.3	99.7	99.0	99.8	97.0	78.8	97.8	94.7
RegMean	78.5	79.6	96.5	89.0	99.0	88.4	71.7	94.9	87.2
Average	70.1	54.3	81.1	60.8	94.4	75.5	68.6	72.7	72.2
TA	61.5	53.8	76.7	72.5	98.3	65.6	47.8	87.1	70.4
TIES	73.2	69.0	88.3	73.4	98.7	79.2	65.9	88.1	79.5

Appendix

Table 8: Task-wise merging results

Method ( $\downarrow$ )	ViT-B/16			ViT-B/32			ViT-L/14
Num. Experts ( $\rightarrow$ )	8	14	20	8	14	20	8	14	20
Zero-shot	55.5	61.6	59.8	48.2	57.9	56.4	65.2	69.6	66.1
Experts	94.7	93.1	93.3	90.4	89.7	90.4	94.1	93.6	94.1
RegMean	87.5	82.2	78.9	83.0	78.7	75.4	90.1	87.6	85.6
Average	72.2	70.3	66.1	65.4	64.9	61.4	79.3	77.9	72.4
TIES	79.5	72.2	66.2	72.8	65.5	58.7	85.2	79.7	74.6

Appendix

Table 9: Comparison between merging methods as the number of experts merged is varied.

Method	Merging FLOPs	Preprocessing FLOPs	# of Exp. Op.
Average	$TN^{2}$	-	0
Task Arith.	$(2T+1)N^{2}$	-	0
RegMean	$(T+3)N^{3}+(2T-2)N^{2}$	$(2L-1)TN^{2}$	1
ACTMat	$(2T+3)N^{3}+(3T-2)N^{2}$	-	1
Iso-C	$23N^{3}+(2T+2)N^{2}+N$	-	1
TSV	$(22T+45)N^{3}+(T+3)N^{2}$	-	T+2

Appendix

Table 10: We report the computational cost of merging

T

models for a single linear layer with equal input and output dimension

N

. For each method we show the merging FLOPs, the pre-processing FLOPs (applicable only to RegMean, where

L

is the number of samples used to estimate the covariance matrices), and the number of expensive operations ( exp. op. ), i.e., SVDs or matrix inverses, which are inherently sequential).

Explore similar work

ReForge: Refining Merged Models with Anchor-Regularized Regression
May 13, 2026Kaiyang Li, Shaobo Han, Qing Su +1Continual Model Merging Multi--Task Learning
School of Computing, University of Connecticut, Storrs, CT 06269 · Optical Networking and Sensing, NEC Labs America, Princeton, NJ 08540
Model Merging: Foundations and Algorithms
May 2, 2026Donato CrisostomiContinual Model Merging Multi--Task Learning
SAPIENZA UNIVERSITY OF ROME · GLADIA research lab · Department of Computer Science +1
Model Merging by Output-Space Projection
May 27, 2026Bethan Evans, Benjamin Etheridge, Stephen Roberts +1Continual Model Merging Multi--Task Learning
Department of Mathematics, University of Oxford, Oxford, UK · Department of Engineering Science, University of Oxford, Oxford, UK.

May 13, 2026, cs.LGJ/K move · Enter open · S save

ReForge: Refining Merged Models with Anchor-Regularized Regression

Kaiyang Li, Shaobo Han, Qing Su, Shihao Ji

School of Computing, University of Connecticut, Storrs, CT 06269 · Optical Networking and Sensing, NEC Labs America, Princeton, NJ 08540

Continual Model Merging Multi--Task Learning Mixture-Of-Experts Architectures Sparse Spatial Anchors Bayesian

Open paper AskPDF arXiv

Model merging aims to combine multiple task-specific expert models into a single model without joint retraining, offering a practical alternative to multi-task learning when data access or computational budget is limited. Existing model merging methods rarely exploit strong merged models as priors for further improvement. To address this limitation, we propose ReForge, a bilevel optimization framework that formulates module-wise refinement as Bayesian linear regression with an anchor-centered prior. The inner level yields a closed-form MAP estimate from unlabeled calibration activations. The outer level uses Bayesian optimization to jointly select heterogeneous regularization strengths and assembly scales using held-out validation data. Furthermore, we develop a data-free variant of ReForge that replaces activation statistics with task-vector Grams, eliminating the need for calibration examples. Across extensive benchmarks, including up to 20-task merging in vision and 5-task merging in language, ReForge consistently outperforms all evaluated plug-and-play anchor baselines (e.g., TA, WUDI-Merging, and TSV). On 20-task ViT-B/32, ReForge improves the strongest evaluated baseline, ISO-CTS, from 77.6% to 82.8% in the data-assisted setting and to 81.5% in the data-free setting. On eight-task ViT-L/14, the data-assisted variant achieves 95.1% mean accuracy, compared with 95.8% for the individual task experts. Our source code will be released soon.

Abstract

Explore similar work

ReForge: Refining Merged Models with Anchor-Regularized Regression

Model Merging: Foundations and Algorithms

Model Merging by Output-Space Projection