CrossMean: Model Merging by Pairing Merged and Expert Activations

Moshe Eliasof

cs.LGOct 10, 2026

CrossMean: Model Merging by Pairing Merged and Expert Activations

Authors: Moshe Eliasof

Organizations: Faculty of Computer and Information Science, Ben-Gurion University of the Negev

Abstract

Model merging combines fine-tuned experts into one network without retraining, and closed-form methods compute each merged layer from statistics of its input activations. However, these statistics are measured on a single expert, while in the merged model, each layer receives activations from the merged layers that precede it. Recent work re-measures these statistics in the merged model but still forms them from a single network. We formulate closed-form methods as a linear system and show that they use the same matrix on both sides. We then introduce CrossMean, which replaces this matrix on one side with a bilinear statistic that pairs merged and expert activations. From the same statistics, we also derive a label-free criterion that indicates the task sets in which accounting for the merged activations improves the merge more. Empirically, task sets with a lower criterion value show a larger improvement. On the eight-task benchmark of each vision backbone that we tested and on BERT encoders, CrossMean improves on merging with statistics measured on the experts, and with all linear layers merged, it also improves on re-measuring these statistics in the merged model. These results suggest that merging benefits from pairing merged and expert activations.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Existing closed-form methods measure the inputs $\mathbf{X}_{t}$ of each layer on expert $t$ , while in the merged model, the layer receives the merged inputs $\tilde{\mathbf{X}}_{t}$ . (b) In the linear system of Equation 1 , the existing activation-based methods set $\mathbf{B}_{t}=\mathbf{A}_{t}$ , a symmetric Gram matrix (gray) , while our CrossMean sets $\mathbf{B}_{t}=\bar{\mathbf{C}}_{t}$ , the bilinear statistic (orange) of Equation 4 , which pairs the merged and the expert inputs. (c) The margin of our CrossMean over RegMean tends to decrease as our alignment criterion $\rho$ of Equation 6 approaches $1$ , both across backbones and numbers of tasks, shown by the open markers, and with only the tasks varied, shown by the filled markers.

Method	$\mathbf{A}_{t}$	$\mathbf{B}_{t}$	Measured on
Weight averaging	$\mathbf{I}$	$\mathbf{I}$	none
Task Arithmetic	$\mathbf{I}$	$\lambda\mathbf{I}$	none
TIES	$\mathrm{diag}\,\mathbf{m}_{t}$	$\lambda\,\mathrm{diag}\,\mathbf{m}_{t}$	none
Fisher merging	$\mathrm{diag}\,\hat{\mathbf{F}}_{t}$	$\mathrm{diag}\,\hat{\mathbf{F}}_{t}$	expert $t$
RegMean	$\mathbf{G}_{t}$	$\mathbf{G}_{t}$	expert $t$
LOT	$\mathbf{G}_{t}$	$\mathbf{G}_{t}$	expert $t$

Table 1: Our formulation of merging methods as the linear system in Equation 1 . All the existing methods in this table put the same matrix on both sides, up to a scalar for the weight-space methods, while our CrossMean does not. Here

\mathbf{X}_{t}

and

\tilde{\mathbf{X}}_{t}

hold the expert and the merged inputs of a layer,

\mathbf{G}_{t}=\mathbf{X}_{t}^{\top}\mathbf{X}_{t}

is the Gram matrix of the expert inputs,

\mathbf{m}_{t}

is the TIES mask of expert

t

, and

\hat{\mathbf{F}}_{t}

is the Fisher information of expert

t

, of which Fisher merging keeps the diagonal. The CrossMean row shows the bilinear statistic

\bar{\mathbf{C}}_{t}

of Equation 4 , which averages the cross statistic of Equation 3 along a path.

Data-free baselines
Method	B/32	B/16	L/14	L/14-336	H/14
Weight averaging	$64.3$	$70.2$	$77.9$	$76.8$	$82.6$
Task Arithmetic	$67.4$	$75.2$	$79.8$	$83.3$	$87.6$
DARE	$67.1$	$75.1$	$80.1$	$83.3$	$87.5$
TIES	$70.6$	$76.4$	$82.6$	$83.7$	$87.7$
Model Stock	$51.2$	$57.6$	$66.4$	$62.5$	$71.1$

Table 2: Our results on the eight-task sets with the MLP coverage of Section 3.1 , in test accuracy (%), averaged over three calibration seeds. ViT-B/32, B/16, and L/14 use QuickGELU, and the other two backbones use GELU. We mark the top three in each column as First , Second , and Third .

Activation-based baselines
Method	B/32	B/16	L/14
RegMean	$80.4$	$84.0$	$89.1$
RegMean++	$82.4$	$85.1$	89.8
Progressive RegMean	$82.5$	$85.2$	89.8
LOT	$80.3$	$85.0$	$89.6$
LOT + Progressive RegMean	82.6	85.3	$89.6$

Table 3: Our results on the eight-task sets with QuickGELU and full coverage, in which all linear layers are merged. Our CrossMean and LOT + CrossMean both improve on all the activation-based baselines in this table on all three backbones, including RegMean++ and LOT. Entries are test accuracies in percent, averaged over three calibration seeds. We mark the top three in each column as First , Second , and Third .

	Low- $\rho$ task set			High- $\rho$ task set
Backbone	$\rho$	Prog. RegMean	CrossMean	$\rho$	Prog. RegMean	CrossMean
B/32	$0.863$	$\mathbf{+5.81}$	$\mathbf{+12.50}$	$0.947$	$+0.47$	$+1.70$
B/16	$0.901$	$\mathbf{+3.71}$	$\mathbf{+11.97}$	$0.963$	$+0.41$	$+0.62$
L/14	$0.934$	$\mathbf{+3.70}$	$\mathbf{+7.01}$	$0.968$	$+0.40$	$-0.20$
L/14-336	$0.950$	$\mathbf{+5.79}$	$\mathbf{+10.50}$	$0.974$	$+0.40$	$-0.19$
H/14	$0.934$	$\mathbf{+5.02}$	$\mathbf{+9.49}$	$0.970$	$+0.33$	$+0.55$

Table 4: Our controlled pairs of task sets. On each of the five backbones, the task set with the lower

\rho

has the larger margins of Progressive RegMean and of our CrossMean over RegMean. The two task sets of each backbone are disjoint eight-task subsets of the twenty-task pool. Margins over RegMean are in percent, averaged over three calibration seeds, and bold marks the larger margin of each pair.

Task set	$N$	RegMean	Progressive RegMean	CrossMean	$\rho$
GLUE-4	$256$	66.9	64.0 $-2.98$	72.0 $+5.11$	$0.942$
GLUE-4	$512$	67.0	64.0 $-2.94$	72.0 $+5.04$	$0.942$
GLUE-4	$2048$	67.0	64.1 $-2.90$	73.3 $+6.29$	$0.943$
GLUE-6	$256$	66.2	64.4 $-1.77$	73.4 $+7.24$	$0.936$

Table 5: Our merges of BERT-base encoders on GLUE show that our CrossMean improves on RegMean in all four settings. Entries are average GLUE validation scores over three calibration draws, with the margin over RegMean in small type,

N

is the number of calibration examples per task, and

\rho

is our alignment criterion of each setting. We mark the top three in each row as First , Second , and Third .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

	Progressive RegMean margin		CrossMean margin		$\rho$
Backbone	G	Q	G	Q	G	Q
B/32	$+2.65$	$+2.59$	$+2.84$	$+2.15$	$0.894$	$0.892$
B/16	$+1.97$	$+1.92$	$+1.12$	$+1.38$	$0.919$	$0.915$
L/14	$+1.37$	$+1.08$	$-0.86$	$+0.19$	$0.931$	$0.928$

Appendix

Table 6: Our comparison of the two activations shows that switching from GELU (G) to QuickGELU (Q) changes our criterion

\rho

by at most

0.004

and the margin of Progressive RegMean over RegMean by at most

0.29\%

, while it increases the accuracy of RegMean and Progressive RegMean by

3.5

4.9\%

. Each row is an eight-task set of public experts, with everything else fixed, and margins are in percent, averaged over three calibration seeds. The margin of Progressive RegMean is over RegMean, and the margin of CrossMean is over Progressive RegMean.

Data-free baselines
	B/32			B/16			L/14			L/14-336			H/14
Method	8	14	20	8	14	20	8	14	20	8	14	20	8	14	20
Weight averaging	$58.8$	$62.7$	$53.7$	$65.0$	$67.8$	$58.0$	$73.3$	$75.6$	$64.5$	$76.8$	$76.6$	$65.1$	$82.6$	$81.8$	$69.6$
Task Arithmetic	$61.6$	$63.9$	$55.3$	$69.3$	$70.5$	$60.0$	$75.0$	$77.1$	$66.4$	$83.3$	$80.7$	$69.0$	$87.6$	$84.6$	$72.4$
DARE	$61.2$	$63.8$	$55.2$	$69.3$	$70.4$	$59.9$	$74.8$	$77.0$	$66.7$	$83.3$	$80.6$	$68.9$	$87.5$	$84.5$	$72.4$
TIES	$64.7$	$65.8$	$56.3$	$70.4$	$71.4$	$61.0$	$77.5$	$78.9$	$67.5$	$83.7$	$81.4$	$69.4$	$87.7$	$84.8$	$73.0$

Appendix

Table 7: In our experiments, CrossMean reaches a higher accuracy than every other method in eleven of the fifteen settings. Entries are test accuracies in percent with GELU and MLP coverage, averaged over three seeds, and the headers indicate the number of tasks

T

. We report the eight-task sets of ViT-B/32, B/16, and L/14 with QuickGELU in Table 2 . We mark the top three in each column as First , Second , and Third .

Backbone	Parameters	$T{=}8$	$T{=}14$	$T{=}20$
ViT-B/32	88M	$+2.65$	$+1.85$	$+2.16$
ViT-B/16	86M	$+1.97$	$+1.95$	$+1.97$
ViT-L/14	304M	$+1.37$	$+1.08$	$+1.27$
ViT-L/14-336	304M	$+1.26$	$+1.46$	$+2.09$
ViT-H/14	632M	$+0.80$	$+0.93$	$+1.40$

Appendix

Table 8: Our measurements of the margin of Progressive RegMean over RegMean in the fifteen settings of the shared-protocol grid, which our criterion

\rho

anticipates in Section 5.2 . Each entry is the margin, in percent, averaged over three calibration seeds, with GELU.

T

is the number of tasks.

ViT-B/32
Method	SUN397	Cars	RESISC45	EuroSAT	SVHN	GTSRB	MNIST	DTD	Avg.
Data-free baselines
Weight averaging	$64.3$	61.2	$67.1$	$68.5$	$64.3$	$54.8$	$86.5$	$47.7$	$64.3$
Task Arithmetic	$63.3$	$59.7$	$67.6$	$70.7$	$73.9$	$62.1$	$92.9$	$48.6$	$67.4$
DARE	$63.3$	$59.6$	$67.0$	$70.0$	$73.4$	$62.2$	$92.6$	$48.7$	$67.1$
TIES	$63.5$	$59.1$	$69.8$	$74.2$	$82.6$	$73.3$	$95.5$	$46.8$	$70.6$

Appendix

Table 9: In our experiments, CrossMean improves on every baseline in task average on ViT-B/32, ViT-B/16, and ViT-L/14. In addition, on all three backbones, it improves on every baseline on RESISC45, EuroSAT, SVHN, GTSRB, and DTD. Entries are test accuracies in percent per task of the eight-task set, with QuickGELU, averaged over three calibration seeds. Avg. is the average that we report in Table 2 . We mark the top three in each column as First , Second , and Third .

ViT-L/14-336
Method	SUN397	Cars	RESISC45	EuroSAT	SVHN	GTSRB	MNIST	DTD	Avg.
Data-free baselines
Weight averaging	$72.4$	$79.4$	$75.9$	$80.6$	$80.0$	$72.8$	$92.5$	$61.1$	$76.8$
Task Arithmetic	$74.0$	$81.5$	$82.5$	$90.9$	$87.6$	$84.7$	97.6	$67.3$	$83.3$
DARE	$73.8$	$81.8$	$82.5$	$90.6$	$87.8$	$84.8$	97.6	$67.3$	$83.3$
TIES	74.8	81.9	$83.7$	$88.0$	$87.3$	88.0	$97.2$	68.8	$83.7$

Appendix

Table 10: In our experiments on ViT-L/14-336 and ViT-H/14, with GELU, CrossMean again improves on every baseline on RESISC45, EuroSAT, SVHN, GTSRB, and DTD, and on ViT-H/14, also in task average. Entries are as in Table 9 . We mark the top three in each column as First , Second , and Third .

Baselines
	ViT-B/32	ViT-B/16	ViT-L/14
LOT	80.32	85.01	89.64
Applied to LOT
LOT + Progressive RegMean	82.61	85.26	89.60
LOT + CrossMean	84.07	87.12	90.14
Improvement over LOT (%)

Appendix

Table 11: In our experiments, LOT + CrossMean improves on LOT on all three backbones, and the improvement decreases as the alignment criterion

\rho

of the task set increases. Entries are test accuracies in percent on the eight-task sets with QuickGELU, averaged over three calibration seeds. We mark the top three in each column as First , Second , and Third . In the last row, we report

\rho

of each task set.

Task set	$\rho$	Progressive RegMean	CrossMean
ViT-B/32, low- $\rho$ task set	$0.8635$	$+5.81$	$+12.50$
ViT-B/32, $T{=}20$	$0.8801$	$+2.16$	$+5.12$
ViT-B/32, $T{=}8$	$0.8943$	$+2.65$	$+5.49$
ViT-B/16, low- $\rho$ task set	$0.9006$	$+3.71$	$+11.97$
ViT-B/32, $T{=}14$	$0.9101$	$+1.85$	$+3.83$
ViT-B/16, $T{=}20$	$0.9105$	$+1.97$	$+4.03$

Appendix

Table 12: Our measurements show that a lower value of the alignment criterion

\rho

is associated with a larger margin of CrossMean and of Progressive RegMean over RegMean, in percent. Over these task sets, the rank correlations are

-0.72

and

-0.73

, respectively, and we report their intervals in Table 13 . The rows are the fifteen settings of the shared-protocol grid and the ten task sets of the controlled pairs, sorted by

\rho

Figure 2: We measure the alignment criterion

\rho

layer by layer on the eight-task set of each backbone, from the weight-averaged model, and the average of each curve is the criterion of Section 4 . Our measurements show that

\rho

varies with depth differently on each backbone. The dashed line marks

\rho=1

, the value of

\rho

when the merged inputs equal the expert inputs.

Panel of Figure 3 — Figure 3: Our merges of twelve random eight-task sets on ViT-B/32, with the backbone and the number of tasks fixed. (a) Task sets with a lower value of our alignment criterion $\rho$ show a larger margin of Progressive RegMean over RegMean. (b) The same margin as a function of task difficulty, namely the mean accuracy of each expert on its own task. Error bars are the standard deviation of the margin over three calibration seeds.

Quantity	$n$	Est.	$95\%$ CI [ $p$ ]
$\rho$ vs. margin of CrossMean , all 25 measured task sets	25	$-0.72$	$[-0.91,-0.40]$ [ $2\times 10^{-4}$ ]
$\rho$ vs. margin of CrossMean , shared-protocol grid	15	$-0.68$	$[-0.95,-0.17]$ [ $0.006$ ]
$\rho$ vs. margin of CrossMean over Progressive RegMean, shared-protocol grid	15	$-0.69$	$[-0.96,-0.19]$ [ $0.006$ ]
$\rho$ vs. margin of Progressive RegMean, all 25 measured task sets	25	$-0.73$	$[-0.90,-0.43]$ [ $10^{-4}$ ]
$\rho$ vs. margin, ViT-B/32 confound study	12	$-0.73$	$[-0.94,-0.20]$ [ $0.010$ ]
Difficulty vs. margin, ViT-B/32 confound study	12	$+0.27$	$[-0.50,+0.79]$ [ $0.408$ ]

Appendix

Table 13: We report every rank correlation that we quote in this paper, with its percentile-bootstrap

95\%

interval over task sets and its two-sided permutation

p

-value, both computed from

10^{4}

resamples. Unless stated otherwise, the margin is that of Progressive RegMean over RegMean.

Change to CrossMean	ViT-B/32	ViT-L/14
None, the unmodified method	$+2.84$	$-0.86$
Statistic
$\beta=0.25$	$-25.92$	$-27.20$
$\beta=0.50$	$-61.46$	$-78.55$
$\beta=1.00$	$-70.61$	$-78.82$
$\gamma=0.15$	$+2.40$	$-0.62$

Appendix

Table 14: Our ablations show that, of the changes that we tested, only symmetrizing the bilinear statistic lowers the margin of CrossMean over Progressive RegMean by more than

1\%

. Each entry is that margin, in percent, with one design choice changed, on the eight-task sets with GELU. Here,

\beta

is the weight of the symmetric part of the statistic,

\gamma

is the weight of the shrinkage toward the Gram matrix of the merged inputs, and

p

is the dropout rate of the calibration tokens. We define each change in Section D.1 .

	Time relative to RegMean			Peak memory (GiB)
Method	B/32, $T{=}8$	B/32, $T{=}20$	H/14, $T{=}8$	B/32, $T{=}8$	H/14, $T{=}8$
Iso-C	$0.3\times$	$0.1\times$	$0.6\times$	$0.34$	$2.36$
RegMean	$1.0\times$	$1.0\times$	$1.0\times$	$1.34$	$9.50$
Progressive RegMean	$24.4\times$	$26.8\times$	$25.9\times$	$0.96$	$6.37$
CrossMean	$31.2\times$	$32.4\times$	$68.8\times$	$0.92$	$6.27$

Appendix

Table 15: Our measurements of the cost of one merge show that CrossMean and Progressive RegMean need less peak memory than RegMean. On one NVIDIA B200, we measure (i) the wall-clock time of one merge relative to RegMean; and (ii) the peak allocated memory.

Figure 4: We measure the accuracy on the eight-task set of ViT-B/32 with GELU as a function of the number

N

of calibration images per task. RegMean (gray) and Progressive RegMean (blue) reach a plateau at

N{=}64

, while our CrossMean (orange) reaches its plateau at about four times as many images.

Model Merging

Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.

See all 191 papers

May 13, 2026

ReForge: Refining Merged Models with Anchor-Regularized Regression

Model merging aims to combine multiple task-specific expert models into a single model without joint retraining, offering a practical alternative to multi-task learning when data access or computational budget is limited. Existing model merging methods rarely exploit strong merged models as priors for further improvement. To address this limitation, we propose ReForge, a bilevel optimization framework that formulates module-wise refinement as Bayesian linear regression with an anchor-centered prior. The inner level yields a closed-form MAP estimate from unlabeled calibration activations. The outer level uses Bayesian optimization to jointly select heterogeneous regularization strengths and assembly scales using held-out validation data. Furthermore, we develop a data-free variant of ReForge that replaces activation statistics with task-vector Grams, eliminating the need for calibration examples. Across extensive benchmarks, including up to 20-task merging in vision and 5-task merging in language, ReForge consistently outperforms all evaluated plug-and-play anchor baselines (e.g., TA, WUDI-Merging, and TSV). On 20-task ViT-B/32, ReForge improves the strongest evaluated baseline, ISO-CTS, from 77.6% to 82.8% in the data-assisted setting and to 81.5% in the data-free setting. On eight-task ViT-L/14, the data-assisted variant achieves 95.1% mean accuracy, compared with 95.8% for the individual task experts. Our source code will be released soon.

Kaiyang Li, Shaobo Han, Qing Su +1

School of Computing, University of Connecticut, Storrs, CT 06269 · Optical Networking and Sensing, NEC Labs America, Princeton, NJ 08540

Model Merging

Ask PDF

Sep 19, 2026

Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion

Model merging consolidates fine-tuned experts into one multi-task model without retraining. All existing data-free methods approach this problem entirely in weight space. Restricted to arithmetic on parameters, these methods never observe how each expert behaves, a signal that only emerges through forward evaluation. Accessing this behavioral signal requires inputs to evaluate on, which the data-free setting prohibits. We propose Merge++, a post-hoc method that addresses this by inverting the expert checkpoints to synthesize task-representative images, then distilling expert knowledge into the merged model using those images. Merge++ requires no additional data beyond the checkpoints themselves. It applies universally across merging algorithms and operates as a complementary refinement stage independent of the underlying weight-space method. The method consistently improves merging algorithms ranging from simple task arithmetic to state-of-the-art spectral methods, with average gains of +2 to +8 points and up to +25.9 on individual configurations.

Aditya Pola, Vineeth N. Balasubramanian

IIT Hyderabad · Microsoft Research, India

Model Merging

Ask PDF

Apr 1, 2026

Model Merging via Data-Free Covariance Estimation

Model merging provides a way of cheaply combining individual models to produce a model that inherits each individual's capabilities. While some merging methods can approach the performance of multitask training, they are often heuristically motivated and lack theoretical justification. A principled alternative is to pose model merging as a layer-wise optimization problem that directly minimizes interference between tasks. However, this formulation requires estimating per-layer covariance matrices from data, which may not be available when performing merging. In contrast, many of the heuristically-motivated methods do not require auxiliary data, making them practically advantageous. In this work, we revisit the interference minimization framework and show that, under certain conditions, covariance matrices can be estimated directly from difference matrices, eliminating the need for data while also reducing computational costs. We validate our approach across vision and language benchmarks on models ranging from 86M parameters to 7B parameters, outperforming previous data-free state-of-the-art merging methods

Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo +2

Mila & DIRO, Universit´e de Montr´eal · University of Toronto & Vector Institute

Multi-Task Learning Language Model Merging Model Merging Covariance Estimation

Ask PDF

Abstract

Model Merging

Explore similar work

ReForge: Refining Merged Models with Anchor-Regularized Regression

Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion

Model Merging via Data-Free Covariance Estimation