Error-Subspace Calibration for Multi-Task Model Merging

Yeseul Cho; Hanseul Cho; Chulhee Yun

cs.LGOct 24, 2026

Error-Subspace Calibration for Multi-Task Model Merging

Authors: Yeseul Cho, Hanseul Cho, Chulhee Yun

Organizations: Kim Jaechul Graduate School of AI, KAIST

Abstract

Model merging combines task-specific expert models into a single model without joint training on their original datasets, but often incurs a performance gap relative to the individual experts. Post-merging calibration can reduce this gap through task-specific representation alignment, but the alignment objective alone does not specify which feature directions to correct. We propose Error-Subspace Calibration (ESCal), a post-merging calibration framework that identifies low-dimensional error subspaces from weight differences between expert and merged models. At each intermediate block boundary, a bridge module projects the merged representation onto the next block's input-projection error subspace and maps these coordinates to an additive correction in the current block's output-projection error subspace. The subspace bases remain fixed, so calibration learns only these linear maps, yielding a parameter-efficient mechanism for feature correction. We find that ESCal brings the merged model closer to the experts in both representations and predictions, indicating more faithful recovery of task-specific behavior. Experiments on vision and natural language understanding benchmarks show that ESCal improves merged-model performance across diverse architectures and merging methods using a small calibration set for each task.

Figures & tables

Panel of Figure 1 — Figure 1: Post-merging calibration of an eight-task Task Arithmetic merge of ViT-B/32 experts. (Left) Test-accuracy gains over the uncalibrated merge (percentage points); “Average” denotes the eight-task mean. (Right) Mean squared error between merged and expert hidden features at depth $\ell$ , averaged over eight tasks; shading indicates standard errors across tasks. IntervMerge uses the selected ViT-B/32–Task Arithmetic setting ( $r=16$ , $\eta=3\times 10^{-4}$ , 1,000 outer iterations). Its results are averaged over five seeds in both panels; for feature error, seed numerators and element counts are pooled within each task before averaging across tasks. ESCal achieves the largest accuracy gains on six tasks and on average, and the lowest feature error at most block boundaries.

Figure 2: Overview of Error-Subspace Calibration ( ESCal ). Task-specific Bridge modules align the frozen merged model with its expert at complete block outputs, after both residual additions. At intermediate boundaries, read and write bases come from merging errors in the next block’s joint QKV input projection and the current block’s MLP output projection, respectively.

Method	SUN397	Cars	RESISC45	EuroSAT	SVHN	GTSRB	MNIST	DTD	Avg.
Pretrained	63.18	59.73	60.33	45.37	31.61	32.56	48.25	43.99	48.13
Expert	75.34	77.73	95.98	99.74	97.46	98.73	99.69	79.36	90.50
Task Arithmetic	55.32	54.98	66.82	77.78	80.21	69.68	97.34	50.37	69.06
w/ Surgery	60.39	52.62	71.38	88.65	81.67	74.27	97.82	52.77	72.45
w/ Surgery V2	70.70	66.41	90.65	99.12	96.61	95.88	99.47	70.90	86.22
w/ IntervMerge	66.35	58.87	85.88	97.61	93.44	87.49	98.62	64.31	81.57

Table 1: Vision model merging results on ViT-B/32. Task-wise test accuracy (%) and average performance across eight image classification tasks.

Method	CoLA	SST-2	MRPC	STS-B	QQP	MNLI	QNLI	RTE	Avg.
Pretrained	0.0000	0.5975	0.1581	-0.0758	0.4633	0.3183	0.5078	0.5271	0.3120
Expert	0.5895	0.9438	0.9091	0.8997	0.8994	0.8718	0.9198	0.7726	0.8507
Task Arithmetic	0.1232	0.6135	0.5500	0.5356	0.5881	0.6270	0.5365	0.5343	0.5135
w/ Surgery	0.2553	0.8532	0.7657	0.7877	0.7793	0.7215	0.7839	0.7184	0.7081
w/ Surgery V2	0.5358	0.9326	0.8908	0.8978	0.8002	0.8404	0.9084	0.7408	0.8184
w/ IntervMerge	0.4647	0.9135	0.8477	0.8706	0.7779	0.7625	0.8524	0.7199	0.7762

Table 2: NLU model merging results on RoBERTa-base. Task-wise performance and average score across eight GLUE tasks.

Basis	SUN397	Cars	RESISC45	EuroSAT	SVHN	GTSRB	MNIST	DTD	Avg.
Random orthonormal	65.42	63.88	88.54	98.61	96.06	95.81	99.32	64.98	84.08
Fine-tuned weights, top- $r$	66.94	65.02	89.85	99.03	96.17	96.32	99.36	65.70	84.80
Task vector, top- $r$	69.29	67.72	93.01	99.44	97.08	97.47	99.53	72.53	87.01
Merging error, top- $r$	70.33	69.02	93.60	99.48	97.10	97.70	99.55	73.52	87.54

Table 3: We compare different basis sources on ViT-B/32 merged with Task Arithmetic using rank

r=128

. We report mean accuracy here, with standard deviations provided in Table 18 in Appendix D.1 .

Read basis	Write basis
Read basis	Top- $r$	Random- $r$	Bottom- $r$
Top- $r$	$\mathbf{87.54\pm 0.07}$	$84.24\pm 0.18$	$81.69\pm 0.12$
Random- $r$	$85.96\pm 0.15$	$82.93\pm 0.22$	$80.89\pm 0.19$
Bottom- $r$	$85.35\pm 0.07$	$82.63\pm 0.20$	$80.86\pm 0.13$

Table 4: We independently vary the read and write directions for ESCal with Task Arithmetic on ViT-B/32 using rank

r=128

. Random-

r

denotes

r

directions selected from the singular-vector basis of the merging error.

Read	Write	$r=64$	$r=128$	$r=256$
Constrained	Constrained	$86.91\pm 0.06$	$87.54\pm 0.07$	$87.62\pm 0.12$
Unconstrained	Constrained	$87.23\pm 0.05$	$87.50\pm 0.04$	$87.62\pm 0.14$
Constrained	Unconstrained	$87.53\pm 0.10$	$87.76\pm 0.04$	$87.75\pm 0.07$

Table 5: Constrained uses a fixed top-

r

error basis. Unconstrained learns the corresponding projection from a random initialization without restricting its directions to the error subspace.

Method	ViT-B/32		ViT-B/16		ViT-L/14		RoBERTa
Method	KL $\downarrow$	Agreement $\uparrow$	KL $\downarrow$	Agreement $\uparrow$	KL $\downarrow$	Agreement $\uparrow$	KL $\downarrow$	Agreement $\uparrow$
Task Arithmetic	0.9383	74.51%	0.7918	79.56%	0.4447	86.83%	0.6926	61.38%
w/ Surgery V2	0.2671	90.70%	0.2054	92.94%	0.1259	95.27%	0.2872	87.76%
w/ FeatCal	0.3204	89.48%	0.2987	90.96%	0.1610	94.40%	0.4697	83.35%
w/ ESCal	0.1821	92.09%	0.1341	94.16%	0.0910	96.09%	0.2439	88.74%

Table 6: Prediction alignment with expert models. KL divergence and prediction agreement are reported across vision and NLU architectures.

Table 9

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Base merging method	ViT-B/32	ViT-B/16	ViT-L/14
Task Arithmetic	$(16,3\!\times\!10^{-4},1000)$	$(16,10^{-3},200)$	$(4,10^{-3},400)$
TIES-Merging	$(16,3\!\times\!10^{-4},500)$	$(16,10^{-3},1000)$	$(16,3\!\times\!10^{-4},1000)$
AdaMerging (tw)	$(16,10^{-3},400)$	$(16,10^{-3},400)$	$(4,3\!\times\!10^{-4},1000)$
AdaMerging (lw)	$(16,3\!\times\!10^{-4},400)$	$(16,10^{-3},100)$	$(16,3\!\times\!10^{-4},100)$
WUDI-Merging	$(16,10^{-4},500)$	$(16,10^{-3},100)$	$(16,3\!\times\!10^{-4},1000)$
ProDistill	$(16,10^{-3},1000)$	$(16,3\!\times\!10^{-4},1000)$	$(16,3\!\times\!10^{-4},500)$

Appendix

Table 9: Selected IntervMerge configurations. Each vision entry is

(r,\eta,K)

, with

K

measured in outer iterations.

Base merging method	Rank $r$	Learning rate $\eta$	Epochs $E$
Task Arithmetic	4	$3\times 10^{-3}$	500
TIES-Merging	16	$10^{-3}$	200
AdaMerging (tw)	16	$3\times 10^{-3}$	500
AdaMerging (lw)	4	$3\times 10^{-3}$	1000
WUDI-Merging	16	$10^{-3}$	1000
ProDistill	16	$3\times 10^{-3}$	1000

Appendix

Table 10: Selected IntervMerge configurations for RoBERTa-base.

Method	SUN397	Cars	RESISC45	EuroSAT	SVHN	GTSRB	MNIST	DTD	Avg.
Task Arithmetic	55.32	54.98	66.82	77.78	80.21	69.68	97.34	50.37	69.06
w/ Surgery	60.39 $\pm$ 0.46	52.62 $\pm$ 0.59	71.38 $\pm$ 1.76	88.65 $\pm$ 0.79	81.67 $\pm$ 1.65	74.27 $\pm$ 0.91	97.82 $\pm$ 0.16	52.77 $\pm$ 2.73	72.45 $\pm$ 0.52
w/ Surgery V2	70.70 $\pm$ 0.19	66.41 $\pm$ 0.49	90.65 $\pm$ 1.23	99.12 $\pm$ 0.18	96.61 $\pm$ 0.07	95.88 $\pm$ 0.62	99.47 $\pm$ 0.06	70.90 $\pm$ 0.75	86.22 $\pm$ 0.17
w/ IntervMerge	66.35 $\pm$ 0.38	58.87 $\pm$ 0.83	85.88 $\pm$ 1.55	97.61 $\pm$ 0.32	93.44 $\pm$ 0.22	87.49 $\pm$ 1.09	98.62 $\pm$ 0.10	64.31 $\pm$ 1.10	81.57 $\pm$ 0.29
w/ FeatCal	69.22	70.31	89.06	98.00	95.01	92.40	99.13	71.11	85.53
w/ ESCal	70.33 $\pm$ 0.22	69.02 $\pm$ 0.19	93.60 $\pm$ 0.36	99.48 $\pm$ 0.07	97.10 $\pm$ 0.05	97.70 $\pm$ 0.22	99.55 $\pm$ 0.03	73.52 $\pm$ 0.44	87.54 $\pm$ 0.07

Appendix

Table 11: Full task-wise test accuracy (%) on ViT-B/32.

Method	SUN397	Cars	RESISC45	EuroSAT	SVHN	GTSRB	MNIST	DTD	Avg.
Task Arithmetic	61.74	65.96	74.15	77.07	88.06	73.92	98.48	53.03	74.05
w/ Surgery	65.25 $\pm$ 0.43	63.49 $\pm$ 1.63	76.73 $\pm$ 1.36	89.67 $\pm$ 0.99	88.03 $\pm$ 0.40	76.13 $\pm$ 2.11	98.39 $\pm$ 0.12	58.62 $\pm$ 2.21	77.04 $\pm$ 0.40
w/ Surgery V2	74.17 $\pm$ 0.16	76.68 $\pm$ 0.75	93.02 $\pm$ 0.59	98.95 $\pm$ 0.24	97.21 $\pm$ 0.06	96.51 $\pm$ 0.62	99.52 $\pm$ 0.04	74.85 $\pm$ 0.95	88.86 $\pm$ 0.17
w/ IntervMerge	70.34 $\pm$ 0.22	69.55 $\pm$ 0.96	88.15 $\pm$ 2.12	97.50 $\pm$ 0.45	95.30 $\pm$ 0.22	86.57 $\pm$ 2.14	99.00 $\pm$ 0.12	67.57 $\pm$ 1.88	84.25 $\pm$ 0.36
w/ FeatCal	71.96 $\pm$ 0.06	79.84 $\pm$ 0.19	90.12 $\pm$ 0.32	97.95 $\pm$ 0.26	96.44 $\pm$ 0.03	93.37 $\pm$ 0.03	99.16 $\pm$ 0.03	73.41 $\pm$ 0.65	87.78 $\pm$ 0.11
w/ ESCal	74.22 $\pm$ 0.27	79.82 $\pm$ 0.54	95.07 $\pm$ 0.26	99.46 $\pm$ 0.09	97.61 $\pm$ 0.08	98.47 $\pm$ 0.26	99.58 $\pm$ 0.02	77.31 $\pm$ 0.63	90.19 $\pm$ 0.20

Appendix

Table 12: Full task-wise test accuracy (%) on ViT-B/16.

Method	SUN397	Cars	RESISC45	EuroSAT	SVHN	GTSRB	MNIST	DTD	Avg.
Task Arithmetic	74.16	82.09	86.67	93.93	87.91	86.77	98.94	65.69	84.52
w/ Surgery	74.71 $\pm$ 0.27	80.50 $\pm$ 0.43	87.89 $\pm$ 1.66	95.47 $\pm$ 0.57	88.82 $\pm$ 0.59	86.87 $\pm$ 1.01	98.94 $\pm$ 0.13	69.84 $\pm$ 1.94	85.38 $\pm$ 0.19
w/ Surgery V2	78.61 $\pm$ 0.28	86.93 $\pm$ 0.31	94.57 $\pm$ 1.00	99.39 $\pm$ 0.04	97.73 $\pm$ 0.04	97.55 $\pm$ 0.54	99.58 $\pm$ 0.03	78.62 $\pm$ 0.99	91.62 $\pm$ 0.11
w/ IntervMerge	76.40 $\pm$ 0.26	83.59 $\pm$ 0.68	91.62 $\pm$ 1.51	98.53 $\pm$ 0.30	96.29 $\pm$ 0.15	92.54 $\pm$ 1.61	99.42 $\pm$ 0.04	74.19 $\pm$ 1.01	89.07 $\pm$ 0.10
w/ FeatCal	79.05 $\pm$ 0.04	90.18 $\pm$ 0.09	94.11 $\pm$ 0.11	99.30 $\pm$ 0.05	96.47 $\pm$ 0.04	96.85 $\pm$ 0.08	99.36 $\pm$ 0.03	78.96 $\pm$ 0.06	91.78 $\pm$ 0.02
w/ ESCal	79.12 $\pm$ 0.17	89.28 $\pm$ 0.16	96.11 $\pm$ 0.28	99.66 $\pm$ 0.05	97.98 $\pm$ 0.03	98.75 $\pm$ 0.23	99.64 $\pm$ 0.02	80.52 $\pm$ 0.28	92.63 $\pm$ 0.05

Appendix

Table 13: Full task-wise test accuracy (%) on ViT-L/14.

Method	CoLA	SST-2	MRPC	STS-B	QQP	MNLI	QNLI	RTE	Avg.
Pretrained	0.0000	0.5975	0.1581	-0.0758	0.4633	0.3183	0.5078	0.5271	0.3120
Expert	0.5895	0.9438	0.9091	0.8997	0.8994	0.8718	0.9198	0.7726	0.8507
Task Arithmetic	0.1232	0.6135	0.5500	0.5356	0.5881	0.6270	0.5365	0.5343	0.5135
w/ Surgery	0.2553 $\pm$ 0.0253	0.8532 $\pm$ 0.0124	0.7657 $\pm$ 0.0155	0.7877 $\pm$ 0.0089	0.7793 $\pm$ 0.0192	0.7215 $\pm$ 0.0118	0.7839 $\pm$ 0.0224	0.7184 $\pm$ 0.0099	0.7081 $\pm$ 0.0088
w/ Surgery V2	0.5358 $\pm$ 0.0106	0.9326 $\pm$ 0.0029	0.8908 $\pm$ 0.0037	0.8978 $\pm$ 0.0018	0.8002 $\pm$ 0.0137	0.8404 $\pm$ 0.0016	0.9084 $\pm$ 0.0019	0.7408 $\pm$ 0.0110	0.8184 $\pm$ 0.0016
w/ IntervMerge	0.4647 $\pm$ 0.0273	0.9135 $\pm$ 0.0049	0.8477 $\pm$ 0.0061	0.8706 $\pm$ 0.0075	0.7779 $\pm$ 0.0135	0.7625 $\pm$ 0.0080	0.8524 $\pm$ 0.0089	0.7199 $\pm$ 0.0098	0.7762 $\pm$ 0.0067

Appendix

Table 14: Full task-wise performance on the NLU benchmark using RoBERTa-base.

Method	GSM8K	MATH	HumanEval	MBPP	Avg.	Norm Avg.
WizardMath-13B	0.6343	0.1468	0.0732	0.2156	0.2675	0.7655
Llama-2-13b-code-alpaca	0.0000	0.0000	0.2195	0.2959	0.1289	0.5000
Task Arithmetic	0.6494	0.1446	0.0122	0.0917	0.2245	0.5936
w/ Surgery	0.6542	0.1468	0.0488	0.1514	0.2503	0.6913
w/ Surgery V2	0.6255	0.1430	0.0793	0.2133	0.2653	0.7606
w/ FeatCal	0.6406	0.1452	0.2073	0.2683	0.3154	0.9625

Appendix

Table 15: Post-merging calibration for LLaMA-2-13B math–code merging.

Method	HumanEval	MBPP	AlpacaEval	Avg.	Norm Avg.
WizardLM-13B-V1.2	0.3293	0.3509	0.1646	0.2816	1.2287
Llama-2-13b-code-alpaca	0.2195	0.2959	0.0196	0.1783	0.7064
Task Arithmetic	0.3232	0.3440	0.1576	0.2749	1.1975
w/ Surgery	0.3171	0.3486	0.1604	0.2754	1.1991
w/ Surgery V2	0.3293	0.3349	0.1615	0.2752	1.2044
w/ FeatCal	0.3110	0.3372	0.1520	0.2667	1.1600

Appendix

Table 16: Post-merging calibration for LLaMA-2-13B instruction–code merging.

Method	GSM8K	MATH	HumanEval	MBPP	AlpacaEval	Avg.	Norm Avg.
WizardLM-13B-V1.2	0.0223	0.0004	0.3293	0.3280	0.1646	0.1689	0.6076
WizardMath-13B	0.6343	0.1468	0.0732	0.2100	0.0487	0.2226	0.6317
Task Arithmetic	0.6614	0.1440	0.2622	0.3000	0.1053	0.2946	0.8749
w/ Surgery	0.6693	0.1358	0.2683	0.3300	0.1101	0.3027	0.8940
w/ Surgery V2	0.6566	0.1434	0.2500	0.3160	0.1149	0.2962	0.8865
w/ FeatCal	0.6454	0.1404	0.2439	0.3040	0.1041	0.2876	0.8548

Appendix

Table 17: Post-merging calibration for LLaMA-2-13B instruction–math merging.

Basis construction	SUN397	Cars	RESISC45	EuroSAT	SVHN	GTSRB	MNIST	DTD	Avg.
Random orthonormal	65.42 $\pm$ 0.27	63.88 $\pm$ 0.55	88.54 $\pm$ 0.57	98.61 $\pm$ 0.23	96.06 $\pm$ 0.10	95.81 $\pm$ 0.22	99.32 $\pm$ 0.05	64.98 $\pm$ 1.02	84.08 $\pm$ 0.15
Fine-tuned weights, top- $r$	66.94 $\pm$ 0.22	65.02 $\pm$ 0.48	89.85 $\pm$ 0.61	99.03 $\pm$ 0.15	96.17 $\pm$ 0.04	96.32 $\pm$ 0.45	99.36 $\pm$ 0.06	65.70 $\pm$ 0.84	84.80 $\pm$ 0.24
Task vector, top- $r$	69.29 $\pm$ 0.15	67.72 $\pm$ 0.67	93.01 $\pm$ 0.45	99.44 $\pm$ 0.11	97.08 $\pm$ 0.04	97.47 $\pm$ 0.28	99.53 $\pm$ 0.04	72.53 $\pm$ 0.39	87.01 $\pm$ 0.09
Merging error, bottom- $r$	62.11 $\pm$ 0.21	62.17 $\pm$ 0.30	82.26 $\pm$ 0.36	97.20 $\pm$ 0.35	94.02 $\pm$ 0.19	91.05 $\pm$ 0.17	98.83 $\pm$ 0.07	59.21 $\pm$ 0.58	80.86 $\pm$ 0.13
Merging error, top- $r$	70.33 $\pm$ 0.22	69.02 $\pm$ 0.19	93.60 $\pm$ 0.36	99.48 $\pm$ 0.07	97.10 $\pm$ 0.05	97.70 $\pm$ 0.22	99.55 $\pm$ 0.03	73.52 $\pm$ 0.44	87.54 $\pm$ 0.07

Appendix

Table 18: Full results for the error-subspace construction ablation. We report per-task and average accuracy (%; mean

\pm

standard deviation) for the basis-source comparison in Table 3 , together with the bottom-

r

merging-error baseline.

	Training ( $T=8$ tasks)		Inference (one task)
Method	Trainable Parameters	# Trainable	Stored Entries	# Stored	BF16 Storage	Overhead
Shared ViT-B/32 image encoder	–	–	$P_{\mathrm{img}}$	87,849,216	167.56 MiB	Reference
Surgery	$2Td_{\mathrm{out}}r_{s}$	131,072	$2d_{\mathrm{out}}r_{s}$	16,384	0.031 MiB	0.02%
Surgery V2	$T(2Ldr_{s}+2d_{\mathrm{out}}r_{s})$	2,490,368	$2Ldr_{s}+2d_{\mathrm{out}}r_{s}$	311,296	0.594 MiB	0.35%
FeatCal	$0$ (closed form)	0	$0$	0	0.000 MiB	0.00%
ESCal	$TLr_{b}^{2}$	1,572,864	$Lr_{b}^{2}+2Ldr_{b}$	2,555,904	4.875 MiB	2.91%

Appendix

Table 19: Trainable parameters across eight tasks and additional storage for single-task inference on ViT-B/32.

	Base construction	Calibration
Base merging method	Time	Surgery	Surgery V2	FeatCal	ESCal
Task Arithmetic	0.09	6.28	13.41	0.63	13.22
TIES-Merging	0.39	5.48	12.05	0.58	12.92
WUDI-Merging	5.61	5.53	12.18	0.53	13.05
AdaMerging (tw)	39.46	6.87	13.69	0.65	14.06
AdaMerging (lw)	40.04	6.96	13.69	0.89	14.05

Appendix

Table 20: Base-model construction and calibration times in minutes on ViT-B/32 with eight tasks and 64 calibration examples per task (single run on an NVIDIA RTX A6000). Calibration times exclude base-model construction; gradient-based calibration uses 200 epochs. The task-wise and layer-wise variants of AdaMerging are denoted by tw and lw, respectively.

	Batch size 1			Batch size 64
Calibration	Image latency (ms)	$\Delta t$ (ms)	Overhead (%)	Batch latency (ms)	Throughput (img/s)	Throughput reduction (%)
None	5.51 $\pm$ 0.02	Ref.	Ref.	14.83 $\pm$ 0.05	4316.5 $\pm$ 15.6	Ref.
Surgery	5.60 $\pm$ 0.03	+0.095	+1.72 [+1.51, +1.95]	14.81 $\pm$ 0.07	4320.1 $\pm$ 20.0	+0.15 [+0.06, +0.25]
Surgery V2	7.27 $\pm$ 0.02	+1.758	+31.91 [+31.60, +32.27]	15.63 $\pm$ 0.04	4095.0 $\pm$ 11.6	+5.04 [+4.99, +5.08]
FeatCal	5.50 $\pm$ 0.02	+0.004	+0.07 [-0.30, +0.48]	14.84 $\pm$ 0.03	4311.4 $\pm$ 8.3	+0.01 [-0.01, +0.04]
ESCal	7.09 $\pm$ 0.02	+1.575	+28.53 [+28.23, +28.84]	15.66 $\pm$ 0.04	4086.9 $\pm$ 10.3	+5.29 [+5.22, +5.35]

Appendix

Table 21: Inference cost relative to the uncalibrated Task Arithmetic ViT-B/32 model. None denotes the model without calibration. Absolute values are mean

\pm

standard deviation.

\Delta t

denotes additional per-image latency. Relative changes are computed against paired uncalibrated runs and reported with 95% bootstrap confidence intervals; they may therefore differ from ratios based on the pooled None row.

Support / task	SUN397	Cars	RESISC45	EuroSAT	SVHN	GTSRB	MNIST	DTD	Avg.
4	66.64 $\pm$ 0.21	63.17 $\pm$ 1.20	85.49 $\pm$ 1.08	91.32 $\pm$ 3.07	93.87 $\pm$ 0.33	92.01 $\pm$ 1.37	97.47 $\pm$ 0.49	60.33 $\pm$ 1.44	81.29 $\pm$ 0.36
8	67.83 $\pm$ 0.42	65.05 $\pm$ 1.21	88.01 $\pm$ 0.91	94.77 $\pm$ 3.28	95.47 $\pm$ 0.33	93.60 $\pm$ 0.75	98.27 $\pm$ 0.64	63.48 $\pm$ 1.61	83.31 $\pm$ 0.49
16	68.71 $\pm$ 0.27	66.50 $\pm$ 0.84	90.03 $\pm$ 0.56	96.72 $\pm$ 0.91	96.02 $\pm$ 0.51	94.87 $\pm$ 0.69	98.79 $\pm$ 0.34	65.49 $\pm$ 0.86	84.64 $\pm$ 0.34
32	69.68 $\pm$ 0.14	67.77 $\pm$ 0.41	92.35 $\pm$ 0.28	99.09 $\pm$ 0.23	96.79 $\pm$ 0.11	96.90 $\pm$ 0.63	99.38 $\pm$ 0.07	69.89 $\pm$ 1.42	86.48 $\pm$ 0.21
64	70.33 $\pm$ 0.22	69.02 $\pm$ 0.19	93.60 $\pm$ 0.36	99.48 $\pm$ 0.07	97.10 $\pm$ 0.05	97.70 $\pm$ 0.22	99.55 $\pm$ 0.03	73.52 $\pm$ 0.44	87.54 $\pm$ 0.07
128	70.66 $\pm$ 0.08	69.70 $\pm$ 0.24	94.09 $\pm$ 0.19	99.57 $\pm$ 0.04	97.22 $\pm$ 0.04	98.18 $\pm$ 0.18	99.64 $\pm$ 0.02	75.46 $\pm$ 0.21	88.07 $\pm$ 0.06

Appendix

Table 22: Effect of the per-task support size on ESCal , ViT-B/32 with a task-arithmetic base. Epochs are held at 200, so the optimizer’s update count scales with the support size. Accuracy is on the full test set; the default setting (

64

) is set in bold.

Model Merging

Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.

See all 193 papers

FeatCal: Feature Calibration for Post-Merging Models
May 13, 2026Yanggan Gu, Shuo Cai, Zihao Wang +7Distribution Shift Model Calibration
The Hong Kong Polytechnic University (PolyU) · The Chinese University of Hong Kong · PolyU-Daya Bay Technology and Innovation Research Institute
Model Merging by Output-Space Projection
May 27, 2026Bethan Evans, Benjamin Etheridge, Stephen Roberts +1Quadratic Programming Language Model Merging
Department of Mathematics, University of Oxford, Oxford, UK · Department of Engineering Science, University of Oxford, Oxford, UK.
Model Merging in the Essential Subspace
Feb 23, 2026Longhua Li, Lei Qi, Qi Tian +1Task Interference Model Merging
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shanghai, China

May 13, 2026J/K move · Enter open · S save

FeatCal: Feature Calibration for Post-Merging Models

Yanggan Gu, Shuo Cai, Zihao Wang, Wenjun Wang+6

The Hong Kong Polytechnic University (PolyU) · The Chinese University of Hong Kong · PolyU-Daya Bay Technology and Innovation Research Institute

Distribution Shift Model Calibration Model Merging

Open paper AskPDF arXiv

Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task experts. We study this performance gap through feature drift, the difference between features produced by the merged model and by the expert on the same input. Our theory decomposes this drift into upstream propagation and local mismatch, tracks how it propagates and combines through later layers in forward order, and links final feature drift to output drift. This view motivates FeatCal, which uses a small calibration set to calibrate the merged model weights layer by layer in forward order, reducing feature drift while staying close to merged weights and preserving the benefits of model merging. FeatCal uses an efficient closed-form solution to update model weights, with no gradient descent, iterative optimization, or extra modules. On the main CLIP and GLUE benchmarks, FeatCal beats Surgery and ProbSurgery, the closest post-merging calibration baselines: 85.5% vs. 77.0%/78.8% on CLIP-ViT-B/32 Task Arithmetic (TA) and 85.2% vs. 83.7%/82.2% on FLAN-T5-base GLUE. On CLIP-ViT-B/32, 8 examples per task reach 82.9%, and 256 examples per task take 53 seconds, about 4x faster than both baselines, showing better sample efficiency and lower calibration cost.

Abstract

Model Merging

Explore similar work

FeatCal: Feature Calibration for Post-Merging Models

Model Merging by Output-Space Projection

Model Merging in the Essential Subspace