MergeHEIR: Mitigating Multimodal Hallucinations as the Tax of Model Merging

Jinyu Li; Hao Fang; Zhiming Zhang; Jiawei Kong; Bin Chen; Shu-Tao Xia

cs.AISep 26, 2026

MergeHEIR: Mitigating Multimodal Hallucinations as the Tax of Model Merging

Authors: Jinyu Li, Hao Fang, Zhiming Zhang, Jiawei Kong, Bin Chen, Shu-Tao Xia

Organizations: Harbin Institute of Technology, Shenzhen · Tsinghua Shenzhen International Graduate School, Tsinghua University

Abstract

Model merging consolidates task-specialized experts into a single deployable model. However, we show that such capability consolidation incurs a merging tax of increased hallucination: across 8 model-merging methods, every merged checkpoint exhibits a higher hallucination rate than the average of its constituent experts. An intuitive approach is to adapt existing hallucination-mitigation methods to the post-merge model, yet this unconstrained adaptation disrupts inherited capabilities, creating a tension between hallucination mitigation and expertise retention. To tackle this challenge, we introduce MergeHEIR, a post-merge adaptation framework designed to reduce this merging tax while preserving expertise inherited from initial experts. Using small expert-task calibration sets, MergeHEIR constructs layer-wise null-space projectors via SVD from task-specific activations collected from the merged checkpoint, and periodically projects the accumulated post-merge displacement onto the resulting null spaces to preserve inherited expertise. Theoretically, we establish minimum-distortion and maximum-dimensionality guarantees, characterize the threshold-controlled adaptation-retention trade-off, and extend perturbation guarantees beyond finite calibration data. Across 24 paired comparisons spanning three MLLM configurations and 8 model-merging methods, MergeHEIR consistently mitigates hallucination while largely preserving inherited expertise, demonstrating a more favorable hallucination-retention trade-off.

Figures & tables

Figure 1: Merging tax manifested as elevated hallucination rates. We evaluate eight merging methods with AMBER ( Wang et al., 2023 ) on Qwen2.5-VL-3B and Qwen2.5-VL-7B. Dashed lines indicate the corresponding expert-average baselines, with HalRates of 33.5 and 20.8, respectively. Every evaluated merged checkpoint lies above its corresponding baseline.

Figure 2: Matched comparison of post-merge adaptation outcomes across three model configurations. HalRate denotes the AMBER hallucination rate, while Expert Avg. is the mean performance across the seven expert-task benchmarks. Unconstrained adaptation achieves comparable hallucination mitigation but incurs a substantially larger reduction in Expert Avg., whereas MergeHEIR better preserves inherited expertise while achieving effective hallucination mitigation.

Figure 3: Overview of MergeHEIR. Four task-specialized experts are first consolidated into a merged checkpoint $\theta_{m}$ . Hallucination-oriented preference optimization produces corrective updates, while expert-task calibration activations define layer-wise null-space projectors. MergeHEIR periodically projects the accumulated parameter displacement onto the corresponding null spaces, reducing hallucination while retaining inherited expertise.

Table 1: Expert-task results on Qwen2.5-VL models.

Table 2: Hallucination-task results on Qwen2.5-VL models.

Figure 4: Constraint ablation on Qwen2.5-VL-7B and InternVL3.5-8B. MergeHEIR better preserves inherited expertise while achieving hallucination mitigation comparable to unconstrained adaptation.

Figure 5: Sensitivity to the null-space threshold

\alpha

across three model configurations. The upper panels report Object HalBench CHAIR s and CHAIR i , while the lower panels report Expert Avg.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Task Category	Size	Training Datasets
VQA	589K	GQA ( Hudson and Manning, 2019 ) , CogVLM-Multiround ( Wang et al., 2024c ) , CogVLM-Singleround ( Wang et al., 2024c )
Geometry	177K	GeoQA+ ( Cao and Xiao, 2022 ) , G-LLaVA ( Gao et al., 2025 )
Chart	219K	ChartQA ( Masry et al., 2022 ) , DVQA ( Kafle et al., 2018 )
OCR	239K	OCRVQA ( Mishra et al., 2019 ) , SynthDoG ( Kim et al., 2022 ) , DocVQA ( Mathew et al., 2021 ) , TextVQA ( Singh et al., 2019 ) , LLaVAR ( Zhang et al., 2023 ) , ST-VQA ( Biten et al., 2019 )

Appendix

Table 3: Summary of the training datasets used to construct the four task-specialized experts.

Merging method	Stage	Chart	Geometry		OCR		VQA		Avg.
		ChartQA	MATH-Vision	Geometry3K	TextVQA	DocVQA	GQA	OKVQA	Avg.
Task Arithmetic	Merged	76.20	23.68	60.23	63.83	56.46	59.96	42.17	54.65
	MergeHEIR	76.08	23.03	60.57	63.62	56.26	60.84	39.61	54.29
TIES Merging	Merged	76.52	24.67	60.07	64.21	56.85	60.93	43.87	55.30
	MergeHEIR	76.76	23.03	59.73	64.50	57.29	61.78	42.71	55.11
DELLA Merging	Merged	77.08	24.67	60.07	64.09	56.63	60.57	43.39	55.21

Appendix

Table 4: Expert-task results on InternVL3.5-8B.

Merging method	Stage	AMBER			Object HalBench		POPE Adversarial		GPT-4 Eval
		CHAIR $\downarrow$	HalRate $\downarrow$	Cog $\downarrow$	CHAIR ${}_{s}\downarrow$	CHAIR ${}_{i}\downarrow$	Acc. $\uparrow$	F1 $\uparrow$	SHR $\downarrow$
Task Arithmetic	Merged	8.3	66.5	7.3	25.3	6.7	73.9	78.6	36.2
	MergeHEIR	7.6	63.1	6.3	24.3	6.7	77.5	80.6	32.8
TIES Merging	Merged	8.1	68.7	7.1	24.0	6.2	76.5	80.0	36.3
	MergeHEIR	7.2	62.3	6.1	20.3	5.3	80.6	82.5	33.6
DELLA Merging	Merged	8.1	68.7	7.3	23.7	6.2	75.3	79.4	36.9

Appendix

Table 5: Hallucination-task results on InternVL3.5-8B.

Merging method	Stage	POPE Adversarial		POPE Popular		POPE Random
		Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$
Task Arithmetic	Merged	82.17	83.47	85.83	86.41	91.96	92.03
	MergeHEIR	85.80	86.05	88.63	88.51	92.23	92.08
TIES Merging	Merged	82.17	83.46	85.77	86.34	91.89	91.96
	MergeHEIR	85.80	86.06	88.70	88.58	92.20	92.05
DELLA Merging	Merged	82.07	83.37	85.97	86.50	91.89	91.96

Appendix

Table 6: Complete POPE results for Qwen2.5-VL-3B.

Merging method	Stage	POPE Adversarial		POPE Popular		POPE Random
		Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$
Task Arithmetic	Merged	83.37	80.70	83.87	81.17	84.09	81.84
	MergeHEIR	86.97	86.00	88.20	87.09	88.93	88.14
TIES Merging	Merged	85.53	84.41	87.27	86.02	87.84	86.91
	MergeHEIR	89.17	88.05	91.07	90.19	92.34	92.06
DELLA Merging	Merged	84.00	81.85	84.67	82.47	85.26	83.46

Appendix

Table 7: Complete POPE results for Qwen2.5-VL-7B.

Merging method	Stage	POPE Adversarial		POPE Popular		POPE Random
		Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$
Task Arithmetic	Merged	73.93	78.61	78.43	81.62	92.20	92.68
	MergeHEIR	77.50	80.64	82.20	84.04	93.40	93.61
TIES Merging	Merged	76.53	80.02	81.50	83.56	92.85	93.13
	MergeHEIR	80.63	82.52	85.73	86.35	94.60	94.48
DELLA Merging	Merged	75.33	79.36	79.73	82.40	92.41	92.79

Appendix

Table 8: Complete POPE results for InternVL3.5-8B.

Model	Setting	Expert Avg. $\uparrow$	AMBER			Object HalBench
Model	Setting		CHAIR $\downarrow$	HalRate $\downarrow$	Cog $\downarrow$	CHAIR ${}_{s}\downarrow$	CHAIR ${}_{i}\downarrow$
Qwen2.5-VL-3B	Merged	60.83	8.8	59.6	7.1	36.0	9.0
	RLAIF-V	60.28	5.3	32.2	3.5	27.7	7.9
	RLAIF-V + HEIR	60.86	5.5	30.8	3.6	28.0	7.7
	OPA-DPO	60.15	4.5	21.2	1.8	21.3	6.1
	OPA-DPO + HEIR	60.90	4.6	21.5	2.0	21.7	6.2

Appendix

Table 9: Compatibility of MergeHEIR with alternative hallucination-oriented preference methods. “+ HEIR” denotes augmenting the corresponding method with the null-space projection mechanism of MergeHEIR.

Figure 6: Visualization results comparing the merged model, MergeHEIR, and unconstrained post-merge adaptation on chart understanding task. Incorrect or visually unsupported response spans are highlighted in red .

Figure 7: Visualization results comparing the merged model, MergeHEIR, and unconstrained post-merge adaptation on geometry reasoning task. Incorrect or visually unsupported response spans are highlighted in red .

Figure 8: Visualization results comparing the merged model, MergeHEIR, and unconstrained post-merge adaptation on OCR task. Incorrect or visually unsupported response spans are highlighted in red .

Figure 9: Visualization results comparing the merged model, MergeHEIR, and unconstrained post-merge adaptation on VQA task. Incorrect or visually unsupported response spans are highlighted in red .

Figure 10: Visualization results comparing the merged model, MergeHEIR, and unconstrained post-merge adaptation on POPE. Hallucinated or visually unsupported content is highlighted in red .

Figure 11: Visualization results comparing the merged model, MergeHEIR, and unconstrained post-merge adaptation on ObjHalBench. Hallucinated objects and visually unsupported scene details are highlighted in red .

Model Merging

Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.

See all 193 papers

Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs
Jul 13, 2026Nikita Kozodoi, Zainab Afolabi, Jack ButlerLLM Fine-Tuning Language Model Merging
1Amazon Web Services
Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion
Sep 19, 2026Aditya Pola, Vineeth N. BalasubramanianModel Merging
IIT Hyderabad · Microsoft Research, India
Model Merging by Output-Space Projection
May 27, 2026Bethan Evans, Benjamin Etheridge, Stephen Roberts +1Quadratic Programming Language Model Merging
Department of Mathematics, University of Oxford, Oxford, UK · Department of Engineering Science, University of Oxford, Oxford, UK.

Jul 13, 2026J/K move · Enter open · S save

Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

Nikita Kozodoi, Zainab Afolabi, Jack Butler

1Amazon Web Services

LLM Fine-Tuning Language Model Merging Model Merging

Open paper AskPDF arXiv

Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training. Standard practice merges experts at their optimal validation loss. We challenge this convention by systematically studying how training duration of domain experts affects the quality of the merged model. We fine-tune experts on five domains (Math, Code, Instruction Following, Multilingual, and Safety) across three model sizes (Qwen 3.5 0.8B, 2B, and 4B), saving checkpoints from 25% to 500% of the optimal training steps and evaluating five merging methods at each duration. Our findings reveal a striking method-dependent pattern: simple averaging degrades sharply with overfitting, while sparsification-based methods achieve their best performance well past the validation optimum. We formalize this through bias-variance decomposition analysis, drawing a parallel to random forests where averaging benefits from high-variance individual learners. These results suggest that training duration and merging method should be chosen jointly rather than independently.

Abstract

Model Merging

Explore similar work

Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion

Model Merging by Output-Space Projection