Model merging consolidates task-specialized experts into a single deployable model. However, we show that such capability consolidation incurs a merging tax of increased hallucination: across 8 model-merging methods, every merged checkpoint exhibits a higher hallucination rate than the average of its constituent experts. An intuitive approach is to adapt existing hallucination-mitigation methods to the post-merge model, yet this unconstrained adaptation disrupts inherited capabilities, creating a tension between hallucination mitigation and expertise retention. To tackle this challenge, we introduce MergeHEIR, a post-merge adaptation framework designed to reduce this merging tax while preserving expertise inherited from initial experts. Using small expert-task calibration sets, MergeHEIR constructs layer-wise null-space projectors via SVD from task-specific activations collected from the merged checkpoint, and periodically projects the accumulated post-merge displacement onto the resulting null spaces to preserve inherited expertise. Theoretically, we establish minimum-distortion and maximum-dimensionality guarantees, characterize the threshold-controlled adaptation-retention trade-off, and extend perturbation guarantees beyond finite calibration data. Across 24 paired comparisons spanning three MLLM configurations and 8 model-merging methods, MergeHEIR consistently mitigates hallucination while largely preserving inherited expertise, demonstrating a more favorable hallucination-retention trade-off.
Figures & tables
Figure 1: Merging tax manifested as elevated hallucination rates. We evaluate eight merging methods with AMBER ( Wang et al., 2023 ) on Qwen2.5-VL-3B and Qwen2.5-VL-7B. Dashed lines indicate the corresponding expert-average baselines, with HalRates of 33.5 and 20.8, respectively. Every evaluated merged checkpoint lies above its corresponding baseline.
Figure 2: Matched comparison of post-merge adaptation outcomes across three model configurations. HalRate denotes the AMBER hallucination rate, while Expert Avg. is the mean performance across the seven expert-task benchmarks. Unconstrained adaptation achieves comparable hallucination mitigation but incurs a substantially larger reduction in Expert Avg., whereas MergeHEIR better preserves inherited expertise while achieving effective hallucination mitigation.
Figure 3: Overview of MergeHEIR. Four task-specialized experts are first consolidated into a merged checkpoint θm . Hallucination-oriented preference optimization produces corrective updates, while expert-task calibration activations define layer-wise null-space projectors. MergeHEIR periodically projects the accumulated parameter displacement onto the corresponding null spaces, reducing hallucination while retaining inherited expertise.
Table 1: Expert-task results on Qwen2.5-VL models.
Table 2: Hallucination-task results on Qwen2.5-VL models.
Figure 4: Constraint ablation on Qwen2.5-VL-7B and InternVL3.5-8B. MergeHEIR better preserves inherited expertise while achieving hallucination mitigation comparable to unconstrained adaptation.
Figure 5: Sensitivity to the null-space threshold α across three model configurations. The upper panels report Object HalBench CHAIR s and CHAIR i , while the lower panels report Expert Avg.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Task Category
Size
Training Datasets
VQA
589K
GQA ( Hudson and Manning, 2019 ) , CogVLM-Multiround ( Wang et al., 2024c ) , CogVLM-Singleround ( Wang et al., 2024c )
Geometry
177K
GeoQA+ ( Cao and Xiao, 2022 ) , G-LLaVA ( Gao et al., 2025 )
Chart
219K
ChartQA ( Masry et al., 2022 ) , DVQA ( Kafle et al., 2018 )
OCR
239K
OCRVQA ( Mishra et al., 2019 ) , SynthDoG ( Kim et al., 2022 ) , DocVQA ( Mathew et al., 2021 ) , TextVQA ( Singh et al., 2019 ) , LLaVAR ( Zhang et al., 2023 ) , ST-VQA ( Biten et al., 2019 )
Appendix
Table 3: Summary of the training datasets used to construct the four task-specialized experts.
Merging method
Stage
Chart
Geometry
OCR
VQA
Avg.
ChartQA
MATH-Vision
Geometry3K
TextVQA
DocVQA
GQA
OKVQA
Task Arithmetic
Merged
76.20
23.68
60.23
63.83
56.46
59.96
42.17
54.65
MergeHEIR
76.08
23.03
60.57
63.62
56.26
60.84
39.61
54.29
TIES Merging
Merged
76.52
24.67
60.07
64.21
56.85
60.93
43.87
55.30
MergeHEIR
76.76
23.03
59.73
64.50
57.29
61.78
42.71
55.11
DELLA Merging
Merged
77.08
24.67
60.07
64.09
56.63
60.57
43.39
55.21
Appendix
Table 4: Expert-task results on InternVL3.5-8B.
Merging method
Stage
AMBER
Object HalBench
POPE Adversarial
GPT-4 Eval
CHAIR ↓
HalRate ↓
Cog ↓
CHAIR s↓
CHAIR i↓
Acc. ↑
F1 ↑
SHR ↓
Task Arithmetic
Merged
8.3
66.5
7.3
25.3
6.7
73.9
78.6
36.2
MergeHEIR
7.6
63.1
6.3
24.3
6.7
77.5
80.6
32.8
TIES Merging
Merged
8.1
68.7
7.1
24.0
6.2
76.5
80.0
36.3
MergeHEIR
7.2
62.3
6.1
20.3
5.3
80.6
82.5
33.6
DELLA Merging
Merged
8.1
68.7
7.3
23.7
6.2
75.3
79.4
36.9
Appendix
Table 5: Hallucination-task results on InternVL3.5-8B.
Merging method
Stage
POPE Adversarial
POPE Popular
POPE Random
Acc. ↑
F1 ↑
Acc. ↑
F1 ↑
Acc. ↑
F1 ↑
Task Arithmetic
Merged
82.17
83.47
85.83
86.41
91.96
92.03
MergeHEIR
85.80
86.05
88.63
88.51
92.23
92.08
TIES Merging
Merged
82.17
83.46
85.77
86.34
91.89
91.96
MergeHEIR
85.80
86.06
88.70
88.58
92.20
92.05
DELLA Merging
Merged
82.07
83.37
85.97
86.50
91.89
91.96
Appendix
Table 6: Complete POPE results for Qwen2.5-VL-3B.
Merging method
Stage
POPE Adversarial
POPE Popular
POPE Random
Acc. ↑
F1 ↑
Acc. ↑
F1 ↑
Acc. ↑
F1 ↑
Task Arithmetic
Merged
83.37
80.70
83.87
81.17
84.09
81.84
MergeHEIR
86.97
86.00
88.20
87.09
88.93
88.14
TIES Merging
Merged
85.53
84.41
87.27
86.02
87.84
86.91
MergeHEIR
89.17
88.05
91.07
90.19
92.34
92.06
DELLA Merging
Merged
84.00
81.85
84.67
82.47
85.26
83.46
Appendix
Table 7: Complete POPE results for Qwen2.5-VL-7B.
Merging method
Stage
POPE Adversarial
POPE Popular
POPE Random
Acc. ↑
F1 ↑
Acc. ↑
F1 ↑
Acc. ↑
F1 ↑
Task Arithmetic
Merged
73.93
78.61
78.43
81.62
92.20
92.68
MergeHEIR
77.50
80.64
82.20
84.04
93.40
93.61
TIES Merging
Merged
76.53
80.02
81.50
83.56
92.85
93.13
MergeHEIR
80.63
82.52
85.73
86.35
94.60
94.48
DELLA Merging
Merged
75.33
79.36
79.73
82.40
92.41
92.79
Appendix
Table 8: Complete POPE results for InternVL3.5-8B.
Model
Setting
Expert Avg. ↑
AMBER
Object HalBench
CHAIR ↓
HalRate ↓
Cog ↓
CHAIR s↓
CHAIR i↓
Qwen2.5-VL-3B
Merged
60.83
8.8
59.6
7.1
36.0
9.0
RLAIF-V
60.28
5.3
32.2
3.5
27.7
7.9
RLAIF-V + HEIR
60.86
5.5
30.8
3.6
28.0
7.7
OPA-DPO
60.15
4.5
21.2
1.8
21.3
6.1
OPA-DPO + HEIR
60.90
4.6
21.5
2.0
21.7
6.2
Appendix
Table 9: Compatibility of MergeHEIR with alternative hallucination-oriented preference methods. “+ HEIR” denotes augmenting the corresponding method with the null-space projection mechanism of MergeHEIR.
Figure 6: Visualization results comparing the merged model, MergeHEIR, and unconstrained post-merge adaptation on chart understanding task. Incorrect or visually unsupported response spans are highlighted in red .
Figure 7: Visualization results comparing the merged model, MergeHEIR, and unconstrained post-merge adaptation on geometry reasoning task. Incorrect or visually unsupported response spans are highlighted in red .
Figure 8: Visualization results comparing the merged model, MergeHEIR, and unconstrained post-merge adaptation on OCR task. Incorrect or visually unsupported response spans are highlighted in red .
Figure 9: Visualization results comparing the merged model, MergeHEIR, and unconstrained post-merge adaptation on VQA task. Incorrect or visually unsupported response spans are highlighted in red .
Figure 10: Visualization results comparing the merged model, MergeHEIR, and unconstrained post-merge adaptation on POPE. Hallucinated or visually unsupported content is highlighted in red .
Figure 11: Visualization results comparing the merged model, MergeHEIR, and unconstrained post-merge adaptation on ObjHalBench. Hallucinated objects and visually unsupported scene details are highlighted in red .
Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training. Standard practice merges experts at their optimal validation loss. We challenge this convention by systematically studying how training duration of domain experts affects the quality of the merged model. We fine-tune experts on five domains (Math, Code, Instruction Following, Multilingual, and Safety) across three model sizes (Qwen 3.5 0.8B, 2B, and 4B), saving checkpoints from 25% to 500% of the optimal training steps and evaluating five merging methods at each duration. Our findings reveal a striking method-dependent pattern: simple averaging degrades sharply with overfitting, while sparsification-based methods achieve their best performance well past the validation optimum. We formalize this through bias-variance decomposition analysis, drawing a parallel to random forests where averaging benefits from high-variance individual learners. These results suggest that training duration and merging method should be chosen jointly rather than independently.
Model merging consolidates fine-tuned experts into one multi-task model without retraining. All existing data-free methods approach this problem entirely in weight space. Restricted to arithmetic on parameters, these methods never observe how each expert behaves, a signal that only emerges through forward evaluation. Accessing this behavioral signal requires inputs to evaluate on, which the data-free setting prohibits. We propose Merge++, a post-hoc method that addresses this by inverting the expert checkpoints to synthesize task-representative images, then distilling expert knowledge into the merged model using those images. Merge++ requires no additional data beyond the checkpoints themselves. It applies universally across merging algorithms and operates as a complementary refinement stage independent of the underlying weight-space method. The method consistently improves merging algorithms ranging from simple task arithmetic to state-of-the-art spectral methods, with average gains of +2 to +8 points and up to +25.9 on individual configurations.
Model merging combines fine-tuned checkpoints into a single multi-task model without retraining. Existing methods - such as task arithmetic, model soups, TIES, and DARE - are computationally efficient and empirically successful, but rely on heuristic design choices and lack formal optimality guarantees. We show that merging can be formulated as a convex quadratic programme over residual updates, yielding weights that minimise a squared-output calibration objective using calibration inputs and fine-tuned model outputs, and subsuming existing methods as special cases. Our framework yields a closed-form diagnostic - the fraction of residual energy captured by a chosen basis - that predicts downstream merge quality using only the calibration set. Empirically, the QP matches or outperforms existing methods in the single-layer setting, and we characterise when the optimal basis provides significant gains over the cheaper diagonal QP. We extend to multi-layer merging via a sequential layer-wise algorithm and demonstrate consistent gains across language and vision benchmarks.
Bethan Evans, Benjamin Etheridge, Stephen Roberts +1
Department of Mathematics, University of Oxford, Oxford, UK · Department of Engineering Science, University of Oxford, Oxford, UK.