Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation

Xingyu Zhu; Kesen Zhao; Liang Yi; Shuo Wang; Zhicai Wang; Beier Zhu; Hanwang Zhang

cs.CVFeb 27, 2026

Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation

Authors: Xingyu Zhu, Kesen Zhao, Liang Yi, Shuo Wang, Zhicai Wang, Beier Zhu, Hanwang Zhang

Organizations: MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University

Abstract

Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning, yet they remain vulnerable to hallucination, where generated content deviates from visual evidence. Existing mitigation strategies either require costly supervision during training or introduce additional latency at inference time. Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding, but they typically inject all tokens indiscriminately, which causes interference from background regions and distracts the model from critical cues. To overcome this challenge, we propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs. AIR consists of two components. Prototype-based token reduction condenses the large pool of visual tokens into a compact subset to suppress redundancy. OT-guided patch reinforcement quantifies the alignment between hidden states and patch embeddings to selectively integrate the most consistent patches into feed-forward layers. As a result, AIR enhances the model's reliance on salient visual information and effectively mitigates hallucination. Extensive experiments across representative MLLMs demonstrate that AIR substantially reduces hallucination while preserving general capabilities, establishing it as an effective solution for building reliable MLLMs.

Figures & tables

Figure 1: Analysis of existing hallucination mitigation strategies in multimodal large language models (MLLMs). (a) existing strategies hallucinate non-existent objects, while AIR produces faithful, image-grounded answers. (b) Similarity across decoder layers between hidden states and different visual tokens, showing that salient patches consistently achieve higher alignment than irrelevant ones. (c) Attention heatmaps comparing existing re-injection with AIR: prior methods spread attention to irrelevant regions, whereas AIR focuses on semantically critical areas. (d) AIR reduces hallucination across benchmarks with minimal impact on general multimodal performance.

Figure 2: Overview of our proposed AIR framework. Given a multimodal input (image and question), visual features are extracted by the visual encoder and aligned with textual embeddings via the projector. Inside the Transformer layers, visual tokens are first compressed through prototype-based selection to remove redundancy, and then reinforced by patch-level alignment using optimal transport. These selective reinforcement strategies enrich the hidden states with salient visual cues, enabling AIR to produce safer, more faithful, and visually grounded responses.

Method	LLaVA-1.5-7B			Qwen-VL-Chat			GLM-4V-9B
Method	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	BLEU $\uparrow$	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	BLEU $\uparrow$	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	BLEU $\uparrow$
Vanilla	22.0 $\uparrow$ 0.0	6.7 $\uparrow$ 0.0	14.5 $\uparrow$ 0.0	20.0 $\uparrow$ 0.0	6.2 $\uparrow$ 0.0	13.5 $\uparrow$ 0.0	13.0 $\uparrow$ 0.0	5.6 $\uparrow$ 0.0	9.8 $\uparrow$ 0.0
VCD	24.6 $\uparrow$ 2.6	7.3 $\uparrow$ 0.6	13.9 $\downarrow$ 0.6	19.2 $\downarrow$ 0.8	5.7 $\downarrow$ 0.5	13.4 $\uparrow$ 0.1	14.8 $\uparrow$ 1.8	6.5 $\uparrow$ 0.9	9.5 $\downarrow$ 0.3
MemVR	21.6 $\downarrow$ 0.4	6.4 $\downarrow$ 0.3	14.4 $\downarrow$ 0.1	20.0 $\uparrow$ 0.0	6.1 $\downarrow$ 0.1	13.3 $\downarrow$ 0.2	13.0 $\uparrow$ 0.0	5.6 $\uparrow$ 0.0	9.8 $\uparrow$ 0.0
VAF	20.4 $\downarrow$ 1.6	6.5 $\downarrow$ 0.2	14.6 $\uparrow$ 0.1	20.6 $\uparrow$ 0.6	6.6 $\uparrow$ 0.4	13.4 $\downarrow$ 0.1	11.6 $\downarrow$ 1.4	5.3 $\downarrow$ 0.3	9.7 $\downarrow$ 0.1
AIR	18.4 $\downarrow$ 3.6	5.7 $\downarrow$ 1.0	14.4 $\downarrow$ 0.1	18.6 $\downarrow$ 1.4	5.9 $\downarrow$ 0.3	13.6 $\uparrow$ 0.1	11.6 $\downarrow$ 1.4	5.3 $\downarrow$ 0.3	9.7 $\downarrow$ 0.1

Table 1: CHAIR evaluation results on MSCOCO dataset of MLLMs with different methods. We use 64 as the maximum token in this experiment. Bold indicates the best result of all methods.

Datasets	Methods	Random		Popular		Adversarial
Datasets	Methods	Accuracy $\uparrow$	F1-score $\uparrow$	Accuracy $\uparrow$	F1-score $\uparrow$	Accuracy $\uparrow$	F1-score $\uparrow$
MSCOCO	Vanilla	83.7 $\uparrow$ 0.0	83.0 $\uparrow$ 0.0	78.2 $\uparrow$ 0.0	78.4 $\uparrow$ 0.0	75.0 $\uparrow$ 0.0	76.0 $\uparrow$ 0.0
	VCD	85.4 $\uparrow$ 1.7	83.7 $\uparrow$ 0.7	84.3 $\uparrow$ 6.1	83.0 $\uparrow$ 4.6	81.8 $\uparrow$ 6.8	80.9 $\uparrow$ 4.9
	MemVR	87.6 $\uparrow$ 3.9	86.2 $\uparrow$ 3.2	86.0 $\uparrow$ 7.8	84.7 $\uparrow$ 6.3	83.5 $\uparrow$ 8.5	82.5 $\uparrow$ 6.5
	VAF	87.6 $\uparrow$ 3.9	86.2 $\uparrow$ 3.2	86.2 $\uparrow$ 8.0	85.0 $\uparrow$ 6.6	83.9 $\uparrow$ 8.9	82.8 $\uparrow$ 6.8
	AIR	89.0 $\uparrow$ 5.3	88.2 $\uparrow$ 5.2	87.1 $\uparrow$ 8.9	86.4 $\uparrow$ 8.0	83.9 $\uparrow$ 8.9	83.6 $\uparrow$ 7.6

Table 2: Performance on POPE benchmark using LLaVA-1.5-7B. Bold numbers indicate the best results. We report accuracy and F1-score under three settings, i.e. , Random , Popular , and Adversarial , to show the robustness of different methods.

	Methods	MME-Hall	Object-Level		Attribute-Level		Cognition	MMBench
	Methods	Total $\uparrow$	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	Score $\uparrow$	Accuracy $\uparrow$
LLaVA-1.5	Vanilla	643.3 $\uparrow$ 0.0	190.0 $\uparrow$ 0.0	155.0 $\uparrow$ 0.0	128.3 $\uparrow$ 0.0	170.0 $\uparrow$ 0.0	357.8 $\uparrow$ 0.0	64.6 $\uparrow$ 0.0
	VCD	613.3 $\downarrow$ 30.	190.0 $\uparrow$ 0.0	140.0 $\downarrow$ 15.	118.3 $\downarrow$ 10.	165.0 $\downarrow$ 5.0	337.1 $\downarrow$ 20.	61.6 $\downarrow$ 3.0
	MemVR	648.3 $\uparrow$ 5.0	190.0 $\uparrow$ 0.0	155.0 $\uparrow$ 0.0	133.3 $\uparrow$ 5.0	170.0 $\uparrow$ 0.0	378.6 $\uparrow$ 20.	64.6 $\uparrow$ 0.0
	VAF	603.3 $\downarrow$ 40.	190.0 $\uparrow$ 0.0	135.0 $\downarrow$ 20.	108.3 $\downarrow$ 20.	170.0 $\uparrow$ 0.0	322.8 $\downarrow$ 35.	14.8 $\downarrow$ 49.
	AIR	638.3 $\downarrow$ 5.0	190.0 $\uparrow$ 0.0	155.0 $\uparrow$ 0.0	123.3 $\downarrow$ 5.0	170.0 $\uparrow$ 0.0	372.5 $\uparrow$ 14.	64.7 $\uparrow$ 0.1

Table 3: Comparison of evaluation results on the MME Hallucination subset and MMBench.

Table 6

Model	Prototype-based	OT-based	CHAIR
Model	Token Reduction	Patch Reinforcement	$\text{CHAIR}_{S}\downarrow$	$\text{CHAIR}_{I}\downarrow$
LLaVA-1.5-7B	✗	✗	22.7	6.7
	✗	✓	22.3	6.5
	✓	✗	20.2	6.2
	✓	✓	19.8	5.8

Table 6: Ablation study of our method on the CHAIR using LLaVA-1.5-7B, where we ablate two alignment components: prototype-based token reduction and OT-based patch reinforcement.

Panel of Figure 6 — Figure 6: Analysis of OT-based versus cosine-based patch selection. (a) OT produces consistently larger margin differentials than cosine. (b) Most patch-level results lie above the $y=x$ line, showing that OT provides clearer separation between patches. (c) Qualitative evidence shows that OT focuses on visually informative regions aligned with the original image.

Model	Avg. Latency $\downarrow$	GPU Memory $\downarrow$	$\text{CHAIR}_{S}$ $\downarrow$	$\text{CHAIR}_{i}$ $\downarrow$
LLaVA-1.5-7B	1.68s	13.5G	22.0	6.7
VAF	1.69s	13.5G	20.4	6.5
MemVR	2.05s	13.6G	21.6	6.4
AIR	2.07s	13.7G	18.4	5.7

Table 7: Comparison of inference speed, GPU memory usage, and hallucination performance on the CHAIR using a single A100 GPU.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Method	LLaVA-1.5-7B				Qwen-VL-Chat
Method	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	BLEU $\uparrow$	Recall $\uparrow$	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	BLEU $\uparrow$	Recall $\uparrow$
Vanilla	6.8 $\uparrow$ 0.0	2.9 $\uparrow$ 0.0	22.9 $\uparrow$ 0.0	52.4 $\uparrow$ 0.0	6.6 $\uparrow$ 0.0	2.9 $\uparrow$ 0.0	21.4 $\uparrow$ 0.0	52.1 $\uparrow$ 0.0
VCD	8.2 $\uparrow$ 1.4	4.0 $\uparrow$ 1.1	21.7 $\downarrow$ 2.2	50.0 $\downarrow$ 2.4	6.6 $\uparrow$ 0.0	2.9 $\uparrow$ 0.0	21.6 $\uparrow$ 0.2	52.0 $\downarrow$ 0.1
MemVR	6.6 $\downarrow$ 0.2	2.9 $\uparrow$ 0.0	22.8 $\downarrow$ 0.1	52.3 $\downarrow$ 0.1	7.0 $\uparrow$ 0.4	3.2 $\uparrow$ 0.3	21.3 $\downarrow$ 0.1	52.3 $\uparrow$ 0.2
VAF	7.0 $\uparrow$ 0.2	3.1 $\uparrow$ 0.2	$23.0$ $\uparrow$ 0.1	52.4 $\uparrow$ 0.0	6.6 $\uparrow$ 0.0	3.0 $\uparrow$ 0.1	21.4 $\uparrow$ 0.0	51.7 $\downarrow$ 0.4
AIR	6.8 $\uparrow$ 0.0	2.9 $\uparrow$ 0.0	23.0 $\uparrow$ 0.1	52.8 $\uparrow$ 0.4	5.8 $\downarrow$ 1.2	2.8 $\downarrow$ 0.1	21.7 $\uparrow$ 0.3	52.0 $\downarrow$ 0.1

Appendix

Table 8: CHAIR evaluation results on the MSCOCO dataset of MLLMs with different methods. We use 32 as the maximum token in this experiment. Bold indicates the best result of all methods.

Datasets	Methods	Random		Popular		Adversarial
Datasets	Methods	Accuracy $\uparrow$	F1-score $\uparrow$	Accuracy $\uparrow$	F1-score $\uparrow$	Accuracy $\uparrow$	F1-score $\uparrow$
MSCOCO	Vanilla	84.5 $\uparrow$ 0.0	81.9 $\uparrow$ 0.0	84.0 $\uparrow$ 0.0	81.4 $\uparrow$ 0.0	83.0 $\uparrow$ 0.0	80.5 $\uparrow$ 0.0
	MemVR	84.7 $\uparrow$ 0.2	82.1 $\uparrow$ 0.2	84.2 $\uparrow$ 0.2	81.6 $\uparrow$ 0.2	83.1 $\uparrow$ 0.1	80.6 $\uparrow$ 0.1
	AIR	84.7 $\uparrow$ 0.2	82.1 $\uparrow$ 0.2	84.2 $\uparrow$ 0.2	81.6 $\uparrow$ 0.2	83.2 $\uparrow$ 0.2	80.6 $\uparrow$ 0.1
A-OKVQA	Vanilla	86.2 $\uparrow$ 0.2	84.5 $\uparrow$ 0.2	86.2 $\uparrow$ 0.2	84.5 $\uparrow$ 0.2	81.2 $\uparrow$ 0.0	80.0 $\uparrow$ 0.0
	MemVR	86.8 $\uparrow$ 0.6	85.3 $\uparrow$ 0.8	86.8 $\uparrow$ 0.6	85.3 $\uparrow$ 0.8	81.6 $\uparrow$ 0.4	80.6 $\uparrow$ 0.6

Appendix

Table 9: Performance on POPE benchmark using Qwen-VL-Chat. Bold numbers indicate the best results. We report accuracy and F1-score under three settings, i.e. , Random , Popular , and Adversarial , to show the robustness of different methods.

Method	MME
Method	Overall $\uparrow$	Perception $\uparrow$	Cognition $\uparrow$
LLaVA-1.5-7B	1863.89	1506.03	357.86
VCD	1822.04 $\downarrow$ 41.85	1484.90 $\downarrow$ 21.13	337.14 $\downarrow$ 20.72
MemVR	1890.95 $\uparrow$ 27.06	1512.38 $\uparrow$ 6.35	378.57 $\uparrow$ 20.71
VAF	1746.55 $\downarrow$ 117.34	1423.69 $\downarrow$ 82.34	322.86 $\downarrow$ 35.00
AIR	1876.67 $\uparrow$ 12.78	1504.17 $\downarrow$ 1.86	372.50 $\uparrow$ 14.64

Appendix

Table 10: Results on the MME dataset. Bold indicates the best result of all methods.

Method	MMBench
Method	AR $\uparrow$	CP $\uparrow$	FP-C $\uparrow$	FP-S $\uparrow$	LR $\uparrow$	RR $\uparrow$	Overall $\uparrow$
LLaVA-1.5-7B	73.37	77.03	57.34	61.92	30.51	53.04	64.60
VCD	68.34 $\downarrow$ 5.03	75.34 $\downarrow$ 1.69	55.24 $\downarrow$ 2.10	62.80 $\uparrow$ 0.88	28.81 $\downarrow$ 1.70	53.04 $\uparrow$ 0.00	61.60 $\downarrow$ 3.00
MemVR	73.37 $\uparrow$ 0.00	77.03 $\uparrow$ 0.00	57.34 $\uparrow$ 0.00	61.92 $\uparrow$ 0.00	30.51 $\uparrow$ 0.00	53.04 $\uparrow$ 0.00	64.60 $\uparrow$ 0.00
VAF	17.08 $\downarrow$ 56.29	21.28 $\downarrow$ 55.75	18.88 $\downarrow$ 38.46	10.92 $\downarrow$ 51.00	5.08 $\downarrow$ 25.43	8.70 $\downarrow$ 44.34	14.78 $\downarrow$ 49.82
AIR	72.36 $\downarrow$ 1.01	77.03 $\uparrow$ 0.00	58.04 $\uparrow$ 0.70	67.92 $\uparrow$ 6.00	31.36 $\uparrow$ 0.85	53.91 $\uparrow$ 0.87	64.69 $\uparrow$ 0.09

Appendix

Table 11: Results on the MMBench dataset. Abbreviations adopted: AR for Attribute Reasoning; CP for Coarse Perception; FP-C for Fine-grained Perception (Cross Instance); FP-S for Fine-grained Perception (Single Instance); LR for Logical Reasoning; RR for Relation Reasoning. Bold indicates the best result of all methods.

Model	fACC $\uparrow$	qACC $\uparrow$	easyA $\uparrow$	hardA $\uparrow$	aACC $\uparrow$
LLaVA-1.5	17.9	8.1	36.0	36.7	41.5
VCD	13.9	11.4	33.00	34.7	41.1
MemVR	17.9	9.0	36.9	37.7	42.5
AIR	19.9	9.3	37.5	38.3	43.2

Appendix

Table 12: Results on HallusionBench.

Model	Attribute	Spatial	Overall
LLaVA-1.5	43.47	56.57	48.68
MemVR	45.38	57.82	49.35
AIR	48.23	59.31	51.26
Qwen-VL-Chat	74.78	68.42	72.25
MemVR	75.31	69.08	73.58
AIR	76.02	69.46	76.23

Appendix

Table 13: Performance on V* Bench.

Model	R $\uparrow$	OCR_S $\uparrow$	OCR_K_R $\uparrow$	OCR_G_S $\uparrow$	Total ↑
LLaVA-1.5	67.6	17.7	21.2	10.0	31.1
VCD	62.2	15.8	17.5	60.0	30.2
MemVR	70.3	23.8	21.2	30.0	32.2
AIR	72.1	24.6	21.5	30.0	34.7

Appendix

Table 15: Results MM-Vet.

Model	Convs $\uparrow$	Detail $\uparrow$	Complex $\uparrow$	All $\uparrow$	Average $\uparrow$
LLaVA-1.5	58.8	52.1	74.6	63.4	64.8
VCD	57.8	50.8	77.9	59.1	63.2
MemVR	63.8	52.6	77.9	64.0	65.2
AIR	65.3	52.7	79.1	64.3	65.8

Appendix

Table 16: Results on LLaVA-Bench (In-the-Wild).

Panel of Figure 10 — Figure 10: Effect of the OT regularization strength $\epsilon$ .

Model	$\text{CHAIR}_{S}\downarrow$	$\text{CHAIR}_{I}\downarrow$	BLEU $\uparrow$
LLaVA-1.5-7B	25.3	11.5	17.3
AIR	22.1	9.5	14.2
Qwen-VL-Chat	23.5	11.9	16.8
AIR	21.4	8.9	13.7

Appendix

Table 17: Results on CHAIR and BLEU under adversarial scenarios.

Model	$\text{CHAIR}_{S}\downarrow$	$\text{CHAIR}_{I}\downarrow$	BLEU $\uparrow$
LLaVA-1.5-7B	22.0	6.7	14.5
random patch	22.3	6.9	13.1
AIR	18.4	5.7	14.4

Appendix

Table 18: Comparison with the random patch baseline.

	$\text{CHAIR}_{S}\downarrow$			$\text{CHAIR}_{I}\downarrow$			Recall $\uparrow$
Model	64	128	256	64	128	256	64	128	256
LLaVA-1.5	22.0	47.5	47.8	6.7	13.1	13.4	66.2	73.1	80.6
MemVR	21.6	46.6	47.2	6.5	13.0	13.2	66.5	73.5	81.0
AIR	18.4	38.1	38.8	5.7	9.3	10.0	66.7	73.9	81.4

Appendix

Table 19: Results on different generation lengths.

Method	Object HallBench		AMBER			HallusionBench
	Resp. $\downarrow$	Ment. $\downarrow$	CHAIR $\downarrow$	Hal. $\downarrow$	Cog. $\downarrow$	Question Acc. $\uparrow$
LLaVA-v1.5-7B	52.7	28.0	8.4	35.5	4.0	46.86
SENTINEL ( Peng et al., 2025 )	4.3	2.6	2.9	14.6	1.2	47.56
+ AIR	3.9	2.2	2.1	13.3	1.1	47.25
LLaVA-v1.5-13B	46.0	23.0	6.9	31.9	3.3	46.43
SENTINEL ( Peng et al., 2025 )	3.3	1.9	2.7	11.7	0.9	46.77

Appendix

Table 20: Comparison with fine-tuned baselines.

Model	$\text{CHAIR}_{S}\downarrow$	$\text{CHAIR}_{I}\downarrow$	$\text{BLEU}\uparrow$
LLaVA-1.5-7B	22.0	6.7	14.5
AIR (noise)	24.5	7.9	13.2
AIR (averaged)	20.8	6.4	14.5
AIR	18.4	5.7	14.4

Appendix

Table 21: Comparison of noise/averaging baselines.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM
Aug 31, 2026Peiyang Xu, Xiaopei Zhu, Jun Zhu +1Object Hallucination in VLMs Multimodal Contrastive Learning
Tsinghua University Beijing, China
Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation
Apr 11, 2026Yebo Wu, Han Jin, Zhijiang Guo +1VLM Hallucination Mitigation Language Model Decoding
State Key Laboratory of IOTSC, University of Macau · Hong Kong University of Science and Technology (Guangzhou)
Adversarial Orthogonal Disentanglement for LVLM Hallucination Mitigation
May 25, 2026Ruoxi Cheng, Haoxuan Ma, Zhengfei Hai +6Adversarial Representation Learning Vision-Language Models
Fudan University · Tencent · Nanjing University +3

Aug 31, 2026J/K move · Enter open · S save

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM

Peiyang Xu, Xiaopei Zhu, Jun Zhu, Xiaolin Hu

Tsinghua University Beijing, China

Object Hallucination in VLMs Multimodal Contrastive Learning VLM Hallucination Mitigation

Open paper AskPDF arXiv

Existing research on object hallucination in multimodal large language models (MLLMs) predominantly attributes the problem to language priors such as over-reliance on textual co-occurrence statistics. We challenge this view by presenting quantitative evidence for a complementary, under-explored cause: visual-origin hallucination, where hallucinations arise from incorrect visual feature extraction and misalignment between image and text embeddings. Through cosine similarity analysis and Smooth Grad-CAM entropy measurements, we show that hallucinated samples exhibit systematically lower image-text similarity (average 0.158 vs. -0.122) and inverted attention patterns, where attention is dispersed when the target object is present but wrongly concentrated when it is absent. Guided by this diagnosis, we propose Adversarial Contrastive Fine-Tuning (ACFT). ACFT uses an Adversarial Hallucination Attribute Flipping (AHAF) procedure, involving minimal, targeted adversarial perturbations that flip an image's hallucination attribute, to construct perfectly aligned positive-negative pairs, which are then used for contrastive fine-tuning. AHAF simultaneously serves as a diagnostic probe, revealing that MLLM visual representations lie dangerously close to hallucination decision boundaries. Requiring only 0.9% of the COCO dataset and adding zero inference overhead, ACFT achieves state-of-the-art performance on POPE, MME, and four description-level hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL. Code is available at https://github.com/zxp555/ACFT_MM

Abstract

VLM Hallucination Mitigation

Explore similar work

Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM

Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation

Adversarial Orthogonal Disentanglement for LVLM Hallucination Mitigation