Kestrel: Grounding Self-Refinement for LVLM Hallucination Mitigation

Jiawei Mao; Hardy Chen; Haoqin Tu; Yuhan Wang; Letian Zhang; Zeyu Zheng; Huaxiu Yao; Zirui Wang

cs.CVMar 17, 2026

Kestrel: Grounding Self-Refinement for LVLM Hallucination Mitigation

Authors: Jiawei Mao, Hardy Chen, Haoqin Tu, Yuhan Wang, Letian Zhang, Zeyu Zheng, Huaxiu Yao, Zirui Wang,

Organizations: UC Santa Cruz · UC Berkeley · UNC-Chapel Hill · Apple

Abstract

Large vision-language models (LVLMs) have become increasingly strong but remain prone to hallucinations in multimodal tasks, which significantly narrows their deployment. As training these LVLMs to avoid hallucinations becomes prohibitively expensive for larger models, training-free methods offer a cheap and flexible solution to this problem, yet existing approaches based on decoding or tool use often bring limited gains and/or weak interpretability. We propose Kestrel, a training-free framework for LVLM hallucination mitigation that combines an explicit visual-grounding agent with evidence-verified self-refinement mechanism. In detail, Kestrel first collects explicit visual evidence and converts tool outputs into reusable and structured textual evidence. Second, to take full advantage of these evidence, Kestrel verifies them via an LVLM judge for evidence checking, then iteratively self-refine answers based on verified evidence to reduce the risk of over-correction. Extensive experiments show that Kestrel improves performance over strong baselines across hallucination benchmarks (e.g., average +3.31% on POPE and +28.34 on MME-Hallucination with Qwen3-VL), while providing transparent verification traces for hallucination diagnosis and analysis -- e.g., both the integrated self-refinement module and grounding agent contributing an average +2.0% gain on POPE.

Figures & tables

Figure 2: Kestrel vs. prior training-free hallucination mitigation methods. By combining an external grounding agent with iterative self-improvement, Kestrel collects explicit visual evidence and further converts tool outputs into structured textual evidence for verification. This design yields more interpretable and stable evidence, reduces overconfident corrections, and avoids biased interpretation that may arise when LVLMs rely only on raw visual evidence compared with prior approaches.

Figure 3: Overview of Kestrel . Given an image-question pair, Kestrel follows a training-free four-stage pipeline for LVLM hallucination mitigation: (1) Initialization , which obtains an initial answer and rewrites it into question-aligned verifiable claims with associated visual entities and claim types; (2) Agent Grounding , which invokes an external SAM3-based grounding agent to collect explicit visual evidence (e.g., segmentation overlays, boxes, and crop-and-zoom views) and convert them into structured textual evidence; (3) Claim-level Verification , which verifies each claim against the cited evidence to produce claim-wise verdicts, confidence scores, and a top-level verification decision; and (4) Self-Refinement , which performs evidence-gated answer updating based on the current and previous verification traces.

Backbone	Method	MS-COCO [ 21 ]			A-OKVQA [ 27 ]			GQA [ 14 ]
Backbone	Method	Rand. $\uparrow$	Pop. $\uparrow$	Adv. $\uparrow$	Rand. $\uparrow$	Pop. $\uparrow$	Adv. $\uparrow$	Rand. $\uparrow$	Pop. $\uparrow$	Adv. $\uparrow$
Qwen3-VL [ 3 ]	Baseline [ 3 ]	89.00	86.92	86.20	92.36	86.67	81.87	90.70	83.63	81.50
	Qwen3-VL agent [ 3 ]	91.03	88.06	86.13	92.87	85.20	78.03	91.41	81.81	78.30
	VCD [ 17 ]	90.40	88.80	87.41	93.53	87.86	82.00	91.56	85.76	81.93
	Woodpecker [ 33 ]	89.97	88.03	87.10	93.23	88.90	83.33	91.27	86.27	82.77
	RITUAL [ 32 ]	86.20	83.67	82.27	87.67	83.50	77.76	86.86	82.30	78.23

Table 1: Results on POPE [ 20 ] benchmark . Higher (

\uparrow

) accuracy indicates better performance. The best results are bolded , and the second-best are underlined .

Figure 4: Prediction transition statistics with Qwen3-VL before and after refinement. Prediction are categorized into four types: correctly preserved, error corrected, over-corrected, and incorrectly preserved. The results show that the refinement process is conservative, retaining most originally correct predictions while correcting a portion of erroneous ones, with limited over-correction. Zoom in for a better view.

Figure 5: Qualitative results of Kestrel . We compare the VQA responses from the regular baseline and our method based on Qwen3-VL. Zoom in for a better view.

Backbone	Method	Object-level		Attribute-level		MME Score $\uparrow$
Backbone	Method	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	MME Score $\uparrow$
Qwen3-VL [ 3 ]	Baseline [ 3 ]	195.00 ( $+0.00$ )	175.00 ( $+0.00$ )	168.33 ( $+0.00$ ) 0	193.33 ( $+0.00$ ) 0	731.66 ( $+0.00$ ) 0
	Qwen3-VL agent [ 3 ]	200.00 ( $+5.00$ )	181.67 ( $+6.67$ )	168.33 ( $+0.00$ ) 0	193.33 ( $+0.00$ ) 0	743.33 ( $+11.67$ ) 0
	VCD [ 17 ]	195.00 ( $+0.00$ )	180.00 ( $+5.00$ ) 0	168.33 ( $+6.73$ ) 0	193.33 ( $+0.00$ )	736.66 ( $+5.00$ )
	Woodpecker [ 33 ]	195.00 ( $+0.00$ )	173.33 ( $+1.67$ ) 0	168.33 ( $+0.00$ ) 0	195.00 ( $+1.67$ )	731.66 ( $+10.00$ )
	RITUAL [ 32 ]	195.00 ( $+0.00$ )	180.00 ( $+5.00$ ) 0	168.33 ( $+0.00$ )	193.33 ( $+0.00$ ) 0	736.66 ( $+5.00$ )

Table 2: Results on MME-Hallucination [ 9 ] benchmark. Higher scores (

\uparrow

) indicate better performance. The best results are bolded , and the second-best are underlined .

Method	Average Latency $\downarrow$	GPU Memory $\downarrow$	Checked Case	POPE (MS-COCO) $\uparrow$
Qwen3-VL [ 3 ]	0.78 s ( $\times$ 1.00)	17428 MB ( $\times$ 1.00)	9000	87.37
Kestrel ( $1^{\text{st}}$ iteration)	12.00 s ( $\times$ 15.38)	21472 MB ( $\times$ 1.23)	9000	89.28
Kestrel ( $2^{\text{nd}}$ iteration)	8.54 s ( $\times$ 10.94)	21296 MB ( $\times$ 1.22)	4978	89.45
Kestrel ( $3^{\text{rd}}$ iteration)	7.12 s ( $\times$ 9.12)	21274 MB ( $\times$ 1.22)	477	89.34
Kestrel	18.75 s ( $\times$ 24.03)	21472 MB ( $\times$ 1.23)	9000	89.45

Table 3: Efficiency comparison. We report the average inference latency per instance, peak GPU memory, and POPE accuracy on MS-COCO. Experiments are conducted on a single NVIDIA A100 GPU. Lower latency/memory is better, while higher POPE accuracy is better.

Figure 6: Human preference comparison across methods. Bars show the fraction of trials in which each method is preferred (n=60). Error bars denote 95 $\%$ Wilson confidence intervals.

Setting	Grounding	Structured Textual	Claim-level	Evidence-gated	Self-	History-	POPE
Setting	Agent	Evidence	Verification	Update	Refinement	Aware	MS-COCO
baseline	✗	✗	✗	✗	✗	✗	87.37
s1	✗	✗	✗	✗	✓	✗	87.44
s2	✓	✗	✗	✗	✗	✗	87.50
s3	✗	✗	✓	✓	✓	✓	87.58
s4	✓	✗	✓	✓	✓	✓	88.61

Table 4: Ablation study of core design components. We evaluate different settings by progressively enabling the key components in Kestrel . Higher scores indicate better performance.

Table 10

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Prompt template for initialization stage.

Panel of Figure 8 — Figure 8: Prompt template for claim-level verification stage.

Panel of Figure 9 — Figure 9: Prompt template for self-refinement stage.

Figure 10: Qualitative results of Kestrel . We compare the VQA responses from the regular baseline and our method based on Qwen3-VL. Zoom in for a better view.

Figure 11: Qualitative results of Kestrel . We compare the VQA responses from the regular baseline and our method based on InternVL-3.5. Zoom in for a better view.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models
Feb 28, 2026April FuCross-Layer Attention Self-Correction in VLMs
Independent Researcher
See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.
Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment
Jul 18, 2026Pengxu Chen, Yao Zhu, Guangming Zhu +4Large Vision-Language Models VLM Hallucination Mitigation
Xidian University Xi’an, China · Tsinghua University Beijing, China · Shanghai Road Transport Development Center Shanghai, China +1

Feb 28, 2026J/K move · Enter open · S save

Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models

April Fu

Independent Researcher

Cross-Layer Attention Self-Correction in VLMs VLM Hallucination Mitigation Vision-Language Models Large Vision-Language Models Transformer Attention

Open paper AskPDF arXiv

Although Large Vision-Language Models (LVLMs) have made substantial progress, hallucination, where generated text is not grounded in the visual input, remains a challenge. As LVLMs become stronger, previously reported hallucination patterns, such as linguistic bias and overthinking phenomenon, become far less consistent, making the corresponding mitigation techniques substantially less effective. In this paper, we introduce an Internal self-Correction mechanism utilizing Layer Attention (ICLA) that operates directly on hidden states during generation. Each layer selectively retrieves information from all preceding layers through a diagonal cross-layer attention mechanism, enabling self-refinement without any external correction signals. With introducing and training only 0.2M and 0.1M additional parameters on LLaVA1.5-7B and Qwen2.5-VL-7B, \ours consistently improves visual grounding across multiple hallucination benchmarks, demonstrating its effectiveness for more advanced LVLMs.

Abstract

VLM Hallucination Mitigation

Explore similar work

Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment