Metric-Driven Reinforcement Learning for Object Hallucination Reduction in LVLM-Based Image Captioning

Thanh Nguyen; Hee Suk Yoon; Tung Minh Luu; Eunseop Yoon; Chang D. Yoo

cs.CVOct 10, 2026

Metric-Driven Reinforcement Learning for Object Hallucination Reduction in LVLM-Based Image Captioning

Authors: Thanh Nguyen, Hee Suk Yoon, Tung Minh Luu, Eunseop Yoon, Chang D. Yoo

Organizations: School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea

Abstract

Large Vision-Language Models (LVLMs) have excelled in joint visual and language understanding, particularly in image captioning. However, LVLMs remain prone to object hallucination, describing non-existent entities in the given image. Previous remedies, including dataset-enhanced supervised training and reinforcement learning from human feedback, show promise but rely heavily on costly human involvement, limiting scalability. To address this, we propose a reinforcement learning framework to mitigate object hallucination in LVLM-based image captioning, driven entirely by natural language processing (NLP) metrics. While optimizing reinforcement learning solely on NLP metrics offers scalability, it also introduces pitfalls: reward hacking, sparse feedback, and costly training. Our framework sidesteps these by introducing an Object Consistency Score (OCS) capturing both correctness and coverage of objects, incorporating KL-regularization for denser token-level guidance, and designing a lightweight PPO variant that fine-tunes adapters atop a frozen LVLM backbone, achieving efficiency and stability. Extensive experiments on the InstructBLIP baseline demonstrate the effectiveness and generalizability of our framework, achieving a substantial hallucination reduction of up to 41% on the COCO dataset and 29% on the Visual Genome. Furthermore, the framework can enhance caption quality by incorporating complementary evaluation metrics such as BERTScore and METEOR. These results reveal a broader insight in multimodal learning: LVLMs can effectively self-correct hallucinations using metric feedback, motivating deeper exploration into scalable reinforcement frameworks that significantly reduce human effort.

Figures & tables

Panel of Figure 1 — Fig. 1 : Quantitative and qualitative comparison between Base (InstructBLIP) and Ours : Chart (a) shows a significant 41% reduction in object hallucination on the COCO dataset using Ours . Figure (b) presents an example where the Base model produces a caption with substantial object hallucination, while the Ours model provides an accurate description without hallucinated objects.

Fig. 2 : Detailed architecture of our framework. Specifically, the Policy Network is crafted by augmenting the shared LVLM with delicately learnable soft prompts. Meanwhile, the Value Network is formed by replacing the LLM Head with a Linear Value head. Notably, all parameters of the LVLM remain shared and frozen, with only a very small fraction (less than 0.01% LVLM weight) of trainable parameters added to the LVLM for the meticulous modeling of the policy network and value network.

Fig. 3 : Simplified overview of the framework’s network components: policy network, value network, and reference network. All networks share the same frozen LVLM as its foundation. The reference network mirrors the LVLM identically, while the value and policy networks incorporate a lightweight adapter into the shared LVLM.

Type	Prompt	InstructBLIP		mPLUG-Owl
Type	Prompt	$\text{CHAIR}_{i}$ (%) $\downarrow$	$\text{CHAIR}_{s}$ (%) $\downarrow$	$\text{CHAIR}_{i}$ (%) $\downarrow$	$\text{CHAIR}_{s}$ (%) $\downarrow$
Short	Generate a short caption of the image.	2.43	3.13	22.81	60.55
Short	Create a textual summary for the image.	4.95	6.51	22.98	61.33
Long	Provide a detailed description of the image.	27.01	60.91	26.03	71.39
Long	Create a detailed textual summary for the image.	25.80	59.11	24.25	66.31

TABLE I : Object Hallucination, gauged by

\text{CHAIR}_{s}

and

\text{CHAIR}_{i}

metrics, across diverse caption types paired with specific input prompts in the COCO test set. These prompts are designed to elicit both short and long captions. Two distinct methods are illustrated: InstructBLIP and mPLUG-Owl.

Method	$\text{CHAIRi}(\%)\downarrow$	$\text{CHAIRs}(\%)\downarrow$	$\text{Coverage}(\%)\uparrow$
mPLUG-Owl	26.2	70.5	76.7
LLaVA	22.5	62.7	79.6
LlaVA-Next	13.0	32.0	68.2
InstructBLIP	25.8	59.1	71.3
Teacher	7.5	36.4	NA
CoT	7.8	35.7	NA

TABLE II : Performance of object hallucination on the COCO dataset. The first row presents the non-fine-tuned LVLM baselines. Starting from the second row, all methods are built upon the InstructBLIP baseline. Specifically, the second row includes hallucination mitigation approaches reported in [ 98 ] . We additionally report Coverage (i.e., the recall percentage of ground-truth objects mentioned) to quantify recall preservation. The third row presents the results of our Hard Prompting strategy, while the final row reports the performance of the baseline fine-tuned using our RL framework.

Method	Captioning Quality
Method	SPICE $(\%)\uparrow$	BLEU $(\%)\uparrow$	BERTScore $(\%)\uparrow$
mPLUG-Owl	12.5	2.7	87.4
LLaVA	13.5	3.0	87.8
LLaVA-Next	13.6	2.9	87.5
InstructBLIP	10.9	1.1	85.8
Hard Prompt	11.1	1.0	85.9

TABLE III : Captioning quality is evaluated using NLP metrics, comparing our approach to other baseline. Ours uses only OCS and KL divergence, while Ours+ incorporates additional metrics: METEOR and BERTScore in the reward function.

Model	$\text{CCEval}_{i}\downarrow$	$\text{CCEval}_{s}\downarrow$
InstructBLIP	22.3	72.0
Ours	15.6	43.0

TABLE IV : Performance on the Visual Genome dataset. Note that for Visual Genome, CCEval is selected as the performance metric instead of CHAIR. CCEval provides a more accurate evaluation by extending CHAIR’s methodology using GPT-assisted parsing for object extraction, specifically tailored to the Visual Genome dataset.

Model	$\text{CHAIR}_{i}\downarrow$	$\text{CHAIR}_{s}\downarrow$	$\text{BERTScore}\uparrow$	$\text{BLEU}\uparrow$
LLaVA	22.5	62.7	87.4	3.0
LLaVA+Ours	7.7	22.2	87.5	3.1

TABLE V : Comparison of LLaVA baseline and LLaVa fine-tuned using our method.

Fig. 4 : Qualitative comparison between the original InstructBLIP model ( Base ) and the fine-tuned InstructBLIP model trained with the proposed framework ( Ours ) on the COCO dataset. Hallucinated objects are highlighted in red . Compared with the baseline, our method generates captions that are more grounded in the visual content and significantly reduces object hallucinations.

Reward	Pre (%) $\uparrow$	Rec (%) $\uparrow$	$\text{CHAIR}_{i}$ (%) $\downarrow$	$\text{CHAIR}_{s}$ (%) $\downarrow$
Base	72.9	71.3	27.1	60.9
CHAIR	93.7	20.6	6.3	14.4
OCS	93.2	70.2	6.8	17.8

TABLE VI : Comparison of Precision (

Pre

) and Recall (

Rec

) between using CHAIR and OCS in the reward function.

Base	OCS	BERTScore	METEOR	$\text{CHAIR}_{i}$	BERTScore	BLUE
$\checkmark$				25.8	85.8	1.1
$\checkmark$	$\checkmark$			6.8	86.9	1.5
$\checkmark$	$\checkmark$	$\checkmark$		6.9	90.5	1.8
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	6.9	92.4	6.6

TABLE VII : The ablation studies examining the impact of OCS, BERTscore and METEOR metrics on the COCO test set

Model	Detection mAP	Top1-Acc	$\text{CHAIR}_{i}$	$\text{CHAIR}_{s}$
Baseline	-	-	25.8	59.1
YOLOV8	53.9	79.0	7.8	25.5
YOLOV11	54.7	79.5	7.8	22.3
GT	-	-	6.8	17.8

TABLE VIII : Effect of object detection accuracy on hallucination reduction. YOLO-based detections greatly reduce hallucination compared to the baseline, with YOLOv11x performing slightly better; ground-truth objects remain most effective

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Parameters
Do sample (greedy decoding)	False
Num beams	1
Max Soft Prompt Length $n$	20
Max Episode Length $H$	150
Mini-Batch Size	64
PPO Learning Rate	0.0002

Appendix

TABLE IX : Training parameters of our framework for the COCO dataset

Object Hallucination in VLMs

Object hallucination in vision-language models is the production of object claims unsupported by visual evidence.

See all 128 papers

CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering
May 6, 2026Qiming Li, Zekai Ye, Xiaocheng Feng +9Hallucination in Language Models Large Vision-Language Models
1Harbin Institute of Technology · 2Peng Cheng Laboratory · 3Harbin Institute of Technology, Shenzhen
Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
May 24, 2026Yuanzhi Xu, Qian Gao, Jun Fan +4VLM Robustness Vision-Language Models
Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China +1
Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
May 20, 2026Meng Shen, Minghao Wu, Deepu RajanVision-Language Models Hallucination in Language Models
Nanyang Technological University · Monash University

May 6, 2026J/K move · Enter open · S save

CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering

Qiming Li, Zekai Ye, Xiaocheng Feng, Weihong Zhong+8

1Harbin Institute of Technology · 2Peng Cheng Laboratory · 3Harbin Institute of Technology, Shenzhen

Hallucination in Language Models Large Vision-Language Models Object Hallucination in VLMs Activation Steering VLM Hallucination Mitigation

Open paper AskPDF arXiv

Although Large Vision-Language Models (LVLMs) have demonstrated remarkable performance on downstream tasks, they frequently produce contents that deviate from visual information, leading to object hallucination. To tackle this, recent works mostly depend on expensive manual annotations and training cost, or decoding strategies which significantly increase inference time. In this work, we observe that LVLMs' attention to visual information is significantly enhanced when answering caption queries compared to non-caption queries. Inspired by this phenomenon, we propose Caption-guided Visual Attention Steering (CAST), a training-free, plug-and-play hallucination mitigation method that leverages the attention activation pattern corresponding to caption queries to enhance LVLMs' visual perception capability. Specifically, we use probing techniques to identify attention heads that are highly sensitive to caption queries and estimate optimized steering directions for their outputs. This steering strengthens LVLM's fine-grained visual perception capabilities, thereby effectively mitigating object hallucination. CAST reduced object hallucination by an average of 6.03% across five widely used LVLMs and five benchmarks including both discriminative and generative tasks, demonstrating state-of-the-art performance while adding little inference cost and preserving other foundational capabilities.

Abstract

Object Hallucination in VLMs

Explore similar work

CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering

Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration

Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens