Re$^3$Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning

Haonan Jia; Shichao Dong; Zenghui Sun; Jiawen Zheng; Ziqi Miao; Gege Shi; Qiuyu Zhao; Jinsong Lan

cs.CVAug 21, 2026

Re $^3$ Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning

Authors: Haonan Jia, Shichao Dong, Zenghui Sun, Jiawen Zheng, Ziqi Miao, Gege Shi, Qiuyu Zhao, Jinsong Lan,

Organizations: Taobao & Tmall Group of Alibaba · The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Artificial Intelligence Laboratory

Abstract

Reinforcement Learning (RL) has demonstrated significant gains in image captioning, yet it is still limited in encouraging Large Vision-Language Models (LVLMs) to explore novel reasoning strategies. This limitation leads to a performance gap between RL and Supervised Fine-Tuning (SFT). In this paper, we argue that multi-modal retrieval can serve as an effective reasoning signal for caption refinement. Based on this insight, we present the Retrieval-Guided Refinement for Image Captioning (Re $^3$ Cap), a retrieval-guided reasoning strategy that enhances image captioning without requiring additional annotations. Instantiated by Caption Refinement Suggester (CRS) and Caption Quality Assessor (CQA), this strategy identifies hallucinations and omissions in image captions, leading to more accurate and detailed descriptions. Extensive experiments demonstrate the superiority of our method in image captioning, even compared with Supervised Fine-Tuning. Especially, Re $^3$ Cap outperforms GRPO with an average improvement of 8.64% in relation reasoning on the COCO-LN500 benchmark.

Figures & tables

Figure 1: Reasoning strategy of Caption Refinement Suggester (CRS) and Caption Quality Assessor (CQA). We leverage k-core subgraph computation to analyze critical textual content in the retrieval results. Specifically, CRS extracts overlapping sentences $S^{k}_{v}$ from the image retrieval results and uses them as signals to guide LVLMs to incorporate these key elements into their generated descriptions. By analyzing the discrepancy between $S^{k}_{v}$ and caption-retrieved results $S^{k}_{c}$ , CQA identifies hallucinations ( $S^{k}_{c}-S^{k}_{vc}$ ) and omissions ( $S^{k}_{v}-S^{k}_{vc}$ ) in this caption.

Figure 2: Overview of Retrieval-Guided Refinement for Image Captioning (Re 3 Cap). Re 3 Cap begins by sampling initial captions $\{c_{i}\}_{i=1}^{M}$ for each image $v$ . Next, it performs image-conditioned retrieval and caption-conditioned retrieval, and leverages k-core analysis to generate guidance $\{f_{i}\}_{i=1}^{M}$ based on Caption Refinement Suggester and Caption Quality Assessor. Finally, the method incorporates the guidance into prompts to obtain refined captions $\{c\textquoteright_{i}\}_{i=1}^{M}$ and optimizes the policy model with reinforcement learning.

Figure 3: Capability boundary analysis with max@ $k$ .

Base Model	Reward	Method	COCO-LN500			DOCCI500
Base Model	Reward	Method	Objects F1	Attributes F1	Relations QA	Objects F1	Attributes F1	Relations QA
LLaVA1.5 -7B	–	$\mathbf{\circ}$ Base	67.56	42.34	14.38	59.41	48.01	9.19
	–	$\mathbf{\circ}$ SFT	73.45	54.25	28.59	68.44	53.93	19.87
	CLIP	$\mathbf{\circ}$ GRPO	66.66	53.15	22.58	61.77	53.62	18.42
	CLIP	$\bullet$ Ours	72.03 ( $\uparrow$ 5.4)	54.73 ( $\uparrow$ 1.6)	30.38 ( $\uparrow$ 7.8)	68.51 ( $\uparrow$ 6.7)	57.68 ( $\uparrow$ 4.1)	23.86 ( $\uparrow$ 5.4)
	SC	$\mathbf{\circ}$ GRPO	69.49	50.44	21.77	62.52	53.80	17.86

Table 1: Performance comparison of Reinforcement Learning with GRPO and Re 3 Cap across multiple reward functions on COCO-LN500 Pont-Tuset et al. (2020) and DOCCI500 Onoe et al. (2024) . Constrained by the reasoning capacity of base models, GRPO struggles to surpass the performance of task-specific SFT models, especially on weaker LVLMs such as LLaVA-1.5-7B Liu et al. (2023) . In contrast, our method introduces a novel reasoning strategy to generate unexplored caption candidates, consistently improving image captioning performance across weaker and stronger base models, including Qwen2-VL-7B Wang et al. (2024b) and Qwen2.5-VL-7B Bai et al. (2025b) . Moreover, using a more accurate reward further improves the performance of ours.

Base Model	Reward	Method	COCO-LN500			DOCCI500
Base Model	Reward	Method	Objects F1	Attributes F1	Relations QA	Objects F1	Attributes F1	Relations QA
Qwen2 -VL-7B	–	$\mathbf{\circ}$ VCD	69.85	48.71	26.77	67.57	53.80	25.03
	–	$\mathbf{\circ}$ INTER	69.81	48.58	26.93	67.42	53.70	24.67
	SC	$\mathbf{\circ}$ SC-Captioner	76.37	57.56	38.51	71.63	57.67	30.51
	SC	$\bullet$ Ours	77.77 ( $\uparrow$ 1.4)	58.79 ( $\uparrow$ 1.2)	44.60 ( $\uparrow$ 6.1)	73.29 ( $\uparrow$ 1.7)	58.88 ( $\uparrow$ 1.2)	36.40 ( $\uparrow$ 5.9)
	CIM	$\mathbf{\circ}$ SFT+CIM	76.65	58.09	42.12	73.87	58.68	36.32

Table 2: Performance comparison with state-of-the-art methods on Qwen2-VL-7B Wang et al. (2024b) over COCO-LN500 Pont-Tuset et al. (2020) and DOCCI500 Onoe et al. (2024) . The results demonstrate the superiority of our proposed reinforcement learning framework when compared with state-of-the-art methods.

Components		Objects	Attributes	Relations
CRS	CQA	F1	F1	QA
✗	✗	75.80	58.22	38.71
✓	✗	76.83	58.98	40.90
✗	✓	77.14	59.04	42.69
✓	✓	78.18	59.28	44.19

Table 3: Ablation study of core components on COCO-LN500 Pont-Tuset et al. (2020) using Qwen2-VL-7B Wang et al. (2024b) with CIM Jia et al. (2026) as the reward function.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Prompt example for initial caption, Re 3 Cap, and relation evaluation.

Base Model	Reward	Method	COCO-LN500			DOCCI500
Base Model	Reward	Method	Objects F1	Attributes F1	Relations QA	Objects F1	Attributes F1	Relations QA
InternVL3 -8B	–	$\mathbf{\circ}$ Base	71.00	50.66	26.44	66.08	53.72	25.11
	–	$\mathbf{\circ}$ SFT	76.42	57.79	42.32	72.72	58.59	35.31
	CLIP	$\mathbf{\circ}$ GRPO	72.14	54.84	30.83	68.70	57.58	28.13
	CLIP	$\bullet$ Ours	74.79 ( $\uparrow$ 2.7)	54.42 ( $\downarrow$ 0.4)	35.46 ( $\uparrow$ 4.6)	71.15 ( $\uparrow$ 2.5)	54.85 ( $\downarrow$ 2.7)	28.81 ( $\uparrow$ 0.7)
	SC	$\mathbf{\circ}$ GRPO	75.77	57.14	35.58	70.72	57.78	28.38

Appendix

Table 4: Performance comparison of reinforcement learning with GRPO and Re 3 Cap on more LVLMs. We evaluate InternVL3-8B Zhu et al. (2025) and Qwen3-VL-8B Bai et al. (2025a) with different reward functions on COCO-LN500 Pont-Tuset et al. (2020) and DOCCI500 Onoe et al. (2024) . Re 3 Cap consistently improves over GRPO across most metrics, especially on the Relations QA task.

Figure 5: Robustness study of the retrieval number $K$ . We evaluate Re 3 Cap with different retrieval numbers $K$ on COCO-LN500 Pont-Tuset et al. (2020) and DOCCI500 Onoe et al. (2024) across multiple LVLMs. The performance remains stable across different values of $K$ , demonstrating that our method is robust to the choice of retrieval number.

Figure 6: Robustness study of the similarity threshold $\tau$ . We evaluate Re 3 Cap with different similarity thresholds $\tau$ for graph edge construction on COCO-LN500 Pont-Tuset et al. (2020) and DOCCI500 Onoe et al. (2024) across multiple LVLMs. The results show only minor variations across different thresholds, indicating that our method is robust to the choice of graph construction threshold.

Image Encoder	Text Encoder	Objects			Attributes			Relations
Image Encoder	Text Encoder	Precision	Recall	F1	Precision	Recall	F1	QA
DINOv3 ViT-L/16	MiniLM	79.24	77.23	77.77	70.48	55.89	59.03	45.13
DINOv3 ViT-L/16	MPNet	80.60	76.88	78.23	71.34	54.82	58.57	44.68
OpenCLIP ViT-H/14	MiniLM	79.77	77.03	77.91	69.99	55.29	58.49	45.04
OpenCLIP ViT-H/14	MPNet	79.62	77.70	78.18	71.00	56.03	59.28	44.19

Appendix

Table 5: Robustness study of diverse encoders on COCO-LN500 Pont-Tuset et al. (2020) using Qwen2-VL-7B Wang et al. (2024b) with CIM Jia et al. (2026) as the reward function. Our method maintains stable performance in image captioning when leveraging various encoders as the retrieval model. The experimental results indicate that our proposed Re 3 Cap is robust to diverse encoders.

Image Captioning

Image captioning generates natural-language descriptions of visual content, with research focused on accuracy, coverage, grounding, and evaluation.

See all 94 papers

CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning
Jun 8, 2026Penghui Yang, Long Xing, Xiaoyi Dong +10Image Captioning Multimodal Pretraining
Tsinghua University · University of Science and Technology of China · Microsoft +4
BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning
May 8, 2026Shaokai Ye, Vasileios Saveris, Yihao Qian +3Multi-Objective Reinforcement Learning Image Captioning
Apple
CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning
Feb 25, 2026Zhijiang Tang, Linhua Wang, Jiaxin Qi +4Image Captioning RL for VLMs
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Zhejiang, China · LLM Team, Shopee Pte. Ltd., Shanghai, China

Jun 8, 2026J/K move · Enter open · S save

CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning

Penghui Yang, Long Xing, Xiaoyi Dong, Yuhang Zang+9

Tsinghua University · University of Science and Technology of China · Microsoft+4

Image Captioning Multimodal Pretraining Video Captioning Reinforcement Learning with Verifiable Rewards Dense Video Captioning RL for VLMs

Open paper AskPDF arXiv

Image and video captioning are fundamental tasks that bridge the visual and linguistic domains, playing a critical role in pre-training Large Vision-Language Models (LVLMs). Current state-of-the-art captioning models are typically trained with Supervised Fine-Tuning (SFT), a paradigm that relies on expensive, non-scalable annotations and often causes models to memorize specific ground-truth answers, limiting their generality and ability to generate diverse, creative descriptions. To overcome these limitations, we propose applying Reinforcement Learning with Verifiable Rewards (RLVR) to the open-ended task of multimodal captioning. We introduce Captioning Reinforcement Learning++ (CapRL++), a novel reference-free training framework that redefines caption quality through its utility: a high-quality caption should enable a non-visual language model to accurately answer questions about the corresponding visual content. CapRL++ employs a decoupled two-stage pipeline where an LVLM generates a caption, and the objective reward is derived from the accuracy of a separate, vision-free LLM answering Multiple-Choice Questions based solely on that caption. Evaluations on more than 20 image and video benchmarks show that CapRL++ improves dense caption quality and strengthens caption-based pretraining across tasks such as spatial and temporal understanding. Pretraining on scalable image and video caption datasets annotated by CapRL++ yields substantial downstream gains. Furthermore, within the Prism Framework for caption quality evaluation, compact models trained with CapRL++ achieve dense captioning performance comparable to substantially larger models such as Qwen2.5-VL-72B and Qwen3-VL-235B-A22B. These results validate that CapRL++ effectively trains models to produce generalizable, high-fidelity descriptions, establishing a robust foundation beyond the limitations of traditional SFT.

Abstract

Image Captioning

Explore similar work

CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning

BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning

CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning