Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning

Haonan Jia; Shichao Dong; Xin Dong; Zenghui Sun; Jin Wang; Jinsong Lan; Xiaoyong Zhu; Bo Zheng

cs.CVMar 2, 2026

Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning

Authors: Haonan Jia, Shichao Dong, Xin Dong, Zenghui Sun, Jin Wang, Jinsong Lan, Xiaoyong Zhu, Bo Zheng,

Organizations: Taobao & Tmall Group of Alibaba · The University of Hong Kong

Abstract

Large Vision-Language Models (LVLMs) often omit or misrepresent critical visual content in generated image captions. Minimizing such information loss will force LVLMs to focus on image details to generate precise descriptions. However, measuring information loss during modality conversion is inherently challenging due to the modal gap between visual content and text output. In this paper, we argue that the quality of an image caption is positively correlated with the similarity between images retrieved via text search using that caption. Based on this insight, we further propose Cross-modal Identity Mapping (CIM), a reinforcement learning framework that enhances image captioning without requiring additional annotations. Specifically, the method quantitatively evaluates the information loss from two perspectives: Gallery Representation Consistency and Query-gallery Image Relevance. Supervised under these metrics, LVLM minimizes information loss and aims to achieve identity mapping from images to captions. The experimental results demonstrate the superior performance of our method in image captioning, even when compared with Supervised Fine-Tuning. Particularly, on the COCO-LN500 benchmark, CIM achieves a 20% improvement in relation reasoning on Qwen2.5-VL-7B.

Figures & tables

Figure 1: Qualitative comparison of retrieved images using different captions. The more detailed the caption, the higher the semantic consistency among the retrieved images. The more accurate the caption, the greater the visual similarity between the retrieved images and the source image.

Figure 2: Verifying the effectiveness of our proposed metrics for quantifying information loss via Pearson correlation. For each LVLM, we plot the correlation between average metrics and the logit of breed classification accuracy. Each point corresponds to one bin, with each bin containing 100 samples. The Pearson correlation coefficient is reported for each model. This correlation verifies the effectiveness of our proposed metrics for quantifying information loss during modality conversion.

Model	Qwen3-VL-8B [ 4 ]	InternVL3-8B [ 85 ]	LLaVA-v1.6-7B [ 42 ]
Species Acc	100.00	100.00	99.98
Breed Acc	40.54	19.29	15.11

Table 1: Evaluation of fine-grained image classification across LVLMs. We report species accuracy and breed accuracy on the Oxford-IIIT Pet dataset [ 50 ] .

Figure 3: The overall architecture of our proposed Cross-modal Identity Mapping (CIM). We first prompt the LVLM to generate image captions in detail. Based on such captions, CIM then retrieve topK relevant images. In this way, the method analyze the Query-gallery Image Relevance (QIR) and Gallery Representation Consistency (GRC) to infer the information loss between the image and its corresponding caption. Finally, by using CIM as reward function, our method forced LVLMs to generate fine-grained and precise captions.

Base Model	Bench	Method	CAPTURE	Objects			Attributes			Relations
Base Model	Bench	Method	CAPTURE	Precision	Recall	F1	Precision	Recall	F1	QA
LLaVA1.5 -7B [ 41 ]	COCO -LN500	$\mathbf{\circ}$ Base	44.75	81.20	59.02	67.56	58.61	37.2	42.34	14.38
	COCO -LN500	$\bullet$ Ours	48.62 ( $\uparrow$ 3.9)	83.84 ( $\uparrow$ 2.6)	60.70 ( $\uparrow$ 1.7)	69.80 ( $\uparrow$ 2.2)	71.24 ( $\uparrow$ 12.6)	48.91 ( $\uparrow$ 11.7)	54.38 ( $\uparrow$ 12.0)	24.98 ( $\uparrow$ 10.6)
	DOCCI 500	$\mathbf{\circ}$ Base	50.72	81.13	48.37	59.41	64.12	39.78	48.01	9.19
	DOCCI 500	$\bullet$ Ours	57.10 ( $\uparrow$ 6.4)	83.12 ( $\uparrow$ 2.0)	52.78 ( $\uparrow$ 4.4)	63.38 ( $\uparrow$ 4.0)	72.06 ( $\uparrow$ 7.9)	47.42 ( $\uparrow$ 7.6)	56.28 ( $\uparrow$ 8.3)	19.87 ( $\uparrow$ 10.7)
Qwen2 -VL-7B [ 71 ]	COCO -LN500	$\mathbf{\circ}$ Base	46.52	81.12	61.82	69.47	66.48	42.86	48.68	20.47

Table 2: Performance of Reinforcement Learning with CIM on the Base Model over COCO-LN500 [ 53 ] and DOCCI500 [ 48 ] .

\uparrow

\downarrow

indicate improvement/degradation over the Base Model (higher is better). The results indicate that post-training the base model with our CIM enhances its ability to generate high-quality captions.

Base Model	Bench	Method	CAPTURE	Objects			Attributes			Relations
Base Model	Bench	Method	CAPTURE	Precision	Recall	F1	Precision	Recall	F1	QA
LLaVA1.5 -7B [ 41 ]	COCO -LN500	$\mathbf{\circ}$ SFT	46.62	77.09	71.16	73.45	66.65	50.57	54.25	28.59
		$\mathbf{\circ}$ SC-Captioner [ 82 ]	47.11	76.43	75.65	75.20	65.79	53.01	55.35	33.63
		$\bullet$ SFT + Ours	47.74( $\uparrow$ 0.6)	79.48 ( $\uparrow$ 3.1)	69.75 ( $\downarrow$ 5.9)	73.77 ( $\downarrow$ 1.4)	70.48 ( $\uparrow$ 4.7)	51.29 ( $\downarrow$ 1.7)	56.04 ( $\uparrow$ 0.7)	33.96 ( $\uparrow$ 0.3)
	DOCCI 500	$\mathbf{\circ}$ SFT	60.92	78.15	62.20	68.44	66.85	46.27	53.93	19.87
		$\mathbf{\circ}$ SC-Captioner [ 82 ]	62.29	77.58	66.05	70.30	67.64	50.70	57.10	22.69

Table 3: Performance of Reinforcement Learning with CIM on the SFT Model over COCO-LN500 [ 53 ] and DOCCI500 [ 48 ] .

\uparrow

\downarrow

indicate improvement/degradation over SC-Captioner [ 82 ] (higher is better). Our method consistently improves relation understanding and attribute precision, with minor trade-offs in object recall.

Figure 4: Ablation study of reward components on COCO-LN500 [ 53 ] using Qwen2.5-VL-7B [ 4 ] .

Figure 5: Scalability study of the retrieval corpus on COCO-LN500 [ 53 ] across multiple LVLMs.

Image Encoder	Text Encoder	CAPTURE	Objects	Attributes	Relations
Image Encoder	Text Encoder	CAPTURE	F1	F1	QA
DINOv3 ViT-L/16 [ 64 ]	MiniLM [ 72 ]	48.17	76.62	59.05	41.03
DINOv3 ViT-L/16 [ 64 ]	MPNet [ 65 ]	48.64	76.32	59.22	42.49
OpenCLIP ViT-H/14 [ 11 ]	MiniLM [ 72 ]	48.99	76.91	58.96	41.16
OpenCLIP ViT-H/14 [ 11 ]	MPNet [ 65 ]	48.93	77.59	58.51	44.15

Table 4: Robustness study of diverse retrieval models on COCO-LN500 [ 53 ] using Qwen2.5-VL-7B [ 4 ] .

Figure 6: Prompts for image captioning, relation evaluation, and hierarchical classification.

Model	BLEU-4	METEOR	CAPTURE	Objects	Attributes	Relations
Model	BLEU-4	METEOR	CAPTURE	F1	F1	QA
$\mathbf{\circ}$ Qwen2-VL-7B [ 71 ]	29.39	16.59	57.96	66.47	52.65	17.57
$\mathbf{\circ}$ ShareCaptioner [ 82 ]	39.09	23.05	57.90	66.05	52.27	19.47
$\mathbf{\circ}$ Gemini-1.5	24.70	16.25	60.34	68.17	55.69	28.90
$\mathbf{\circ}$ Claude-3.7	41.36	20.17	61.02	69.48	55.57	28.78
$\bullet$ SFT + Ours	45.16	24.88	64.31	73.87	58.68	36.32

Table 5: Performance comparison of open-source and closed-source models on the DOCCI500 [ 48 ] benchmark. Best results are highlighted in bold. Our model achieves the best performance across all metrics, surpassing both open‑source and closed‑source models.

Base Model	Benchmark	Method	BLEU-4	METEOR	CAPTURE	Objects			Attributes			Relations
Base Model	Benchmark	Method	BLEU-4	METEOR	CAPTURE	Precision	Recall	F1	Precision	Recall	F1	QA
Qwen2-VL [ 71 ]	COCO-LN500	$\mathbf{\circ}$ 7B-Base	39.57	20.42	46.52	81.12	61.82	69.47	66.48	42.86	48.68	20.47
		$\mathbf{\circ}$ 72B-Base	39.37	19.77	47.20	82.52	62.89	70.72	69.36	42.78	49.38	27.46
		$\bullet$ 7B-Ours	39.63	27.06	48.64	80.90	72.23	75.80	72.45	54.08	58.22	38.71
	DOCCI500	$\mathbf{\circ}$ 7B-Base	29.39	16.59	57.96	83.69	56.79	66.47	69.96	43.27	52.65	17.57
		$\mathbf{\circ}$ 72B-Base	30.79	17.57	59.56	84.02	60.21	69.14	71.63	44.31	53.92	27.21

Table 6: Performance of Reinforcement Learning with CIM on Base Models over COCO-LN500 [ 53 ] and DOCCI500 [ 48 ] . Best results are highlighted in bold. The results indicate that post-training the base model with our CIM enhances its ability to generate high-quality captions, and even 7B/8B models can outperform much larger baselines.

Base Model	Benchmark	Method	BLEU-4	METEOR	CAPTURE	Objects			Attributes			Relations
Base Model	Benchmark	Method	BLEU-4	METEOR	CAPTURE	Precision	Recall	F1	Precision	Recall	F1	QA
Qwen2-VL-7B [ 71 ]	COCO-LN500	$\mathbf{\circ}$ Base	39.57	20.42	46.52	81.12	61.82	69.47	66.48	42.86	48.68	20.47
		$\mathbf{\circ}$ SFT	34.93	25.97	47.63	78.03	70.42	73.43	67.91	53.51	56.31	30.06
		$\mathbf{\circ}$ SC-Captioner [ 82 ]	32.75	26.17	47.92	78.58	72.09	74.60	69.01	54.03	56.93	32.01
		$\bullet$ SFT + Ours	32.34	26.50	48.36	78.76	72.93	75.13	69.72	55.23	58.10	36.76
	DOCCI500	$\mathbf{\circ}$ Base	29.39	16.59	57.96	83.69	56.79	66.47	69.96	43.27	52.65	17.57

Table 7: Performance of Reinforcement Learning with CIM on the SFT Model, trained on the DOCCI [ 48 ] training set, over COCO-LN500 [ 53 ] and DOCCI500 [ 48 ] . Best results are highlighted in bold. Our model outperforms both the SFT model and SC-Captioner [ 82 ] .

Figure 7: Qualitative comparison between the base model and the base model trained with reinforcement learning using CIM. Green annotations indicate correct descriptions. Our method enhances the base model’s ability to produce more detailed captions.

Figure 8: Qualitative comparison among the base model, the SFT model, and the SFT model trained with reinforcement learning using CIM. Green annotations indicate correct descriptions, while red annotations indicate incorrect descriptions. Compared with the base model and the SFT model, our method produces more detailed captions while maintaining the precision of caption.

Image Captioning

Image captioning generates natural-language descriptions of visual content, with research focused on accuracy, coverage, grounding, and evaluation.

See all 94 papers

Re $^3$ Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning
Aug 21, 2026Haonan Jia, Shichao Dong, Zenghui Sun +7Image Captioning RL for VLMs
Taobao & Tmall Group of Alibaba · The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Artificial Intelligence Laboratory
CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning
Jun 8, 2026Penghui Yang, Long Xing, Xiaoyi Dong +10Image Captioning Multimodal Pretraining
Tsinghua University · University of Science and Technology of China · Microsoft +4
CycleCap: Improving VLMs Captioning Performance via Self-Supervised Cycle Consistency Fine-Tuning
Mar 18, 2026Marios Krestenitis, Christos Tzelepis, Konstantinos Ioannidis +5Cycle Consistency Image Captioning
Queen Mary, University of London · Centre for Research and Technology Hellas · City St George’s, University of London

Aug 21, 2026J/K move · Enter open · S save

Re $^3$ Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning

Haonan Jia, Shichao Dong, Zenghui Sun, Jiawen Zheng+6

Taobao & Tmall Group of Alibaba · The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Artificial Intelligence Laboratory

Image Captioning RL for VLMs

Open paper AskPDF arXiv

Reinforcement Learning (RL) has demonstrated significant gains in image captioning, yet it is still limited in encouraging Large Vision-Language Models (LVLMs) to explore novel reasoning strategies. This limitation leads to a performance gap between RL and Supervised Fine-Tuning (SFT). In this paper, we argue that multi-modal retrieval can serve as an effective reasoning signal for caption refinement. Based on this insight, we present the Retrieval-Guided Refinement for Image Captioning (Re

^3

Cap), a retrieval-guided reasoning strategy that enhances image captioning without requiring additional annotations. Instantiated by Caption Refinement Suggester (CRS) and Caption Quality Assessor (CQA), this strategy identifies hallucinations and omissions in image captions, leading to more accurate and detailed descriptions. Extensive experiments demonstrate the superiority of our method in image captioning, even compared with Supervised Fine-Tuning. Especially, Re

^3

Cap outperforms GRPO with an average improvement of 8.64% in relation reasoning on the COCO-LN500 benchmark.

Abstract

Image Captioning

Explore similar work

Re3^33Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning

CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning

CycleCap: Improving VLMs Captioning Performance via Self-Supervised Cycle Consistency Fine-Tuning

Re $^3$ Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning