ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs

Zitong Xu; Huiyu Duan; Shengyao Qin; Guangyu Yang; Guangji Ma; Xiongkuo Min; Ke Gu; Guangtao Zhai

cs.CVApr 4, 2026

ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs

Authors: Zitong Xu, Huiyu Duan, Shengyao Qin, Guangyu Yang, Guangji Ma, Xiongkuo Min, Ke Gu, Guangtao Zhai,

Organizations: Shanghai Jiao Tong University · University of Electronic and Science Technology of China · Beijing University of Technology · Institut Universitaire de France (IUF), University of Nantes

Abstract

Recent advances in multimodal large language models (MLLMs) have greatly improved image understanding and captioning capabilities. However, existing image captioning benchmarks typically suffer from limited diversity in caption length, the absence of recent advanced MLLMs, and insufficient human annotations, which potentially introduces bias and limits the ability to comprehensively assess the performance of modern MLLMs. To address these limitations, we present a new large-scale image captioning benchmark, termed, ICBench, which covers 12 content categories and consists of both short and long captions generated by 10 advanced MLLMs on 2K images, resulting in 40K captions in total. We conduct extensive human subjective studies to obtain mean opinion scores (MOSs) across fine-grained evaluation dimensions, where short captions are assessed in terms of fluency, relevance, and conciseness, while long captions are evaluated based on fluency, relevance, and completeness. Furthermore, we propose an automated evaluation metric, ITIScore, based on an image-to-text-to-image framework, which measures caption quality through reconstruction consistency. Experimental results demonstrate strong alignment between our automatic metric and human judgments, as well as robust zero-shot generalization ability on other public captioning datasets. Both the dataset and model will be released upon publication.

Figures & tables

Figure 1. Overview of our ICBench. (a) We first collect 2,040 source images across 12 fine-grained tasks. Then 10 advanced MLLMs are applied to generate 20,400 short captions and 20,400 long captions. (b) 122K MOSs are collected from 15 annotators across 4 evaluation dimensions.

Databases	Annotations	Images	Image Source	Short Captions	Long Captions	Models	Dimensions
Composite ( Aditya et al., 2015 )	59,925	3,995	Public Dataset	11,985	✗	2	Relevance
Flick8K-Expert ( Hodosh et al., 2013 )	28,320	5,664	Public Dataset	5,664	✗	✗	Relevance
Flick8K-CF ( Hodosh et al., 2013 )	143,490	1,000	Public Dataset	47,830	✗	✗	Relevance
Polaris ( Wada et al., 2024 )	1,048,160	13,691	Public Dataset	131,020	✗	10	Relevance
MMHE ( Ohi et al., 2024 )	18,000	400	Public Dataset	1,200	✗	10	Correctness, Completeness, Clarity, Fluency, and Conciseness
THumB ( Kasai et al., 2022 )	16,000	500	Public Dataset	2,000	✗	4	Precision, Recall

Table 1. Comparison of image captioning evaluation benchmarks.

Figure 2. MOS distribution of short caption and long caption across different evaluation dimensions.

Figure 3. Performance comparison of different MLLMs on short captioning in terms of fluency, relevance, and conciseness, and on long captioning in terms of fluency, relevance, and completeness across different image contents.

Figure 4. Overview of our ITIScore. Given an image and its caption, a pretrained generative model reconstructs an image from the caption. The original image, reconstructed image, and caption are jointly fed into a multimodal large language model to obtain a unified representation. A lightweight MLP scoring head then predicts the mean score and uncertainty for each evaluation dimension, enabling uncertainty-aware multi-dimensional caption quality assessment.

Subset	Short Caption						Long Caption
Dimension	Fluency		Relevance		Conciseness		Fluency		Relevance		Completeness		Overall
Method/Metric	$\tau_{b}$	$\tau_{c}$	$\tau_{b}$	$\tau_{c}$	$\tau_{b}$	$\tau_{c}$	$\tau_{b}$	$\tau_{c}$	$\tau_{b}$	$\tau_{c}$	$\tau_{b}$	$\tau_{c}$	$\tau_{b}$	$\tau_{c}$
UMIC ( Lee et al., 2021 )	32.17	31.94	18.43	17.21	6.88	5.67	29.76	28.62	7.12	6.97	8.83	7.69	17.20	16.35
CLIP-S ( Hessel et al., 2021 )	35.42	35.31	19.81	18.73	7.91	6.87	30.61	29.53	10.02	9.91	9.79	8.72	18.93	18.17
PAC-S ( Sarto et al., 2023 )	40.47	39.32	21.42	20.82	10.23	6.68	32.79	31.63	13.47	12.31	10.87	9.76	21.54	20.09
FLEUR ( Lee et al., 2024 )	36.18	35.77	22.69	21.56	6.13	5.03	31.53	30.42	11.83	10.72	8.12	7.01	19.41	18.42

Table 2. The Kendall correlation coefficient

\tau_{b}

and

\tau_{c}

between human judgments and various automatic metrics on ICBench. The first part is for reference-free methods, and the second part is advanced MLLMs. The best results are highlighted in red , and the second-best results are highlighted in blue .

Dimensions	Fluency				Relevance				Conciseness				Overall Rank
Models/Metrics	Human	Ours	GPT-5 ( OpenAI, 2025 )	FLEUR ( Lee et al., 2024 )	Human	Ours	GPT-5 ( OpenAI, 2025 )	FLEUR ( Lee et al., 2024 )	Human	Ours	GPT-5 ( OpenAI, 2025 )	FLEUR ( Lee et al., 2024 )	Human	Ours
ChatGPT-5 ( OpenAI, 2025 )	80.13	63.37	57.76	69.12	88.55	62.32	61.29	69.80	70.78	59.84	49.78	57.30	1	1
Gemini-3.0-Flash ( Google, 2025 )	74.36	60.94	55.49	67.20	84.57	60.53	58.25	67.17	77.97	58.72	54.41	61.41	2	2
InternVL3.5 ( Wang et al., 2025b )	70.67	60.04	52.88	65.69	81.81	58.99	50.38	60.03	68.52	56.71	45.34	56.11	3	3
Qwen3-VL ( Yang et al., 2025b )	65.04	59.17	51.45	61.92	77.42	56.90	52.34	64.91	73.60	57.46	51.42	54.70	4	4
DeepSeekVL2 ( Liu et al., 2024a )	67.69	58.25	54.06	63.96	79.52	58.70	54.67	63.94	66.44	55.49	46.65	54.38	5	5

Table 3. Comparisons of the alignment between different evaluation methods and human perception in short captions. The best results are highlighted in red , and the second-best results are highlighted in blue . The Spearman Rank Correlation Coefficient (SRCC) is reported.

Dimensions	Fluency				Relevance				Completeness				Overall Rank
Models/Metrics	Human	Ours	GPT-5 ( OpenAI, 2025 )	FLEUR ( Lee et al., 2024 )	Human	Ours	GPT-5 ( OpenAI, 2025 )	FLEUR ( Lee et al., 2024 )	Human	Ours	GPT-5 ( OpenAI, 2025 )	FLEUR ( Lee et al., 2024 )	Human	Ours
ChatGPT-5 ( OpenAI, 2025 )	74.62	50.73	51.72	64.54	76.57	50.44	63.53	52.74	62.83	47.88	56.31	47.69	1	1
Gemini-3.0-Flash ( Google, 2025 )	68.53	48.44	52.97	61.07	70.40	48.96	58.36	51.86	73.00	49.37	62.27	45.15	2	2
InternVL3.5 ( Wang et al., 2025b )	64.45	47.92	55.66	59.00	66.31	47.47	60.13	53.46	56.64	44.38	52.55	45.36	3	4
Qwen3-VL ( Yang et al., 2025b )	58.21	46.83	49.56	56.79	62.90	46.44	56.73	44.50	66.90	46.12	59.10	43.68	4	3
DeepSeekVL2 ( Liu et al., 2024a )	61.16	45.70	50.75	51.05	59.98	43.03	51.05	49.66	53.91	42.57	54.70	44.02	5	5

Table 4. Comparisons of the alignment between different evaluation methods and human perception in long captions. The best results are highlighted in red , and the second-best results are highlighted in blue . The Spearman Rank Correlation Coefficient (SRCC) is reported.

Backbone $\&$ Strategy					Short Caption			Long Caption
MLLM	Generation Model	Scoring Head	$\mathcal{L}_{dim}$	$\mathcal{L}_{agg}$	Fluency	Relevance	Conciseness	Fluency	Relevance	Completeness
InternVL3.5 (8B) ( Wang et al., 2025b )					47.57	44.38	37.96	46.73	40.49	43.24
InternVL3.5 (8B) ( Wang et al., 2025b )	Qwen-Image ( Wu et al., 2025 )				47.55	51.60	42.06	47.20	50.71	45.28
InternVL3.5 (8B) ( Wang et al., 2025b )		✓	Guassian NLL	✓	60.05	56.32	58.04	61.26	54.16	56.20
InternVL3.5 (8B) ( Wang et al., 2025b )	Qwen-Image ( Wu et al., 2025 )	✓	MSE	✓	59.57	56.55	57.28	60.84	55.82	56.50
InternVL3.5 (8B) ( Wang et al., 2025b )	Qwen-Image ( Wu et al., 2025 )	✓	L1	✓	58.84	56.46	56.70	60.00	55.52	56.45

Table 5. Ablation study on our ICBench with different backbones and training strategies. The Kendall correlation coefficient

\tau_{b}

with human judgments is reported.

Dataset	Composites ( Aditya et al., 2015 )		Flickr8k-CF ( Hodosh et al., 2013 )		Flickr8k-Expert ( Hodosh et al., 2013 )
Method/Metric	$\tau_{b}$	$\tau_{c}$	$\tau_{b}$	$\tau_{c}$	$\tau_{b}$	$\tau_{c}$
BLEU ( Papineni et al., 2002 )	29.0	31.3	32.2	32.3	17.9	9.3
ROUGE ( Lin, 2004 )	30.0	32.4	32.1	32.3	19.9	10.3
METEOR ( Banerjee and Lavie, 2005 )	36.0	38.9	41.5	41.8	22.2	11.5
CIDEr ( Vedantam et al., 2015 )	34.9	37.7	43.6	43.9	24.6	12.7
SPICE ( Anderson et al., 2016 )	38.8	40.3	51.7	44.9	24.4	12.0

Table 6. Comparison of zero-shot results for Kendall correlation coefficient

\tau_{b}

and

\tau_{c}

between human judgments and various automatic metrics on Composites ( Aditya et al., 2015 ) , Flickr8k-CF ( Hodosh et al., 2013 ) , Flickr8k-Expert ( Hodosh et al., 2013 ) datasets that have single evaluation dimension. The first part is for reference-based methods, the second part is for reference-free methods, and the third part is advanced MLLMs. The best results are highlighted in red , and the second-best results are highlighted in blue .

Dataset	MMHE ( Ohi et al., 2024 )			LongCap-Arena ( Liu et al., 2024c )
Method/Dimension	Flu.	Com.	Con.	Flu.	Desc.	Rel.
BLEU ( Papineni et al., 2002 )	8.8	8.7	10.7	14.5	30.3	6.3
ROUGE ( Lin, 2004 )	8.5	7.4	11.6	6.2	8.3	5.7
METEOR ( Banerjee and Lavie, 2005 )	9.1	9.8	6.6	4.6	11.2	8.1
CIDEr ( Vedantam et al., 2015 )	14.4	9.8	9.6	3.2	5.5	5.1
BERT-S ( Zhang et al., 2020 )	11.2	12.4	13.2	7.3	18.6	8.4

Table 7. Comparison of zero-shot results for Kendall correlation coefficient

\tau_{b}

between human judgments and various automatic metrics on MMSE ( Ohi et al., 2024 ) and LongCap-Arena ( Liu et al., 2024c ) datasets that have multiple evaluation dimensions. The first part is for reference-based methods, the second part is for reference-free methods, and the third part is advanced MLLMs. Flu: Fluency, Com: Completeness, Con: Conciseness, Desc: Descriptiveness, Rel: Relevance. The best results are highlighted in red , and the second-best results are highlighted in blue .

Image Captioning Evaluation

See all 16 papers

BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model
May 20, 2026Gonçalo Gomes, Bruno Martins, Chrysoula ZervaVLM Evaluation Image Captioning
Instituto Superior Técnico, University of Lisbon · INESC-ID · Instituto de Telecomunicações
BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning
May 8, 2026Shaokai Ye, Vasileios Saveris, Yihao Qian +3Multi-Objective Reinforcement Learning Image Captioning
Apple
Multimodal Language Models as Text-to-Image Model Evaluators
May 1, 2025Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat +5LLM-as-a-Judge Benchmark Design
1FAIR at Meta · University of Texas at Austin · ∗Work done at Meta +3

May 20, 2026J/K move · Enter open · S save

BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model

Gonçalo Gomes, Bruno Martins, Chrysoula Zerva

Instituto Superior Técnico, University of Lisbon · INESC-ID · Instituto de Telecomunicações

VLM Evaluation Image Captioning Vision-Language Models Image Captioning Evaluation

Open paper AskPDF arXiv

Image captioning evaluation remains a significant challenge, as vision-language models evolve toward more challenging capabilities such as generating long-form and context-rich descriptions. State-of-the-art evaluation metrics involve extensive computational costs associated with the use of Large Language Models (LLMs) as judges, or instead suffer from the limitations of standard CLIP-based encoders, such as strict token limits, lack of fine-grained sensitivity, or lack of compositional generalization by treating captions as ``bags-of-words.'' We propose a new learned metric that tackles the aforementioned challenges, based on a lightweight cross-encoder that is initialized from a visual question-answering model checkpoint, balancing a strong weight initialization with computational efficiency. Our training scheme uses a carefully assembled data mixture for supervised learning, featuring adversarial LLM-based data augmentations to enhance model sensitivity to fine-grained visual-linguistic errors. We also introduce a new benchmark designed to assess detailed captioning evaluation across diverse scenarios. Experimental results demonstrate that the proposed metric achieves state-of-the-art performance while maintaining the efficiency required for large-scale benchmarking, quality-aware decoding, or reward guidance.

Abstract

Image Captioning Evaluation

Explore similar work

BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model

BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning

Multimodal Language Models as Text-to-Image Model Evaluators