MUNIChus: Multilingual News Image Captioning Benchmark

Yuji Chen; Alistair Plum; Hansi Hettiarachchi; Diptesh Kanojia; Saroj Basnet; Marcos Zampieri; Tharindu Ranasinghe

cs.CLMar 11, 2026

MUNIChus: Multilingual News Image Captioning Benchmark

Authors: Yuji Chen, Alistair Plum, Hansi Hettiarachchi, Diptesh Kanojia, Saroj Basnet, Marcos Zampieri, Tharindu Ranasinghe

Organizations: Lancaster University, UK · University of Luxembourg, Luxembourg · University of Surrey, UK · George Mason University, USA

Abstract

The goal of news image captioning is to generate captions by integrating news article content with corresponding images, highlighting the relationship between textual context and visual elements. The majority of research on news image captioning focuses on English, primarily because datasets in other languages are scarce. To address this limitation, we create the first multilingual news image captioning benchmark, MUNIChus, comprising 9 languages, including several low-resource languages such as Sinhala and Urdu. We evaluate various state-of-the-art neural news image captioning models on MUNIChus and find that news image captioning remains challenging. We also make MUNIChus publicly available with over 20 models already benchmarked. MUNIChus opens new avenues for further advancements in developing and evaluating multilingual news image captioning models.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of news and generic image captions for two different images. The generic image captions were generated by BLIP Li et al. (2022) .

	Arabic	Chinese	English	French	Hindi	Indonesian	Japanese	Sinhala	Urdu	All
Family	Afro-Asiatic (Semitic)	Sino-Tibetan ∗	Indo-European (Germanic)	Indo-European (Romance)	Indo-European (Indo-Aryan)	Austronesian	Japonic †	Indo-European (Indo-Aryan)	Indo-European (Indo-Aryan)	—
$\|$ Train $\|$	5,119	9,389	79,195	10,247	12,566	12,137	7,641	2,418	6,602	145,314
$\|$ Test $\|$	999	999	1,000	999	1,000	1,000	1,000	998	998	8,993
Unique Articles	2,289	2,922	38,558	2,853	2,760	1,952	3,805	1,046	2,478	58,663
Avg Images/Article	2.67	3.56	2.08	3.94	4.92	6.73	2.27	3.27	3.07	3.61
Avg Content Tokens	1,010	1,519	461	1,510	1,968	1,794	1,287	1,194	1,792	1,412

Table 1: Dataset statistics for MUNIChus across nine languages showing language family, number of images in train/test splits, unique articles, and average token counts for images per article, content, captions, and titles. Languages are categorized as high-resource , mid-resource , and low-resource following Joshi et al. (2020) . ∗ Chinese tokenised using Jieba. † Japanese tokenised using MeCab.

Panel of Figure 2 — Figure 2: Prompt template used for zero-shot image captioning. The {language} placeholder is replaced with the target language name (e.g., “English”, “Arabic”) at inference time.

	High-resource			Mid-resource			Low-resource				High-resource			Mid-resource			Low-resource
	En	Fr	Zh	Ar	Hi	Ja	Id	Si	Ur	All	En	Fr	Zh	Ar	Hi	Ja	Id	Si	Ur	All
section 3.1.1 Zero-shot
Aya-vision-8b	2.52	2.19	2.19	1.53	2.20	4.39	2.75	0.33	1.30	2.16	28.13	14.71	17.45	16.78	18.23	24.61	16.89	6.72	12.84	17.37
Aya-vision-32b	3.13	3.05	2.47	1.99	2.52	6.83	3.02	0.13	1.95	2.78	35.55	18.12	25.13	22.42	25.17	46.16	24.94	9.00	20.28	25.20
GPT-4o	2.54	1.71	1.48	2.52	3.54	3.26	2.36	1.15	2.18	2.31	39.64	11.31	22.29	29.21	39.13	28.18	24.21	16.42	26.19	26.25
Llama-3.2-11B	2.36	2.41	0.94	1.09	1.77	3.41	2.16	0.24	1.64	1.78	19.37	12.41	8.69	11.59	17.19	16.95	13.39	4.62	15.61	13.31

Table 2: BLEU-4 (left-side) Papineni et al. (2002) and CIDEr (right-side) Oliveira dos Santos et al. (2021) evaluation scores across different languages. The best result for each language (any method) is in bold .

Panel of Figure 3 — Figure 3: Comparison of the actual news image caption and the generated caption by the best model - GPT-4o random few-shot approach.

Panel of Figure 4 — Figure 4: Comparison of similar images retrieved for the similar shot approach. The first image is the test instance, and the rest of the images are the most similar images retrieved from the training set.

Image Captioning

Image captioning generates natural-language descriptions of visual content, with research focused on accuracy, coverage, grounding, and evaluation.

See all 96 papers

VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics
Sep 29, 2026Bo Lv, Mao Zheng, Zheng Li +3Multilingual VLM Evaluation Rubric-Based Evaluation
Foundation Model Department, Tencent
MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation
Mar 25, 2026Gengluo Li, Chengquan Zhang, Yupu Liang +9Image Text Translation Machine Translation
Institute of Information Engineering, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Tencent +3
A Benchmarking Methodology to Assess Open-Source Video Large Language Models in Automatic Captioning of News Videos
Mar 29, 2026David Miranda Paredes, Jose M. Saavedra, Marcelo PizarroVideo Caption Evaluation Video-Language Model Evaluation
Facultad de Ingenier´ıa y Ciencias Aplicadas, Universidad de los Andes, Chile, Av. Monse˜nor ´Alvaro del Portillo 12455, Santiago, 100190, RM, Chile.

Sep 29, 2026J/K move · Enter open · S save

VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics

Bo Lv, Mao Zheng, Zheng Li, Fangxu Liu+2

Foundation Model Department, Tencent

Multilingual VLM Evaluation Rubric-Based Evaluation MT Evaluation

Open paper AskPDF arXiv

Image translation is a fundamental capability of multimodal models for multilingual applications, requiring visual understanding and meaning preservation across languages. However, existing benchmarks have limited language coverage and often lack explicit image-specific evaluation criteria, making it difficult to comprehensively assess this capability. To systematically evaluate this capability, we introduce VISTA-Bench, covering 22 languages and 10 domains, and develop an image-specific rubric evaluation protocol. The benchmark combines sampling for language and scenario coverage with model-assisted, human-verified annotations that group related text into coherent semantic units and provide multilingual reference translations. The rubrics specify essential content, semantic relations, and acceptable translation variants, yielding separate output-based scores for translation quality and the preservation of visual and knowledge-dependent information. We conduct extensive evaluations of 16 mainstream models, including 12 multimodal models and four text-input models, and provide systematic analyses across languages, domains, and evaluation dimensions.

Abstract

Image Captioning

Explore similar work

VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics

MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation

A Benchmarking Methodology to Assess Open-Source Video Large Language Models in Automatic Captioning of News Videos