ImgMT: Wikipedia-based Dataset For Evaluating Text-in-Image Translation

Šárka Uramová; Petar Kirilov Petrov; Josef Jon; Ondřej Bojar; Michal Novák

cs.CLOct 26, 2026

ImgMT: Wikipedia-based Dataset For Evaluating Text-in-Image Translation

Authors: Šárka Uramová, Petar Kirilov Petrov, Josef Jon, Ondřej Bojar, Michal Novák

Organizations: Charles University, Faculty of Mathematics and Physics, Prague, Czech Republic

Abstract

Text-in-image translation aims to translate all textual content embedded in an image. In Text Image Translation (TIT), the output is translated text; In-Image Machine Translation (IIMT) additionally renders the translation back into the image. We introduce ImgMT, a massively multilingual dataset and benchmark mined from localized SVG illustrations on Wikimedia Commons, focusing on diagrams, maps, and infographics. ImgMT contains 35,612 aligned image pairs grouped into 1,855 image sets, spanning 129 languages, 20+ writing systems, and 4,886 language pairs. We propose a multi-level evaluation protocol that measures source-side text-region detection and OCR accuracy alongside target-side translation quality. We benchmark several multimodal large language models and a dedicated TIT system in English->X and X->English settings, and find that the source language and writing system substantially affect TIT quality. Finally, we show that ImgMT is not well suited for isolating the contribution of visual information in multimodal translation.

Figures & tables

Figure 1: ImgMT languages across different scripts.

Figure 2: Text Image Translation evaluation diagram.

Category	Count	Share
Proper name	126,835	60.2%
Common noun phrase	79,175	37.6%
Statement without finite verb	1,964	0.9%
List of common noun phrases	1,003	0.5%
Single clause	943	0.4%
List of proper names	493	0.2%

Table 1: Distribution of string types (LLM-based categorization). Shares are computed over the consensus subset.

Figure 3: Pipeline overview for generating the multilingual ImgMT dataset

Alignment Method	Train	Dev	Test
1. svg_switch	6,514	238	328
2. svg_id_match	366	41	28
3. unamb_spatial_overlap	8,842	1,223	1,065
4. amb_spatial_overlap	12,771	2,042	2,154
Theoretically achievable image pairs			118,000
Total JSON metadata files			35,612

Table 2: Success of text alignment methods across dataset splits. The methods are carried out in the specified order and an image pair is counted with the method which succeeds first.

Bounding boxes	Count			Area
Bounding boxes	P	R	F1	P	R	F1
All boxes	96.2%	90.6%	91.5%	94.9%	86.8%	89.0%
Matched boxes	96.2%	90.2%	91.3%	94.9%	86.6%	88.9%
Correctly matched	92.7%	86.9%	88.3%	91.6%	83.3%	86.0%

Table 3: Manual evaluation of dataset quality. Rows correspond to (i) all detected text-field boxes ( All boxes ), (ii) boxes for which a one-to-one cross-image correspondence exists ( Matched boxes ), and (iii) boxes whose correspondence is the correct cross-lingual match ( Correctly matched ). Columns report precision/recall/F1 for box count (BBC) and area overlap (BBA).

Model	chrF	COMET	CER	BBA	BBC
Qwen3.5	59.86	0.827	0.313	0.453	0.746
Kimi K3	61.30	0.834	0.174	0.620	0.829
GPT-5.6 Luna	59.72	0.826	0.224	0.655	0.838
Claude Haiku 4.5	39.78	0.681	0.822	0.203	0.542
AnyTrans	57.50	0.803	0.161	0.620	0.813

Table 4: Translation and OCR performance of multimodal LLMs on English-source language pairs.

Source Script	$N$	CER ( $\Delta$ )	BBA ( $\Delta$ )	BBC ( $\Delta$ )	chrF ( $\Delta$ )	COMET ( $\Delta$ )
Latin	255	-0.106	+0.028	+0.011	+7.38	+0.035
Dravidian	49	+0.359	-0.076	+0.005	-26.07	-0.129
Cyrillic	48	+0.094	+0.015	+0.002	+8.90	+0.073
Devanagari	25	-0.002	-0.006	+0.027	-3.12	-0.001
Bengali-Assamese	22	-0.025	-0.035	-0.068	-15.80	-0.098
Arabic	13	-0.129	-0.253	-0.229	-21.78	-0.132

Table 5: Overall absolute scores, with per-script deltas shown in parentheses (row minus overall). Green indicates improvement (for CER: lower is better), red indicates degradation.

Model	Image	No image	Text-only image
Claude Haiku 4.5	57.94	58.04	57.60
GPT-5.6 Luna	62.30	62.29	62.46
Kimi K3	62.95	62.76	63.08
Qwen3.5	57.43	60.18 ∗	57.84

Table 6: chrF scores on the test split, computed with unaligned (line-order) matching, with paired bootstrap significance testing against the baseline condition. ∗ means significant at

\alpha=0.05

. All of the conditions use the gold, extracted texts from the SVGs, Image additionally uses the source image, Text-only image additionally uses only the text layer of the input image, and No image translates the extracted text without ever seeing the image.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Lang	Writing System	Pairs	Targets	Top Paired Languages
en	Latin	5,089	121	es (386), fr (385), ru (325), de (271), it (187)
es	Latin	3,461	122	en (386), fr (274), pt (191), ru (177), de (171)
fr	Latin	3,253	124	en (385), es (274), de (184), ru (166), it (150)
ru	Cyrillic	2,591	117	en (325), es (177), fr (166), de (139), it (106)
de	Latin	2,568	121	en (271), fr (184), es (171), ru (139), it (95)
it	Latin	2,229	114	en (187), es (169), fr (150), ru (106), de (95)

Appendix

Table 7: Overview of all languages covered in the dataset, their writing systems, total translation pairs, number of unique paired target languages, and top paired counterpart languages.

Figure 6: A screenshot of the tool used for manual annotation

Image Text Translation

See all 9 papers

IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translation
Mar 11, 2026Jiahao Lyu, Pei Fu, Zhenhang Li +7Image Text Translation Multimodal MT
Institute of Information Engineering, Chinese Academy of Science, China · MiLM Plus, Xiaomi Inc, China · School of Cyber Security, University of Chinese Academy of Science, China +3
MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation
Mar 25, 2026Gengluo Li, Chengquan Zhang, Yupu Liang +9Image Text Translation Machine Translation
Institute of Information Engineering, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Tencent +3
Global-Local Dual Perception for MLLMs in High-Resolution Text-Rich Image Translation
Feb 25, 2026Junxin Lu, Tengfei Song, Zhanglin Wu +9Image Text Translation Machine Translation
School of Communication and Information Engineering, Shanghai University, Shanghai 200444, China · Huawei Translation Services Center, China · School of Computer Science and Technology, East China Normal University, Shanghai 200062, China +1

Mar 11, 2026J/K move · Enter open · S save

IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translation

Jiahao Lyu, Pei Fu, Zhenhang Li, Weichao Zeng+6

Institute of Information Engineering, Chinese Academy of Science, China · MiLM Plus, Xiaomi Inc, China · School of Cyber Security, University of Chinese Academy of Science, China+3

Image Text Translation Multimodal MT Machine Translation MT Evaluation

Open paper AskPDF arXiv

End-to-end In-Image Machine Translation (IIMT) aims to convert text embedded within an image into a target language while preserving the original visual context, layout, and rendering style. However, existing IIMT benchmarks are largely synthetic and thus fail to reflect real-world complexity, while current evaluation protocols focus on single-modality metrics and overlook cross-modal faithfulness between rendered text and model outputs. To address these shortcomings, we present In-image Machine Translation Benchmark (IMTBench), a new benchmark of 2,500 image translation samples covering four practical scenarios and nine languages. IMTBench supports multi-aspect evaluation, including translation quality, background preservation, overall image quality, and a cross-modal alignment score that measures consistency between the translated text produced by the model and the text rendered in the translated image. We benchmark strong commercial cascade systems, and both closed- and open-source unified multi-modal models, and observe large performance gaps across scenarios and languages, especially on natural scenes and resource-limited languages, highlighting substantial headroom for end-to-end image text translation. We hope IMTBench establishes a standardized benchmark to accelerate progress in this emerging task.

Abstract

Image Text Translation

Explore similar work

IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translation

MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation

Global-Local Dual Perception for MLLMs in High-Resolution Text-Rich Image Translation