IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translation

Jiahao Lyu; Pei Fu; Zhenhang Li; Weichao Zeng; Shaojie Zhang; Jiahui Yang; Can Ma; Yu Zhou

cs.CVMar 11, 2026

IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translation

Authors: Jiahao Lyu, Pei Fu, Zhenhang Li, Weichao Zeng, Shaojie Zhang, Jiahui Yang, Can Ma, Yu Zhou,

Organizations: Institute of Information Engineering, Chinese Academy of Science, China · MiLM Plus, Xiaomi Inc, China · School of Cyber Security, University of Chinese Academy of Science, China · Binghamton University, USA ·

Abstract

End-to-end In-Image Machine Translation (IIMT) aims to convert text embedded within an image into a target language while preserving the original visual context, layout, and rendering style. However, existing IIMT benchmarks are largely synthetic and thus fail to reflect real-world complexity, while current evaluation protocols focus on single-modality metrics and overlook cross-modal faithfulness between rendered text and model outputs. To address these shortcomings, we present In-image Machine Translation Benchmark (IMTBench), a new benchmark of 2,500 image translation samples covering four practical scenarios and nine languages. IMTBench supports multi-aspect evaluation, including translation quality, background preservation, overall image quality, and a cross-modal alignment score that measures consistency between the translated text produced by the model and the text rendered in the translated image. We benchmark strong commercial cascade systems, and both closed- and open-source unified multi-modal models, and observe large performance gaps across scenarios and languages, especially on natural scenes and resource-limited languages, highlighting substantial headroom for end-to-end image text translation. We hope IMTBench establishes a standardized benchmark to accelerate progress in this emerging task.

Figures & tables

Figure 1: Comparison of existing IIMT benchmarks (Segpixels [ 35 ] , Translatotrion-V [ 11 ] ,IIMT30k [ 37 ] , PRIM [ 36 ] ) and our proposed IMTBench.

TIT Datasets
Dataset	Number	Languages	Parallel	Modality For Eval.	Real Scene
OCRMT-30K [ 54 ]	1.2k	2	✗	Text-Only	✓
MTIT6 [ 26 ]	6k	4	✗	Text-Only	✓
AibTrans [ 41 ]	7k	8	✓	Text-Only	✓
MIT-10M ⋆ [ 14 ]	10.4k	14	✓	Text-Only	✓
IIMT Datasets

Table 1: The comparison between the multi-modal translation dataset. ⋆ indicates the original paper reports the instance number, rather than the number of images.

Figure 2: Overview of the IMTBench dataset construction pipeline. The curation process consists of two main branches. The top branch Document $\&$ Web focuses on multilingual document translation with structured layouts. The medium branch Scene emphasizes instruction-driven editing of scene text in natural images. The bottom PowerPoint focuses on translation in presentation slide scenarios. All branches converge to form the final IMTBench dataset, which supports comprehensive evaluation of in-image machine translation across diverse scenarios.

Figure 3: Data samples of IMTBench, which includes 4 main scenarios, 9 languages and 2500 pairs with detailed annotations.

Figure 4: Dataset statistics of IMTBench. The balanced coverage of diverse scenarios and languages enables comprehensive evaluation of In-Image machine translation systems under varied visual and linguistic conditions.

Figure 5: Example illustrating the automatic evaluation metrics in IMTBench. The left column shows the source image and the ground-truth translated image. The three predicted results (from left to right) are generated by GPT-Image, Qwen-Image, and Tencent. For each prediction, we report metric scores and reasoning.

Metrics		Proprietary		Open-source				Cascaded
		GPT	Gemini	Qwen-Image	Janus-Pro	Bagel	UniWorld	Tencent	Youdao
Doc.	$S_{\text{text}}$	61.0	62.9	62.6	30.5	30.3	48.3	63.1	60.8
	$S_{\text{bg}}$	57.8	81.5	48.8	45.4	72.6	65.5	88.2	87.6
	$S_{\text{align}}$	17.4	1.27	6.7	10.1	9.7	4.0	90.1	92.4
	$S_{\text{vis}}$	72.1	70.3	75.6	57.9	67.5	68.4	75.5	73.7
Web.	$S_{\text{text}}$	79.7	76.8	74.2	25.8	28.6	59.4	77.2	73.1

Table 2: Results on IMTBench (%). We denote COMET as

S_{\text{text}}

, LPIPS as

S_{\text{bg}}

, alignment as

S_{\text{align}}

, and harmonization as

S_{\text{vis}}

. Best in each row is in bold.

Figure 6: Performance comparison of different UMMs across multiple languages. Results on the left panel show performance when varying the target language, while results on the right panel illustrate performance when varying the source output language. We show the number label of Tencent, GPT-Image, and Janus.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Word clouds in various languages of IMTBench.

Figure 8: Token length distribution across nine languages in IMTBench.

Image Text Translation

See all 8 papers

MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation
Mar 25, 2026Gengluo Li, Chengquan Zhang, Yupu Liang +9Image Text Translation Machine Translation
Institute of Information Engineering, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Tencent +3
UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translation
Jun 23, 2026Jiahao Lyu, Pei Fu, Zhenhang Li +6Image Text Translation Unified Multimodal Models
Institute of Information Engineering, Chinese Academy of Science, China · MiLM Plus, Xiaomi Inc, China · School of Cyber Security, University of Chinese Academy of Science, China +2
VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics
Sep 29, 2026Bo Lv, Mao Zheng, Zheng Li +3Multilingual VLM Evaluation Rubric-Based Evaluation
Foundation Model Department, Tencent

Mar 25, 2026J/K move · Enter open · S save

MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation

Gengluo Li, Chengquan Zhang, Yupu Liang, Huawen Shen+8

Institute of Information Engineering, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Tencent+3

Image Text Translation Machine Translation Multilingual VLM Evaluation MT Evaluation Multimodal CoT Reasoning

Open paper AskPDF arXiv

End-to-end text-image machine translation (TIMT), which directly translates textual content in images across languages, is crucial for real-world multilingual scene understanding. Despite advances in vision-language large models (VLLMs), robustness across diverse visual scenes and low-resource languages remains underexplored due to limited evaluation resources. We present MMTIT-Bench, a human-verified multilingual and multi-scenario benchmark with 1,400 images spanning fourteen non-English and non-Chinese languages and diverse settings such as documents, scenes, and web images, enabling rigorous assessment of end-to-end TIMT. Beyond benchmarking, we study how reasoning-oriented data design improves translation. Although recent VLLMs have begun to incorporate long Chain-of-Thought (CoT) reasoning, effective thinking paradigms for TIMT are still immature: existing designs either cascade parsing and translation in a sequential manner or focus on language-only reasoning, overlooking the visual cognition central to VLLMs. We propose Cognition-Perception-Reasoning for Translation (CPR-Trans), a data paradigm that integrates scene cognition, text perception, and translation reasoning within a unified reasoning process. Using a VLLM-driven data generation pipeline, CPR-Trans provides structured, interpretable supervision that aligns perception with reasoning. Experiments on 3B and 7B models show consistent gains in accuracy and interpretability. We will release MMTIT-Bench to promote the multilingual and multi-scenario TIMT research upon acceptance.

Abstract

Image Text Translation

Explore similar work

MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation

UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translation

VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics