VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics

Bo Lv; Mao Zheng; Zheng Li; Fangxu Liu; Mingrui Sun; Tao Chen

cs.CVSep 29, 2026

VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics

Authors: Bo Lv, Mao Zheng, Zheng Li, Fangxu Liu, Mingrui Sun, Tao Chen

Organizations: Foundation Model Department, Tencent

Abstract

Image translation is a fundamental capability of multimodal models for multilingual applications, requiring visual understanding and meaning preservation across languages. However, existing benchmarks have limited language coverage and often lack explicit image-specific evaluation criteria, making it difficult to comprehensively assess this capability. To systematically evaluate this capability, we introduce VISTA-Bench, covering 22 languages and 10 domains, and develop an image-specific rubric evaluation protocol. The benchmark combines sampling for language and scenario coverage with model-assisted, human-verified annotations that group related text into coherent semantic units and provide multilingual reference translations. The rubrics specify essential content, semantic relations, and acceptable translation variants, yielding separate output-based scores for translation quality and the preservation of visual and knowledge-dependent information. We conduct extensive evaluations of 16 mainstream models, including 12 multimodal models and four text-input models, and provide systematic analyses across languages, domains, and evaluation dimensions.

Figures & tables

Figure 1: Overview of VISTA-Bench. Stages 1–3 construct language–scenario coverage, human-reviewed semantic blocks, and multilingual references with image-specific rubrics. Stage 4 separates the model’s image–target input from evaluator-only annotations, then converts evidence-based judgments into separate importance-weighted V, T, and K scores.

Dataset	Test Size	Languages (source $\rightarrow$ target)	# Scenarios	Evaluation
Vistra	772	En $\rightarrow$ De/Es/Ru/Zh	No scenario breakdown	BLEU, chrF, COMET
MMTIT-Bench	1,400	14 languages $\rightarrow$ En/Zh	3 (broad types)	COMET + VLLM judge
PRIM	340	En $\rightarrow$ De/Fr/Cs/Ru/Ro	No scenario breakdown	BLEU, COMET
IIMT30K	2,740	En $\leftrightarrow$ De	No scenario breakdown	BLEU, COMET
PATIMT-Bench	1,200	En $\leftrightarrow$ Zh	10 (6 eval. groups)	BLEU, COMET
VISTA-Bench (Ours)	2,228	22 languages $\rightarrow$ 22 languages	100+ (10 domains)	Image-specific rubric judge

Table 1: Comparison of image translation benchmarks and evaluation protocols.

Figure 2: Human revision rates for ten selected widely used languages.

Figure 3: Rubric structure for an automatic-gate notice. Only T2 is expanded, with condensed wording. The knowledge dimension is inapplicable to this image.

Image-input: closed-weight models
Model	Overall	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	55.88	51.00	60.03	53.44	54.38	61.84	60.94	58.92	58.59	58.35	63.55
Claude Sonnet 5	69.43	64.71	71.78	65.71	71.14	74.18	77.94	67.08	71.38	78.13	77.82
Muse Spark 1.3	71.56	64.09	71.98	70.13	71.99	77.43	77.59	73.70	76.86	72.75	73.95
Qwen3.8 Max	72.67	66.00	71.94	72.57	72.89	76.20	81.07	74.67	75.57	76.43	75.40
Gemini 3.8 Flash	74.20	69.10	75.62	74.57	76.22	79.52	79.12	72.64	76.34	78.40	78.17

Table 2: Translation-quality (T) results. Overall denotes global T, not the composite score. Global T averages directions for image-input models and tasks for text-input models. Tinted cells mark the highest score within each group and column.

Figure 4: Source-language translation profiles for six multimodal models. Each point is the mean T for one source language over its target-language directions. Dashed lines mark the per-model average across 22 languages; teal and orange denote scores above and below that average.

Figure 5: Paired T–V (left) and T–K (right) scores, ordered by increasing T from top to bottom. Lines connect the same model’s scores; columns report values to two decimals. Correlations use all 12 models.

Pearson $r=0.99996$ Spearman $\rho=1.0000$ MAE $=0.88$
Model	Full set (10,920)	Selected set (2,228)	$\lvert\Delta T\rvert$
GLM-4.1V 9B	30.7	30.0	0.7
Gemma-4 E4B	20.0	19.5	0.5
Claude Sonnet 5	70.2	69.4	0.8
GPT-6 Astra	79.1	77.6	1.5

Table 3: Translation-quality scores on the full and selected sets. Statistics are computed from the reported one-decimal scores.

Figure 6: Human–judge agreement and score deviations. Left: each point is one candidate model’s mean T under an automatic judge versus Human; the dashed line marks equal scores. Right: automatic-minus-human differences. Correlations use four model-level means, not individual images.

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Source-image counts across 22 languages. Each of the 2,228 images is counted once; simplified and traditional Chinese are combined. Sector areas are proportional to image counts, and the legend reports absolute counts.

ID	English source	French reference
1	5 grams	5 grammes
2	TRINOLONE ORAL PASTE	PÂTE BUCCALE TRINOLONE
3	Triamcinolone Acetonide in Dental Paste	Acétonide de triamcinolone dans une pâte dentaire
4	NIDA PHARMA INCORPORATION	NIDA PHARMA INCORPORATION
5	Wang Noi, Ayutthaya, Thailand.	Wang Noi, Ayutthaya, Thaïlande.
6	NIDA	NIDA

Appendix

Table B1: Complete source blocks and stored French references for the package label.

Field	Complete content
V1 / Visual understanding
Metadata	Importance: core; check type: coverage; source blocks: 1, 2, 3, 4, 5, 6
Expected 1	The output gives a meaning-bearing representation of every source block: the quantity, product name, ingredient-and-dosage-form line, manufacturer, location, and standalone NIDA identifier.
Expected 2	The output preserves the meaningful grouping of the package label, while allowing blocks to be merged, reordered, or laid out naturally in the target language.
Accepted 1	Repeated NIDA text may be represented together with the manufacturer name or separately as the logo, provided both the company identifier and standalone identifier remain recoverable.
Forbidden 1	Omitting any source block or treating visible label text as decorative.

Appendix

Table B2: Complete shared rubric criteria for the English-to-French package example.

Field	Complete content
TOL1
Source blocks	4
Term links	Empty
Knowledge links	Empty
Rule	“NIDA PHARMA INCORPORATION” has no single required official translation. Accept retention, reasonable transliteration, semantic translation, or a readable target-language rendering, provided NIDA remains identifiable and the organization is understood as a pharmaceutical company or corporation.
French forms	Empty

Appendix

Table B3: Complete tolerances and their French accepted forms. Empty lists have no prescribed French surface form.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	47.84	53.52	52.11	59.52	54.49	56.78	60.55	56.52	45.87	68.36
Claude Sonnet 5	68.75	70.56	60.98	69.47	70.74	80.30	59.81	69.70	79.02	79.42
Muse Spark 1.3	64.45	71.65	65.71	70.75	74.88	74.24	73.39	73.79	72.02	70.85
Qwen3.8 Max	66.86	77.01	74.74	73.56	71.96	79.37	75.42	74.40	74.95	78.97
Gemini 3.8 Flash	69.30	78.49	76.21	76.10	81.22	79.31	70.11	75.40	74.65	78.63

Appendix

Table C1: Domain-level T for translations into English, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	51.99	62.06	53.85	53.89	58.47	67.32	63.22	61.58	56.54	62.79
Claude Sonnet 5	62.26	69.55	67.30	69.31	80.39	81.80	65.21	73.34	76.83	78.43
Muse Spark 1.3	65.40	70.50	74.25	74.07	78.49	85.75	78.11	80.21	75.27	69.65
Qwen3.8 Max	64.05	68.54	72.73	75.86	79.96	78.34	78.09	79.46	72.52	72.46
Gemini 3.8 Flash	71.65	72.82	75.59	75.70	81.69	72.47	74.12	77.51	78.87	78.03

Appendix

Table C2: Domain-level T for translations into Czech, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	46.92	59.29	54.80	54.31	58.69	55.42	60.78	57.96	60.71	66.01
Claude Sonnet 5	58.88	66.76	72.90	69.90	70.98	66.78	61.28	72.60	80.59	80.16
Muse Spark 1.3	60.23	67.23	75.64	70.51	74.10	69.53	73.77	75.58	68.24	75.62
Qwen3.8 Max	66.11	70.08	73.15	74.48	74.00	77.90	72.67	73.70	76.11	76.02
Gemini 3.8 Flash	63.00	74.57	77.07	77.90	77.34	71.88	75.14	75.22	79.38	84.81

Appendix

Table C3: Domain-level T for translations into German, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	56.10	54.62	50.34	51.26	65.73	52.69	61.21	60.52	54.21	68.22
Claude Sonnet 5	60.27	66.03	64.22	66.57	69.34	76.75	68.29	72.37	81.99	77.89
Muse Spark 1.3	65.38	69.98	72.79	69.38	74.58	80.39	73.83	80.25	73.88	76.19
Qwen3.8 Max	67.17	68.66	77.26	72.23	72.29	79.99	75.04	76.98	73.49	78.32
Gemini 3.8 Flash	66.86	75.20	73.29	72.01	78.35	80.04	73.47	74.07	79.48	79.66

Appendix

Table C4: Domain-level T for translations into Dutch, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	50.71	62.74	46.81	53.15	58.91	60.68	58.64	56.77	55.56	61.20
Claude Sonnet 5	64.53	74.04	64.49	67.63	76.33	74.34	66.20	64.84	77.63	76.93
Muse Spark 1.3	61.45	73.92	69.17	73.87	77.84	78.52	72.06	73.72	73.87	75.56
Qwen3.8 Max	69.23	72.73	69.81	73.29	79.27	80.97	75.17	72.23	80.17	76.89
Gemini 3.8 Flash	74.79	79.27	71.94	78.02	81.11	68.26	70.79	75.56	81.31	79.50

Appendix

Table C5: Domain-level T for translations into French, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	51.15	52.68	56.30	56.37	53.11	65.54	58.94	58.35	63.38	66.80
Claude Sonnet 5	66.45	69.15	67.83	74.95	74.13	80.47	66.09	70.73	79.89	79.18
Muse Spark 1.3	67.49	73.43	72.80	71.16	80.29	83.58	74.11	76.06	71.88	76.86
Qwen3.8 Max	69.62	69.15	71.37	74.12	72.68	83.46	73.94	69.47	82.89	76.20
Gemini 3.8 Flash	68.04	70.85	76.88	73.64	81.73	83.18	73.64	74.75	79.56	75.38

Appendix

Table C6: Domain-level T for translations into Italian, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	51.28	48.55	46.54	58.18	63.60	60.88	58.51	57.93	54.90	61.65
Claude Sonnet 5	65.55	72.13	59.53	70.83	76.28	75.98	66.52	71.48	77.16	73.35
Muse Spark 1.3	60.98	69.00	70.18	67.79	77.37	75.48	70.90	75.89	71.66	60.01
Qwen3.8 Max	65.61	69.48	73.88	72.35	75.04	83.35	71.26	72.71	79.08	75.27
Gemini 3.8 Flash	71.03	72.99	76.13	77.58	80.03	75.98	70.48	70.00	76.38	74.80

Appendix

Table C7: Domain-level T for translations into Spanish, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	54.00	64.66	56.22	53.17	65.77	50.64	54.25	56.66	61.39	57.18
Claude Sonnet 5	66.51	75.33	62.60	66.91	70.62	73.48	65.38	68.36	74.60	76.38
Muse Spark 1.3	67.39	79.13	75.69	69.49	75.55	77.22	73.60	74.56	71.93	71.77
Qwen3.8 Max	66.97	76.50	68.87	71.68	72.79	76.09	72.64	68.61	67.48	75.23
Gemini 3.8 Flash	72.07	81.57	71.01	75.71	71.89	71.52	67.38	76.17	79.43	80.05

Appendix

Table C8: Domain-level T for translations into Portuguese, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	54.57	61.43	52.45	48.44	65.02	76.85	59.35	61.19	48.24	65.06
Claude Sonnet 5	69.07	72.55	67.34	72.84	72.40	83.04	64.93	68.21	78.38	78.51
Muse Spark 1.3	72.48	72.49	72.37	72.93	78.86	82.16	75.75	77.95	67.31	78.90
Qwen3.8 Max	68.21	68.37	72.60	72.42	76.56	92.32	77.98	75.57	76.09	78.96
Gemini 3.8 Flash	73.16	76.83	76.66	74.05	78.47	88.45	71.77	72.39	75.95	77.14

Appendix

Table C9: Domain-level T for translations into Turkish, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	50.20	61.95	53.10	58.97	65.55	58.42	55.05	59.78	61.44	65.03
Claude Sonnet 5	62.29	67.75	70.08	72.00	75.56	75.34	63.80	68.95	76.96	77.02
Muse Spark 1.3	61.18	65.10	70.86	73.91	76.56	67.28	76.05	80.02	67.75	75.86
Qwen3.8 Max	63.48	71.16	74.45	74.67	73.70	77.44	70.45	74.02	72.91	74.00
Gemini 3.8 Flash	69.92	74.67	77.94	76.62	78.13	76.79	67.29	76.75	77.25	81.77

Appendix

Table C10: Domain-level T for translations into Indonesian, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	51.21	73.24	51.26	52.13	70.45	61.59	57.03	59.64	67.70	65.06
Claude Sonnet 5	68.08	73.69	66.45	71.08	76.03	78.30	69.83	73.02	80.82	77.61
Muse Spark 1.3	64.72	78.90	71.04	74.03	80.35	82.11	74.53	75.04	66.87	74.00
Qwen3.8 Max	66.87	75.71	75.40	72.61	84.72	83.99	75.63	74.11	80.56	80.01
Gemini 3.8 Flash	65.56	81.84	74.21	74.04	80.90	78.35	77.55	79.70	79.03	79.59

Appendix

Table C11: Domain-level T for translations into Malay, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	53.32	56.50	53.91	50.16	65.23	63.77	63.54	55.64	56.24	56.42
Claude Sonnet 5	66.51	67.63	68.32	69.65	80.13	78.09	69.57	70.40	81.15	74.65
Muse Spark 1.3	61.23	65.25	64.92	67.27	77.24	76.28	70.10	72.68	67.62	69.99
Qwen3.8 Max	67.54	68.42	72.56	70.14	81.52	82.58	77.54	78.94	74.66	73.99
Gemini 3.8 Flash	70.57	69.75	79.60	74.73	79.57	85.75	74.64	75.48	77.65	71.80

Appendix

Table C12: Domain-level T for translations into Vietnamese, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	54.88	57.79	56.77	51.38	58.78	61.18	59.48	60.75	53.89	65.41
Claude Sonnet 5	60.93	67.96	69.45	68.12	72.99	86.11	67.28	70.69	77.26	82.91
Muse Spark 1.3	62.91	65.77	71.11	64.69	75.48	77.11	74.43	74.60	75.42	77.68
Qwen3.8 Max	61.38	68.84	74.70	71.06	75.61	82.84	73.93	76.62	72.80	79.36
Gemini 3.8 Flash	71.06	73.62	76.99	75.04	77.44	81.45	73.82	73.36	79.85	80.45

Appendix

Table C13: Domain-level T for translations into Russian, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	50.62	55.95	57.08	51.68	66.99	61.70	58.62	61.54	58.84	68.56
Claude Sonnet 5	63.65	70.97	69.23	72.50	75.55	84.77	67.02	75.66	76.18	75.81
Muse Spark 1.3	63.24	73.03	72.51	74.59	80.24	85.12	75.10	78.42	76.73	74.38
Qwen3.8 Max	64.33	73.02	75.20	73.47	77.18	79.43	73.70	79.68	77.69	72.90
Gemini 3.8 Flash	67.49	76.53	81.04	79.00	73.27	78.63	72.70	76.95	80.42	81.25

Appendix

Table C14: Domain-level T for translations into Ukrainian, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	50.59	70.51	52.63	54.33	66.52	61.41	60.13	58.03	69.43	70.26
Claude Sonnet 5	61.33	77.74	65.57	72.09	76.00	76.31	67.75	70.54	81.41	76.22
Muse Spark 1.3	64.52	76.66	73.66	70.98	73.69	73.14	75.25	79.23	76.07	78.77
Qwen3.8 Max	70.83	78.18	77.24	68.66	76.21	81.53	72.65	75.52	74.95	76.47
Gemini 3.8 Flash	68.80	79.58	72.17	73.15	78.15	77.95	70.25	75.27	81.22	79.73

Appendix

Table C15: Domain-level T for translations into Arabic, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	49.07	63.30	54.12	60.05	72.39	65.82	55.80	61.39	54.78	65.79
Claude Sonnet 5	62.91	73.90	63.77	73.05	70.51	78.62	72.10	75.49	73.38	79.31
Muse Spark 1.3	60.96	76.87	72.31	77.09	77.91	81.77	68.83	82.72	59.65	68.85
Qwen3.8 Max	66.41	79.03	75.59	74.55	76.20	81.52	74.41	77.18	72.32	74.67
Gemini 3.8 Flash	72.05	78.47	78.14	78.01	81.75	81.13	76.46	81.03	76.41	74.14

Appendix

Table C16: Domain-level T for translations into Hebrew, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	52.15	62.03	61.15	56.58	57.41	62.59	57.82	65.83	58.81	70.12
Claude Sonnet 5	66.79	71.19	69.95	74.08	74.48	72.47	68.37	78.04	77.44	85.33
Muse Spark 1.3	70.52	67.92	64.14	71.20	72.62	76.14	75.32	76.01	79.36	78.47
Qwen3.8 Max	71.47	72.91	74.82	76.61	79.04	77.72	73.31	75.71	82.70	76.22
Gemini 3.8 Flash	71.52	75.70	68.60	71.74	78.74	84.33	71.16	79.70	74.76	78.48

Appendix

Table C17: Domain-level T for translations into Hindi, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	54.15	73.71	64.82	66.25	67.65	67.15	67.63	65.10	63.20	70.33
Claude Sonnet 5	71.29	74.75	75.34	72.94	80.63	79.49	73.41	76.86	83.64	86.43
Muse Spark 1.3	68.34	80.47	80.05	78.21	79.96	79.83	82.45	80.02	76.65	83.20
Qwen3.8 Max	67.95	60.17	62.74	64.68	68.66	71.01	76.46	69.94	74.66	71.80
Gemini 3.8 Flash	71.61	74.54	81.12	80.50	84.25	85.57	79.48	78.53	82.72	84.25

Appendix

Table C18: Domain-level T for translations into Burmese, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	52.03	56.75	63.56	50.12	56.46	63.22	61.72	61.54	72.64	71.60
Claude Sonnet 5	66.55	68.81	70.45	73.12	65.33	80.62	68.11	72.11	78.66	79.09
Muse Spark 1.3	61.69	67.80	70.92	74.39	76.71	76.59	73.17	79.17	78.89	78.18
Qwen3.8 Max	64.33	67.85	75.37	72.98	73.78	84.80	73.88	80.17	76.18	75.33
Gemini 3.8 Flash	69.62	75.62	77.33	75.53	79.46	83.78	77.94	78.91	80.37	77.32

Appendix

Table C19: Domain-level T for translations into Thai, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	55.09	71.89	52.86	53.73	69.54	57.69	64.59	60.96	59.78	58.80
Claude Sonnet 5	67.25	76.49	69.70	69.54	75.52	77.56	69.45	73.16	73.53	73.76
Muse Spark 1.3	69.30	74.77	71.80	69.89	76.75	75.52	71.80	78.12	77.71	72.08
Qwen3.8 Max	68.85	75.25	80.05	71.42	78.23	82.11	73.18	81.40	84.17	77.47
Gemini 3.8 Flash	69.92	73.14	72.34	75.80	79.13	80.72	71.46	76.64	78.01	74.87

Appendix

Table C20: Domain-level T for translations into Japanese, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	52.59	60.55	53.10	53.14	62.94	60.83	57.85	57.44	49.98	66.25
Claude Sonnet 5	66.30	77.85	67.88	70.04	74.13	83.75	67.74	72.53	71.24	80.76
Muse Spark 1.3	62.99	71.06	73.24	72.31	72.03	76.10	71.67	78.63	65.54	75.46
Qwen3.8 Max	67.83	70.18	73.07	70.33	72.88	84.41	75.60	76.92	67.01	77.25
Gemini 3.8 Flash	70.37	77.34	77.53	78.38	74.21	81.42	67.57	80.98	77.32	80.40

Appendix

Table C21: Domain-level T for translations into Korean, averaged over available source languages.

Closed-weight models
Model	Transport	Dining	Products	Tourism	Health care	Safety	Social Media	Publi- cations	Finance	Public Services
GPT-5.6 Luna	49.37	56.22	53.68	58.68	67.49	60.49	55.79	58.46	58.75	58.38
Claude Sonnet 5	65.09	71.39	57.18	70.30	69.78	73.55	67.91	68.84	75.54	78.87
Muse Spark 1.3	63.50	74.83	67.63	72.17	75.90	76.83	73.36	76.09	75.93	76.73
Qwen3.8 Max	63.24	70.70	69.89	73.30	74.88	80.43	76.25	74.63	75.96	71.67
Gemini 3.8 Flash	64.06	72.17	73.37	76.63	77.34	75.43	70.11	74.29	74.91	76.04

Appendix

Table C22: Domain-level T for translations into Chinese, averaged over available source languages.

Src $\backslash$ Tgt	en	cs	de	nl	fr	it	es	pt	tr	id	ms	vi	ru	uk	ar	he	hi	my	th	ja	ko	zh
en	–	18.4	26.2	23.3	15.8	19.2	23.0	19.8	18.4	22.7	19.3	19.9	22.6	26.4	23.1	17.1	18.8	27.6	23.1	24.2	20.6	21.3
cs	22.3	–	17.0	15.4	27.3	23.2	19.2	18.9	17.6	19.4	21.6	21.5	27.3	30.4	22.7	18.4	24.5	27.0	27.9	16.2	14.5	16.0
de	14.4	13.5	–	19.0	17.1	19.7	17.1	16.3	15.2	21.7	15.8	15.7	22.7	16.6	16.9	20.3	18.6	17.6	27.2	17.4	13.5	16.4
nl	23.0	22.3	15.8	–	20.0	16.8	11.1	16.3	21.5	19.3	16.3	9.7	20.1	11.2	12.2	22.5	18.1	22.7	26.5	16.5	15.5	18.2
fr	31.5	20.5	22.0	32.3	–	28.3	25.6	22.1	30.9	26.2	26.9	30.5	26.4	26.5	20.9	23.5	28.5	28.5	32.0	23.4	23.7	24.8
it	19.2	16.2	16.6	20.1	20.9	–	14.6	19.0	17.6	19.4	22.3	22.6	21.0	16.0	19.4	18.1	16.0	17.0	21.7	20.9	22.8	17.6

Appendix

Table C23: Direction-level translation quality for Gemma-4 E4B. Rows are source languages and columns are target languages.

Src $\backslash$ Tgt	en	cs	de	nl	fr	it	es	pt	tr	id	ms	vi	ru	uk	ar	he	hi	my	th	ja	ko	zh
en	–	40.7	34.2	38.2	35.5	41.0	35.8	31.5	37.2	31.3	33.0	32.1	36.7	40.4	30.3	36.9	37.5	30.5	35.1	35.9	42.9	33.6
cs	32.0	–	30.5	29.9	33.1	30.1	28.4	37.7	30.3	21.6	32.0	27.3	36.4	40.2	36.3	37.2	38.5	40.3	38.1	26.1	26.6	38.7
de	35.8	33.6	–	47.3	41.1	35.3	39.0	38.4	32.8	35.8	35.6	32.0	27.8	39.1	31.3	40.1	40.2	37.2	40.9	36.5	35.6	34.7
nl	58.4	54.0	48.6	–	41.6	51.8	39.2	51.3	39.2	43.2	36.4	41.0	40.0	46.3	28.6	35.9	49.2	52.7	47.6	43.0	43.0	45.2
fr	44.0	36.9	43.4	43.1	–	40.8	42.4	51.0	39.3	42.3	44.7	33.5	49.6	38.1	40.1	41.7	52.8	47.4	43.9	51.7	47.1	37.9
it	32.5	34.3	35.8	32.8	35.7	–	38.9	41.2	31.8	37.1	34.6	36.7	34.5	28.1	32.3	33.7	30.2	36.3	36.3	33.5	34.0	35.1

Appendix

Table C24: Direction-level translation quality for GLM-4.1V 9B. Rows are source languages and columns are target languages.

Src $\backslash$ Tgt	en	cs	de	nl	fr	it	es	pt	tr	id	ms	vi	ru	uk	ar	he	hi	my	th	ja	ko	zh
en	–	49.4	58.4	55.8	49.9	49.9	55.8	53.1	56.3	55.8	51.4	51.1	59.4	55.0	62.0	54.8	57.5	61.6	67.8	57.8	63.2	55.1
cs	61.1	–	58.7	69.1	64.0	59.8	56.9	66.4	71.1	61.7	71.1	59.0	68.4	73.4	71.1	61.5	66.9	75.5	61.8	61.8	65.0	65.3
de	39.0	50.9	–	55.8	54.5	54.5	52.3	49.8	47.7	64.2	51.8	54.2	52.9	57.4	49.0	53.0	55.6	52.6	60.6	51.0	50.7	56.2
nl	75.1	83.5	77.9	–	75.9	82.5	68.4	78.9	82.6	76.7	85.3	71.9	77.8	79.0	77.8	76.2	79.5	82.1	78.0	80.6	78.3	74.5
fr	60.3	68.7	64.3	57.9	–	57.0	66.2	60.4	67.5	59.1	65.0	70.0	61.3	69.0	62.4	69.8	76.5	66.3	64.3	67.2	69.7	65.3
it	46.1	57.5	57.9	48.4	53.4	–	48.0	52.4	50.3	54.8	54.3	54.6	57.5	50.3	56.7	55.8	54.2	65.0	54.7	54.0	50.7	52.2

Appendix

Table C25: Direction-level translation quality for GLM-5.3 Flash. Rows are source languages and columns are target languages.

Src $\backslash$ Tgt	en	cs	de	nl	fr	it	es	pt	tr	id	ms	vi	ru	uk	ar	he	hi	my	th	ja	ko	zh
en	–	58.2	52.0	54.6	48.1	53.5	52.4	50.8	42.4	53.0	58.1	45.8	48.3	56.1	61.8	59.1	47.8	64.6	61.2	52.2	53.5	55.0
cs	66.5	–	60.0	71.5	72.4	62.2	65.2	66.8	74.0	61.9	61.9	69.6	62.5	68.7	68.2	70.3	69.2	83.8	71.5	68.7	65.3	67.1
de	45.7	47.5	–	60.3	53.6	51.5	58.7	47.9	52.9	59.6	53.7	52.3	53.5	63.0	52.4	49.4	54.0	57.4	57.1	49.3	49.6	49.5
nl	57.8	72.2	68.5	–	70.7	64.5	63.1	56.0	69.6	73.8	68.7	70.5	72.8	63.1	63.3	75.5	69.8	71.6	64.8	66.9	46.2	64.7
fr	62.5	67.9	67.5	65.9	–	58.8	63.8	68.3	65.7	69.2	71.0	67.0	61.9	66.7	69.8	73.4	68.4	73.3	67.8	62.3	62.4	64.6
it	55.9	61.0	60.8	54.7	59.0	–	56.9	54.8	59.9	62.7	57.1	59.7	59.2	55.8	53.9	59.1	56.2	65.1	64.4	62.4	58.2	62.3

Appendix

Table C26: Direction-level translation quality for GPT-5.6 Luna. Rows are source languages and columns are target languages.

Src $\backslash$ Tgt	en	cs	de	nl	fr	it	es	pt	tr	id	ms	vi	ru	uk	ar	he	hi	my	th	ja	ko	zh
en	–	52.8	50.7	55.9	55.2	57.1	47.5	50.8	48.6	48.6	57.9	54.5	59.1	53.2	61.7	64.0	55.8	59.4	57.4	46.2	48.1	54.1
cs	72.6	–	69.6	78.1	81.1	80.3	70.6	84.3	76.0	73.7	75.7	73.1	78.5	77.7	80.9	73.2	85.1	79.6	76.8	71.7	75.8	76.6
de	46.6	57.6	–	58.7	56.4	54.4	60.7	50.3	57.1	66.9	58.5	52.8	60.7	61.8	59.1	58.0	54.3	61.5	68.3	54.3	61.7	58.6
nl	71.5	70.1	67.0	–	65.0	88.8	67.5	69.7	72.6	75.7	69.2	63.5	79.5	75.2	66.1	71.7	83.4	84.2	78.7	76.7	68.2	72.8
fr	68.6	70.1	63.8	72.4	–	65.5	61.0	64.0	71.7	65.0	71.0	68.5	64.0	75.0	67.1	72.5	70.7	76.8	72.4	64.0	66.6	63.9
it	52.8	61.1	52.8	57.2	58.9	–	56.5	44.9	55.7	60.7	55.0	57.4	57.5	46.7	47.6	52.9	50.5	62.7	55.5	58.5	49.7	51.9

Appendix

Table C27: Direction-level translation quality for DeepSeek Flash. Rows are source languages and columns are target languages.

Src $\backslash$ Tgt	en	cs	de	nl	fr	it	es	pt	tr	id	ms	vi	ru	uk	ar	he	hi	my	th	ja	ko	zh
en	–	63.1	60.9	56.9	57.8	65.2	55.0	57.8	52.1	66.8	62.6	61.7	60.5	64.8	62.0	65.3	59.3	62.1	62.2	67.8	60.8	60.4
cs	67.7	–	61.8	64.6	64.4	65.8	61.9	55.5	58.9	65.5	69.8	60.3	61.5	65.9	73.0	61.0	68.3	71.5	61.7	68.7	54.4	63.7
de	56.3	53.6	–	63.0	56.4	55.0	59.9	60.7	61.5	65.6	57.2	57.0	56.0	62.0	59.5	53.3	64.8	62.2	63.5	57.7	54.7	59.3
nl	67.2	68.2	59.0	–	67.3	64.8	77.3	68.1	70.1	68.5	71.2	70.7	72.9	67.8	65.3	78.3	74.8	79.4	84.2	85.1	73.4	74.7
fr	62.9	67.4	63.5	57.0	–	55.9	61.6	52.3	65.1	64.7	71.4	61.0	65.2	60.7	66.7	60.4	67.0	73.8	73.2	62.8	62.3	60.6
it	59.3	60.5	62.9	65.2	62.6	–	57.5	59.9	69.1	57.9	58.6	59.4	65.1	53.3	56.8	58.1	56.4	70.4	62.3	62.4	56.5	55.5

Appendix

Table C28: Direction-level translation quality for Qwen3.8 27B. Rows are source languages and columns are target languages.

Src $\backslash$ Tgt	en	cs	de	nl	fr	it	es	pt	tr	id	ms	vi	ru	uk	ar	he	hi	my	th	ja	ko	zh
en	–	65.4	61.3	57.6	54.5	61.9	52.7	57.9	56.3	56.5	60.7	68.8	63.9	68.0	63.6	64.0	58.2	72.3	63.8	51.8	58.4	58.5
cs	83.6	–	84.3	80.1	74.0	81.2	78.5	82.5	86.8	76.6	85.1	75.7	81.4	80.9	83.0	80.5	78.2	83.4	81.8	85.1	79.8	78.4
de	57.1	65.3	–	61.1	65.7	63.7	61.3	57.2	61.0	67.9	59.7	62.1	64.1	69.3	55.6	65.1	68.8	76.0	62.1	65.0	62.8	54.9
nl	79.4	84.9	84.2	–	93.0	94.0	84.3	79.9	92.7	87.7	90.5	92.7	91.2	88.9	85.1	90.9	89.5	85.9	87.5	86.1	82.6	83.4
fr	69.3	76.5	75.4	66.7	–	67.7	76.3	74.6	69.3	67.0	81.3	74.2	67.6	67.4	75.2	69.9	79.8	79.4	77.8	75.3	75.1	64.6
it	63.2	67.5	63.7	69.1	68.3	–	67.5	58.8	60.5	68.8	70.3	68.1	66.8	65.9	65.4	67.2	66.8	75.5	69.2	68.3	71.2	60.0

Appendix

Table C29: Direction-level translation quality for Claude Sonnet 5. Rows are source languages and columns are target languages.

Src $\backslash$ Tgt	en	cs	de	nl	fr	it	es	pt	tr	id	ms	vi	ru	uk	ar	he	hi	my	th	ja	ko	zh
en	–	64.5	59.5	61.0	61.8	65.5	50.0	62.4	63.7	63.5	64.2	58.3	65.1	70.6	68.6	68.4	66.4	75.3	68.6	60.2	64.2	58.9
cs	82.7	–	80.7	81.2	88.3	80.8	85.4	89.8	95.9	81.7	89.0	82.4	87.2	94.8	83.5	83.2	86.4	84.3	88.4	79.8	75.5	87.5
de	46.3	61.3	–	62.4	64.4	62.5	56.2	58.1	67.2	67.4	60.2	62.2	62.9	70.4	63.0	63.1	61.6	70.2	63.9	56.0	58.9	62.9
nl	81.4	79.8	75.1	–	83.3	81.9	86.6	85.0	85.3	77.4	80.8	84.9	82.5	77.3	87.5	85.9	76.3	86.1	79.1	83.9	81.8	83.7
fr	64.4	86.3	73.1	75.7	–	76.7	78.0	71.9	81.9	76.5	78.1	64.8	82.5	79.4	77.1	77.4	81.7	83.4	79.7	76.5	72.0	71.7
it	60.9	72.1	68.1	70.3	68.4	–	65.8	60.9	64.0	66.9	68.6	65.8	63.8	70.7	69.6	71.3	64.8	67.0	67.2	76.2	62.6	64.2

Appendix

Table C30: Direction-level translation quality for Muse Spark 1.3. Rows are source languages and columns are target languages.

Src $\backslash$ Tgt	en	cs	de	nl	fr	it	es	pt	tr	id	ms	vi	ru	uk	ar	he	hi	my	th	ja	ko	zh
en	–	74.9	69.4	62.2	64.5	70.2	61.6	60.7	64.7	66.0	73.0	67.4	73.3	63.9	69.3	76.2	68.2	63.5	73.3	79.6	64.7	65.6
cs	85.4	–	76.7	80.2	85.2	82.6	77.2	88.6	86.1	83.0	91.9	85.2	85.5	87.0	89.8	88.5	87.2	75.2	86.9	82.6	80.9	88.9
de	60.8	67.9	–	72.3	77.2	68.9	63.5	65.0	69.5	72.0	71.2	70.3	65.2	75.1	72.0	63.7	69.9	68.3	70.6	70.3	70.2	70.4
nl	83.2	83.7	80.5	–	83.6	77.0	88.2	81.8	80.5	80.7	85.1	85.1	89.8	82.6	80.3	82.9	85.0	70.3	78.7	86.3	78.4	77.9
fr	75.1	77.2	74.7	74.5	–	65.8	74.0	70.4	79.9	72.9	78.2	74.9	72.8	75.1	77.2	78.7	79.6	64.3	74.9	75.4	73.1	66.7
it	72.6	72.3	74.2	75.4	69.0	–	69.0	65.4	76.4	70.5	68.7	73.4	72.3	74.2	72.8	73.3	70.1	66.0	78.5	77.0	71.7	68.4

Appendix

Table C31: Direction-level translation quality for Qwen3.8 Max. Rows are source languages and columns are target languages.

Src $\backslash$ Tgt	en	cs	de	nl	fr	it	es	pt	tr	id	ms	vi	ru	uk	ar	he	hi	my	th	ja	ko	zh
en	–	69.4	60.0	68.1	67.0	71.1	57.5	65.0	61.8	66.1	67.9	65.8	64.0	67.4	64.7	78.4	61.5	73.7	74.8	67.7	68.6	63.3
cs	88.1	–	88.6	84.0	83.7	81.5	90.0	86.5	87.0	87.0	85.3	83.3	87.4	86.7	82.0	89.8	89.0	88.8	84.8	85.5	86.7	84.8
de	65.5	71.9	–	75.2	77.2	66.3	68.8	69.7	66.3	73.9	68.0	64.2	67.0	76.7	64.4	70.0	71.8	78.9	71.8	72.5	66.8	68.7
nl	88.6	93.8	93.5	–	93.5	89.4	87.0	87.8	86.7	97.8	93.8	87.9	98.0	89.7	88.2	83.3	85.8	86.0	81.6	80.4	94.4	82.2
fr	73.7	80.9	79.6	75.6	–	70.8	79.7	73.7	79.4	73.9	79.3	83.1	72.8	77.7	84.7	83.6	80.1	83.0	81.3	76.2	77.4	74.9
it	68.3	74.5	70.8	76.8	72.4	–	69.5	73.9	69.1	72.9	69.8	73.5	69.1	76.1	69.3	70.7	71.6	76.5	72.4	74.8	73.0	67.9

Appendix

Table C32: Direction-level translation quality for Gemini 3.8 Flash. Rows are source languages and columns are target languages.

Src $\backslash$ Tgt	en	cs	de	nl	fr	it	es	pt	tr	id	ms	vi	ru	uk	ar	he	hi	my	th	ja	ko	zh
en	–	74.0	70.7	66.7	73.8	79.9	63.8	76.8	72.9	73.8	75.8	73.0	73.3	70.9	77.9	74.8	61.4	67.5	75.2	69.4	68.5	74.4
cs	90.9	–	82.8	85.6	92.3	84.0	90.3	81.8	92.2	89.1	86.5	85.5	94.2	88.4	84.4	81.7	83.9	92.9	88.2	86.7	79.4	91.4
de	64.9	74.1	–	67.9	68.8	68.3	66.0	62.7	64.3	74.2	70.6	61.6	77.1	76.5	65.9	74.6	74.2	71.1	73.6	64.6	67.1	70.8
nl	82.3	86.4	84.8	–	92.6	86.3	75.8	90.6	90.7	91.4	78.9	90.6	86.7	90.9	89.2	81.6	88.4	76.4	86.3	86.0	88.5	83.6
fr	75.3	88.2	75.0	80.8	–	80.9	81.8	80.2	73.6	78.9	79.4	79.9	82.5	75.8	84.1	82.7	82.7	91.0	81.7	73.4	85.8	77.3
it	71.3	73.6	76.4	76.2	68.9	–	68.0	62.5	73.8	73.1	64.5	72.6	68.1	68.0	73.5	72.8	68.1	70.2	75.6	71.2	74.3	66.9

Appendix

Table C33: Direction-level translation quality for Doubao Seed 2.1 Pro. Rows are source languages and columns are target languages.

Src $\backslash$ Tgt	en	cs	de	nl	fr	it	es	pt	tr	id	ms	vi	ru	uk	ar	he	hi	my	th	ja	ko	zh
en	–	81.9	79.3	78.5	70.3	77.3	82.6	76.4	83.6	76.9	78.1	80.1	79.6	77.9	79.3	79.7	75.1	84.7	82.3	78.4	76.7	78.5
cs	90.9	–	82.7	92.0	92.1	86.5	91.5	89.5	87.8	92.0	87.8	87.6	83.4	94.0	87.6	90.6	88.0	89.1	87.4	84.6	93.1	88.3
de	73.9	75.0	–	75.9	73.1	68.8	74.2	62.2	79.9	76.8	72.1	72.4	77.2	83.2	73.3	69.3	72.6	85.1	75.3	78.9	71.2	74.1
nl	86.4	83.9	88.6	–	85.8	87.0	79.6	91.8	89.9	90.7	84.2	86.2	84.3	80.8	88.5	85.5	85.8	90.7	92.1	86.5	81.9	89.4
fr	76.6	85.6	79.0	80.1	–	74.2	79.0	88.8	88.7	80.2	82.8	82.9	80.1	80.0	82.0	84.4	84.5	85.9	85.1	82.5	77.7	80.0
it	69.1	76.0	77.2	79.4	79.6	–	76.7	69.1	74.3	74.5	76.6	76.4	78.7	82.5	73.8	76.0	78.1	79.7	76.1	79.9	74.5	76.2

Appendix

Table C34: Direction-level translation quality for GPT-6 Astra. Rows are source languages and columns are target languages.

	GPT-5.6 Luna			Claude Sonnet 5
Source language	T	K	V	T	K	V
English	53.73	82.94	59.80	60.77	86.78	57.40
Czech	67.96	76.06	74.83	80.99	87.00	78.48
German	53.29	89.32	58.94	63.14	84.02	65.22
Dutch	66.38	72.53	71.70	87.33	87.40	76.93
French	66.59	89.26	65.32	72.89	90.41	58.43

Appendix

Table C35: Source-language T, K, and V for GPT-5.6 Luna and Claude Sonnet 5.

	Muse Spark 1.3			Qwen3.8 Max
Source language	T	K	V	T	K	V
English	63.84	81.83	64.79	68.20	84.55	74.87
Czech	85.17	94.60	85.71	84.50	92.85	87.79
German	61.96	84.86	61.95	69.25	93.89	75.31
Dutch	82.18	86.92	81.34	81.98	85.79	80.35
French	76.60	94.72	66.25	74.07	93.83	73.39

Appendix

Table C36: Source-language T, K, and V for Muse Spark 1.3 and Qwen3.8 Max.

	Gemini 3.8 Flash			Doubao Seed 2.1 Pro
Source language	T	K	V	T	K	V
English	66.86	83.81	70.67	72.12	88.55	81.98
Czech	86.21	96.00	86.54	87.25	96.30	89.02
German	70.28	86.24	75.82	69.47	96.84	83.53
Dutch	89.02	85.83	78.72	86.09	92.94	83.57
French	78.16	90.46	75.94	80.53	93.75	82.36

Appendix

Table C37: Source-language T, K, and V for Gemini 3.8 Flash and Doubao Seed 2.1 Pro.

	GPT-6 Astra			Gemma-4 E4B
Source language	T	K	V	T	K	V
English	78.92	88.64	91.33	21.47	41.16	16.88
Czech	88.88	98.60	91.76	21.34	29.24	15.02
German	74.48	93.65	87.49	17.74	27.28	11.96
Dutch	86.64	82.22	80.39	17.90	48.85	24.21
French	81.91	92.90	86.47	26.43	28.76	23.43

Appendix

Table C38: Source-language T, K, and V for GPT-6 Astra and Gemma-4 E4B.

	GLM-4.1V 9B			GLM-5.3 Flash
Source language	T	K	V	T	K	V
English	35.74	63.93	39.42	56.24	82.04	59.21
Czech	32.92	40.99	47.09	65.22	70.29	70.43
German	36.66	58.27	40.70	53.03	84.82	57.28
Dutch	44.58	78.07	58.60	78.22	96.82	66.36
French	43.43	74.67	39.38	65.15	88.42	56.96

Appendix

Table C39: Source-language T, K, and V for GLM-4.1V 9B and GLM-5.3 Flash.

	DeepSeek Flash			Qwen3.8 27B
Source language	T	K	V	T	K	V
English	54.21	83.91	68.58	61.20	82.60	65.76
Czech	76.71	85.25	83.57	64.10	81.65	80.63
German	58.01	89.89	64.80	59.01	92.96	68.48
Dutch	73.19	85.29	78.24	71.82	80.92	80.94
French	68.31	87.55	65.85	63.60	89.95	58.89

Appendix

Table C40: Source-language T, K, and V for DeepSeek Flash and Qwen3.8 27B.

Explore similar work

Comparative Evaluation of Machine Translation Systems on Images with Text
May 28, 2026Blai Puchol, Sergio Gómez González, Miguel Domingo +1Image-Text Pairs Unpaired Image-To-Image Translation
PRHLT Research Center - Universitat Polit`ecnica de Val`encia, Camino de Vera, s/n - Valencia · ValgrAI - Valencian Graduate School and Research Network for Artificial Intelligence, Camino de Vera s/n
Lost in Translation: Do LVLM Judges Generalize Across Languages?
Apr 21, 2026Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5Multilingual Benchmark Large Language Model Judges
York University · University of Alberta · Nanyang Technological University +1
IFMTBench: A Comprehensive Benchmark for Multilingual Translation Instruction Following
May 27, 2026Mingrui Sun, Mao Zheng, Zheng Li +1Multilingual Benchmark Instruction
Large Language Model Department, Tencent

May 28, 2026, cs.CLJ/K move · Enter open · S save

Comparative Evaluation of Machine Translation Systems on Images with Text

Blai Puchol, Sergio Gómez González, Miguel Domingo, Francisco Casacuberta

PRHLT Research Center - Universitat Polit`ecnica de Val`encia, Camino de Vera, s/n - Valencia · ValgrAI - Valencian Graduate School and Research Network for Artificial Intelligence, Camino de Vera s/n

Image-Text Pairs Unpaired Image-To-Image Translation Machine Translation Multimodal Large Language Models Optical Character Recognition Text Analysis and Detection

Open paper AskPDF arXiv

This work presents a comparative evaluation of machine translation systems applied to images containing textual information, a task that lies at the intersection of computer vision and natural language processing. The study compares three main paradigms: modular pipelines that separate text detection, recognition, and translation; multi-modal large language models (MLLMs) capable of processing both image and text jointly; and an end-to-end model, Translatotron-V, which directly generates translated images. The modular systems employ state-of-the-art OCR (docTR) combined with multilingual LLMs such as Llama and EuroLLM, while the evaluated MLLMs include different configurations of Gemini 2.5. Experiments were conducted on parallel multilingual datasets covering multiple language pairs, with evaluation based on BLEU, chrF, and TER metrics. The results show that modular pipelines outperform the end-to-end approach, while MLLMs achieve the best overall performance, demonstrating superior flexibility and contextual understanding. These findings underscore the effectiveness of multi-modal reasoning for image-to-text translation and provide a solid foundation for future research on integrating visual understanding and language generation in multilingual settings.

Abstract

Explore similar work

Comparative Evaluation of Machine Translation Systems on Images with Text

Lost in Translation: Do LVLM Judges Generalize Across Languages?

IFMTBench: A Comprehensive Benchmark for Multilingual Translation Instruction Following