Image translation is a fundamental capability of multimodal models for multilingual applications, requiring visual understanding and meaning preservation across languages. However, existing benchmarks have limited language coverage and often lack explicit image-specific evaluation criteria, making it difficult to comprehensively assess this capability. To systematically evaluate this capability, we introduce VISTA-Bench, covering 22 languages and 10 domains, and develop an image-specific rubric evaluation protocol. The benchmark combines sampling for language and scenario coverage with model-assisted, human-verified annotations that group related text into coherent semantic units and provide multilingual reference translations. The rubrics specify essential content, semantic relations, and acceptable translation variants, yielding separate output-based scores for translation quality and the preservation of visual and knowledge-dependent information. We conduct extensive evaluations of 16 mainstream models, including 12 multimodal models and four text-input models, and provide systematic analyses across languages, domains, and evaluation dimensions.
Figures & tables
Figure 1: Overview of VISTA-Bench. Stages 1–3 construct language–scenario coverage, human-reviewed semantic blocks, and multilingual references with image-specific rubrics. Stage 4 separates the model’s image–target input from evaluator-only annotations, then converts evidence-based judgments into separate importance-weighted V, T, and K scores.
Dataset
Test Size
Languages (source → target)
# Scenarios
Evaluation
Vistra
772
En → De/Es/Ru/Zh
No scenario breakdown
BLEU, chrF, COMET
MMTIT-Bench
1,400
14 languages → En/Zh
3 (broad types)
COMET + VLLM judge
PRIM
340
En → De/Fr/Cs/Ru/Ro
No scenario breakdown
BLEU, COMET
IIMT30K
2,740
En ↔ De
No scenario breakdown
BLEU, COMET
PATIMT-Bench
1,200
En ↔ Zh
10 (6 eval. groups)
BLEU, COMET
VISTA-Bench (Ours)
2,228
22 languages → 22 languages
100+ (10 domains)
Image-specific rubric judge
Table 1: Comparison of image translation benchmarks and evaluation protocols.
Figure 2: Human revision rates for ten selected widely used languages.
Figure 3: Rubric structure for an automatic-gate notice. Only T2 is expanded, with condensed wording. The knowledge dimension is inapplicable to this image.
Model
Overall
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Image-input: closed-weight models
GPT-5.6 Luna
55.88
51.00
60.03
53.44
54.38
61.84
60.94
58.92
58.59
58.35
63.55
Claude Sonnet 5
69.43
64.71
71.78
65.71
71.14
74.18
77.94
67.08
71.38
78.13
77.82
Muse Spark 1.3
71.56
64.09
71.98
70.13
71.99
77.43
77.59
73.70
76.86
72.75
73.95
Qwen3.8 Max
72.67
66.00
71.94
72.57
72.89
76.20
81.07
74.67
75.57
76.43
75.40
Gemini 3.8 Flash
74.20
69.10
75.62
74.57
76.22
79.52
79.12
72.64
76.34
78.40
78.17
Table 2: Translation-quality (T) results. Overall denotes global T, not the composite score. Global T averages directions for image-input models and tasks for text-input models. Tinted cells mark the highest score within each group and column.
Figure 4: Source-language translation profiles for six multimodal models. Each point is the mean T for one source language over its target-language directions. Dashed lines mark the per-model average across 22 languages; teal and orange denote scores above and below that average.
Figure 5: Paired T–V (left) and T–K (right) scores, ordered by increasing T from top to bottom. Lines connect the same model’s scores; columns report values to two decimals. Correlations use all 12 models.
Model
Full set (10,920)
Selected set (2,228)
∣ΔT∣
GLM-4.1V 9B
30.7
30.0
0.7
Gemma-4 E4B
20.0
19.5
0.5
Claude Sonnet 5
70.2
69.4
0.8
GPT-6 Astra
79.1
77.6
1.5
Pearson r=0.99996 Spearman ρ=1.0000 MAE =0.88
Table 3: Translation-quality scores on the full and selected sets. Statistics are computed from the reported one-decimal scores.
Figure 6: Human–judge agreement and score deviations. Left: each point is one candidate model’s mean T under an automatic judge versus Human; the dashed line marks equal scores. Right: automatic-minus-human differences. Correlations use four model-level means, not individual images.
Appendix figures & tables44 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Source-image counts across 22 languages. Each of the 2,228 images is counted once; simplified and traditional Chinese are combined. Sector areas are proportional to image counts, and the legend reports absolute counts.
ID
English source
French reference
1
5 grams
5 grammes
2
TRINOLONE ORAL PASTE
PÂTE BUCCALE TRINOLONE
3
Triamcinolone Acetonide in Dental Paste
Acétonide de triamcinolone dans une pâte dentaire
4
NIDA PHARMA INCORPORATION
NIDA PHARMA INCORPORATION
5
Wang Noi, Ayutthaya, Thailand.
Wang Noi, Ayutthaya, Thaïlande.
6
NIDA
NIDA
Appendix
Table B1: Complete source blocks and stored French references for the package label.
The output gives a meaning-bearing representation of every source block: the quantity, product name, ingredient-and-dosage-form line, manufacturer, location, and standalone NIDA identifier.
Expected 2
The output preserves the meaningful grouping of the package label, while allowing blocks to be merged, reordered, or laid out naturally in the target language.
Accepted 1
Repeated NIDA text may be represented together with the manufacturer name or separately as the logo, provided both the company identifier and standalone identifier remain recoverable.
Forbidden 1
Omitting any source block or treating visible label text as decorative.
Appendix
Table B2: Complete shared rubric criteria for the English-to-French package example.
Field
Complete content
TOL1
Source blocks
4
Term links
Empty
Knowledge links
Empty
Rule
“NIDA PHARMA INCORPORATION” has no single required official translation. Accept retention, reasonable transliteration, semantic translation, or a readable target-language rendering, provided NIDA remains identifiable and the organization is understood as a pharmaceutical company or corporation.
French forms
Empty
Appendix
Table B3: Complete tolerances and their French accepted forms. Empty lists have no prescribed French surface form.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
47.84
53.52
52.11
59.52
54.49
56.78
60.55
56.52
45.87
68.36
Claude Sonnet 5
68.75
70.56
60.98
69.47
70.74
80.30
59.81
69.70
79.02
79.42
Muse Spark 1.3
64.45
71.65
65.71
70.75
74.88
74.24
73.39
73.79
72.02
70.85
Qwen3.8 Max
66.86
77.01
74.74
73.56
71.96
79.37
75.42
74.40
74.95
78.97
Gemini 3.8 Flash
69.30
78.49
76.21
76.10
81.22
79.31
70.11
75.40
74.65
78.63
Appendix
Table C1: Domain-level T for translations into English, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
51.99
62.06
53.85
53.89
58.47
67.32
63.22
61.58
56.54
62.79
Claude Sonnet 5
62.26
69.55
67.30
69.31
80.39
81.80
65.21
73.34
76.83
78.43
Muse Spark 1.3
65.40
70.50
74.25
74.07
78.49
85.75
78.11
80.21
75.27
69.65
Qwen3.8 Max
64.05
68.54
72.73
75.86
79.96
78.34
78.09
79.46
72.52
72.46
Gemini 3.8 Flash
71.65
72.82
75.59
75.70
81.69
72.47
74.12
77.51
78.87
78.03
Appendix
Table C2: Domain-level T for translations into Czech, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
46.92
59.29
54.80
54.31
58.69
55.42
60.78
57.96
60.71
66.01
Claude Sonnet 5
58.88
66.76
72.90
69.90
70.98
66.78
61.28
72.60
80.59
80.16
Muse Spark 1.3
60.23
67.23
75.64
70.51
74.10
69.53
73.77
75.58
68.24
75.62
Qwen3.8 Max
66.11
70.08
73.15
74.48
74.00
77.90
72.67
73.70
76.11
76.02
Gemini 3.8 Flash
63.00
74.57
77.07
77.90
77.34
71.88
75.14
75.22
79.38
84.81
Appendix
Table C3: Domain-level T for translations into German, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
56.10
54.62
50.34
51.26
65.73
52.69
61.21
60.52
54.21
68.22
Claude Sonnet 5
60.27
66.03
64.22
66.57
69.34
76.75
68.29
72.37
81.99
77.89
Muse Spark 1.3
65.38
69.98
72.79
69.38
74.58
80.39
73.83
80.25
73.88
76.19
Qwen3.8 Max
67.17
68.66
77.26
72.23
72.29
79.99
75.04
76.98
73.49
78.32
Gemini 3.8 Flash
66.86
75.20
73.29
72.01
78.35
80.04
73.47
74.07
79.48
79.66
Appendix
Table C4: Domain-level T for translations into Dutch, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
50.71
62.74
46.81
53.15
58.91
60.68
58.64
56.77
55.56
61.20
Claude Sonnet 5
64.53
74.04
64.49
67.63
76.33
74.34
66.20
64.84
77.63
76.93
Muse Spark 1.3
61.45
73.92
69.17
73.87
77.84
78.52
72.06
73.72
73.87
75.56
Qwen3.8 Max
69.23
72.73
69.81
73.29
79.27
80.97
75.17
72.23
80.17
76.89
Gemini 3.8 Flash
74.79
79.27
71.94
78.02
81.11
68.26
70.79
75.56
81.31
79.50
Appendix
Table C5: Domain-level T for translations into French, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
51.15
52.68
56.30
56.37
53.11
65.54
58.94
58.35
63.38
66.80
Claude Sonnet 5
66.45
69.15
67.83
74.95
74.13
80.47
66.09
70.73
79.89
79.18
Muse Spark 1.3
67.49
73.43
72.80
71.16
80.29
83.58
74.11
76.06
71.88
76.86
Qwen3.8 Max
69.62
69.15
71.37
74.12
72.68
83.46
73.94
69.47
82.89
76.20
Gemini 3.8 Flash
68.04
70.85
76.88
73.64
81.73
83.18
73.64
74.75
79.56
75.38
Appendix
Table C6: Domain-level T for translations into Italian, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
51.28
48.55
46.54
58.18
63.60
60.88
58.51
57.93
54.90
61.65
Claude Sonnet 5
65.55
72.13
59.53
70.83
76.28
75.98
66.52
71.48
77.16
73.35
Muse Spark 1.3
60.98
69.00
70.18
67.79
77.37
75.48
70.90
75.89
71.66
60.01
Qwen3.8 Max
65.61
69.48
73.88
72.35
75.04
83.35
71.26
72.71
79.08
75.27
Gemini 3.8 Flash
71.03
72.99
76.13
77.58
80.03
75.98
70.48
70.00
76.38
74.80
Appendix
Table C7: Domain-level T for translations into Spanish, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
54.00
64.66
56.22
53.17
65.77
50.64
54.25
56.66
61.39
57.18
Claude Sonnet 5
66.51
75.33
62.60
66.91
70.62
73.48
65.38
68.36
74.60
76.38
Muse Spark 1.3
67.39
79.13
75.69
69.49
75.55
77.22
73.60
74.56
71.93
71.77
Qwen3.8 Max
66.97
76.50
68.87
71.68
72.79
76.09
72.64
68.61
67.48
75.23
Gemini 3.8 Flash
72.07
81.57
71.01
75.71
71.89
71.52
67.38
76.17
79.43
80.05
Appendix
Table C8: Domain-level T for translations into Portuguese, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
54.57
61.43
52.45
48.44
65.02
76.85
59.35
61.19
48.24
65.06
Claude Sonnet 5
69.07
72.55
67.34
72.84
72.40
83.04
64.93
68.21
78.38
78.51
Muse Spark 1.3
72.48
72.49
72.37
72.93
78.86
82.16
75.75
77.95
67.31
78.90
Qwen3.8 Max
68.21
68.37
72.60
72.42
76.56
92.32
77.98
75.57
76.09
78.96
Gemini 3.8 Flash
73.16
76.83
76.66
74.05
78.47
88.45
71.77
72.39
75.95
77.14
Appendix
Table C9: Domain-level T for translations into Turkish, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
50.20
61.95
53.10
58.97
65.55
58.42
55.05
59.78
61.44
65.03
Claude Sonnet 5
62.29
67.75
70.08
72.00
75.56
75.34
63.80
68.95
76.96
77.02
Muse Spark 1.3
61.18
65.10
70.86
73.91
76.56
67.28
76.05
80.02
67.75
75.86
Qwen3.8 Max
63.48
71.16
74.45
74.67
73.70
77.44
70.45
74.02
72.91
74.00
Gemini 3.8 Flash
69.92
74.67
77.94
76.62
78.13
76.79
67.29
76.75
77.25
81.77
Appendix
Table C10: Domain-level T for translations into Indonesian, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
51.21
73.24
51.26
52.13
70.45
61.59
57.03
59.64
67.70
65.06
Claude Sonnet 5
68.08
73.69
66.45
71.08
76.03
78.30
69.83
73.02
80.82
77.61
Muse Spark 1.3
64.72
78.90
71.04
74.03
80.35
82.11
74.53
75.04
66.87
74.00
Qwen3.8 Max
66.87
75.71
75.40
72.61
84.72
83.99
75.63
74.11
80.56
80.01
Gemini 3.8 Flash
65.56
81.84
74.21
74.04
80.90
78.35
77.55
79.70
79.03
79.59
Appendix
Table C11: Domain-level T for translations into Malay, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
53.32
56.50
53.91
50.16
65.23
63.77
63.54
55.64
56.24
56.42
Claude Sonnet 5
66.51
67.63
68.32
69.65
80.13
78.09
69.57
70.40
81.15
74.65
Muse Spark 1.3
61.23
65.25
64.92
67.27
77.24
76.28
70.10
72.68
67.62
69.99
Qwen3.8 Max
67.54
68.42
72.56
70.14
81.52
82.58
77.54
78.94
74.66
73.99
Gemini 3.8 Flash
70.57
69.75
79.60
74.73
79.57
85.75
74.64
75.48
77.65
71.80
Appendix
Table C12: Domain-level T for translations into Vietnamese, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
54.88
57.79
56.77
51.38
58.78
61.18
59.48
60.75
53.89
65.41
Claude Sonnet 5
60.93
67.96
69.45
68.12
72.99
86.11
67.28
70.69
77.26
82.91
Muse Spark 1.3
62.91
65.77
71.11
64.69
75.48
77.11
74.43
74.60
75.42
77.68
Qwen3.8 Max
61.38
68.84
74.70
71.06
75.61
82.84
73.93
76.62
72.80
79.36
Gemini 3.8 Flash
71.06
73.62
76.99
75.04
77.44
81.45
73.82
73.36
79.85
80.45
Appendix
Table C13: Domain-level T for translations into Russian, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
50.62
55.95
57.08
51.68
66.99
61.70
58.62
61.54
58.84
68.56
Claude Sonnet 5
63.65
70.97
69.23
72.50
75.55
84.77
67.02
75.66
76.18
75.81
Muse Spark 1.3
63.24
73.03
72.51
74.59
80.24
85.12
75.10
78.42
76.73
74.38
Qwen3.8 Max
64.33
73.02
75.20
73.47
77.18
79.43
73.70
79.68
77.69
72.90
Gemini 3.8 Flash
67.49
76.53
81.04
79.00
73.27
78.63
72.70
76.95
80.42
81.25
Appendix
Table C14: Domain-level T for translations into Ukrainian, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
50.59
70.51
52.63
54.33
66.52
61.41
60.13
58.03
69.43
70.26
Claude Sonnet 5
61.33
77.74
65.57
72.09
76.00
76.31
67.75
70.54
81.41
76.22
Muse Spark 1.3
64.52
76.66
73.66
70.98
73.69
73.14
75.25
79.23
76.07
78.77
Qwen3.8 Max
70.83
78.18
77.24
68.66
76.21
81.53
72.65
75.52
74.95
76.47
Gemini 3.8 Flash
68.80
79.58
72.17
73.15
78.15
77.95
70.25
75.27
81.22
79.73
Appendix
Table C15: Domain-level T for translations into Arabic, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
49.07
63.30
54.12
60.05
72.39
65.82
55.80
61.39
54.78
65.79
Claude Sonnet 5
62.91
73.90
63.77
73.05
70.51
78.62
72.10
75.49
73.38
79.31
Muse Spark 1.3
60.96
76.87
72.31
77.09
77.91
81.77
68.83
82.72
59.65
68.85
Qwen3.8 Max
66.41
79.03
75.59
74.55
76.20
81.52
74.41
77.18
72.32
74.67
Gemini 3.8 Flash
72.05
78.47
78.14
78.01
81.75
81.13
76.46
81.03
76.41
74.14
Appendix
Table C16: Domain-level T for translations into Hebrew, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
52.15
62.03
61.15
56.58
57.41
62.59
57.82
65.83
58.81
70.12
Claude Sonnet 5
66.79
71.19
69.95
74.08
74.48
72.47
68.37
78.04
77.44
85.33
Muse Spark 1.3
70.52
67.92
64.14
71.20
72.62
76.14
75.32
76.01
79.36
78.47
Qwen3.8 Max
71.47
72.91
74.82
76.61
79.04
77.72
73.31
75.71
82.70
76.22
Gemini 3.8 Flash
71.52
75.70
68.60
71.74
78.74
84.33
71.16
79.70
74.76
78.48
Appendix
Table C17: Domain-level T for translations into Hindi, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
54.15
73.71
64.82
66.25
67.65
67.15
67.63
65.10
63.20
70.33
Claude Sonnet 5
71.29
74.75
75.34
72.94
80.63
79.49
73.41
76.86
83.64
86.43
Muse Spark 1.3
68.34
80.47
80.05
78.21
79.96
79.83
82.45
80.02
76.65
83.20
Qwen3.8 Max
67.95
60.17
62.74
64.68
68.66
71.01
76.46
69.94
74.66
71.80
Gemini 3.8 Flash
71.61
74.54
81.12
80.50
84.25
85.57
79.48
78.53
82.72
84.25
Appendix
Table C18: Domain-level T for translations into Burmese, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
52.03
56.75
63.56
50.12
56.46
63.22
61.72
61.54
72.64
71.60
Claude Sonnet 5
66.55
68.81
70.45
73.12
65.33
80.62
68.11
72.11
78.66
79.09
Muse Spark 1.3
61.69
67.80
70.92
74.39
76.71
76.59
73.17
79.17
78.89
78.18
Qwen3.8 Max
64.33
67.85
75.37
72.98
73.78
84.80
73.88
80.17
76.18
75.33
Gemini 3.8 Flash
69.62
75.62
77.33
75.53
79.46
83.78
77.94
78.91
80.37
77.32
Appendix
Table C19: Domain-level T for translations into Thai, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
55.09
71.89
52.86
53.73
69.54
57.69
64.59
60.96
59.78
58.80
Claude Sonnet 5
67.25
76.49
69.70
69.54
75.52
77.56
69.45
73.16
73.53
73.76
Muse Spark 1.3
69.30
74.77
71.80
69.89
76.75
75.52
71.80
78.12
77.71
72.08
Qwen3.8 Max
68.85
75.25
80.05
71.42
78.23
82.11
73.18
81.40
84.17
77.47
Gemini 3.8 Flash
69.92
73.14
72.34
75.80
79.13
80.72
71.46
76.64
78.01
74.87
Appendix
Table C20: Domain-level T for translations into Japanese, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
52.59
60.55
53.10
53.14
62.94
60.83
57.85
57.44
49.98
66.25
Claude Sonnet 5
66.30
77.85
67.88
70.04
74.13
83.75
67.74
72.53
71.24
80.76
Muse Spark 1.3
62.99
71.06
73.24
72.31
72.03
76.10
71.67
78.63
65.54
75.46
Qwen3.8 Max
67.83
70.18
73.07
70.33
72.88
84.41
75.60
76.92
67.01
77.25
Gemini 3.8 Flash
70.37
77.34
77.53
78.38
74.21
81.42
67.57
80.98
77.32
80.40
Appendix
Table C21: Domain-level T for translations into Korean, averaged over available source languages.
Model
Transport
Dining
Products
Tourism
Health care
Safety
Social Media
Publi- cations
Finance
Public Services
Closed-weight models
GPT-5.6 Luna
49.37
56.22
53.68
58.68
67.49
60.49
55.79
58.46
58.75
58.38
Claude Sonnet 5
65.09
71.39
57.18
70.30
69.78
73.55
67.91
68.84
75.54
78.87
Muse Spark 1.3
63.50
74.83
67.63
72.17
75.90
76.83
73.36
76.09
75.93
76.73
Qwen3.8 Max
63.24
70.70
69.89
73.30
74.88
80.43
76.25
74.63
75.96
71.67
Gemini 3.8 Flash
64.06
72.17
73.37
76.63
77.34
75.43
70.11
74.29
74.91
76.04
Appendix
Table C22: Domain-level T for translations into Chinese, averaged over available source languages.
Src \ Tgt
en
cs
de
nl
fr
it
es
pt
tr
id
ms
vi
ru
uk
ar
he
hi
my
th
ja
ko
zh
en
–
18.4
26.2
23.3
15.8
19.2
23.0
19.8
18.4
22.7
19.3
19.9
22.6
26.4
23.1
17.1
18.8
27.6
23.1
24.2
20.6
21.3
cs
22.3
–
17.0
15.4
27.3
23.2
19.2
18.9
17.6
19.4
21.6
21.5
27.3
30.4
22.7
18.4
24.5
27.0
27.9
16.2
14.5
16.0
de
14.4
13.5
–
19.0
17.1
19.7
17.1
16.3
15.2
21.7
15.8
15.7
22.7
16.6
16.9
20.3
18.6
17.6
27.2
17.4
13.5
16.4
nl
23.0
22.3
15.8
–
20.0
16.8
11.1
16.3
21.5
19.3
16.3
9.7
20.1
11.2
12.2
22.5
18.1
22.7
26.5
16.5
15.5
18.2
fr
31.5
20.5
22.0
32.3
–
28.3
25.6
22.1
30.9
26.2
26.9
30.5
26.4
26.5
20.9
23.5
28.5
28.5
32.0
23.4
23.7
24.8
it
19.2
16.2
16.6
20.1
20.9
–
14.6
19.0
17.6
19.4
22.3
22.6
21.0
16.0
19.4
18.1
16.0
17.0
21.7
20.9
22.8
17.6
Appendix
Table C23: Direction-level translation quality for Gemma-4 E4B. Rows are source languages and columns are target languages.
Src \ Tgt
en
cs
de
nl
fr
it
es
pt
tr
id
ms
vi
ru
uk
ar
he
hi
my
th
ja
ko
zh
en
–
40.7
34.2
38.2
35.5
41.0
35.8
31.5
37.2
31.3
33.0
32.1
36.7
40.4
30.3
36.9
37.5
30.5
35.1
35.9
42.9
33.6
cs
32.0
–
30.5
29.9
33.1
30.1
28.4
37.7
30.3
21.6
32.0
27.3
36.4
40.2
36.3
37.2
38.5
40.3
38.1
26.1
26.6
38.7
de
35.8
33.6
–
47.3
41.1
35.3
39.0
38.4
32.8
35.8
35.6
32.0
27.8
39.1
31.3
40.1
40.2
37.2
40.9
36.5
35.6
34.7
nl
58.4
54.0
48.6
–
41.6
51.8
39.2
51.3
39.2
43.2
36.4
41.0
40.0
46.3
28.6
35.9
49.2
52.7
47.6
43.0
43.0
45.2
fr
44.0
36.9
43.4
43.1
–
40.8
42.4
51.0
39.3
42.3
44.7
33.5
49.6
38.1
40.1
41.7
52.8
47.4
43.9
51.7
47.1
37.9
it
32.5
34.3
35.8
32.8
35.7
–
38.9
41.2
31.8
37.1
34.6
36.7
34.5
28.1
32.3
33.7
30.2
36.3
36.3
33.5
34.0
35.1
Appendix
Table C24: Direction-level translation quality for GLM-4.1V 9B. Rows are source languages and columns are target languages.
Src \ Tgt
en
cs
de
nl
fr
it
es
pt
tr
id
ms
vi
ru
uk
ar
he
hi
my
th
ja
ko
zh
en
–
49.4
58.4
55.8
49.9
49.9
55.8
53.1
56.3
55.8
51.4
51.1
59.4
55.0
62.0
54.8
57.5
61.6
67.8
57.8
63.2
55.1
cs
61.1
–
58.7
69.1
64.0
59.8
56.9
66.4
71.1
61.7
71.1
59.0
68.4
73.4
71.1
61.5
66.9
75.5
61.8
61.8
65.0
65.3
de
39.0
50.9
–
55.8
54.5
54.5
52.3
49.8
47.7
64.2
51.8
54.2
52.9
57.4
49.0
53.0
55.6
52.6
60.6
51.0
50.7
56.2
nl
75.1
83.5
77.9
–
75.9
82.5
68.4
78.9
82.6
76.7
85.3
71.9
77.8
79.0
77.8
76.2
79.5
82.1
78.0
80.6
78.3
74.5
fr
60.3
68.7
64.3
57.9
–
57.0
66.2
60.4
67.5
59.1
65.0
70.0
61.3
69.0
62.4
69.8
76.5
66.3
64.3
67.2
69.7
65.3
it
46.1
57.5
57.9
48.4
53.4
–
48.0
52.4
50.3
54.8
54.3
54.6
57.5
50.3
56.7
55.8
54.2
65.0
54.7
54.0
50.7
52.2
Appendix
Table C25: Direction-level translation quality for GLM-5.3 Flash. Rows are source languages and columns are target languages.
Src \ Tgt
en
cs
de
nl
fr
it
es
pt
tr
id
ms
vi
ru
uk
ar
he
hi
my
th
ja
ko
zh
en
–
58.2
52.0
54.6
48.1
53.5
52.4
50.8
42.4
53.0
58.1
45.8
48.3
56.1
61.8
59.1
47.8
64.6
61.2
52.2
53.5
55.0
cs
66.5
–
60.0
71.5
72.4
62.2
65.2
66.8
74.0
61.9
61.9
69.6
62.5
68.7
68.2
70.3
69.2
83.8
71.5
68.7
65.3
67.1
de
45.7
47.5
–
60.3
53.6
51.5
58.7
47.9
52.9
59.6
53.7
52.3
53.5
63.0
52.4
49.4
54.0
57.4
57.1
49.3
49.6
49.5
nl
57.8
72.2
68.5
–
70.7
64.5
63.1
56.0
69.6
73.8
68.7
70.5
72.8
63.1
63.3
75.5
69.8
71.6
64.8
66.9
46.2
64.7
fr
62.5
67.9
67.5
65.9
–
58.8
63.8
68.3
65.7
69.2
71.0
67.0
61.9
66.7
69.8
73.4
68.4
73.3
67.8
62.3
62.4
64.6
it
55.9
61.0
60.8
54.7
59.0
–
56.9
54.8
59.9
62.7
57.1
59.7
59.2
55.8
53.9
59.1
56.2
65.1
64.4
62.4
58.2
62.3
Appendix
Table C26: Direction-level translation quality for GPT-5.6 Luna. Rows are source languages and columns are target languages.
Src \ Tgt
en
cs
de
nl
fr
it
es
pt
tr
id
ms
vi
ru
uk
ar
he
hi
my
th
ja
ko
zh
en
–
52.8
50.7
55.9
55.2
57.1
47.5
50.8
48.6
48.6
57.9
54.5
59.1
53.2
61.7
64.0
55.8
59.4
57.4
46.2
48.1
54.1
cs
72.6
–
69.6
78.1
81.1
80.3
70.6
84.3
76.0
73.7
75.7
73.1
78.5
77.7
80.9
73.2
85.1
79.6
76.8
71.7
75.8
76.6
de
46.6
57.6
–
58.7
56.4
54.4
60.7
50.3
57.1
66.9
58.5
52.8
60.7
61.8
59.1
58.0
54.3
61.5
68.3
54.3
61.7
58.6
nl
71.5
70.1
67.0
–
65.0
88.8
67.5
69.7
72.6
75.7
69.2
63.5
79.5
75.2
66.1
71.7
83.4
84.2
78.7
76.7
68.2
72.8
fr
68.6
70.1
63.8
72.4
–
65.5
61.0
64.0
71.7
65.0
71.0
68.5
64.0
75.0
67.1
72.5
70.7
76.8
72.4
64.0
66.6
63.9
it
52.8
61.1
52.8
57.2
58.9
–
56.5
44.9
55.7
60.7
55.0
57.4
57.5
46.7
47.6
52.9
50.5
62.7
55.5
58.5
49.7
51.9
Appendix
Table C27: Direction-level translation quality for DeepSeek Flash. Rows are source languages and columns are target languages.
Src \ Tgt
en
cs
de
nl
fr
it
es
pt
tr
id
ms
vi
ru
uk
ar
he
hi
my
th
ja
ko
zh
en
–
63.1
60.9
56.9
57.8
65.2
55.0
57.8
52.1
66.8
62.6
61.7
60.5
64.8
62.0
65.3
59.3
62.1
62.2
67.8
60.8
60.4
cs
67.7
–
61.8
64.6
64.4
65.8
61.9
55.5
58.9
65.5
69.8
60.3
61.5
65.9
73.0
61.0
68.3
71.5
61.7
68.7
54.4
63.7
de
56.3
53.6
–
63.0
56.4
55.0
59.9
60.7
61.5
65.6
57.2
57.0
56.0
62.0
59.5
53.3
64.8
62.2
63.5
57.7
54.7
59.3
nl
67.2
68.2
59.0
–
67.3
64.8
77.3
68.1
70.1
68.5
71.2
70.7
72.9
67.8
65.3
78.3
74.8
79.4
84.2
85.1
73.4
74.7
fr
62.9
67.4
63.5
57.0
–
55.9
61.6
52.3
65.1
64.7
71.4
61.0
65.2
60.7
66.7
60.4
67.0
73.8
73.2
62.8
62.3
60.6
it
59.3
60.5
62.9
65.2
62.6
–
57.5
59.9
69.1
57.9
58.6
59.4
65.1
53.3
56.8
58.1
56.4
70.4
62.3
62.4
56.5
55.5
Appendix
Table C28: Direction-level translation quality for Qwen3.8 27B. Rows are source languages and columns are target languages.
Src \ Tgt
en
cs
de
nl
fr
it
es
pt
tr
id
ms
vi
ru
uk
ar
he
hi
my
th
ja
ko
zh
en
–
65.4
61.3
57.6
54.5
61.9
52.7
57.9
56.3
56.5
60.7
68.8
63.9
68.0
63.6
64.0
58.2
72.3
63.8
51.8
58.4
58.5
cs
83.6
–
84.3
80.1
74.0
81.2
78.5
82.5
86.8
76.6
85.1
75.7
81.4
80.9
83.0
80.5
78.2
83.4
81.8
85.1
79.8
78.4
de
57.1
65.3
–
61.1
65.7
63.7
61.3
57.2
61.0
67.9
59.7
62.1
64.1
69.3
55.6
65.1
68.8
76.0
62.1
65.0
62.8
54.9
nl
79.4
84.9
84.2
–
93.0
94.0
84.3
79.9
92.7
87.7
90.5
92.7
91.2
88.9
85.1
90.9
89.5
85.9
87.5
86.1
82.6
83.4
fr
69.3
76.5
75.4
66.7
–
67.7
76.3
74.6
69.3
67.0
81.3
74.2
67.6
67.4
75.2
69.9
79.8
79.4
77.8
75.3
75.1
64.6
it
63.2
67.5
63.7
69.1
68.3
–
67.5
58.8
60.5
68.8
70.3
68.1
66.8
65.9
65.4
67.2
66.8
75.5
69.2
68.3
71.2
60.0
Appendix
Table C29: Direction-level translation quality for Claude Sonnet 5. Rows are source languages and columns are target languages.
Src \ Tgt
en
cs
de
nl
fr
it
es
pt
tr
id
ms
vi
ru
uk
ar
he
hi
my
th
ja
ko
zh
en
–
64.5
59.5
61.0
61.8
65.5
50.0
62.4
63.7
63.5
64.2
58.3
65.1
70.6
68.6
68.4
66.4
75.3
68.6
60.2
64.2
58.9
cs
82.7
–
80.7
81.2
88.3
80.8
85.4
89.8
95.9
81.7
89.0
82.4
87.2
94.8
83.5
83.2
86.4
84.3
88.4
79.8
75.5
87.5
de
46.3
61.3
–
62.4
64.4
62.5
56.2
58.1
67.2
67.4
60.2
62.2
62.9
70.4
63.0
63.1
61.6
70.2
63.9
56.0
58.9
62.9
nl
81.4
79.8
75.1
–
83.3
81.9
86.6
85.0
85.3
77.4
80.8
84.9
82.5
77.3
87.5
85.9
76.3
86.1
79.1
83.9
81.8
83.7
fr
64.4
86.3
73.1
75.7
–
76.7
78.0
71.9
81.9
76.5
78.1
64.8
82.5
79.4
77.1
77.4
81.7
83.4
79.7
76.5
72.0
71.7
it
60.9
72.1
68.1
70.3
68.4
–
65.8
60.9
64.0
66.9
68.6
65.8
63.8
70.7
69.6
71.3
64.8
67.0
67.2
76.2
62.6
64.2
Appendix
Table C30: Direction-level translation quality for Muse Spark 1.3. Rows are source languages and columns are target languages.
Src \ Tgt
en
cs
de
nl
fr
it
es
pt
tr
id
ms
vi
ru
uk
ar
he
hi
my
th
ja
ko
zh
en
–
74.9
69.4
62.2
64.5
70.2
61.6
60.7
64.7
66.0
73.0
67.4
73.3
63.9
69.3
76.2
68.2
63.5
73.3
79.6
64.7
65.6
cs
85.4
–
76.7
80.2
85.2
82.6
77.2
88.6
86.1
83.0
91.9
85.2
85.5
87.0
89.8
88.5
87.2
75.2
86.9
82.6
80.9
88.9
de
60.8
67.9
–
72.3
77.2
68.9
63.5
65.0
69.5
72.0
71.2
70.3
65.2
75.1
72.0
63.7
69.9
68.3
70.6
70.3
70.2
70.4
nl
83.2
83.7
80.5
–
83.6
77.0
88.2
81.8
80.5
80.7
85.1
85.1
89.8
82.6
80.3
82.9
85.0
70.3
78.7
86.3
78.4
77.9
fr
75.1
77.2
74.7
74.5
–
65.8
74.0
70.4
79.9
72.9
78.2
74.9
72.8
75.1
77.2
78.7
79.6
64.3
74.9
75.4
73.1
66.7
it
72.6
72.3
74.2
75.4
69.0
–
69.0
65.4
76.4
70.5
68.7
73.4
72.3
74.2
72.8
73.3
70.1
66.0
78.5
77.0
71.7
68.4
Appendix
Table C31: Direction-level translation quality for Qwen3.8 Max. Rows are source languages and columns are target languages.
Src \ Tgt
en
cs
de
nl
fr
it
es
pt
tr
id
ms
vi
ru
uk
ar
he
hi
my
th
ja
ko
zh
en
–
69.4
60.0
68.1
67.0
71.1
57.5
65.0
61.8
66.1
67.9
65.8
64.0
67.4
64.7
78.4
61.5
73.7
74.8
67.7
68.6
63.3
cs
88.1
–
88.6
84.0
83.7
81.5
90.0
86.5
87.0
87.0
85.3
83.3
87.4
86.7
82.0
89.8
89.0
88.8
84.8
85.5
86.7
84.8
de
65.5
71.9
–
75.2
77.2
66.3
68.8
69.7
66.3
73.9
68.0
64.2
67.0
76.7
64.4
70.0
71.8
78.9
71.8
72.5
66.8
68.7
nl
88.6
93.8
93.5
–
93.5
89.4
87.0
87.8
86.7
97.8
93.8
87.9
98.0
89.7
88.2
83.3
85.8
86.0
81.6
80.4
94.4
82.2
fr
73.7
80.9
79.6
75.6
–
70.8
79.7
73.7
79.4
73.9
79.3
83.1
72.8
77.7
84.7
83.6
80.1
83.0
81.3
76.2
77.4
74.9
it
68.3
74.5
70.8
76.8
72.4
–
69.5
73.9
69.1
72.9
69.8
73.5
69.1
76.1
69.3
70.7
71.6
76.5
72.4
74.8
73.0
67.9
Appendix
Table C32: Direction-level translation quality for Gemini 3.8 Flash. Rows are source languages and columns are target languages.
Src \ Tgt
en
cs
de
nl
fr
it
es
pt
tr
id
ms
vi
ru
uk
ar
he
hi
my
th
ja
ko
zh
en
–
74.0
70.7
66.7
73.8
79.9
63.8
76.8
72.9
73.8
75.8
73.0
73.3
70.9
77.9
74.8
61.4
67.5
75.2
69.4
68.5
74.4
cs
90.9
–
82.8
85.6
92.3
84.0
90.3
81.8
92.2
89.1
86.5
85.5
94.2
88.4
84.4
81.7
83.9
92.9
88.2
86.7
79.4
91.4
de
64.9
74.1
–
67.9
68.8
68.3
66.0
62.7
64.3
74.2
70.6
61.6
77.1
76.5
65.9
74.6
74.2
71.1
73.6
64.6
67.1
70.8
nl
82.3
86.4
84.8
–
92.6
86.3
75.8
90.6
90.7
91.4
78.9
90.6
86.7
90.9
89.2
81.6
88.4
76.4
86.3
86.0
88.5
83.6
fr
75.3
88.2
75.0
80.8
–
80.9
81.8
80.2
73.6
78.9
79.4
79.9
82.5
75.8
84.1
82.7
82.7
91.0
81.7
73.4
85.8
77.3
it
71.3
73.6
76.4
76.2
68.9
–
68.0
62.5
73.8
73.1
64.5
72.6
68.1
68.0
73.5
72.8
68.1
70.2
75.6
71.2
74.3
66.9
Appendix
Table C33: Direction-level translation quality for Doubao Seed 2.1 Pro. Rows are source languages and columns are target languages.
Src \ Tgt
en
cs
de
nl
fr
it
es
pt
tr
id
ms
vi
ru
uk
ar
he
hi
my
th
ja
ko
zh
en
–
81.9
79.3
78.5
70.3
77.3
82.6
76.4
83.6
76.9
78.1
80.1
79.6
77.9
79.3
79.7
75.1
84.7
82.3
78.4
76.7
78.5
cs
90.9
–
82.7
92.0
92.1
86.5
91.5
89.5
87.8
92.0
87.8
87.6
83.4
94.0
87.6
90.6
88.0
89.1
87.4
84.6
93.1
88.3
de
73.9
75.0
–
75.9
73.1
68.8
74.2
62.2
79.9
76.8
72.1
72.4
77.2
83.2
73.3
69.3
72.6
85.1
75.3
78.9
71.2
74.1
nl
86.4
83.9
88.6
–
85.8
87.0
79.6
91.8
89.9
90.7
84.2
86.2
84.3
80.8
88.5
85.5
85.8
90.7
92.1
86.5
81.9
89.4
fr
76.6
85.6
79.0
80.1
–
74.2
79.0
88.8
88.7
80.2
82.8
82.9
80.1
80.0
82.0
84.4
84.5
85.9
85.1
82.5
77.7
80.0
it
69.1
76.0
77.2
79.4
79.6
–
76.7
69.1
74.3
74.5
76.6
76.4
78.7
82.5
73.8
76.0
78.1
79.7
76.1
79.9
74.5
76.2
Appendix
Table C34: Direction-level translation quality for GPT-6 Astra. Rows are source languages and columns are target languages.
GPT-5.6 Luna
Claude Sonnet 5
Source language
T
K
V
T
K
V
English
53.73
82.94
59.80
60.77
86.78
57.40
Czech
67.96
76.06
74.83
80.99
87.00
78.48
German
53.29
89.32
58.94
63.14
84.02
65.22
Dutch
66.38
72.53
71.70
87.33
87.40
76.93
French
66.59
89.26
65.32
72.89
90.41
58.43
Appendix
Table C35: Source-language T, K, and V for GPT-5.6 Luna and Claude Sonnet 5.
Muse Spark 1.3
Qwen3.8 Max
Source language
T
K
V
T
K
V
English
63.84
81.83
64.79
68.20
84.55
74.87
Czech
85.17
94.60
85.71
84.50
92.85
87.79
German
61.96
84.86
61.95
69.25
93.89
75.31
Dutch
82.18
86.92
81.34
81.98
85.79
80.35
French
76.60
94.72
66.25
74.07
93.83
73.39
Appendix
Table C36: Source-language T, K, and V for Muse Spark 1.3 and Qwen3.8 Max.
Gemini 3.8 Flash
Doubao Seed 2.1 Pro
Source language
T
K
V
T
K
V
English
66.86
83.81
70.67
72.12
88.55
81.98
Czech
86.21
96.00
86.54
87.25
96.30
89.02
German
70.28
86.24
75.82
69.47
96.84
83.53
Dutch
89.02
85.83
78.72
86.09
92.94
83.57
French
78.16
90.46
75.94
80.53
93.75
82.36
Appendix
Table C37: Source-language T, K, and V for Gemini 3.8 Flash and Doubao Seed 2.1 Pro.
GPT-6 Astra
Gemma-4 E4B
Source language
T
K
V
T
K
V
English
78.92
88.64
91.33
21.47
41.16
16.88
Czech
88.88
98.60
91.76
21.34
29.24
15.02
German
74.48
93.65
87.49
17.74
27.28
11.96
Dutch
86.64
82.22
80.39
17.90
48.85
24.21
French
81.91
92.90
86.47
26.43
28.76
23.43
Appendix
Table C38: Source-language T, K, and V for GPT-6 Astra and Gemma-4 E4B.
GLM-4.1V 9B
GLM-5.3 Flash
Source language
T
K
V
T
K
V
English
35.74
63.93
39.42
56.24
82.04
59.21
Czech
32.92
40.99
47.09
65.22
70.29
70.43
German
36.66
58.27
40.70
53.03
84.82
57.28
Dutch
44.58
78.07
58.60
78.22
96.82
66.36
French
43.43
74.67
39.38
65.15
88.42
56.96
Appendix
Table C39: Source-language T, K, and V for GLM-4.1V 9B and GLM-5.3 Flash.
DeepSeek Flash
Qwen3.8 27B
Source language
T
K
V
T
K
V
English
54.21
83.91
68.58
61.20
82.60
65.76
Czech
76.71
85.25
83.57
64.10
81.65
80.63
German
58.01
89.89
64.80
59.01
92.96
68.48
Dutch
73.19
85.29
78.24
71.82
80.92
80.94
French
68.31
87.55
65.85
63.60
89.95
58.89
Appendix
Table C40: Source-language T, K, and V for DeepSeek Flash and Qwen3.8 27B.
This work presents a comparative evaluation of machine translation systems applied to images containing textual information, a task that lies at the intersection of computer vision and natural language processing. The study compares three main paradigms: modular pipelines that separate text detection, recognition, and translation; multi-modal large language models (MLLMs) capable of processing both image and text jointly; and an end-to-end model, Translatotron-V, which directly generates translated images. The modular systems employ state-of-the-art OCR (docTR) combined with multilingual LLMs such as Llama and EuroLLM, while the evaluated MLLMs include different configurations of Gemini 2.5. Experiments were conducted on parallel multilingual datasets covering multiple language pairs, with evaluation based on BLEU, chrF, and TER metrics. The results show that modular pipelines outperform the end-to-end approach, while MLLMs achieve the best overall performance, demonstrating superior flexibility and contextual understanding. These findings underscore the effectiveness of multi-modal reasoning for image-to-text translation and provide a solid foundation for future research on integrating visual understanding and language generation in multilingual settings.
Blai Puchol, Sergio Gómez González, Miguel Domingo +1
PRHLT Research Center - Universitat Polit`ecnica de Val`encia, Camino de Vera, s/n - Valencia · ValgrAI - Valencian Graduate School and Research Network for Artificial Intelligence, Camino de Vera s/n
Automatic evaluators such as reward models play a central role in the alignment and evaluation of large vision-language models (LVLMs). Despite their growing importance, these evaluators are almost exclusively assessed on English-centric benchmarks, leaving open the question of how well these evaluators generalize across languages. To answer this question, we introduce MM-JudgeBench, the first large-scale benchmark for multilingual and multimodal judge model evaluation, which includes over 60K pairwise preference instances spanning 25 typologically diverse languages. MM-JudgeBench integrates two complementary subsets: a general vision-language preference evaluation subset extending VL-RewardBench, and a chart-centric visual-text reasoning subset derived from OpenCQA, enabling systematic analysis of reward models (i.e., LVLM judges) across diverse settings. We additionally release a multilingual training set derived from MM-RewardBench, disjoint from our evaluation data, to support domain adaptation. By evaluating 22 LVLMs (15 open-source, 7 proprietary), we uncover substantial cross-lingual performance variance in our proposed benchmark. Our analysis further shows that model size and architecture are poor predictors of multilingual robustness, and that even state-of-the-art LVLM judges exhibit inconsistent behavior across languages. Together, these findings expose fundamental limitations of current reward modeling and underscore the necessity of multilingual, multimodal benchmarks for developing reliable automated evaluators.
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5
York University · University of Alberta · Nanyang Technological University +1
Modern translation workflows demand more than semantic equivalence. Users routinely require models to preserve JSON or HTML schemas, honor curated glossaries, disambiguate with provided context, and match prescribed registers, often several at once. Conventional metrics such as BLEU and xCOMET capture semantic fidelity but provide little signal on constraint adherence, while general instruction following benchmarks ignore the cross-lingual nature of translation. We introduce \bench, a benchmark for multilingual translation instruction following covering seven languages, with 4,506 single-constraint and 2,838 multi-constraint items spanning six constraint dimensions and five compositional patterns with instructions issued in all seven languages. Constraints are split into a gating subset verified by deterministic checkers and a continuous subset scored by a rubric-based LLM judge, combined under a multiplicative rule that resists reward hacking. Evaluating 15 models reveals systematic gaps that prior protocols miss: Instruction following scales with size more sharply than translation quality, glossary and structured-format constraints dominate the difficulty gradient, and general instruction following rankings correlate only weakly with translation behavior. Our benchmark are available at https://github.com/Tencent-Hunyuan/Hy-MT2/tree/main/IFMTBench.