Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss

Ronald Skorobogat; Ameya Prabhu; Matthias Bethge

cs.CLApr 14, 2026

Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss

Authors: Ronald Skorobogat, Ameya Prabhu, Matthias Bethge

Organizations: Tübingen AI Center, University of Tübingen

Abstract

Multilingual benchmarks guide the development of frontier models. Yet multilingual evaluations reported by frontier models are structured similar to popular reasoning and knowledge benchmarks, but across many languages. We show such benchmarks, and consequently multilingual evaluations, measure mathematical reasoning and factual recall, not multilingual proficiency. For example, thinking variants dramatically outperform instruct variants on these benchmarks, yet often perform worse on real-world multilingual tasks, such as LMArena. We propose a simple alternative: evaluate multilingual capability via round-trip translation. Given text in a source language, translate it to a target language and back; semantic gaps between the original and result expose failures in multilingual generation capabilities. Round-trip translation correlates almost perfectly (\r{ho} = 0.94) with user ratings on LMArena with our benchmark, requires no human reference translations, and does not require a more capable multilingual judge than tested models. Lastly, we introduce Lost in Translation (LiT), a challenging round-trip translation benchmark spanning widely spoken languages worldwide, for realistic evaluation of multilingual frontier models.

Figures & tables

General Knowledge Multiple-Choice
Benchmark	Contamination-Free	Challenging	Linguistic Diversity	NLG	Efficiency	Ground Truth-Free
Include	✓	✓	✓	✗	✗	✗
MMMLU	✗	✗	✗	✗	✗	✗
Global MMLU	✗	✗	✗	✗	✗	✗
Mathematical Reasoning
MT-AIME24	✗	✓	✗	✓	✗	✗

Table 1: Benchmark comparison across six evaluation criteria. We compare nine multilingual benchmarks across six evaluation criteria including: contamination-Free, challenging (not saturated yet, difficult for frontier models), linguistic diversity (covers diverse language families and scripts), NLG (tests natural language generation), Efficiency (low computational cost), and ground truth-free (without per-sample human annotation).

Panel of Figure 1 — Figure 1: Multilingual benchmarks correlate poorly with human preferences. We show benchmark scores against LMArena Elo ratings for six frontier open-source models, each in Thinking and Non-Thinking variants. (a) MT-AIME24 ( Son et al., 2025 ) shows near-zero correlation ( $\rho=-0.09$ ), with thinking variants dramatically outperforming on the benchmark but perform no better on LMArena; (b) INCLUDE ( Romanou et al., 2025 ) shows moderate negative correlation ( $\rho=-0.26$ ), with similar Thinking vs. Non-Thinking disconnect; (c) Round trip translation using LiT with percentage of MQM scores at least 80 correlates almost perfectly with LMArena ( $\rho=0.94$ , with one-sided permutation test p=0.008), with minimal gap between thinking and instruct variants of a model.

Panel of Figure 2 — Figure 2: Multilingual benchmarks track English reasoning performance, and models reason in English regardless of input language. (a) MT-AIME24 scores correlate strongly with English AIME25 performance ( $\rho=0.94$ , one-sided permutation test p=0.008), indicating the benchmark primarily measures mathematical reasoning ability. (b) INCLUDE scores correlate strongly with English MMLU-Pro ( $\rho=0.83$ , one-sided permutation test p=0.029), indicating the benchmark primarily measures factual knowledge. (c) Models overwhelmingly default to English for internal reasoning (y-axis) even when answering in the target language (x-axis). This rules out the possibility that benchmark errors reflect multilingual reasoning failures.

Panel of Figure 3 — Figure 3: Errors on multilingual benchmarks stem from reasoning and knowledge gaps—not language comprehension failures. We manually categorize all errors made by Qwen3-235B-A22B-Thinking across 11 languages. (a) On MT-AIME24, errors are overwhelmingly logical (arithmetic mistakes, flawed reasoning steps) rather than semantic (misunderstanding the translated question). (b) On INCLUDE, errors are predominantly factual (wrong facts, knowledge gaps) rather than semantic (misunderstanding the translated question). Both patterns confirm that these benchmarks measure reasoning and factual recall – not multilingual proficiency.

Model	Average	Abstracts		Pragmatics					Informal
	Average	Humanities	STEM	Core Semantics	Discourse	Implicit	Inference	Social
Gemini-3-Flash (No-Thinking)	87.2 [0.25em] $\pm$ 0.7	90.0 [0.25em] $\pm$ 1.9	85.6 [0.25em] $\pm$ 1.8	87.5 [0.25em] $\pm$ 1.9	89.3 [0.25em] $\pm$ 1.5	86.9 [0.25em] $\pm$ 2.3	88.5 [0.25em] $\pm$ 1.6	89.7 [0.25em] $\pm$ 1.6	80.3 [0.25em] $\pm$ 2.2
Qwen3.5-397B (Thinking)	73.8 [0.25em] $\pm$ 1.0	70.0 [0.25em] $\pm$ 3.7	88.1 [0.25em] $\pm$ 2.7	81.2 [0.25em] $\pm$ 2.5	75.6 [0.25em] $\pm$ 3.2	76.2 [0.25em] $\pm$ 3.1	78.4 [0.25em] $\pm$ 2.0	75.9 [0.25em] $\pm$ 2.5	45.0 [0.25em] $\pm$ 3.5
Gemma-4-31B (Instruct)	73.0 [0.25em] $\pm$ 1.0	71.2 [0.25em] $\pm$ 3.4	76.2 [0.25em] $\pm$ 4.4	78.1 [0.25em] $\pm$ 2.3	75.0 [0.25em] $\pm$ 2.8	69.0 [0.25em] $\pm$ 3.1	79.8 [0.25em] $\pm$ 1.7	77.7 [0.25em] $\pm$ 1.6	56.9 [0.25em] $\pm$ 2.7
Gemma-4-31B (Thinking)	71.9 [0.25em] $\pm$ 1.0	70.6 [0.25em] $\pm$ 3.2	76.2 [0.25em] $\pm$ 3.4	75.5 [0.25em] $\pm$ 2.6	74.4 [0.25em] $\pm$ 2.5	70.8 [0.25em] $\pm$ 2.9	74.0 [0.25em] $\pm$ 2.5	77.7 [0.25em] $\pm$ 1.8	55.9 [0.25em] $\pm$ 2.9
GLM-5 (Thinking)	70.3 [0.25em] $\pm$ 1.1	66.2 [0.25em] $\pm$ 4.0	74.4 [0.25em] $\pm$ 2.9	77.1 [0.25em] $\pm$ 3.1	77.4 [0.25em] $\pm$ 1.4	71.4 [0.25em] $\pm$ 3.3	77.4 [0.25em] $\pm$ 1.9	72.8 [0.25em] $\pm$ 3.0	45.9 [0.25em] $\pm$ 3.2

Table 2: LiT benchmark by linguistic category under

MQM_{\geq 80}

. We report the percentage of translations with MQM

\geq 80

, aggregated within each category over eight translation sequences. Each cell shows the mean and the bootstrap-estimated standard error, computed via sentence-level resampling after averaging across sequences. Higher is better. Informal text is the hardest category overall (30.8 avg), while Core Semantics is the easiest (61.5).

Model	E. Asia	C. Europe	N. East	Avg
Gemini-3-Flash (No-Think)	95.0	98.0	98.0	97.0
GLM-5 (Thinking)	93.5	93.5	82.5	89.8
Qwen3.5-397B (Thinking)	91.0	92.0	85.5	89.5
Gemma-4-31B (Instruct)	90.5	90.5	86.0	89.0
Gemma-4-31B (Thinking)	93.5	87.5	86.0	89.0
GLM-4.7 (Thinking)	92.0	90.0	80.5	87.5

Table 3:

MQM_{\geq 80}

collapses on low-resource language sequences. We report

MQM_{\geq 80}

rates, the percentage of subset examples with MQM

\geq 80

, across the same eight language sequences grouped by resource availability. (a) High-resource sequences remain solvable for the strongest frontier models, with Gemini-3-Flash averaging 97.0 and reaching 98.0 on both Central European and Near Eastern chains. (b) Medium-resource sequences remain relatively stable for the best models, though separation widens below the frontier tier. (c) Low-resource sequences collapse sharply: only Gemini-3-Flash sustains a substantial pass rate, averaging 59.2 across African and South American chains; the next-best model drops to 32.0. (d) Global averages across these eight sequences show Gemini-3-Flash far ahead of all competitors.

Model	Average	Conceptual &	Idioms &	Polysemy &	Register &	Syntactic Complexity
	Average	Abstract Nuance	Cultural Metaphors	Lexical Ambiguity	Tone Shifts	& Garden Paths
Gemini-3-Flash (No-Thinking)	71.0 [0.25em] $\pm$ 2.1	83.3 [0.25em] $\pm$ 3.4	65.6 [0.25em] $\pm$ 5.9	68.8 [0.25em] $\pm$ 5.4	60.4 [0.25em] $\pm$ 5.3	77.1 [0.25em] $\pm$ 2.5
Qwen3.5-397B (Thinking)	63.5 [0.25em] $\pm$ 2.0	75.0 [0.25em] $\pm$ 3.6	62.5 [0.25em] $\pm$ 5.9	59.4 [0.25em] $\pm$ 4.2	58.3 [0.25em] $\pm$ 3.4	62.5 [0.25em] $\pm$ 4.2
GLM-5 (Thinking)	59.8 [0.25em] $\pm$ 2.0	74.0 [0.25em] $\pm$ 2.7	54.2 [0.25em] $\pm$ 4.0	50.0 [0.25em] $\pm$ 6.9	57.3 [0.25em] $\pm$ 3.5	63.5 [0.25em] $\pm$ 4.0
Gemma-4-31B (Thinking)	55.8 [0.25em] $\pm$ 2.4	75.0 [0.25em] $\pm$ 3.6	47.9 [0.25em] $\pm$ 6.0	54.2 [0.25em] $\pm$ 6.2	55.2 [0.25em] $\pm$ 4.7	46.9 [0.25em] $\pm$ 5.9
Kimi-K2 (Thinking)	54.6 [0.25em] $\pm$ 1.8	67.7 [0.25em] $\pm$ 2.3	49.0 [0.25em] $\pm$ 4.8	55.2 [0.25em] $\pm$ 3.6	50.0 [0.25em] $\pm$ 5.1	51.0 [0.25em] $\pm$ 3.3

Table 4:

MQM_{\geq 80}

on the Backtranslation robustness subset, which targets classical round-trip translation challenges: idioms, polysemy, register shifts, and garden-path syntax. We report the percentage of sentence-sequence translations with MQM

\geq 80

, aggregated within each category over the same eight translation sequences. Each cell shows the mean and bootstrap-estimated standard error, computed via sentence-level resampling after averaging across sequences. Higher is better. Each category contains 12 source sentences. Idioms & Cultural Metaphors is the hardest category overall (35.3 avg), while Conceptual & Abstract Nuance is the easiest (54.8).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

LiT Category	Spearman $\rho$	$p$ -value
Average	0.946	$3.11\times 10^{-10}$
Humanities	0.917	$1.27\times 10^{-8}$
STEM	0.901	$5.98\times 10^{-8}$
Core Semantics	0.959	$2.46\times 10^{-11}$
Discourse Coherence	0.953	$8.98\times 10^{-11}$
Implicit Content	0.954	$7.78\times 10^{-11}$

Appendix

Table 5: Rank correlation between LiT categories and the robustness benchmark under

MQM_{\geq 80}

. We report Spearman rank correlations between model performance on each LiT category and the robustness benchmark average, using two-sided tests over all models.

Model	Average	Abstracts		Pragmatics					Informal
	Average	Humanities	STEM	Core Semantics	Discourse	Implicit	Inference	Social
Gemini-3-Flash (No-Thinking)	86.9 [0.25em] $\pm$ 0.3	88.6 [0.25em] $\pm$ 0.8	86.4 [0.25em] $\pm$ 0.9	88.8 [0.25em] $\pm$ 0.8	87.7 [0.25em] $\pm$ 0.8	85.2 [0.25em] $\pm$ 1.0	87.5 [0.25em] $\pm$ 0.8	87.5 [0.25em] $\pm$ 0.7	83.8 [0.25em] $\pm$ 0.8
Qwen3.5-397B (Thinking)	78.1 [0.25em] $\pm$ 0.6	76.9 [0.25em] $\pm$ 2.1	87.5 [0.25em] $\pm$ 1.1	85.2 [0.25em] $\pm$ 1.1	80.8 [0.25em] $\pm$ 1.4	80.0 [0.25em] $\pm$ 1.8	79.3 [0.25em] $\pm$ 1.5	79.0 [0.25em] $\pm$ 1.2	56.5 [0.25em] $\pm$ 2.2
Gemma-4-31B (Instruct)	75.2 [0.25em] $\pm$ 0.5	77.4 [0.25em] $\pm$ 1.5	82.0 [0.25em] $\pm$ 1.8	75.6 [0.25em] $\pm$ 1.6	77.8 [0.25em] $\pm$ 1.6	72.3 [0.25em] $\pm$ 1.8	76.8 [0.25em] $\pm$ 1.2	75.3 [0.25em] $\pm$ 1.1	64.7 [0.25em] $\pm$ 1.6
Gemma-4-31B (Thinking)	73.7 [0.25em] $\pm$ 0.6	77.2 [0.25em] $\pm$ 1.5	81.6 [0.25em] $\pm$ 1.7	74.9 [0.25em] $\pm$ 1.8	75.0 [0.25em] $\pm$ 1.5	71.3 [0.25em] $\pm$ 2.3	74.0 [0.25em] $\pm$ 1.5	73.6 [0.25em] $\pm$ 1.3	62.4 [0.25em] $\pm$ 1.9
GLM-5 (Thinking)	73.5 [0.25em] $\pm$ 0.7	76.1 [0.25em] $\pm$ 2.1	79.9 [0.25em] $\pm$ 2.1	79.9 [0.25em] $\pm$ 1.5	77.8 [0.25em] $\pm$ 1.3	74.0 [0.25em] $\pm$ 2.3	74.1 [0.25em] $\pm$ 1.7	74.6 [0.25em] $\pm$ 1.8	51.4 [0.25em] $\pm$ 2.3

Appendix

Table 6: LiT benchmark by linguistic category under MQM. We report mean MQM scores (higher is better), aggregated within each category over the same eight translation sequences. Each cell shows the mean and the bootstrap-estimated standard error, computed via sentence-level resampling after averaging across sequences. Informal text is the hardest category overall (36.7 avg), while Core Semantics is the easiest (64.4).

Model	Average	Abstracts		Pragmatics					Informal
	Average	Humanities	STEM	Core Semantics	Discourse	Implicit	Inference	Social
Gemini-3-Flash (No-Thinking)	89.1 [0.25em] $\pm$ 0.2	91.2 [0.25em] $\pm$ 0.6	90.5 [0.25em] $\pm$ 0.7	89.6 [0.25em] $\pm$ 0.8	88.9 [0.25em] $\pm$ 0.9	86.2 [0.25em] $\pm$ 0.9	88.5 [0.25em] $\pm$ 0.6	88.2 [0.25em] $\pm$ 0.5	89.5 [0.25em] $\pm$ 0.5
Qwen3.5-397B (Thinking)	83.9 [0.25em] $\pm$ 0.4	85.2 [0.25em] $\pm$ 1.2	91.1 [0.25em] $\pm$ 0.7	86.8 [0.25em] $\pm$ 0.8	84.9 [0.25em] $\pm$ 0.9	81.9 [0.25em] $\pm$ 1.8	82.5 [0.25em] $\pm$ 0.9	83.1 [0.25em] $\pm$ 0.7	75.7 [0.25em] $\pm$ 1.5
Gemma-4-31B (Instruct)	80.9 [0.25em] $\pm$ 0.3	83.1 [0.25em] $\pm$ 1.0	87.6 [0.25em] $\pm$ 1.0	79.9 [0.25em] $\pm$ 0.9	80.7 [0.25em] $\pm$ 1.1	76.3 [0.25em] $\pm$ 1.2	80.1 [0.25em] $\pm$ 0.7	79.4 [0.25em] $\pm$ 0.6	80.1 [0.25em] $\pm$ 0.6
GLM-5 (Thinking)	80.7 [0.25em] $\pm$ 0.4	82.7 [0.25em] $\pm$ 1.0	86.4 [0.25em] $\pm$ 1.0	82.9 [0.25em] $\pm$ 1.1	82.6 [0.25em] $\pm$ 0.8	77.8 [0.25em] $\pm$ 1.7	80.0 [0.25em] $\pm$ 0.8	79.5 [0.25em] $\pm$ 0.9	73.4 [0.25em] $\pm$ 1.5
Qwen3.5-397B (Instruct)	80.1 [0.25em] $\pm$ 0.4	83.6 [0.25em] $\pm$ 1.6	85.0 [0.25em] $\pm$ 1.7	80.4 [0.25em] $\pm$ 1.5	78.9 [0.25em] $\pm$ 1.1	77.2 [0.25em] $\pm$ 1.6	79.7 [0.25em] $\pm$ 0.9	78.6 [0.25em] $\pm$ 0.7	77.4 [0.25em] $\pm$ 0.8

Appendix

Table 7: LiT benchmark by linguistic category under raw Judge Score. We report mean judge scores (higher is better), aggregated within each category over the same eight translation sequences. Each cell shows the mean and the bootstrap-estimated standard error, computed via sentence-level resampling after averaging across sequences. Informal text is the hardest category overall (63.3 avg), while STEM is the easiest (71.3).

Model	E. Asia	C. Europe	N. East	Avg
Gemini-3-Flash (No-Think)	90.0	91.6	91.6	91.0
Qwen3.5-397B (Thinking)	89.3	90.3	87.1	88.9
GLM-5 (Thinking)	90.4	89.7	85.4	88.5
Gemma-4-31B (Thinking)	89.6	88.0	86.8	88.1
Gemma-4-31B (Instruct)	88.6	88.2	86.0	87.6
Qwen3.5-397B (Instruct)	88.1	89.3	85.2	87.5

Appendix

Table 8: Raw MQM scores similarly collapse on low-resource languages. We report the raw MQM scores across eight language sequences grouped by resource availability. (a) High-resource sequences (East Asian, Central European, Near Eastern): top frontier models remain strong, with Gemini-3-Flash leading at 91.0 average and GLM-5 (Thinking) taking the East Asian column at 90.4. (b) Medium-resource sequences (Southeast Asian, South Asian, North European): performance remains stable for the strongest models, with Gemini-3-Flash averaging 89.9. (c) Low-resource sequences (African, South American): catastrophic collapse remains, with only Gemini-3-Flash (75.4) maintaining clearly usable performance; the next-best model (Qwen3.5-397B (Thinking)) drops to 43.7. (d) Global averages across these eight sequences show Gemini-3-Flash holding roughly a 10.5-point lead over the next competitor.

Model	E. Asia	C. Europe	N. East	Avg
Gemini-3-Flash (No-Think)	91.2	94.2	93.5	93.0
Gemma-4-31B (Thinking)	91.1	91.3	90.3	90.9
Qwen3.5-397B (Thinking)	91.0	91.5	90.0	90.8
Qwen3.5-397B (Instruct)	89.7	91.9	90.1	90.6
GLM-5 (Thinking)	91.5	91.3	88.7	90.5
Gemma-4-31B (Instruct)	90.5	90.9	89.9	90.4

Appendix

Table 9: Judge scores also drop on low-resource languages. We report judge scores across the same eight language sequences on the 200-example subset, grouped by resource availability. (a) High-resource sequences remain strong for frontier models, with Gemini-3-Flash averaging 93.0 and taking both Central Europe (94.2) and Near East (93.5), while GLM-5 (Thinking) narrowly leads East Asia (91.5). (b) Medium-resource sequences remain comparatively stable for the strongest models, with Gemini-3-Flash averaging 91.6. (c) Low-resource sequences show a substantial drop, with Gemini-3-Flash still clearly first at 79.4 average, while the next-best model, Qwen3.5-397B (Thinking), falls to 64.4. (d) Global averages across these eight sequences preserve the same ranking pattern, with Gemini-3-Flash leading at 89.1 and holding a 6.0-point lead over the next competitor.

Panel of Figure 4 — Figure 4: Qwen-3 and GPT-OSS models default to English reasoning on MT-AIME24. We analyze the language used for answers (left column) and reasoning traces (right column) across 11 languages. (a-b) Qwen-3-32B answers in the target language inconsistently but reasons almost entirely in English. (c-d) Qwen-3-235B-Thinking answers consistently in the target language (100%) but still reasons predominantly in English, especially for Swahili. (e-f) GPT-OSS-20B shows mixed answering behavior but reasons 93–100% in English. (g-h) GPT-OSS-120B answers mostly in the target language but reasons almost entirely in English. These patterns confirm that mathematical reasoning occurs in English regardless of input language.

Panel of Figure 5 — Figure 5: GLM-4.7 and MiMo-V2-Flash show contrasting reasoning language patterns on MT-AIME24. (a-b) GLM-4.7 answers in mixed languages across different inputs but reasons overwhelmingly in English (93–100% for most languages), with slight exceptions for Swahili and Chinese. (c-d) MiMo-V2-Flash displays the most diverse reasoning behavior: it reasons natively 33–100% of the time depending on the language, making it an outlier among tested models. However, this native reasoning does not translate to better benchmark performance, further suggesting that MT-AIME24 measures reasoning ability rather than multilingual proficiency.

Panel of Figure 6 — Figure 6: Qwen-3 and GPT-OSS models reason almost entirely in English on INCLUDE despite answering in target languages. (a-b) Qwen-3-32B answers consistently in the target language but reasons in English 89–100% of the time. (c-d) Qwen-3-235B-Thinking achieves 100% target-language answering and 100% target-language reasoning—a unique pattern among tested models. (e-f) GPT-OSS-20B answers mostly in the target language (89–98%) but reasons in English 92–100% of the time. (g-h) GPT-OSS-120B shows similar patterns with 91–99% English reasoning. The disconnect between answer language and reasoning language explains why INCLUDE performance tracks English knowledge benchmarks.

Panel of Figure 7 — Figure 7: GLM-4.7 reasons in English while MiMo-V2-Flash shows mixed reasoning patterns on INCLUDE. (a-b) GLM-4.7 answers in the target language 87–100% of the time and reasons in English 92–100% of the time across languages. Russian (87%) and Telugu (12%) show the most answer-language variation. (c-d) MiMo-V2-Flash displays highly variable reasoning behavior: it reasons natively 33–100% of the time depending on the language, with particularly high native reasoning for Telugu (100%), Bengali (60%), and Spanish (84%). This variability makes MiMo-V2-Flash an interesting case study for understanding how reasoning language affects downstream task performance.

Panel of Figure 8 — Figure 8: Error analysis across four additional models confirms MT-AIME24 errors are logical, not linguistic. We categorize errors for four models on MT-AIME24 across 11 languages. (a-d) Across all models, errors are predominantly logical (blue) or arithmetic (green) rather than semantic (red). The semantic error rate rarely exceeds 25% for any language-model combination. For several languages (e.g., French in panels a and c), 100% of errors stem from reasoning failures. This consistent pattern across diverse model architectures confirms that MT-AIME24 does not effectively measure multilingual comprehension.

Panel of Figure 9 — Figure 9: Error analysis on INCLUDE confirms errors are factual and knowledge-based, not linguistic. We categorize errors for four models across eight languages on INCLUDE. (a-d) Across all models, the dominant error type is factual (blue), accounting for 81–98% of errors depending on language and model. Semantic errors (red) rarely exceed 10% for any configuration. Regional knowledge gaps (orange) contribute modestly (6–14% in some cases). Hallucinations appear occasionally but are not the primary failure mode. This consistent pattern confirms that INCLUDE measures factual knowledge coverage rather than multilingual comprehension ability.

Model	Temp.	Top-p	Effort
Gemini-3-Flash (No-Thinking)	1.0	0.95	–
GLM-4.7	1.0	0.95	–
GLM-5 (Thinking)	1.0	0.95	–
GLM-5 (Instruct)	1.0	0.95	–
Qwen3-235B (Thinking)	0.6	0.95	–
Qwen3-235B (Instruct)	0.7	0.8	–

Appendix

Table 10: Hyperparameter details for round-trip translation.

Model	Temp.	Top-p	Max Tokens		Effort
Model	Temp.	Top-p	AIME24	Include	Effort
Gemini-3-Flash	1.0	0.95	38,912	32,768	–
GLM-4.7	1.0	0.95	38,912	32,768	–
Qwen3-235B (Thinking)	0.6	0.95	38,912	32,768	–
Kimi-K2 (Thinking)	1.0	1.0	131,072	32,768	–
DeepSeek-V3.2-Exp (Think)	1.0	0.95	38,912	32,768	–

Appendix

Table 11: Hyperparameter details for MT-AIME24 and INCLUDE

Multilingual Language Model Evaluation

Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.

See all 482 papers

Oct 14, 2026

Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation

Large language models (LLMs) are increasingly deployed across languages, yet their safety and reasoning capabilities remain predominantly evaluated in English. Translating existing benchmarks offers a scalable approach to multilingual evaluation, but preserving task-specific meaning across languages presents significant challenges, particularly for harmful requests, mathematical reasoning, and conversational interactions. We introduce a reasoning-guided translation framework for constructing multilingual evaluations designed to preserve the structure and intended meaning of diverse benchmark tasks. Our approach separates translation strategy selection from large-scale generation, combining reasoning-guided configuration selection with task-aware validation, targeted repair, and independent quality screening. Using this framework, we develop a source-aligned evaluation collection spanning seven safety, conversational, and mathematical reasoning resources across 11 languages. The resulting collection achieves 99.94% coverage of intended item--language pairs in the two subsets requiring repair. Evaluations of open-source LLMs reveal distinct language-dependent vulnerabilities, including inconsistent refusal behaviour and substantial reasoning degradation in lower-resource languages. These findings highlight the importance of task-sensitive benchmark construction and systematic quality assessment for understanding multilingual model behaviour beyond English.

Zahraa Al Sahili, Ioannis Patras, Matthew Purver

School of Electronic Engineering and Computer Science, Queen Mary University of London, London, UK · Department of Knowledge Technologies, Jožef Stefan Institute, Ljubljana, Slovenia

Multilingual Language Model Evaluation Language Model Safety Evaluation LLM Safety Benchmarks Cross-Lingual Reasoning in Language Models

Ask PDF

Aug 4, 2026

M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task in a language rather than whether it commands the language itself, conflating fluency with proficiency. We introduce M-GATE (Multilingual Grammar, Accuracy in Translation, and Efficiency), a benchmark of linguistic proficiency spanning 30 typologically diverse languages from high- to low-resource. M-GATE comprises three tasks: grammatical error detection on linguist-crafted, adversarially selected sentences that turn on hard, language-specific phenomena; round-trip translation of shared English sources across 29 target languages, scored by a three-provider LLM judge panel validated against professional annotators; and a supplementary tokenizer-efficiency measure. We evaluate over 50 models in more than 80 configurations. Fluency and proficiency come apart sharply: models that translate competently sit near chance on the adversarial grammar items, the best reaching a Matthews correlation coefficient (MCC) of only 0.36, and their errors lean systematically toward under-flagging, accepting ungrammatical text rather than raising false alarms. Translation quality closely tracks a language's share of pretraining data (r = 0.86 against log Common Crawl share), producing a steep low-resource penalty that is nonetheless narrowing with successive model releases. Enabling reasoning reliably improves translation, while its effect on error detection is smaller and for some models negative, so the best configuration is task-dependent. To resist contamination, test items are kept private behind a continuously updated public leaderboard, with illustrative examples released (https://m-gate.ai).

Tomáš Burkert, Angelika Peljak-Łapińska, David Zelený

RWS TrainAI

Multilingual Language Model Evaluation Low-Resource Language Processing MT Evaluation

Ask PDF

Jun 5, 2026

mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?

We introduce mmPISA-bench, a compact high-quality multilingual reasoning benchmark derived from the OECD Programme for International Student Assessment (PISA). The benchmark consists of 25 multiple-choice questions that require reasoning in order to be answered correctly. Each question is provided in official human translations to 43 languages and complemented with machine-translated counterparts (i.e., 2,150 data points in total). We evaluate two mainstream proprietary LLMs across languages, reasoning effort levels, and translation types in terms of their ability to answer the questions correctly. Our results show that modern LLMs can reason effectively across all evaluated languages, achieve accuracy comparable to human test-takers, with some performance variations across covered languages. We further find that machine-translated questions do not degrade accuracy relative to official human translations which suggests that high-quality machine translation (synthetic data) might often be adequate for large-scale multilingual reasoning evaluations where official translations are not available. Finally, we analyze token usage and related inference cost and find that LLMs usage in some languages is simultaneously more expensive and less accurate.

Yerzhan Sapenov, Jaromir Savelka

School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA

Multilingual Language Models Multilingual Language Model Evaluation Educational Assessment Cross-Lingual Reasoning in Language Models Multilingual QA

Ask PDF

Abstract

Multilingual Language Model Evaluation

Explore similar work

Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation

M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?