Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite

Klaudia Thellmann; Bernhard Stadler; Michael Färber

cs.CLApr 2, 2026

Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite

Authors: Klaudia Thellmann, Bernhard Stadler, Michael Färber

Organizations: TU Dresden and ScaDS.AI · InfAI e.V.

Abstract

Machine-translated benchmark datasets reduce costs and offer scale, but noise, loss of structure, and uneven quality weaken confidence. What matters is not merely whether we can translate, but also whether we can measure and verify translation reliability at scale. We study translation quality in the EU20 benchmark suite, which comprises five established benchmarks translated into 20 languages, via a three-step automated quality assurance approach: (i) a structural corpus audit with targeted fixes; (ii) quality profiling using a neural metric (COMET, reference-free and reference-based) with translation service comparisons (DeepL / ChatGPT / Google); and (iii) an LLM-based span-level translation error landscape. Trends are consistent: datasets with lower COMET scores exhibit a higher share of accuracy/mistranslation errors at span level (notably HellaSwag; ARC is comparatively clean). Reference-based COMET on MMLU against human-edited samples points in the same direction. We release cleaned/corrected versions of the EU20 datasets, and code for reproducibility. In sum, automated quality assurance offers practical, scalable indicators that help prioritize review -- complementing, not replacing, human gold standards.

Figures & tables

Dataset	Split	$\mathbf{N_{\text{en}}}$	$\mathbf{N_{T}}^{\phantom{\ast\ast}}$	$\mathbf{N_{C}}$	$\mathbf{N_{L}}$
ARC	train	3,370	6,420 ${}^{\ast\phantom{\ast}}$	8	6,420
ARC	val	869	17,380 ${}^{\phantom{\ast\ast}}$	17	0
ARC	test	3,548	70,960 ${}^{\phantom{\ast\ast}}$	109	0
GSM8K	train	7,473	2,288 ${}^{\phantom{\ast\ast}}$	0	2,288
GSM8K	test	1,319	26,380 ${}^{\phantom{\ast\ast}}$	0	0
He.Sw.	train	39,905	1,980 ∗∗	10	1,980

Table 1:

\mathbf{N_{\text{en}}}

: #English samples.

\mathbf{N_{T}}

: #Non-English samples.

\mathbf{N_{C}}

: #Non-English samples with missing content.

\mathbf{N_{L}}

: #Non-English samples with other Non-English version(s) missing. ∗ 3,060 samples from the respective other subset. ∗∗ All samples present translated from validation (val) split.

Figure 1: EU20 reference-free quality landscape. Left: median xCOMET-XXL per language $\times$ dataset on a unified $[0,1]$ scale; short in-cell tick encodes IQR ( $Q_{3}{-}Q_{1}$ ). Middle: median target-side sentence length (words). Right: Spearman correlation ( $\rho$ ) between length and score (negative $\rho$ indicates lower scores for longer outputs). Rows are aligned across panels and sorted by the language-wise median across datasets.

Figure 2: EU20 vs. OKAPI xCOMET-XXL reference-free quality comparison per language $\times$ dataset. Cells report the median difference $\Delta=\mathrm{median}(EU20)-\mathrm{median}(Okapi)$ on the paired overlap and the win-rate (% items where EU20 $>$ Okapi). Positive $\Delta$ favors EU20.

Figure 3: Critical-difference (CD) diagram on MMLU (ref-free). Points are systems’ average ranks across five languages (lower is better). Thin bars show Nemenyi intervals (avg

\pm

CD/2;

\alpha{=}0.05

k{=}3

N{=}5

). A grey bridge links systems that are not significantly different (no bridge = significant).

lang	$m_{\mathrm{EU20}}$	$m_{\mathrm{Okapi}}$	$m_{\mathrm{Global}}$	items	$r_{\mathrm{EU20}}$	$r_{\mathrm{Okapi}}$	$r_{\mathrm{Global}}$
de	.96	.94	.95	2378	1	3	2
es	.93	.92	.91	2378	1	2	3
fr	.91	.89	.88	2378	1	2	3
it	.92	.90	.91	2378	1	3	2
ro	.93	.91	.88	2378	1	2	3

Table 2: Per-language medians and ranks on the triple overlap (MMLU, ref-free). For each language, we report median xCOMET-XXL for EU20, Okapi, and Global-MMLU, the number of common items, and per-language ranks (1 = best).

lang	$m_{\mathrm{EU20}}^{\mathrm{ref}}$	$m_{\mathrm{Okapi}}^{\mathrm{ref}}$	$\Delta_{\mathrm{ref}}$	CI[low,high]	items
it	.92	.89	.029	[.0195, .0398]	2342
de	.96	.93	.026	[.0218, .0320]	2342
fr	.90	.88	.025	[.0150, .0344]	2342
ro	.92	.91	.015	[.0070, .0232]	2342
es	.91	.91	-.004	[-.0237, .0158]	516

Table 3: Per-language reference-based medians and deltas on MMLU. For each language, we report

\mathrm{median}(EU20_{\text{ref}})

\mathrm{median}(Okapi_{\text{ref}})

\Delta_{\text{ref}}

, 95% CI, and

n

of common items.

Figure 4:

\Delta_{\text{ref}}

(EU20

-

Okapi) of reference-based xCOMET-XXL on MMLU (reference = Global-MMLU). Bars show

\Delta_{\text{ref}}

per language with 95% paired bootstrap CIs. Zero line indicates parity. Sorted by

\Delta_{\text{ref}}

. Common items only.

Figure 5: EU20 error overview per $\text{language}\times\text{dataset}$ . Each cell shows four horizontal bars: A+M , F , O , and Clean . Error rates per 1,000 items.

	AM	F	O	CLEAN
	maj/min	maj/min	maj/min	-/-
arc	73.9/26.1	3.7/96.3	0.9/99.1	72.6
gsm8k	83.4/16.6	6.1/93.9	2.2/97.8	56.7
hellaswag	87.6/12.4	2.0/98.0	4.1/95.9	19.3
mmlu	81.9/18.1	5.1/94.9	9.4/90.6	66.3
truthfulqa	75.4/24.6	7.3/92.7	1.4/98.6	65.2

Table 4: Share of major/minor ( maj / min ) severities (in %) among majority-agreed errors per category ( A+M F , O , CLEAN )

Explore similar work

Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets
Feb 25, 2026Hanna Yukhymenko, Anton Alexandrov, Martin VechevMultilingual Language Model Evaluation MT Evaluation
INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich
Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
May 24, 2026Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber +1Multilingual Language Model Evaluation MT Evaluation
TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon +1
XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics
Apr 16, 2026Jingxuan Liu, Zhi Qu, Jin Tei +3MT Evaluation
Nara Institute of Science and Technology, Japan. · Fudan University, China.

Feb 25, 2026J/K move · Enter open · S save

Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets

Hanna Yukhymenko, Anton Alexandrov, Martin Vechev

INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich

Multilingual Language Model Evaluation MT Evaluation

Open paper AskPDF arXiv

The reliability of multilingual Large Language Model (LLM) evaluation is currently compromised by the inconsistent quality of translated benchmarks. Existing resources often suffer from semantic drift and context loss, which can lead to misleading performance metrics. In this work, we present a fully automated framework designed to address these challenges by enabling scalable, high-quality translation of datasets and benchmarks. We demonstrate that adapting test-time compute scaling strategies, specifically Universal Self-Improvement (USI) and our proposed multi-round ranking method, T-RANK, allows for significantly higher quality outputs compared to traditional pipelines. Our framework ensures that benchmarks preserve their original task structure and linguistic nuances during localization. We apply this approach to translate popular benchmarks and datasets into eight Eastern and Southern European languages (Ukrainian, Bulgarian, Slovak, Romanian, Lithuanian, Estonian, Turkish, Greek). Evaluations using both reference-based metrics and LLM-as-a-judge show that our translations surpass existing resources, resulting in more accurate downstream model assessment. We release both the framework and the improved benchmarks to facilitate robust and reproducible multilingual AI development.

Abstract

Explore similar work

Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics