Gold vs.~Translation: Are We Measuring Language, Knowledge, or Locality in Translated Benchmarks?

Klaudia-Doris Thellmann; Bernhard Stadler; Dan Saattrup Nielsen; Michael Färber; Jens Lehmann

cs.CLOct 28, 2026

Gold vs.~Translation: Are We Measuring Language, Knowledge, or Locality in Translated Benchmarks?

Authors: Klaudia-Doris Thellmann, Bernhard Stadler, Dan Saattrup Nielsen, Michael Färber, Jens Lehmann

Organizations: TUD Dresden University of Technology and ScaDS.AI · Alexandra Institute

Abstract

Translated versions of English-origin benchmarks such as MMLU are widely used to evaluate large language models in non-English languages, yet it remains unclear whether they measure the same constructs as natively created benchmarks. We frame this as a proxy validity question and compare DeepL-translated MMLU proxies against six native benchmarks in seven pairings across five Nordic and Baltic languages, evaluating 29 models on each pairing. Beyond ranking agreement, we test incremental proxy validity: does the translated score add predictive value for native performance after controlling for model size and language proficiency? Proxy validity varies substantially across benchmark pairings. Applying the same robustness criteria to all seven pairings, we find four robust effects, two marginal effects, and one null result. The strongest evidence occurs for curriculum-overlap benchmarks such as Skolprov ( $ΔR^2$ =.128, 95% CI [.038,.253]), whereas translated MMLU provides no significant incremental value for Sweden-specific factual knowledge in SwedishFacts (p=.242). Because SwedishFacts is also the only open-ended benchmark, we cannot fully separate the effects of local content and response format. Overall, the results indicate a clear pattern: translated benchmarks provide a better proxy when their source content overlaps with the native evaluation target, while locally anchored knowledge is less reliably captured.

Figures & tables

Native (Gold)	$n$	MMLU-Proxy	$n$
Exam-ET	1,614	ET	2,800
INCLUDE-ET	224	ET	2,800
Skolprov	545	SV	2,800
SwedishFacts	1,292	SV	2,800
DanishCitizenTest	668	DA	2,800
NRK-Quiz	3,600	NB	2,701

Table 1: Dataset overview and native-vs-proxy comparison pairings. Each native benchmark is paired with the DeepL-translated MMLU in the same language (G-MMLUX subset). Proxies sourced from EU20-MMLU for ET, LT, SV, DA and from m_mmlu for NB.

	Required knowledge					Explicit CS/CA
L	none	reg.	cult.	dial.	ties	CS	CA
DA	4	85	6	0	5	97	3
ET	77	19	3	1	0	22	78
LT	34	41	12	0	13	66	34
NB	23	30	41	2	4	75	25
SV	45	2	52	0	1	59	41

Table 2: Three-annotator majority reference for 100 native items per language (500 total). Required knowledge and explicit CS/CA (culturally sensitive / culturally agnostic) are independently annotated. Required-knowledge counts exclude 23 unresolved 1–1–1 ties. Explicit CS/CA has a unique majority for all 500 items.

	n	Acc.	CS		CA
			P	R	P	R
GlobalMMLU-Lite	217	73.3	79.5	61.7	69.4	84.5
Native (5 langs)	500	90.6	89.8	96.2	92.4	80.7

Table 3: Binary CS/CA agreement between GPT-5.2 and reference labels (%). CS/CA denotes culturally sensitive / culturally agnostic. For GlobalMMLU-Lite, the CS/CA reference follows the documented rule over human required_knowledge votes. For Native-500, the reference is the majority of three independent explicit CS/CA annotations per item.

Set	Class	P	R	F1
GlobalMMLU ( $n=217$ )	cultural	59.5	75.8	66.7
	regional	82.5	55.9	66.7
	none	76.1	82.9	79.4
	dialect	100	50.0	66.7
Native ( $n=477$ )	cultural	78.6	57.9	66.7
	regional	76.4	93.2	84.0

Table 4: Per-class precision (P), recall (R), and F1 (%) for required_knowledge predictions by GPT-5.2. Native metrics are computed on the 477 items with a unique three-annotator majority. Confusion matrices and tie handling are in Appendix C.5 .

Native	L	$\tau_{b}$ [95% CI]	T5	$\boldsymbol{\tau}_{\textbf{CA}}$	$\boldsymbol{\tau}_{\textbf{CS}}$
Skolprov	SV	.89 [.78, .96]	5/5	.88	.77
Exam-ET	ET	.88 [.78, .96]	5/5	.83	.88
INCLUDE	LT	.87 [.79, .93]	5/5	.82	.85
NRK-Quiz	NB	.85 [.76, .92]	4/5	.82	.88
DanishCT	DA	.82 [.73, .90]	3/5	.66	.84
INCLUDE	ET	.72 [.58, .83]	3/5	.68	.72

Table 5: Native-vs-proxy ranking consistency.

\tau_{b}

: Kendall’s

\tau_{b}

on overall accuracy across 29 models. T5 : top-5 model overlap.

\tau_{\text{CA}}

\tau_{\text{CS}}

\tau_{b}

restricted to culturally agnostic / sensitive items. All

p<.001

. Full statistics in Appendix D.4 .

Native	$\boldsymbol{\Delta R^{2}}$ [95% CI]	$\boldsymbol{\beta_{\text{proxy}}}$ [95% CI]	Class
Skolprov	.128 [.038,.253]	1.06 [.80,1.19]	robust
INCLUDE-LT	.076 [.021,.148]	0.94 [.67,1.13]	robust
NRK-Quiz	.041 [.009,.087]	0.46 [.29,.59]	robust
Exam-ET	.024 [.010,.044]	0.52 [.34,.85]	robust
INCLUDE-ET	.059 [.000,.199]	0.84 [ $-.02$ ,1.30]	marginal
DanishCT	.009 [.000,.027]	0.31 [.01,.57]	marginal

Table 6: Incremental proxy validity on all items. Brackets show 95% model-bootstrap CIs. Classes follow the fixed evidence definition in the text. Full details are in Appendix D.6 . The

\Delta R^{2}

CI is not used as a null test because in-sample

\Delta R^{2}\geq 0

by construction.

Figure 1: Proxy validity as a function of cultural sensitivity. Each point represents one native-vs-proxy pairing. (a) Partial correlation after residualizing on controls. (b) Incremental variance (

\Delta R^{2}

) from the nested

F

-test. The dashed line shows the MC-only linear trend (6 pairings; descriptive and not statistically significant at

n=6

). DanishCitizenTest (95% CS, MC format) already shows weak incremental validity. SwedishFacts lies close to the descriptive MC extrapolation in panel (b), which is consistent with a locality/content-overlap interpretation, but the design does not isolate locality from its open-ended response format.

Native	$\boldsymbol{\beta_{\text{proxy}}}$ [95% CI]	$\boldsymbol{p_{F}}$
Skolprov	1.17 [.88,1.28]	$<10^{-12}$
INCLUDE-LT	1.08 [.84,1.30]	$<10^{-9}$
INCLUDE-ET	1.06 [.36,1.42]	$<10^{-5}$
NRK-Quiz	0.54 [.34,.68]	$<10^{-6}$
Exam-ET	0.59 [.39,.95]	$<10^{-7}$
DanishCT	0.57 [.34,.92]	$<10^{-3}$

Table 7: Sensitivity analysis adding English G-MMLUX as a third control. Brackets are 95% percentile model-bootstrap CIs. The main evidence classes remain those in Table 6 . Full

\Delta R^{2}

, regression, and bootstrap statistics are in Appendix D.8 .

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Direct	Approx.	No-equiv.
GlobalMMLU	2,850	—	—
GlobalMMLU-Lite	217	—	—
Skolprov	—	465	80
Exam-ET	—	1,130	484
DanishCitizenTest	—	668	—
SwedishFacts	—	—	1,292

Appendix

Table 8: Items per mapping confidence level. Direct : 1:1 MMLU subject match. Approx. : plausible subject mapping with granularity or scope mismatch. No-equiv. : no MMLU subject equivalent; mapped at category level only. Datasets without native subject labels (bottom block) are classified entirely via item-level annotation.

Native Subject	English
Skolprov (SV)
kunskapsprov_läkare	Prof. Knowl.: Physician
kunskapsprov_tandläkare	Prof. Knowl.: Dentist
kunskapsprov_audionom	Prof. Knowl.: Audiologist
kunskapsprov_apotekare	Prof. Knowl.: Pharmacist
matematik_och_fysikprovet	Math. & Physics Test

Appendix

Table 9: Native exam subject labels and English translations.

Dataset	L	Items	Subj.	M-subj.	M-cat.	Ann.
GlobalMMLU	EN	2,850	57	57	6	—
G-MMLU-Lite	EN	217	43	43	6	6
Skolprov	SV	545	6	9	3	5
Exam-ET	ET	1,614	8	13	3	5
DanishCT	DA	668	2	2	1	6
SwedishFacts	SV	1,292	—	—	—	6

Appendix

Table 10: Dataset overview. Subj. : number of native subject labels. M-subj. : number of distinct MMLU subjects covered by the mapping. M-cat. : number of MMLU categories covered. Ann. : number of distinct subject_category labels assigned by GPT-5.2 annotation. Datasets without native subject labels rely on item-level annotation for category assignment.

Dataset	Native Subject	MMLU Subject(s)	Cat.	Conf.	$n$
Skolprov (SV)
	kunskapsprov_läkare	professional_medicine, clinical_knowledge, college_medicine	MED	appr.	132
	matematik_och_fysikprovet	high_school_mathematics, high_school_physics, college_physics	STEM	appr.	125
	kunskapsprov_audionom	clinical_knowledge, anatomy	MED	appr.	85
	högskoleprovet	miscellaneous	OTH	no-eq.	80
	kunskapsprov_tandläkare	professional_medicine, clinical_knowledge, anatomy	MED	appr.	70

Appendix

Table 11: Native subject to MMLU mapping. Cat. : mapped MMLU category (STEM, HUM=Humanities, SOC=Social Sciences, MED=Medical, BUS=Business, OTH=Other). Conf. : mapping confidence— appr. =approximate, no-eq. =no equivalent. GlobalMMLU and GlobalMMLU-Lite are omitted as their subjects map directly (1:1) to MMLU.

n

: number of items.

Dataset	Lang.	Items	CA	CS	Required Knowledge	Time-sens.
GlobalMMLU	EN	2,850	2,058	792	none=9505; regional=1809; cultural=1253; dialect=84	Yes=428
GlobalMMLU-Lite	EN	217	121	96	none=134; cultural=42; regional=40; dialect=1	Yes=32
Skolprov	SV	545	467	78	none=529; regional=14; dialect=2	Yes=66
Exam-ET	ET	1,614	886	728	none=1120; regional=485; cultural=9	Yes=229
DanishCitizenTest	DA	668	31	637	regional=558; cultural=77; none=33	Yes=127
SwedishFacts	SV	1,292	28	1,264	regional=641; cultural=622; none=27; dialect=2	Yes=185

Appendix

Table 12: Cultural annotation statistics. CA : culturally agnostic items. CS : culturally sensitive items. Required Knowledge : distribution of annotated knowledge types (note: GlobalMMLU counts reflect individual annotator votes, not aggregated labels). Time-sens. : items with time-sensitive content. The annotation-derived counts are used only to characterize the benchmark collections. The annotation procedure, reliability analysis, and meta-evaluation are documented separately in Appendix C .

Dataset	Native Subject	Our Cat.	GPT-5.2 Majority	$n$	Agree %	Agreement
Native exam datasets (approximate / no-equivalent mappings)
Skolprov	matematik_och_fysikprovet	STEM	STEM	125	100.0	✓
Skolprov	kunskapsprov_läkare	MED	MED	132	85.6	✓
Skolprov	kunskapsprov_tandläkare	MED	MED	70	100.0	✓
Skolprov	kunskapsprov_audionom	MED	MED	85	82.4	✓
Skolprov	kunskapsprov_apotekare	MED	MED	53	71.7	✓

Appendix

Table 13: Cross-validation of our category mapping against GPT-5.2 subject_category annotations. Our Cat. : category assigned by our mapping. GPT-5.2 Majority : most frequent category in the independent annotation. Agree % : proportion of items where our category matches the annotation. Overall agreement across 3,044 items: 85.1% . The bottom block shows disagreements originating from the MMLU taxonomy itself (law

\rightarrow

humanities vs. social sciences), not from our mapping procedure. Category abbreviations: STEM, HUM=Humanities, SOC=Social Sciences, MED=Medical, BUS=Business, OTH=Other.

Figure 2: Argilla annotation interface shown through a completed Lithuanian item about Napoleonic rule in France. The interface includes the item text in the target language, guideline and navigation controls, required-knowledge selection, time-sensitivity flag, trigger-span highlighting, conditional culture/region/country metadata, explicit CS/CA label, subject category, explanation field, and submission controls. In this example, the annotator selected geographic/regional knowledge, marked no time sensitivity, highlighted the geographic trigger span “Prancūzijoje” (France), selected region Europe and country France (FR) , assigned the CS label, chose Humanities as the subject category, and provided a short explanation before submitting.

Panel of Figure 3 — Figure 3: Full system prompt for automatic cultural sensitivity annotation via GPT-5.2. The placeholder {LANG} is replaced with the target language code. All output fields are in English; only span extractions appear in the original language. The prompt is also available in the accompanying code repository.

L	3/3 agree [95% CI]	Pairwise [95% CI]
DA	.84 [.77,.91]	.89 [.85,.94]
ET	.91 [.85,.96]	.94 [.90,.97]
LT	.80 [.72,.88]	.87 [.81,.92]
NB	.65 [.56,.74]	.77 [.71,.83]
SV	.83 [.76,.90]	.89 [.84,.93]

Appendix

Table 14: Raw/pre-QA inter-annotator agreement for the independently annotated explicit CS/CA field. Each language has three annotators and 100 items. Pairwise is mean pairwise agreement over the three annotator pairs. Brackets show 95% percentile bootstrap CIs from 5,000 item resamples.

L	3/3	2/1	1/1/1	Unique
DA	63	32	5	95
ET	81	19	0	100
LT	36	51	13	87
NB	37	59	4	96
SV	52	47	1	99
All	269	208	23	477

Appendix

Table 15: Raw/pre-QA agreement structure for four-way required_knowledge . A 1/1/1 row is an unresolved three-way tie among the three annotators. These 23 items are excluded from the primary four-way meta-evaluation.

Split	n	none	cult.	reg.	dial.	CA	CS
Dev	174	103	15	55	1	90	84
Test	217	123	33	59	2	110	107
Total	391	226	48	114	3	200	191

Appendix

Table 16: GlobalMMLU-Lite dev/test split after human label aggregation and tie resolution. Columns show required_knowledge counts after majority-vote aggregation and provided CS/CA labels.

H $\downarrow$ \ G $\rightarrow$	cult.	dial.	none	reg.	$\Sigma$
cultural	25	0	8	0	33
dialect	0	1	1	0	2
none	14	0	102	7	123
regional	3	0	23	33	59
$\Sigma$	42	1	134	40	217

Appendix

Table 17: Required-knowledge confusion matrix on the GlobalMMLU-Lite test split (

n=217

). H=human, G=GPT. Overall accuracy: 74.2%.

Human $\downarrow$ / GPT $\rightarrow$	CS	CA	$\boldsymbol{\Sigma}$
CS	307	12	319
CA	35	146	181
$\Sigma$	342	158	500

Appendix

Table 18: Explicit CS/CA confusion matrix on Native-500. Human gold is the majority of the three independently annotated explicit CS/CA labels. GPT uses its explicit cultural_sensitivity_label . Accuracy is 90.6%; macro-F1 is 89.5%.

H $\downarrow$ \ G $\rightarrow$	cult.	dial.	none	reg.	$\Sigma$
cultural	66	1	3	44	114
dialect	0	3	0	0	3
none	10	0	166	7	183
regional	8	0	4	165	177
$\Sigma$	84	4	173	216	477

Appendix

Table 19: Required-knowledge confusion matrix on the 477 Native-500 items with a unique three-annotator majority. Overall accuracy: 83.9%; macro-F1: 82.4%. Counting all 23 unresolved human ties as errors gives a conservative lower bound of 400/500=80.0%.

Construction	Acc.	Role
Human explicit vs. GPT explicit	90.6%	primary; downstream-aligned
Human explicit vs. GPT RK-derived	92.8%	sensitivity
Human RK-derived vs. GPT RK-derived	94.4%	legacy diagnostic

Appendix

Table 20: Native-500 CS/CA meta-evaluation under different constructions. “RK-derived” maps none

\to

CA and non- none

\to

CS. The primary result compares the independently annotated explicit human field with GPT’s explicit field.

Lang.	Acc.	Errors	CS $\to$ CA	CA $\to$ CS
DA	92%	8	5	3
ET	90%	10	0	10
LT	92%	8	2	6
NB	86%	14	2	12
SV	93%	7	3	4

Appendix

Table 21: Explicit CS/CA performance by language on Native-500. CS

\to

CA denotes a human-CS/GPT-CA miss. CA

\to

CS denotes a human-CA/GPT-CS false alarm. NB is the weakest language-level case (86% accuracy; 12/14 errors are CA

\to

CS).

L	Resolved $n$	Correct	Accuracy	Ties
DA	95	83	87.4%	5
ET	100	96	96.0%	0
LT	87	77	88.5%	13
NB	96	72	75.0%	4
SV	99	72	72.7%	1
All	477	400	83.9%	23

Appendix

Table 22: Four-way required_knowledge accuracy by language using only items with a unique three-annotator majority.

Cat.	Model	B	BA
M ( $\geq$ 70B)
	c4ai-command-a-03-2025-111B	111	111
	Apertus-70B-Instruct-2509	70	70
	Llama-4-Scout-17B-16E	109	17
	Mistral-Large-2411	123	123
	Llama-3.1-70B-Instruct	70	70

Appendix

Table 23: Models evaluated in our study. B : total parameters (billions). BA : active parameters (billions; differs from B for mixture-of-experts models). All evaluated checkpoints are instruction-tuned. Model labels are shortened, with exact identifiers in the released harness configuration. Regressions use B (total parameters). Inference uses NVIDIA H100 GPUs (94 GiB HBM2e). M-category models use 4-way tensor parallelism, others use a single GPU.

Group	Dataset	Lang.	Items	CA	CS	Balanced	Used in
Translated proxy benchmarks (MMLU)
MMLU-only	G-MMLUX	DA	2,800	2,059	741	741+741	A, B
MMLU-only	G-MMLUX	EN	2,800	2,059	741	741+741	A
MMLU-only	G-MMLUX	ET	2,800	2,059	741	741+741	A, B
MMLU-only	G-MMLUX	LT	2,800	2,059	741	741+741	A, B
MMLU-only	G-MMLUX	NB	2,701	2,008	693	693+693	A, B

Appendix

Table 24: Data foundation for the proxy validity analysis. Items : unique annotated items. CA / CS : culturally agnostic/sensitive item counts. Balanced : items per class in the balanced CA/CS subsets used for Block A (

\min(n_{\text{CA}},n_{\text{CS}})

per group

\times

language). Used in : A = Block A (CA vs. CS ranking), B = Block B (native vs. proxy ranking), Incr. = incremental proxy validity (§ 5.3 ). † DA non-MMLU excluded from Block A interpretation due to insufficient CA items (

n=31

). For ET and SV, multiple native datasets are pooled before balancing.

Group	L	$n_{\text{CA}}$	$n_{\text{CS}}$	$\tau_{b}$	[95% CI]	$p$
MMLU-only
	DA	741	741	0.91	[.82, .97]	$<$ 10 -11
	EN	741	741	0.91	[.85, .97]	$<$ 10 -11
	ET	741	741	0.91	[.84, .97]	$<$ 10 -11
	LT	741	741	0.94	[.86, .98]	$<$ 10 -12
	NB	693	693	0.91	[.82, .97]	$<$ 10 -11

Appendix

Table 25: Block A: CA vs. CS ranking consistency (Kendall’s

\tau_{b}

) within MMLU-only and non-MMLU native benchmarks, using balanced CA/CS subsets. † DA non-MMLU: excluded from main-text interpretation due to insufficient CA items (

n_{\text{CA}}=31

;

{\approx}

1 item per model).

Native	L	Sub.	$\boldsymbol{n}_{\textbf{nat}}$	$\boldsymbol{n}_{\textbf{prx}}$	$\tau_{b}$	[95% CI]	$\boldsymbol{p}$	T3	T5
Skolprov	SV	ALL	545	2800	.89	[.78, .96]	$<$ 10 -11	1/3	5/5
		CA	467	2059	.88	[.78, .95]	$<$ 10 -10	2/3	4/5
		CS	78	741	.77	[.66, .86]	$<$ 10 -8	1/3	4/5
Exam-ET	ET	ALL	1614	2800	.88	[.78, .96]	$<$ 10 -10	2/3	5/5
		CA	886	2059	.83	[.72, .91]	$<$ 10 -9	1/3	4/5
		CS	728	741	.88	[.78, .95]	$<$ 10 -10	2/3	5/5

Appendix

Table 26: Block B: full native-vs-proxy ranking consistency. Sub. : subset (ALL, CA-only, CS-only).

n_{\text{nat}}

n_{\text{prx}}

: number of items on native/proxy side. T3 / T5 : top-3/top-5 model overlap. The proxy is always G-MMLUX in the corresponding language. All comparisons use 29 models.

Native	L	$\boldsymbol{R^{2}_{\text{base}}}$	$\boldsymbol{R^{2}_{\text{ext}}}$	$\boldsymbol{\Delta R^{2}}$ [95% CI]	$\boldsymbol{\beta_{\text{prx}}}$ [95% CI]	$\boldsymbol{p_{F}}$	LOFO	Class
Skolprov	SV	.856	.984	.128 [.038,.253]	1.056 [.798,1.187]	$<10^{-13}$	17/17	robust
INCLUDE	LT	.890	.965	.076 [.021,.148]	0.938 [.667,1.126]	$<10^{-7}$	17/17	robust
NRK-Quiz	NB	.934	.975	.041 [.009,.087]	0.463 [.289,.592]	$<10^{-6}$	17/17	robust
Exam-ET	ET	.954	.979	.024 [.010,.044]	0.515 [.340,.852]	$<10^{-5}$	17/17	robust
INCLUDE	ET	.818	.877	.059 [.000,.199]	0.842 [ $-.021$ ,1.300]	.002	16/17	marginal
DanishCT	DA	.951	.960	.009 [.000,.027]	0.309 [.006,.575]	.027	14/17	marginal

Appendix

Table 27: Full all-item incremental proxy-validity results with model-bootstrap uncertainty. The baseline model predicts native performance from

\log(\text{total parameters})

and Belebele. The extended model adds the same-language translated MMLU proxy. LOFO reports how many of the 17 leave-one-family-out refits retain positive

\beta_{\text{proxy}}

with nominal

p<.05

. The evidence class follows the fixed definition in the main text.

		CA items		CS items
Native	L	$\boldsymbol{\Delta R^{2}}$	$\boldsymbol{p}$	$\boldsymbol{\Delta R^{2}}$	$\boldsymbol{p}$
Skolprov	SV	.140	$<$ 10 -13	.060	$<$ 10 -3
INCLUDE	LT	.095	$<$ 10 -5	.071	$<$ 10 -7
INCLUDE	ET	.059	.003	.046	.010
NRK-Quiz	NB	.048	$<$ 10 -4	.044	$<$ 10 -7
Exam-ET	ET	.031	$<$ 10 -3	.015	$<$ 10 -4

Appendix

Table 28: CS/CA-stratified incremental proxy validity.

\Delta R^{2}

and

p

from the nested

F

-test, computed separately on CA and CS items of each native benchmark. Proxy scores use the corresponding CA/CS split of G-MMLUX.

Native	L	Sub.	$\boldsymbol{r_{\text{partial}}}$	$\boldsymbol{p}$
Skolprov	SV	ALL	.943	$<10^{-14}$
		CA	.945	$<10^{-14}$
		CS	.670	$<10^{-4}$
INCLUDE	LT	ALL	.828	$<10^{-7}$
		CA	.747	$<10^{-5}$
		CS	.825	$<10^{-7}$

Appendix

Table 29: Partial correlations between native and proxy scores after residualizing on the main controls.

n=29

for all entries.

Native	L	$\boldsymbol{R^{2}_{\text{base}}}$	$\boldsymbol{R^{2}_{\text{ext}}}$	$\boldsymbol{\Delta R^{2}}$ [95% CI]	$\boldsymbol{\beta_{\text{prx}}}$ [95% CI]	$\boldsymbol{p_{F}}$
Skolprov	SV	.882	.987	.105 [.018,.224]	1.166 [.883,1.285]	$<10^{-12}$
INCLUDE	LT	.890	.977	.087 [.024,.160]	1.082 [.843,1.298]	$<10^{-9}$
INCLUDE	ET	.846	.934	.087 [.006,.233]	1.057 [.365,1.418]	$<10^{-5}$
NRK-Quiz	NB	.940	.977	.037 [.006,.080]	0.537 [.342,.679]	$<10^{-6}$
Exam-ET	ET	.956	.986	.030 [.013,.051]	0.587 [.393,.947]	$<10^{-7}$
DanishCT	DA	.952	.971	.020 [.003,.045]	0.572 [.340,.916]	$<10^{-3}$

Appendix

Table 30: Sensitivity analysis with English G-MMLUX as an additional control. Brackets show 95% percentile model-bootstrap CIs. This specification is a robustness check against general MMLU-content familiarity and is not used to define the main robust/marginal/null evidence classes.

Pairing	Sig.	Drops	Class
Skolprov	17/17	none	robust
INCLUDE-LT	17/17	none	robust
NRK-Quiz-QA	17/17	none	robust
Exam-ET	17/17	none	robust
INCLUDE-ET	16/17	Qwen (.102)	marginal
DanishCT	14/17	Cohere (.110); Llama (.072); Qwen (.096)	marginal

Appendix

Table 31: LOFO stability across all 17 model-family exclusions. Sig. reports how many refits retain a positive

\beta_{\text{proxy}}

with nominal

p<.05

. Drops lists the family exclusions for which this criterion is not met. These nominal tests are not multiplicity-corrected and are used only as a conservative stability requirement for the robust label.

Benchmark Validity

Benchmark validity concerns whether evaluation tasks, scores, and rankings support the capability claims researchers make about AI systems.

See all 177 papers

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
May 24, 2026Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber +1Multilingual Language Model Evaluation MT Evaluation
TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon +1
M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models
Aug 4, 2026Tomáš Burkert, Angelika Peljak-Łapińska, David ZelenýMultilingual Language Model Evaluation Low-Resource Language Processing
RWS TrainAI
mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?
Jun 5, 2026Yerzhan Sapenov, Jaromir SavelkaMultilingual Language Models Multilingual Language Model Evaluation
School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA

May 24, 2026J/K move · Enter open · S save

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber, Jens Lehmann

TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon+1

Multilingual Language Model Evaluation MT Evaluation

Open paper AskPDF arXiv

Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored, raising concerns about the reliability and comparability of multilingual evaluation. We address two practical gaps: (i) how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline (xCOMET-XXL) match expert human span annotations on benchmark translations, and (ii) how strongly translation errors (as opposed to source-side issues in the English original) explain accuracy drops on translated benchmarks. We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy even after controlling for English correctness and source-side anomalies.

Abstract

Benchmark Validity

Explore similar work

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?