Translated versions of English-origin benchmarks such as MMLU are widely used to evaluate large language models in non-English languages, yet it remains unclear whether they measure the same constructs as natively created benchmarks. We frame this as a proxy validity question and compare DeepL-translated MMLU proxies against six native benchmarks in seven pairings across five Nordic and Baltic languages, evaluating 29 models on each pairing. Beyond ranking agreement, we test incremental proxy validity: does the translated score add predictive value for native performance after controlling for model size and language proficiency? Proxy validity varies substantially across benchmark pairings. Applying the same robustness criteria to all seven pairings, we find four robust effects, two marginal effects, and one null result. The strongest evidence occurs for curriculum-overlap benchmarks such as Skolprov (ΔR2=.128, 95% CI [.038,.253]), whereas translated MMLU provides no significant incremental value for Sweden-specific factual knowledge in SwedishFacts (p=.242). Because SwedishFacts is also the only open-ended benchmark, we cannot fully separate the effects of local content and response format. Overall, the results indicate a clear pattern: translated benchmarks provide a better proxy when their source content overlaps with the native evaluation target, while locally anchored knowledge is less reliably captured.
Figures & tables
Native (Gold)
n
MMLU-Proxy
n
Exam-ET
1,614
ET
2,800
INCLUDE-ET
224
ET
2,800
Skolprov
545
SV
2,800
SwedishFacts
1,292
SV
2,800
DanishCitizenTest
668
DA
2,800
NRK-Quiz
3,600
NB
2,701
Table 1: Dataset overview and native-vs-proxy comparison pairings. Each native benchmark is paired with the DeepL-translated MMLU in the same language (G-MMLUX subset). Proxies sourced from EU20-MMLU for ET, LT, SV, DA and from m_mmlu for NB.
Required knowledge
Explicit CS/CA
L
none
reg.
cult.
dial.
ties
CS
CA
DA
4
85
6
0
5
97
3
ET
77
19
3
1
0
22
78
LT
34
41
12
0
13
66
34
NB
23
30
41
2
4
75
25
SV
45
2
52
0
1
59
41
Table 2: Three-annotator majority reference for 100 native items per language (500 total). Required knowledge and explicit CS/CA (culturally sensitive / culturally agnostic) are independently annotated. Required-knowledge counts exclude 23 unresolved 1–1–1 ties. Explicit CS/CA has a unique majority for all 500 items.
n
Acc.
CS
CA
P
R
P
R
GlobalMMLU-Lite
217
73.3
79.5
61.7
69.4
84.5
Native (5 langs)
500
90.6
89.8
96.2
92.4
80.7
Table 3: Binary CS/CA agreement between GPT-5.2 and reference labels (%). CS/CA denotes culturally sensitive / culturally agnostic. For GlobalMMLU-Lite, the CS/CA reference follows the documented rule over human required_knowledge votes. For Native-500, the reference is the majority of three independent explicit CS/CA annotations per item.
Set
Class
P
R
F1
GlobalMMLU ( n=217 )
cultural
59.5
75.8
66.7
regional
82.5
55.9
66.7
none
76.1
82.9
79.4
dialect
100
50.0
66.7
Native ( n=477 )
cultural
78.6
57.9
66.7
regional
76.4
93.2
84.0
Table 4: Per-class precision (P), recall (R), and F1 (%) for required_knowledge predictions by GPT-5.2. Native metrics are computed on the 477 items with a unique three-annotator majority. Confusion matrices and tie handling are in Appendix C.5 .
Native
L
τb [95% CI]
T5
τCA
τCS
Skolprov
SV
.89 [.78, .96]
5/5
.88
.77
Exam-ET
ET
.88 [.78, .96]
5/5
.83
.88
INCLUDE
LT
.87 [.79, .93]
5/5
.82
.85
NRK-Quiz
NB
.85 [.76, .92]
4/5
.82
.88
DanishCT
DA
.82 [.73, .90]
3/5
.66
.84
INCLUDE
ET
.72 [.58, .83]
3/5
.68
.72
Table 5: Native-vs-proxy ranking consistency. τb : Kendall’s τb on overall accuracy across 29 models. T5 : top-5 model overlap. τCA / τCS : τb restricted to culturally agnostic / sensitive items. All p<.001 . Full statistics in Appendix D.4 .
Native
ΔR2 [95% CI]
βproxy [95% CI]
Class
Skolprov
.128 [.038,.253]
1.06 [.80,1.19]
robust
INCLUDE-LT
.076 [.021,.148]
0.94 [.67,1.13]
robust
NRK-Quiz
.041 [.009,.087]
0.46 [.29,.59]
robust
Exam-ET
.024 [.010,.044]
0.52 [.34,.85]
robust
INCLUDE-ET
.059 [.000,.199]
0.84 [ −.02 ,1.30]
marginal
DanishCT
.009 [.000,.027]
0.31 [.01,.57]
marginal
Table 6: Incremental proxy validity on all items. Brackets show 95% model-bootstrap CIs. Classes follow the fixed evidence definition in the text. Full details are in Appendix D.6 . The ΔR2 CI is not used as a null test because in-sample ΔR2≥0 by construction.
Figure 1: Proxy validity as a function of cultural sensitivity. Each point represents one native-vs-proxy pairing. (a) Partial correlation after residualizing on controls. (b) Incremental variance ( ΔR2 ) from the nested F -test. The dashed line shows the MC-only linear trend (6 pairings; descriptive and not statistically significant at n=6 ). DanishCitizenTest (95% CS, MC format) already shows weak incremental validity. SwedishFacts lies close to the descriptive MC extrapolation in panel (b), which is consistent with a locality/content-overlap interpretation, but the design does not isolate locality from its open-ended response format.
Native
βproxy [95% CI]
pF
Skolprov
1.17 [.88,1.28]
<10−12
INCLUDE-LT
1.08 [.84,1.30]
<10−9
INCLUDE-ET
1.06 [.36,1.42]
<10−5
NRK-Quiz
0.54 [.34,.68]
<10−6
Exam-ET
0.59 [.39,.95]
<10−7
DanishCT
0.57 [.34,.92]
<10−3
Table 7: Sensitivity analysis adding English G-MMLUX as a third control. Brackets are 95% percentile model-bootstrap CIs. The main evidence classes remain those in Table 6 . Full ΔR2 , regression, and bootstrap statistics are in Appendix D.8 .
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Direct
Approx.
No-equiv.
GlobalMMLU
2,850
—
—
GlobalMMLU-Lite
217
—
—
Skolprov
—
465
80
Exam-ET
—
1,130
484
DanishCitizenTest
—
668
—
SwedishFacts
—
—
1,292
Appendix
Table 8: Items per mapping confidence level. Direct : 1:1 MMLU subject match. Approx. : plausible subject mapping with granularity or scope mismatch. No-equiv. : no MMLU subject equivalent; mapped at category level only. Datasets without native subject labels (bottom block) are classified entirely via item-level annotation.
Native Subject
English
Skolprov (SV)
kunskapsprov_läkare
Prof. Knowl.: Physician
kunskapsprov_tandläkare
Prof. Knowl.: Dentist
kunskapsprov_audionom
Prof. Knowl.: Audiologist
kunskapsprov_apotekare
Prof. Knowl.: Pharmacist
matematik_och_fysikprovet
Math. & Physics Test
Appendix
Table 9: Native exam subject labels and English translations.
Dataset
L
Items
Subj.
M-subj.
M-cat.
Ann.
GlobalMMLU
EN
2,850
57
57
6
—
G-MMLU-Lite
EN
217
43
43
6
6
Skolprov
SV
545
6
9
3
5
Exam-ET
ET
1,614
8
13
3
5
DanishCT
DA
668
2
2
1
6
SwedishFacts
SV
1,292
—
—
—
6
Appendix
Table 10: Dataset overview. Subj. : number of native subject labels. M-subj. : number of distinct MMLU subjects covered by the mapping. M-cat. : number of MMLU categories covered. Ann. : number of distinct subject_category labels assigned by GPT-5.2 annotation. Datasets without native subject labels rely on item-level annotation for category assignment.
Table 12: Cultural annotation statistics. CA : culturally agnostic items. CS : culturally sensitive items. Required Knowledge : distribution of annotated knowledge types (note: GlobalMMLU counts reflect individual annotator votes, not aggregated labels). Time-sens. : items with time-sensitive content. The annotation-derived counts are used only to characterize the benchmark collections. The annotation procedure, reliability analysis, and meta-evaluation are documented separately in Appendix C .
Table 13: Cross-validation of our category mapping against GPT-5.2 subject_category annotations. Our Cat. : category assigned by our mapping. GPT-5.2 Majority : most frequent category in the independent annotation. Agree % : proportion of items where our category matches the annotation. Overall agreement across 3,044 items: 85.1% . The bottom block shows disagreements originating from the MMLU taxonomy itself (law → humanities vs. social sciences), not from our mapping procedure. Category abbreviations: STEM, HUM=Humanities, SOC=Social Sciences, MED=Medical, BUS=Business, OTH=Other.
Figure 2: Argilla annotation interface shown through a completed Lithuanian item about Napoleonic rule in France. The interface includes the item text in the target language, guideline and navigation controls, required-knowledge selection, time-sensitivity flag, trigger-span highlighting, conditional culture/region/country metadata, explicit CS/CA label, subject category, explanation field, and submission controls. In this example, the annotator selected geographic/regional knowledge, marked no time sensitivity, highlighted the geographic trigger span “Prancūzijoje” (France), selected region Europe and country France (FR) , assigned the CS label, chose Humanities as the subject category, and provided a short explanation before submitting.
Figure 3: Full system prompt for automatic cultural sensitivity annotation via GPT-5.2. The placeholder {LANG} is replaced with the target language code. All output fields are in English; only span extractions appear in the original language. The prompt is also available in the accompanying code repository.
L
3/3 agree [95% CI]
Pairwise [95% CI]
DA
.84 [.77,.91]
.89 [.85,.94]
ET
.91 [.85,.96]
.94 [.90,.97]
LT
.80 [.72,.88]
.87 [.81,.92]
NB
.65 [.56,.74]
.77 [.71,.83]
SV
.83 [.76,.90]
.89 [.84,.93]
Appendix
Table 14: Raw/pre-QA inter-annotator agreement for the independently annotated explicit CS/CA field. Each language has three annotators and 100 items. Pairwise is mean pairwise agreement over the three annotator pairs. Brackets show 95% percentile bootstrap CIs from 5,000 item resamples.
L
3/3
2/1
1/1/1
Unique
DA
63
32
5
95
ET
81
19
0
100
LT
36
51
13
87
NB
37
59
4
96
SV
52
47
1
99
All
269
208
23
477
Appendix
Table 15: Raw/pre-QA agreement structure for four-way required_knowledge . A 1/1/1 row is an unresolved three-way tie among the three annotators. These 23 items are excluded from the primary four-way meta-evaluation.
Split
n
none
cult.
reg.
dial.
CA
CS
Dev
174
103
15
55
1
90
84
Test
217
123
33
59
2
110
107
Total
391
226
48
114
3
200
191
Appendix
Table 16: GlobalMMLU-Lite dev/test split after human label aggregation and tie resolution. Columns show required_knowledge counts after majority-vote aggregation and provided CS/CA labels.
H ↓ \ G →
cult.
dial.
none
reg.
Σ
cultural
25
0
8
0
33
dialect
0
1
1
0
2
none
14
0
102
7
123
regional
3
0
23
33
59
Σ
42
1
134
40
217
Appendix
Table 17: Required-knowledge confusion matrix on the GlobalMMLU-Lite test split ( n=217 ). H=human, G=GPT. Overall accuracy: 74.2%.
Human ↓ / GPT →
CS
CA
Σ
CS
307
12
319
CA
35
146
181
Σ
342
158
500
Appendix
Table 18: Explicit CS/CA confusion matrix on Native-500. Human gold is the majority of the three independently annotated explicit CS/CA labels. GPT uses its explicit cultural_sensitivity_label . Accuracy is 90.6%; macro-F1 is 89.5%.
H ↓ \ G →
cult.
dial.
none
reg.
Σ
cultural
66
1
3
44
114
dialect
0
3
0
0
3
none
10
0
166
7
183
regional
8
0
4
165
177
Σ
84
4
173
216
477
Appendix
Table 19: Required-knowledge confusion matrix on the 477 Native-500 items with a unique three-annotator majority. Overall accuracy: 83.9%; macro-F1: 82.4%. Counting all 23 unresolved human ties as errors gives a conservative lower bound of 400/500=80.0%.
Construction
Acc.
Role
Human explicit vs. GPT explicit
90.6%
primary; downstream-aligned
Human explicit vs. GPT RK-derived
92.8%
sensitivity
Human RK-derived vs. GPT RK-derived
94.4%
legacy diagnostic
Appendix
Table 20: Native-500 CS/CA meta-evaluation under different constructions. “RK-derived” maps none → CA and non- none → CS. The primary result compares the independently annotated explicit human field with GPT’s explicit field.
Lang.
Acc.
Errors
CS → CA
CA → CS
DA
92%
8
5
3
ET
90%
10
0
10
LT
92%
8
2
6
NB
86%
14
2
12
SV
93%
7
3
4
Appendix
Table 21: Explicit CS/CA performance by language on Native-500. CS → CA denotes a human-CS/GPT-CA miss. CA → CS denotes a human-CA/GPT-CS false alarm. NB is the weakest language-level case (86% accuracy; 12/14 errors are CA → CS).
L
Resolved n
Correct
Accuracy
Ties
DA
95
83
87.4%
5
ET
100
96
96.0%
0
LT
87
77
88.5%
13
NB
96
72
75.0%
4
SV
99
72
72.7%
1
All
477
400
83.9%
23
Appendix
Table 22: Four-way required_knowledge accuracy by language using only items with a unique three-annotator majority.
Cat.
Model
B
BA
M ( ≥ 70B)
c4ai-command-a-03-2025-111B
111
111
Apertus-70B-Instruct-2509
70
70
Llama-4-Scout-17B-16E
109
17
Mistral-Large-2411
123
123
Llama-3.1-70B-Instruct
70
70
Appendix
Table 23: Models evaluated in our study. B : total parameters (billions). BA : active parameters (billions; differs from B for mixture-of-experts models). All evaluated checkpoints are instruction-tuned. Model labels are shortened, with exact identifiers in the released harness configuration. Regressions use B (total parameters). Inference uses NVIDIA H100 GPUs (94 GiB HBM2e). M-category models use 4-way tensor parallelism, others use a single GPU.
Group
Dataset
Lang.
Items
CA
CS
Balanced
Used in
Translated proxy benchmarks (MMLU)
MMLU-only
G-MMLUX
DA
2,800
2,059
741
741+741
A, B
MMLU-only
G-MMLUX
EN
2,800
2,059
741
741+741
A
MMLU-only
G-MMLUX
ET
2,800
2,059
741
741+741
A, B
MMLU-only
G-MMLUX
LT
2,800
2,059
741
741+741
A, B
MMLU-only
G-MMLUX
NB
2,701
2,008
693
693+693
A, B
Appendix
Table 24: Data foundation for the proxy validity analysis. Items : unique annotated items. CA / CS : culturally agnostic/sensitive item counts. Balanced : items per class in the balanced CA/CS subsets used for Block A ( min(nCA,nCS) per group × language). Used in : A = Block A (CA vs. CS ranking), B = Block B (native vs. proxy ranking), Incr. = incremental proxy validity (§ 5.3 ). † DA non-MMLU excluded from Block A interpretation due to insufficient CA items ( n=31 ). For ET and SV, multiple native datasets are pooled before balancing.
Group
L
nCA
nCS
τb
[95% CI]
p
MMLU-only
DA
741
741
0.91
[.82, .97]
< 10 -11
EN
741
741
0.91
[.85, .97]
< 10 -11
ET
741
741
0.91
[.84, .97]
< 10 -11
LT
741
741
0.94
[.86, .98]
< 10 -12
NB
693
693
0.91
[.82, .97]
< 10 -11
Appendix
Table 25: Block A: CA vs. CS ranking consistency (Kendall’s τb ) within MMLU-only and non-MMLU native benchmarks, using balanced CA/CS subsets. † DA non-MMLU: excluded from main-text interpretation due to insufficient CA items ( nCA=31 ; ≈ 1 item per model).
Native
L
Sub.
nnat
nprx
τb
[95% CI]
p
T3
T5
Skolprov
SV
ALL
545
2800
.89
[.78, .96]
< 10 -11
1/3
5/5
CA
467
2059
.88
[.78, .95]
< 10 -10
2/3
4/5
CS
78
741
.77
[.66, .86]
< 10 -8
1/3
4/5
Exam-ET
ET
ALL
1614
2800
.88
[.78, .96]
< 10 -10
2/3
5/5
CA
886
2059
.83
[.72, .91]
< 10 -9
1/3
4/5
CS
728
741
.88
[.78, .95]
< 10 -10
2/3
5/5
Appendix
Table 26: Block B: full native-vs-proxy ranking consistency. Sub. : subset (ALL, CA-only, CS-only). nnat / nprx : number of items on native/proxy side. T3 / T5 : top-3/top-5 model overlap. The proxy is always G-MMLUX in the corresponding language. All comparisons use 29 models.
Native
L
Rbase2
Rext2
ΔR2 [95% CI]
βprx [95% CI]
pF
LOFO
Class
Skolprov
SV
.856
.984
.128 [.038,.253]
1.056 [.798,1.187]
<10−13
17/17
robust
INCLUDE
LT
.890
.965
.076 [.021,.148]
0.938 [.667,1.126]
<10−7
17/17
robust
NRK-Quiz
NB
.934
.975
.041 [.009,.087]
0.463 [.289,.592]
<10−6
17/17
robust
Exam-ET
ET
.954
.979
.024 [.010,.044]
0.515 [.340,.852]
<10−5
17/17
robust
INCLUDE
ET
.818
.877
.059 [.000,.199]
0.842 [ −.021 ,1.300]
.002
16/17
marginal
DanishCT
DA
.951
.960
.009 [.000,.027]
0.309 [.006,.575]
.027
14/17
marginal
Appendix
Table 27: Full all-item incremental proxy-validity results with model-bootstrap uncertainty. The baseline model predicts native performance from log(total parameters) and Belebele. The extended model adds the same-language translated MMLU proxy. LOFO reports how many of the 17 leave-one-family-out refits retain positive βproxy with nominal p<.05 . The evidence class follows the fixed definition in the main text.
CA items
CS items
Native
L
ΔR2
p
ΔR2
p
Skolprov
SV
.140
< 10 -13
.060
< 10 -3
INCLUDE
LT
.095
< 10 -5
.071
< 10 -7
INCLUDE
ET
.059
.003
.046
.010
NRK-Quiz
NB
.048
< 10 -4
.044
< 10 -7
Exam-ET
ET
.031
< 10 -3
.015
< 10 -4
Appendix
Table 28: CS/CA-stratified incremental proxy validity. ΔR2 and p from the nested F -test, computed separately on CA and CS items of each native benchmark. Proxy scores use the corresponding CA/CS split of G-MMLUX.
Native
L
Sub.
rpartial
p
Skolprov
SV
ALL
.943
<10−14
CA
.945
<10−14
CS
.670
<10−4
INCLUDE
LT
ALL
.828
<10−7
CA
.747
<10−5
CS
.825
<10−7
Appendix
Table 29: Partial correlations between native and proxy scores after residualizing on the main controls. n=29 for all entries.
Native
L
Rbase2
Rext2
ΔR2 [95% CI]
βprx [95% CI]
pF
Skolprov
SV
.882
.987
.105 [.018,.224]
1.166 [.883,1.285]
<10−12
INCLUDE
LT
.890
.977
.087 [.024,.160]
1.082 [.843,1.298]
<10−9
INCLUDE
ET
.846
.934
.087 [.006,.233]
1.057 [.365,1.418]
<10−5
NRK-Quiz
NB
.940
.977
.037 [.006,.080]
0.537 [.342,.679]
<10−6
Exam-ET
ET
.956
.986
.030 [.013,.051]
0.587 [.393,.947]
<10−7
DanishCT
DA
.952
.971
.020 [.003,.045]
0.572 [.340,.916]
<10−3
Appendix
Table 30: Sensitivity analysis with English G-MMLUX as an additional control. Brackets show 95% percentile model-bootstrap CIs. This specification is a robustness check against general MMLU-content familiarity and is not used to define the main robust/marginal/null evidence classes.
Pairing
Sig.
Drops
Class
Skolprov
17/17
none
robust
INCLUDE-LT
17/17
none
robust
NRK-Quiz-QA
17/17
none
robust
Exam-ET
17/17
none
robust
INCLUDE-ET
16/17
Qwen (.102)
marginal
DanishCT
14/17
Cohere (.110); Llama (.072); Qwen (.096)
marginal
Appendix
Table 31: LOFO stability across all 17 model-family exclusions. Sig. reports how many refits retain a positive βproxy with nominal p<.05 . Drops lists the family exclusions for which this criterion is not met. These nominal tests are not multiplicity-corrected and are used only as a conservative stability requirement for the robust label.
Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored, raising concerns about the reliability and comparability of multilingual evaluation. We address two practical gaps: (i) how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline (xCOMET-XXL) match expert human span annotations on benchmark translations, and (ii) how strongly translation errors (as opposed to source-side issues in the English original) explain accuracy drops on translated benchmarks. We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy even after controlling for English correctness and source-side anomalies.
Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber +1
TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon +1
Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task in a language rather than whether it commands the language itself, conflating fluency with proficiency. We introduce M-GATE (Multilingual Grammar, Accuracy in Translation, and Efficiency), a benchmark of linguistic proficiency spanning 30 typologically diverse languages from high- to low-resource. M-GATE comprises three tasks: grammatical error detection on linguist-crafted, adversarially selected sentences that turn on hard, language-specific phenomena; round-trip translation of shared English sources across 29 target languages, scored by a three-provider LLM judge panel validated against professional annotators; and a supplementary tokenizer-efficiency measure. We evaluate over 50 models in more than 80 configurations. Fluency and proficiency come apart sharply: models that translate competently sit near chance on the adversarial grammar items, the best reaching a Matthews correlation coefficient (MCC) of only 0.36, and their errors lean systematically toward under-flagging, accepting ungrammatical text rather than raising false alarms. Translation quality closely tracks a language's share of pretraining data (r = 0.86 against log Common Crawl share), producing a steep low-resource penalty that is nonetheless narrowing with successive model releases. Enabling reasoning reliably improves translation, while its effect on error detection is smaller and for some models negative, so the best configuration is task-dependent. To resist contamination, test items are kept private behind a continuously updated public leaderboard, with illustrative examples released (https://m-gate.ai).
Tomáš Burkert, Angelika Peljak-Łapińska, David Zelený
We introduce mmPISA-bench, a compact high-quality multilingual reasoning benchmark derived from the OECD Programme for International Student Assessment (PISA). The benchmark consists of 25 multiple-choice questions that require reasoning in order to be answered correctly. Each question is provided in official human translations to 43 languages and complemented with machine-translated counterparts (i.e., 2,150 data points in total). We evaluate two mainstream proprietary LLMs across languages, reasoning effort levels, and translation types in terms of their ability to answer the questions correctly. Our results show that modern LLMs can reason effectively across all evaluated languages, achieve accuracy comparable to human test-takers, with some performance variations across covered languages. We further find that machine-translated questions do not degrade accuracy relative to official human translations which suggests that high-quality machine translation (synthetic data) might often be adequate for large-scale multilingual reasoning evaluations where official translations are not available. Finally, we analyze token usage and related inference cost and find that LLMs usage in some languages is simultaneously more expensive and less accurate.
Yerzhan Sapenov, Jaromir Savelka
School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA