The State and Fate of Multilingual, Contextual Evaluation in the NLP World

Manan Uppadhyay; Himanshu Beniwal; Prashant Kodali; Sunayana Sitaram

cs.CLOct 27, 2026

The State and Fate of Multilingual, Contextual Evaluation in the NLP World

Authors: Manan Uppadhyay, Himanshu Beniwal, Prashant Kodali, Sunayana Sitaram

Organizations: Microsoft Research India · IIT Gandhinagar

Abstract

Multilingual evaluation benchmarks are the primary instrument for assessing whether large language models generalize beyond English, yet the adequacy of these benchmarks has received little systematic scrutiny. We present a data-driven audit of 51 recent multilingual benchmarks spanning 242 datasets and 219 languages, organized around three pillars: coverage, representativeness, and rigor. Our analysis reveals that coverage is wide but thin with 36% of evaluated languages appearing in only a single benchmark, entire regions (Oceania, the Americas, Central Asia) are near-zero, and a stark task equity gap leaves low-resource languages evaluated on only 1-3 task categories versus 14 for high-resource languages. Representativeness is structurally compromised: translation from English remains the dominant construction strategy where 56% of all dataset-language instances are translated introducing artifacts and English-centric framing, while culturally grounded content is concentrated in a handful of community-driven benchmarks with narrow language scope. The ecosystem thus forces a trade-off between breadth and validity. Rigor is undermined by benchmark contamination, including translated benchmark leakage and parallel corpus overlap that evade surface-form detection. We synthesize these findings into concrete recommendations for building evaluation frameworks that are natively constructed, culturally grounded, contamination-aware, and designed to serve the communities whose languages they claim to evaluate. The dashboard with all the metadata collected can be found at https://aka.ms/multilingual-evals.

Figures & tables

Figure 1: Languages ranked by the number of datasets in which they appear. The long tail reveals that 36% of evaluated languages appear in only a single dataset .

Panel of Figure 2 — Figure 2: Typological concentration in multilingual benchmarks. (a) Indo-European dominates at 38% of all languages, with many families represented by fewer than 5 languages. (b) Latin script accounts for the majority; widely-used scripts such as Ge’ez, Khmer, and Myanmar are barely represented.

Figure 3: Task coverage by resource level. Each cell shows the percentage of a resource level’s dataset–language rows devoted to that task category. Left-Behinds are overwhelmingly evaluated on Translation (29%) and Sentiment/Classification (25%); specialised tasks such as Coding, Safety, Instruction Following, each account for $\leq$ 1% of their evaluation, whereas Winners are evaluated more evenly across all 15 categories.

Panel of Figure 4 — Figure 4: Representativeness of multilingual benchmarks by world region. (a) Europe and East Asia are the most translation-dependent; Sub-Saharan Africa has the highest share of natively authored content. (b) Community-driven benchmarks in Sub-Saharan Africa and South Asia achieve the highest cultural grounding rates; broad-coverage suites score lower.

Dataset	Task	Prior Work	STAMP	DCQ
Paws-X	Paraphrase Det.	9/10	67.9%	32.6%
Mgsm	Math	—	45.5%	30.9%
XCopa	Commonsense	9/10	—	—
XQuAD	QA	9/10	—	—
AfriMGSM	Math	—	21.0%	18.4%
Xnli	NLI	7/10	—	—

Table 1: Summary of contamination evidence across 14 multilingual datasets. Prior Work : fraction of models flagged as contaminated by surface-form and overlap methods ( Ahuja et al., 2024a ; Ahuja et al., 2024b ; Yao et al., 2024 ) (details in Table 2 ). STAMP : overall contamination rate (% of model–language pairs flagged at

p{<}0.05

) from our analysis using STAMP ( Rastogi et al., 2025 ) (details in Table 4 ). DCQ : mean maximum recognition rate across models and languages using the Data Contamination Quiz ( ) ; 20% is the random-chance floor (details in Table 5 ).

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Human review interface showing extracted dataset properties with evidence passages and confidence ratings, alongside structured dropdowns for reviewer adjudication of translation status, native annotator involvement, and cultural grounding.

Figure 6: Number of unique languages covered by world region. Takeaway : Coverage is concentrated in Sub-Saharan Africa, Europe, and South Asia; Oceania remains near-zero.

Figure 7: Resource-level distribution of all evaluated languages. Takeaway : 65% of evaluated languages fall in the two lowest resource classes (levels 0–1), yet depth of evaluation remains concentrated in high-resource languages.

Figure 8: Task category coverage by world region (number of datatset evaluating each task). Takeaway : QA/Knowledge dominates everywhere; specialized tasks (safety, reasoning, coding) are unevenly distributed, with Central Asia and MENA consistently thinnest.

Figure 9: Distribution of dataset construction methods across 242 unique datasets. 35% of datasets are translated from English; only 53% are originally authored in the target language(s).

Figure 10: Translation status of dataset–language instances by Joshi resource level. Takeaway : Left-Behinds have the lowest native data share; the most under-resourced languages are predominantly evaluated through translated datasets.

Figure 11: Native vs. non-native annotator status by region. Takeaway : Annotator provenance is under-documented; where reported, South Asia and Central Asia lead in native-speaker annotation, while the Americas lag.

Figure 12: Multilingual evaluation transparency across 23 recent model releases (15 open-weight, 8 closed). Left: Reporting rate across five transparency dimensions—only 65% report any multilingual evaluation, and just 13% report contamination results. Right: Breakdown by model type—closed models disclose significantly less across all dimensions.

	Llama-3.1-8B	Llama-3.1-8B-it	Mistral-7B-v0.3	Mistral-7B-v0.3-it	Gemma-2-9B-it	Gemma-2-9B	Aya-23-8B	Gemma-7B-it	Llama-2-7B-it	Mistral-7B-v0.1-it
Flores	✗	✗	✗	✗	✗	✗	✗	–	–	–
Paws-X	✗	✗	✗	✗	✗	✗	✓	✗	✗	✗
XCopa	✗	✗	✗	✗	✗	✗	✓	✗	✗	✗
XLSum	✓	✓	✗	✗	✗	✗	✗	–	–	–
Xnli	✗	✗	✗	✗	✗	✗	✗	✓	✓	✓
XQuad	✗	✗	✗	✗	✗	✗	✗	✓	✗	✗

Appendix

Table 2: Benchmark contamination presence across all evaluated models based on Ahuja et al. (2024a) , Ahuja et al. (2024b) , and Yao et al. (2024) . ✗ = contaminated , ✓ = not contaminated , – = not evaluated.

	Phi2	Phi3	Phi3.5-mini	Phi3.5-MoE	GrinMoE	Abel-7B	LLaMA2	Mistral	Qwen2	GLM4	LLaMA3	Reflection
	2.7B	3.8B	3.8B	3.8B $\times$ 16	3.8B $\times$ 16	7B	7B	7B	7B	9B	70B	70B
MMLU	23.83	67.27	68.64	76.62	77.55	47.08	44.88	57.29	69.05	67.36	78.55	75.83
MMLU-g	25.02	85.29	87.00	91.65	92.83	68.37	72.87	82.71	89.23	84.91	92.17	88.37
difference	1.20	18.02	18.36	15.03	15.28	21.29	27.99	25.42	20.18	17.55	13.62	12.54
ARC-C	42.92	80.20	59.56	54.78	63.57	50.34	36.18	64.08	84.81	86.35	61.52	56.74
ARC-C-g	47.27	92.15	93.94	96.50	96.25	66.04	44.71	85.75	95.22	91.81	95.99	94.45

Appendix

Table 3: Detecting inadvertent contamination in popular open-source LLMs from Yao et al. (2024) . Each benchmark is tested in its original form and a paraphrased variant (-g); the difference row shows the generalizability gap. Bold values indicate significantly lower generalizability, implying potential contamination.

Model	PAW-SX	Global MMLU	IndicParam	MILU	MMMLU	INCLUDE	FLORES	MGSM	AFRiMGSM	Overall
Aya-Expanse-32B	100% (4/4)	0% (0/42)	0% (0/6)	0% (0/11)	0% (0/14)	0% (0/44)	4% (1/28)	55% (6/11)	16% (3/19)	7.8%
Gemma-3-12B	50% (2/4)	12% (5/40)	0% (0/7)	0% (0/11)	14% (2/14)	5% (2/44)	0% (1/204)	55% (6/11)	16% (3/19)	5.9%
Tiny-Aya-Global	100% (4/4)	0% (0/42)	0% (0/12)	0% (0/11)	0% (0/14)	0% (0/44)	2% (5/204)	55% (6/11)	26% (5/19)	5.5%
Aya-Expanse-8B	75% (3/4)	0% (0/42)	10% (1/10)	0% (0/11)	0% (0/14)	0% (0/44)	1% (2/204)	64% (7/11)	26% (5/19)	5.0%
Qwen3-4B	100% (4/4)	0% (0/42)	8% (1/12)	0% (0/11)	0% (0/14)	0% (0/44)	0% (1/204)	36% (4/11)	21% (4/19)	3.9%
Phi-4	25% (1/4)	0% (0/42)	8% (1/12)	0% (0/11)	0% (0/14)	2% (1/44)	1% (2/204)	18% (2/11)	21% (4/19)	3.0%

Appendix

Table 4: Contamination rate per model and dataset. Each cell shows the fraction of languages flagged as contaminated (

p<0.05

), with the count in parentheses. Bold indicates the dataset is contaminated as a whole (Fisher’s combined

p<0.05

Model	AfriMGSM	Global MMLU	INCLUDE	IndicParam	MGSM	MILU	MMMLU	PAWS-X
Aya-Expanse-32B	21.3	28.7	29.7	16.2	47.4	27.0	31.8	54.4
Aya-Expanse-8B	10.8	12.6	11.5	8.7	22.3	13.6	12.4	38.9
Tiny-Aya-Global	27.5	21.0	19.1	15.1	34.6	16.7	22.8	11.4
Gemma-3-12B	39.4	21.5	40.6	36.0	56.6	36.6	19.7	41.3
Llama-3.1-8B	9.2	8.7	11.1	9.9	17.9	9.8	7.3	33.2
Phi-4	2.3	22.6	10.6	13.2	10.6	24.6	36.8	20.6

Appendix

Table 5: DCQ maximum recognition rate (%) by model and benchmark, averaged over each benchmark’s language partitions. The random-chance floor is 20%. The maximum rate selects the best raw hit rate across the tested answer positions and is therefore an optimistic estimate.

Benchmark	English	Non-English Mean
AfriMGSM	45.7%	15.3%
Global MMLU	26.1%	16.6%
MGSM	44.0%	29.6%
MILU	37.9%	18.6%
PAWS-X	47.6%	30.1%

Appendix

Table 6: DCQ maximum recognition rate averaged across the seven models for English and non-English language partitions. INCLUDE, IndicParam, and MMMLU are omitted because the evaluated partitions do not include English.

Type	Name	Release Date
Model	Aya-Expanse-32B	December 2024
	Aya-Expanse-8B	December 2024
	Tiny-Aya-Global	February 2026
	Gemma-3-12B	March 2025
	Llama-3.1-8B	July 2024
	Phi-4	December 2024

Appendix

Table 7: Release dates of models and datasets/benchmarks used in this study.

Figure 13: Landing page of the interactive dashboard showing summary statistics (51 benchmarks, 219 languages, 29 families, 29 scripts, 10 regions, 15 task types) and navigation to the four main views: Coverage, Representativeness, Benchmark Explorer, and Language Lookup.

Figure 14: Coverage Analysis view with interactive drill-down. Clicking on a bar in the language long-tail chart opens a panel listing the specific benchmarks that cover languages appearing in exactly that number of benchmarks, along with matching languages for each benchmark.

Figure 15: Benchmark Explorer view. Users can search, sort by column, and filter by region. Expanding a row reveals the paper reference, description, year, benchmark type, task categories, and full language family breakdown.

#	Benchmark	Task Categories	Year	Venue	Citation
1	MMTEB	Classification / QA / Knowledge (550 tasks)	2025	ICLR	( Enevoldsen et al., 2025 )
2	Belebele	QA / Knowledge	2024	ACL	( Bandarkar et al., 2024 )
3	Aya_Evaluation_Suite	Generation	2024	ACL	( Singh et al., 2024 )
4	FLORES-101	Translation	2022	TACL	( Goyal et al., 2022 )
5	XTREME_UP	Linguistic Probing, Other, QA / Knowledge, …	2023	ACL	( Ruder et al., 2023 )
6	MEGA	NER, NLI / Inference, QA / Knowledge, …	2023	EMNLP	( Ahuja et al., 2023 )

Appendix

Table 8: Comprehensive overview of all 51 multilingual evaluation benchmarks surveyed.

Multilingual Language Model Evaluation

Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.

See all 485 papers

Apr 22, 2026

The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models

Evaluating the multilingual and multicultural capabilities of Large Language Models (LLMs) is essential for their global utility. However, current benchmarks face three critical limitations: (1) fragmented evaluation dimensions that often neglect deep cultural nuances; (2) insufficient language coverage in subjective tasks relying on low-quality machine translation; and (3) shallow analysis that lacks diagnostic depth beyond simple rankings. To address these, we introduce GaoYao, a comprehensive benchmark with 182.3k samples, 26 languages and 51 nations/areas. First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers (General Multilingual, Cross-cultural, Monocultural) and nine cognitive sub-layers. Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective benchmarks into 19 languages and synthesizing cross-cultural test sets for 34 cultures, surpassing prior coverage by up to 111%. Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs. Our findings reveal significant geographical performance disparities and distinct gaps between tasks, offering a reliable map for future work. We release the benchmark (https://github.com/lunyiliu/GaoYao).

Yilun Liu, Chunguang Zhao, Mengyao Piao +14

Huawei, China · Fudan University, China

Multilingual Language Model Evaluation LLM Evaluation Cross-Cultural Language Model Evaluation Cultural Knowledge in Language Models

Ask PDF

Oct 14, 2026

Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation

Large language models (LLMs) are increasingly deployed across languages, yet their safety and reasoning capabilities remain predominantly evaluated in English. Translating existing benchmarks offers a scalable approach to multilingual evaluation, but preserving task-specific meaning across languages presents significant challenges, particularly for harmful requests, mathematical reasoning, and conversational interactions. We introduce a reasoning-guided translation framework for constructing multilingual evaluations designed to preserve the structure and intended meaning of diverse benchmark tasks. Our approach separates translation strategy selection from large-scale generation, combining reasoning-guided configuration selection with task-aware validation, targeted repair, and independent quality screening. Using this framework, we develop a source-aligned evaluation collection spanning seven safety, conversational, and mathematical reasoning resources across 11 languages. The resulting collection achieves 99.94% coverage of intended item--language pairs in the two subsets requiring repair. Evaluations of open-source LLMs reveal distinct language-dependent vulnerabilities, including inconsistent refusal behaviour and substantial reasoning degradation in lower-resource languages. These findings highlight the importance of task-sensitive benchmark construction and systematic quality assessment for understanding multilingual model behaviour beyond English.

Zahraa Al Sahili, Ioannis Patras, Matthew Purver

School of Electronic Engineering and Computer Science, Queen Mary University of London, London, UK · Department of Knowledge Technologies, Jožef Stefan Institute, Ljubljana, Slovenia

Multilingual Language Model Evaluation Language Model Safety Evaluation LLM Safety Benchmarks Cross-Lingual Reasoning in Language Models

Ask PDF

Mar 27, 2026

Toward Culturally Grounded Natural Language Processing

Multilingual NLP is often treated as a route to global inclusion, but linguistic coverage and cultural competence frequently diverge. This paper synthesizes over 50 papers spanning multilingual performance inequality, cross-lingual transfer, culture-aware evaluation, cultural alignment, multimodal benchmarks, benchmark-design critique, and community-grounded data practices. Across this literature, training data coverage remains important, but outcomes are also shaped by tokenization, prompt language, translated benchmark design, culturally grounded supervision, modality, and who authors or validates evaluation data. We argue that culturally grounded NLP should move beyond treating languages as isolated rows in benchmark tables and instead model communicative ecologies: the institutions, scripts, domains, modalities, and communities through which language is used. We propose a layered evaluation and reporting agenda centered on representation audits, mixed elicitation, ecological validity, community validation, adaptation provenance, within-language variation, and maintenance of living cultural resources.

Sina Bagheri Nezhad

Independent Researcher, Seattle, WA, USA

Cross-Cultural Language Model Evaluation Cultural Knowledge in Language Models

Ask PDF

Abstract

Multilingual Language Model Evaluation

Explore similar work

The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models

Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation

Toward Culturally Grounded Natural Language Processing