cs.CLOct 27, 2026
SaveThe State and Fate of Multilingual, Contextual Evaluation in the NLP World
Organizations: Microsoft Research India · IIT Gandhinagar
Abstract
Multilingual evaluation benchmarks are the primary instrument for assessing whether large language models generalize beyond English, yet the adequacy of these benchmarks has received little systematic scrutiny. We present a data-driven audit of 51 recent multilingual benchmarks spanning 242 datasets and 219 languages, organized around three pillars: coverage, representativeness, and rigor. Our analysis reveals that coverage is wide but thin with 36% of evaluated languages appearing in only a single benchmark, entire regions (Oceania, the Americas, Central Asia) are near-zero, and a stark task equity gap leaves low-resource languages evaluated on only 1-3 task categories versus 14 for high-resource languages. Representativeness is structurally compromised: translation from English remains the dominant construction strategy where 56% of all dataset-language instances are translated introducing artifacts and English-centric framing, while culturally grounded content is concentrated in a handful of community-driven benchmarks with narrow language scope. The ecosystem thus forces a trade-off between breadth and validity. Rigor is undermined by benchmark contamination, including translated benchmark leakage and parallel corpus overlap that evade surface-form detection. We synthesize these findings into concrete recommendations for building evaluation frameworks that are natively constructed, culturally grounded, contamination-aware, and designed to serve the communities whose languages they claim to evaluate. The dashboard with all the metadata collected can be found at https://aka.ms/multilingual-evals.
Figures & tables
Figure 1: Languages ranked by the number of datasets in which they appear. The long tail reveals that 36% of evaluated languages appear in only a single dataset .
Figure 2: Typological concentration in multilingual benchmarks. (a) Indo-European dominates at 38% of all languages, with many families represented by fewer than 5 languages. (b) Latin script accounts for the majority; widely-used scripts such as Ge’ez, Khmer, and Myanmar are barely represented.
Figure 3: Task coverage by resource level. Each cell shows the percentage of a resource level’s dataset–language rows devoted to that task category. Left-Behinds are overwhelmingly evaluated on Translation (29%) and Sentiment/Classification (25%); specialised tasks such as Coding, Safety, Instruction Following, each account for 1% of their evaluation, whereas Winners are evaluated more evenly across all 15 categories.
Figure 4: Representativeness of multilingual benchmarks by world region. (a) Europe and East Asia are the most translation-dependent; Sub-Saharan Africa has the highest share of natively authored content. (b) Community-driven benchmarks in Sub-Saharan Africa and South Asia achieve the highest cultural grounding rates; broad-coverage suites score lower.
| Dataset | Task | Prior Work | STAMP | DCQ |
| Paws-X | Paraphrase Det. | 9/10 | 67.9% | 32.6% |
| Mgsm | Math | — | 45.5% | 30.9% |
| XCopa | Commonsense | 9/10 | — | — |
| XQuAD | QA | 9/10 | — | — |
| AfriMGSM | Math | — | 21.0% | 18.4% |
| Xnli | NLI | 7/10 | — | — |
Table 1: Summary of contamination evidence across 14 multilingual datasets. Prior Work : fraction of models flagged as contaminated by surface-form and overlap methods ( Ahuja et al., 2024a ; Ahuja et al., 2024b ; Yao et al., 2024 ) (details in Table 2 ). STAMP : overall contamination rate (% of model–language pairs flagged at ) from our analysis using STAMP ( Rastogi et al., 2025 ) (details in Table 4 ). DCQ : mean maximum recognition rate across models and languages using the Data Contamination Quiz ( ) ; 20% is the random-chance floor (details in Table 5 ).
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Human review interface showing extracted dataset properties with evidence passages and confidence ratings, alongside structured dropdowns for reviewer adjudication of translation status, native annotator involvement, and cultural grounding.
Figure 6: Number of unique languages covered by world region. Takeaway : Coverage is concentrated in Sub-Saharan Africa, Europe, and South Asia; Oceania remains near-zero.
Figure 7: Resource-level distribution of all evaluated languages. Takeaway : 65% of evaluated languages fall in the two lowest resource classes (levels 0–1), yet depth of evaluation remains concentrated in high-resource languages.
Figure 8: Task category coverage by world region (number of datatset evaluating each task). Takeaway : QA/Knowledge dominates everywhere; specialized tasks (safety, reasoning, coding) are unevenly distributed, with Central Asia and MENA consistently thinnest.
Figure 9: Distribution of dataset construction methods across 242 unique datasets. 35% of datasets are translated from English; only 53% are originally authored in the target language(s).
Figure 10: Translation status of dataset–language instances by Joshi resource level. Takeaway : Left-Behinds have the lowest native data share; the most under-resourced languages are predominantly evaluated through translated datasets.
Figure 11: Native vs. non-native annotator status by region. Takeaway : Annotator provenance is under-documented; where reported, South Asia and Central Asia lead in native-speaker annotation, while the Americas lag.
Figure 12: Multilingual evaluation transparency across 23 recent model releases (15 open-weight, 8 closed). Left: Reporting rate across five transparency dimensions—only 65% report any multilingual evaluation, and just 13% report contamination results. Right: Breakdown by model type—closed models disclose significantly less across all dimensions.
| Llama-3.1-8B | Llama-3.1-8B-it | Mistral-7B-v0.3 | Mistral-7B-v0.3-it | Gemma-2-9B-it | Gemma-2-9B | Aya-23-8B | Gemma-7B-it | Llama-2-7B-it | Mistral-7B-v0.1-it | |
| Flores | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | – | – | – |
| Paws-X | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
| XCopa | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
| XLSum | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | – | – | – |
| Xnli | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ |
| XQuad | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
Table 2: Benchmark contamination presence across all evaluated models based on Ahuja et al. (2024a) , Ahuja et al. (2024b) , and Yao et al. (2024) . ✗ = contaminated , ✓ = not contaminated , – = not evaluated.
| Phi2 | Phi3 | Phi3.5-mini | Phi3.5-MoE | GrinMoE | Abel-7B | LLaMA2 | Mistral | Qwen2 | GLM4 | LLaMA3 | Reflection | |
| 2.7B | 3.8B | 3.8B | 3.8B 16 | 3.8B 16 | 7B | 7B | 7B | 7B | 9B | 70B | 70B | |
| MMLU | 23.83 | 67.27 | 68.64 | 76.62 | 77.55 | 47.08 | 44.88 | 57.29 | 69.05 | 67.36 | 78.55 | 75.83 |
| MMLU-g | 25.02 | 85.29 | 87.00 | 91.65 | 92.83 | 68.37 | 72.87 | 82.71 | 89.23 | 84.91 | 92.17 | 88.37 |
| difference | 1.20 | 18.02 | 18.36 | 15.03 | 15.28 | 21.29 | 27.99 | 25.42 | 20.18 | 17.55 | 13.62 | 12.54 |
| ARC-C | 42.92 | 80.20 | 59.56 | 54.78 | 63.57 | 50.34 | 36.18 | 64.08 | 84.81 | 86.35 | 61.52 | 56.74 |
| ARC-C-g | 47.27 | 92.15 | 93.94 | 96.50 | 96.25 | 66.04 | 44.71 | 85.75 | 95.22 | 91.81 | 95.99 | 94.45 |
Table 3: Detecting inadvertent contamination in popular open-source LLMs from Yao et al. (2024) . Each benchmark is tested in its original form and a paraphrased variant (-g); the difference row shows the generalizability gap. Bold values indicate significantly lower generalizability, implying potential contamination.
| Model | PAW-SX | Global MMLU | IndicParam | MILU | MMMLU | INCLUDE | FLORES | MGSM | AFRiMGSM | Overall |
| Aya-Expanse-32B | 100% (4/4) | 0% (0/42) | 0% (0/6) | 0% (0/11) | 0% (0/14) | 0% (0/44) | 4% (1/28) | 55% (6/11) | 16% (3/19) | 7.8% |
| Gemma-3-12B | 50% (2/4) | 12% (5/40) | 0% (0/7) | 0% (0/11) | 14% (2/14) | 5% (2/44) | 0% (1/204) | 55% (6/11) | 16% (3/19) | 5.9% |
| Tiny-Aya-Global | 100% (4/4) | 0% (0/42) | 0% (0/12) | 0% (0/11) | 0% (0/14) | 0% (0/44) | 2% (5/204) | 55% (6/11) | 26% (5/19) | 5.5% |
| Aya-Expanse-8B | 75% (3/4) | 0% (0/42) | 10% (1/10) | 0% (0/11) | 0% (0/14) | 0% (0/44) | 1% (2/204) | 64% (7/11) | 26% (5/19) | 5.0% |
| Qwen3-4B | 100% (4/4) | 0% (0/42) | 8% (1/12) | 0% (0/11) | 0% (0/14) | 0% (0/44) | 0% (1/204) | 36% (4/11) | 21% (4/19) | 3.9% |
| Phi-4 | 25% (1/4) | 0% (0/42) | 8% (1/12) | 0% (0/11) | 0% (0/14) | 2% (1/44) | 1% (2/204) | 18% (2/11) | 21% (4/19) | 3.0% |
Table 4: Contamination rate per model and dataset. Each cell shows the fraction of languages flagged as contaminated ( ), with the count in parentheses. Bold indicates the dataset is contaminated as a whole (Fisher’s combined ).
| Model | AfriMGSM | Global MMLU | INCLUDE | IndicParam | MGSM | MILU | MMMLU | PAWS-X |
| Aya-Expanse-32B | 21.3 | 28.7 | 29.7 | 16.2 | 47.4 | 27.0 | 31.8 | 54.4 |
| Aya-Expanse-8B | 10.8 | 12.6 | 11.5 | 8.7 | 22.3 | 13.6 | 12.4 | 38.9 |
| Tiny-Aya-Global | 27.5 | 21.0 | 19.1 | 15.1 | 34.6 | 16.7 | 22.8 | 11.4 |
| Gemma-3-12B | 39.4 | 21.5 | 40.6 | 36.0 | 56.6 | 36.6 | 19.7 | 41.3 |
| Llama-3.1-8B | 9.2 | 8.7 | 11.1 | 9.9 | 17.9 | 9.8 | 7.3 | 33.2 |
| Phi-4 | 2.3 | 22.6 | 10.6 | 13.2 | 10.6 | 24.6 | 36.8 | 20.6 |
Table 5: DCQ maximum recognition rate (%) by model and benchmark, averaged over each benchmark’s language partitions. The random-chance floor is 20%. The maximum rate selects the best raw hit rate across the tested answer positions and is therefore an optimistic estimate.
| Benchmark | English | Non-English Mean |
| AfriMGSM | 45.7% | 15.3% |
| Global MMLU | 26.1% | 16.6% |
| MGSM | 44.0% | 29.6% |
| MILU | 37.9% | 18.6% |
| PAWS-X | 47.6% | 30.1% |
Table 6: DCQ maximum recognition rate averaged across the seven models for English and non-English language partitions. INCLUDE, IndicParam, and MMMLU are omitted because the evaluated partitions do not include English.
| Type | Name | Release Date |
| Model | Aya-Expanse-32B | December 2024 |
| Aya-Expanse-8B | December 2024 | |
| Tiny-Aya-Global | February 2026 | |
| Gemma-3-12B | March 2025 | |
| Llama-3.1-8B | July 2024 | |
| Phi-4 | December 2024 |
Table 7: Release dates of models and datasets/benchmarks used in this study.
Figure 13: Landing page of the interactive dashboard showing summary statistics (51 benchmarks, 219 languages, 29 families, 29 scripts, 10 regions, 15 task types) and navigation to the four main views: Coverage, Representativeness, Benchmark Explorer, and Language Lookup.
Figure 14: Coverage Analysis view with interactive drill-down. Clicking on a bar in the language long-tail chart opens a panel listing the specific benchmarks that cover languages appearing in exactly that number of benchmarks, along with matching languages for each benchmark.
Figure 15: Benchmark Explorer view. Users can search, sort by column, and filter by region. Expanding a row reveals the paper reference, description, year, benchmark type, task categories, and full language family breakdown.
| # | Benchmark | Task Categories | Year | Venue | Citation |
| 1 | MMTEB | Classification / QA / Knowledge (550 tasks) | 2025 | ICLR | ( Enevoldsen et al., 2025 ) |
| 2 | Belebele | QA / Knowledge | 2024 | ACL | ( Bandarkar et al., 2024 ) |
| 3 | Aya_Evaluation_Suite | Generation | 2024 | ACL | ( Singh et al., 2024 ) |
| 4 | FLORES-101 | Translation | 2022 | TACL | ( Goyal et al., 2022 ) |
| 5 | XTREME_UP | Linguistic Probing, Other, QA / Knowledge, … | 2023 | ACL | ( Ruder et al., 2023 ) |
| 6 | MEGA | NER, NLI / Inference, QA / Knowledge, … | 2023 | EMNLP | ( Ahuja et al., 2023 ) |
Table 8: Comprehensive overview of all 51 multilingual evaluation benchmarks surveyed.
More in this topic
Multilingual Language Model Evaluation
Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.
See all 485 papers