Multilingual Large Language Models do not comprehend all natural languages to equal degrees

Natalia Moskvina; Raquel Montero; Masaya Yoshida; Ferdy Hubers; Paolo Morosi; Walid Irhaymi; Jin Yan; Tamara Serrano

cs.CLFeb 23, 2026

Multilingual Large Language Models do not comprehend all natural languages to equal degrees

Authors: Natalia Moskvina, Raquel Montero, Masaya Yoshida, Ferdy Hubers, Paolo Morosi, Walid Irhaymi, Jin Yan, Tamara Serrano, +3 more

Organizations: Universitat Autònoma de Barcelona · Institució Catalana de Recerca i Estudis Avançats (ICREA) · Radboud University Nijmegen · Humboldt-Universität zu Berlin

Abstract

Large Language Models (LLMs) play a critical role in how humans access information. While their core use relies on comprehending written requests, our understanding of this ability is currently limited, because most benchmarks evaluate LLMs in high-resource languages predominantly spoken by Western, Educated, Industrialised, Rich, and Democratic (WEIRD) communities. The default assumption is that English is the best-performing language for LLMs, while smaller, low-resource languages are linked to less reliable outputs, even in multilingual, state-of-the-art models. To track variation in the comprehension abilities of LLMs, we prompt 3 popular models on a language comprehension task across 12 languages, representing the Indo-European, Afro-Asiatic, Turkic, Sino-Tibetan, and Japonic language families. Our results suggest that the models exhibit remarkable linguistic accuracy across typologically diverse languages, yet they fall behind human baselines in all of them, albeit to different degrees. Contrary to what was expected, English is not the best-performing language, as it was systematically outperformed by several Romance languages, even lower-resource ones. We frame the results by discussing the role of several factors that drive LLM performance, such as tokenization, language distance from Spanish and English, size of training data, and data origin in high- vs. low-resource languages and WEIRD vs. non-WEIRD communities.

Explore similar work

May 27, 2026

DEPART: DEcomposing PARiTy across Multilingual LLMs

Multilingual Large Language Models (mLLMs) leaderboards report per-language accuracy but rarely explain why disparities emerge, leaving systemic biases unattributed and offering practitioners no actionable levers. We first establish that these gaps are systematic rather than artifacts of sampling noise via distribution-free Friedman and Kruskal--Wallis tests, then introduce a two-step Bayesian hierarchical framework that decomposes multilingual performance variance into interpretable components. First, isolating the variance attributable to language identity, we show that observable language features (script, family, typological distance) explain

R^2_{\text{ling}} = 79\%

of this variance on understanding tasks and

92\%

on reasoning, with a model's internal representational similarity to English emerging as the dominant predictor across both task buckets. Second, decomposing the full (model

\times

benchmark

\times

language) cube, we find that NLU and reasoning have fundamentally divergent variance profiles: model identity dominates understanding (

66.7\%

of variance), whereas the benchmark

\times

model interaction dominates reasoning (

46.3\%

). Together these results recast multilingual evaluation from passive performance mapping into an explainable, diagnostic framework with concrete levers for targeting the root drivers of language disparity.

Manan Uppadhyay, Prashant Kodali, Pranjal Chitale +3

Microsoft Research India · IIT Gandhinagar

Multilingual Language Model Evaluation Cross-Lingual Reasoning in Language Models Language Model Bias Evaluation Linguistic Bias in Language Models

Ask PDF

Apr 14, 2026

English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training

Despite the widespread multilingual deployment of large language models, post-training pipelines remain predominantly English-centric, contributing to performance disparities across languages. We present a systematic, controlled study of the interplay between training language coverage, model scale, and task domain, based on 220 supervised fine-tuning runs on parallel translated multilingual data mixtures spanning mathematical reasoning and API calling tasks, with models up to 8B parameters. We find that English-only post-training is typically suboptimal: incorporating even a single non-English language improves both English performance and cross-lingual generalization. Increasing language diversity during post-training generally yields further gains, particularly for low-resource languages, while performance on high-resource languages tends to plateau rather than degrade. Moreover, greater language diversity enables strong zero-shot transfer to unseen languages, reducing the need for direct inclusion, though gains remain limited for typologically distant, low-resource languages.

Mehak Dhaliwal, Shashwat Chaurasia, Yao Qin +2

UC Santa Barbara · Amazon

Cross-Lingual Reasoning in Language Models Low-Resource Language Processing Language Model Post-Training Cross-Lingual Transfer

Ask PDF

Mar 22, 2026

Left Behind: Cross-Lingual Transfer as a Bridge for Low-Resource Languages in Large Language Models

We investigate how large language models perform on low-resource languages by benchmarking eight LLMs across five experimental conditions in English, Kazakh, and Mongolian. Using 50 hand-crafted questions spanning factual, reasoning, technical, and culturally grounded categories, we evaluate 2,000 responses on accuracy, fluency, and completeness. We find a consistent performance gap of 13.8-16.7 percentage points between English and low-resource language conditions, with models maintaining surface-level fluency while producing significantly less accurate content. Cross-lingual transfer-prompting models to reason in English before translating back-yields selective gains for bilingual architectures (+2.2pp to +4.3pp) but provides no benefit to English-dominant models. Our results demonstrate that current LLMs systematically underserve low-resource language communities, and that effective mitigation strategies are architecture-dependent rather than universal.

Abdul-Salem Beibitkhan

North American University

Low-Resource Language Processing Multilingual Language Model Evaluation Cross-Lingual Transfer LLM Prompting

Ask PDF