Multilingual Language Model Evaluation

Follow this topic Ask about this topic

The field studies whether language models understand and respond reliably across languages, rather than assuming results in English transfer. Researchers build native-speaker-authored and translated benchmarks, use controlled prompt comparisons, and probe linguistic features such as morphology, pragmatics, and tokenization. Evaluations span reasoning, translation, tool use, safety, bias, and calibration, combining automatic metrics with human judgments and internal-model analyses. A central tension is that translation artifacts, evaluator bias, and aggregate scores can hide language-specific failures; performance remains uneven, especially for lower-resource languages and cultural contexts.

Latest papers

488 papers69 papers in the last four weeks, up 41% on the four weeks before. 0.4% of all new papers.

Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages
Oct 6, 2025Tarek Naous, Anagha Savit, Carlos Rafael Catalan +17Multilingual Language Model Evaluation Cultural Bias in Language Models
Georgia Institute of Technology · Samsung R&D Institute Philippines · Sungkyunkwan University +6
Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models
Oct 3, 2025Ej Zhou, Caiqi Zhang, Tiancheng Hu +4Multilingual Language Models Multilingual Language Model Evaluation
Language Technology Lab, University of Cambridge
How Well Does Mathematical Reasoning Transfer Across Languages in English-Centric LLMs?
Oct 2, 2025Wen Yang, Junhong Wu, Chong Li +2Cross-Lingual Reasoning in Language Models Cross-Lingual Transfer
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Wuhan AI Research
One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoning
Sep 25, 2025Sualeha Farid, Jayden Lin, Zean Chen +2Multilingual Language Model Evaluation Moral Reasoning in Language Models
University of Michigan, Ann Arbor, USA
Simulating Student-Tutor Interactions with LLMs for Multilingual Math Feedback
Jun 5, 2025Junior Cedric Tonga, KV Aditya Srivatsa, Kaushal Kumar Maurya +2Educational Feedback Generation Generative AI in Education
Mohamed bin Zayed University of Artificial Intelligence
In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models
Mar 3, 2025David Ponce, Thierry EtchegoyhenMultilingual Language Models Multilingual Language Model Evaluation
Vicomtech Foundation, Basque Research and Technology Alliance (BRTA) · University of the Basque Country - EHU
Leveraging LLMs for Translating and Classifying Mental Health Data
Oct 16, 2024Konstantinos Skianis, A. Seza Doğruöz, John PavlopoulosDepression Severity Estimation Clinical NLP
Department of Computer Science and Engineering, University of Ioannina, Greece · LT3, IDLab, Universiteit Gent, Belgium · Department of Informatics, Athens University of Economics and Business, Greece +1
Building Multilingual Datasets for Predicting Mental Health Severity through LLMs: Prospects and Challenges
Sep 25, 2024Konstantinos Skianis, John Pavlopoulos, A. Seza DoğruözMultilingual Language Model Evaluation Clinical Language Model Evaluation
Department of Computer Science and Engineering, University of Ioannina, Greece · Department of Informatics, Athens University of Economics and Business, Greece · Archimedes/AthenaRC, Greece +1

Oct 6, 2025J/K move · Enter open · S save

Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages

Tarek Naous, Anagha Savit, Carlos Rafael Catalan, Geyang Guo+16

Georgia Institute of Technology · Samsung R&D Institute Philippines · Sungkyunkwan University+6

Multilingual Language Model Evaluation Cultural Bias in Language Models Language Model Bias Evaluation Linguistic Bias in Language Models Cultural Knowledge in Language Models

Open paper AskPDF arXiv

As Large Language Models (LLMs) develop stronger multilingual capabilities, their sensitivity to culturally diverse entities becomes increasingly important. Prior work by Naous et al. (2024) has shown that LLMs often favor Western-associated entities in Arabic. Due to the lack of entity-centric multilingual benchmarks, it remains unclear if such biases also manifest in various non-Western languages. In this paper, we introduce Camellia, a benchmark for evaluating entity-centric cultural biases in nine Asian languages, spanning six Asian cultures. Camellia includes 19,530 manually annotated entities associated with the covered Asian or Western cultures, as well as 2,173 masked contexts for these entities derived from social media posts. Using Camellia, we evaluate cultural biases in four recent multilingual LLMs across three tasks: cultural context adaptation, sentiment association, and entity extractive QA. Our analyses show that LLMs struggle with cultural adaptation across these languages, with performance differing across models developed in different regions. We further observe that different LLM families can hold distinct biases, reflected in the ways they link cultures to particular sentiments. Lastly, we find that LLMs can struggle with context understanding in some Asian languages, creating performance gaps between cultures in entity extraction.