Multilingual Language Model Evaluation

Follow this topic Ask about this topic

The field studies whether language models understand and respond reliably across languages, rather than assuming results in English transfer. Researchers build native-speaker-authored and translated benchmarks, use controlled prompt comparisons, and probe linguistic features such as morphology, pragmatics, and tokenization. Evaluations span reasoning, translation, tool use, safety, bias, and calibration, combining automatic metrics with human judgments and internal-model analyses. A central tension is that translation artifacts, evaluator bias, and aggregate scores can hide language-specific failures; performance remains uneven, especially for lower-resource languages and cultural contexts.

Latest papers

488 papers69 papers in the last four weeks, up 41% on the four weeks before. 0.4% of all new papers.

Towards Cross-lingual Values Judgment: A Consensus-Pluralism Perspective
Feb 19, 2026Yukun Chen, Xinyu Zhang, Boyi Deng +6Cross-Cultural Language Model Evaluation Multilingual Language Model Evaluation
State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Tongyi Lab, Alibaba Group +1
ALPS: A Diagnostic Challenge Set for Arabic Linguistic & Pragmatic Reasoning
Feb 19, 2026Hussein S. Al-Olimat, Ahmad AlshareefArabic NLP Arabic Language Models
Independent Researchers
IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages
Feb 18, 2026Priyaranjan Pattnayak, Sanchari ChowdhuriAdversarial Robustness of Language Models LLM Safety Benchmarks
Oracle America Inc.
Evaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmark
Feb 18, 2026Charalampos Mastrokostas, Nikolaos Giarelis, Nikos KaracapilidisQuestion Answering Multilingual Language Model Evaluation
Industrial Management and Information Systems Lab, MEAD, University of Patras, Rio Patras, Greece
IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models
Feb 18, 2026Saurabh Bharti, Gaurav Azad, Abhinaw Jagtap +1Education LLM Evaluation
Department of Computer Sciencce and Engineering, Chhattisgarh Swami Vivekanand Technical University Bhilai, India
MultiCW: A Large-Scale Balanced Benchmark Dataset for Training Robust Check-Worthiness Detection Models
Feb 18, 2026Martin Hyben, Sebastian Kula, Jan Cegin +3Automated Fact-Checking Language Model Robustness
Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia · West Pomeranian University of Technology in Szczecin, Szczecin, Poland
Under-resourced studies of under-resourced languages: lemmatization and POS-tagging with LLM annotators for historical Armenian, Georgian, Greek and Syriac
Feb 17, 2026Chahan Vidal-Gorène, Bastien Kindt, Florian CafieroLow-Resource Language Processing Morphosyntactic Tagging
CJM, LIPN · LIPN, CNRS UMR 7030, France · École nationale des chartes, Université Paris-Sciences-et-Lettres (PSL), France +3
Alignment as Iatrogenesis: Pastoral Power, Collective Pathology, and the Structural Limits of Monolingual Safety Evaluation
Feb 17, 2026Hiroki FukuiLLM Alignment Language Model Safety Evaluation
Department of Neuropsychiatry, Graduate School of Medicine, Kyoto University
Physical Commonsense Reasoning for Lower-Resourced Languages and Dialects: a Study on Basque
Feb 16, 2026Jaione Bengoetxea, Itziar Gonzalez-Dios, Rodrigo AgerriLLM Reasoning Low-Resource Language Processing
HiTZ Center - Ixa, University of the Basque Country UPV/EHU
Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography
Feb 16, 2026Gianluca Vico, Jindřich LibovickýTokenization in Language Models Low-Resource Language Processing
Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Malostranské náměstí 25, 118 00 Praha, Czech Republic
Large Language Models for Math Education in Low-Resource Languages: A Study in Sinhala and Tamil
Feb 16, 2026Sukumar Kishanthan, Kumar Thushalika, Buddhi Jayasekara +1Numerical Reasoning in Language Models Cross-Lingual Reasoning in Language Models
Department of Computer Science and Engineering University of Moratuwa Moratuwa, Sri Lanka · Department of Electrical and Information Engineering University of Ruhuna Galle, Sri Lanka · Faculty of Information Technology and Communication Sciences Tampere University Tampere, Finland +1
SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models
Feb 3, 2026Omar Abdelnasser, Fatemah Alharbi, Khaled Khasawneh +2LLM Safety Benchmarks Multilingual Language Model Evaluation
Compumacy for Artificial Intelligence Solutions, Cairo, Egypt · Computer Science Department, College of Computer Science and Engineering, Taibah University, Yanbu 46522, KSA · Electrical and Computer Engineering Department, George Mason University, VA, USA +2
MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation
Jan 29, 2026Tianyi Xu, Kosei Uemura, Alfred Malengo Kondoro +6Numerical Reasoning in Language Models Mathematical Reasoning Benchmarks
McGill University · Mila-Quebec AI Institute · University of Toronto +7
The Effect of Scripts and Formats on LLM Numeracy
Jan 21, 2026Varshini Reddy, Craig W. Schmidt, Seth Ebner +3Numerical Reasoning in Language Models Multilingual Language Model Evaluation
Kensho Technologies · Ben-Gurion University · MIT Cambridge, MA
Obscuring Data Contamination Through Translation: Evidence from Arabic Corpora
Jan 21, 2026Chaymaa Abbas, Nour Shammaa, Mariette AwadData Contamination in Language Models Multilingual Language Model Evaluation
Department of Electrical and Computer Engineering, Maroun Semaan Faculty of Engineering and Architecture, American University of Beirut
The Language You Ask In: Language-Conditioned Ideological Divergence in LLM Analysis of Contested Political Documents
Jan 17, 2026Oleg SmirnovMultilingual Language Model Evaluation Political Bias in Language Models
Microsoft / Stockholm, Sweden
Preserving Fairness and Safety in Quantized LLMs Through Critical Weight Protection
Jan 17, 2026Muhammad Alif Al Hakim, Alfan Farizki Wicaksono, Fajri KotoLLM Quantization Multilingual Language Model Evaluation
Faculty of Computer Science, Universitas Indonesia · Mohamed bin Zayed University of Artificial Intelligence
Untangling Input Language from Reasoning Language: A Diagnostic Framework for Cross-Lingual Moral Alignment in LLMs
Jan 15, 2026Nan Li, Bo Kang, Tijl De BieMultilingual Language Model Evaluation LLM Alignment
IDLab, Department of Electronics and Information Systems Ghent University, Belgium
BizFinBench.v2: Towards Reliable LLMs in Finance via Real-User Data and Offline/Online Bilingual Evaluation
Jan 10, 2026Xin Guo, Rongjunchen Zhang, Guilong Lu +4Financial Services Multilingual Language Model Evaluation
HiThink Research · Shanghai University of Finance and Economics
What Matters When Building Universal Multilingual Named Entity Recognition Models?
Jan 9, 2026Jonas Golde, Patrick Haller, Alan AkbikNamed Entity Recognition Multilingual Language Models
Humboldt Universität zu Berlin
Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models
Jan 8, 2026Zheng Luo, T Pranav Kutralingam, Ogochukwu N Okoani +3Multilingual Language Model Evaluation LLM Tool Use
University of Southern California · 2Arizona State University
The Language of Bargaining: Linguistic Effects in LLM Negotiations
Jan 7, 2026Stuti Sinha, Himanshu Kumar, Aryan Raju Mandapati +2Multilingual Language Model Evaluation Multi-Agent Negotiation
BITS Pilani
Toward Global Large Language Models in Medicine
Jan 5, 2026Rui Yang, Huitao Li, Weihao Xuan +47Medical LLMs Multilingual Language Model Evaluation
Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results
Nov 7, 2025Jan-Thorsten Peter, David Vilar, Tobias Domhan +2Mathematical Reasoning Benchmarks Multilingual Language Model Evaluation
Google
MultiZebraLogic: A Multilingual Logical Reasoning Benchmark
Nov 5, 2025Sofie Helene Bruun, Dan Saattrup SmartLogical Reasoning Multilingual Language Model Evaluation
The Alexandra Institute Rued Langgaards Vej 7, 5D, 2300 Copenhagen S, Denmark
MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts
Nov 1, 2025Naoto Iwase, Hiroki Okuyama, Junichiro IwasawaMultilingual Language Model Evaluation Healthcare
Preferred Networks, Inc., Tokyo, Japan · School of Medicine, Nagoya University, Nagoya, Japan
KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs
Oct 21, 2025Donghyeon Ko, Kyubyung Chae, Yeguk Jin +6Multilingual Language Model Evaluation LLM Evaluation
Naver Cloud · Graduate School of Data Science, Seoul National University
Is Multilingual LLM Watermarking Truly Multilingual? Scaling Robustness to 100+ Languages via Back-Translation
Oct 20, 2025Asim Mohamed, Martin GubriMultilingual Language Model Evaluation Back-Translation
Parameter Lab · École Polytechnique, IP Paris, CNRS
Zoom In Disparities in Healthcare LLM Q&A
Oct 20, 2025Ipek Baris Schlicht, Burcu Sayin, Zhixue Zhao +5Multilingual Language Model Evaluation Knowledge Augmentation for Language Models
Universitat Politècnica de València, Valencia, Spain · University of Trento, Trento, Italy · University of Sheffield, Sheffield, UK +5
Evaluating Large Language Models on Urdu Idioms
Oct 20, 2025Muhammad Farmal Khan, Mousumi AkterMultilingual Language Model Evaluation Low-Resource Language Processing
Technical University Dortmund · Research Center Trustworthy Data Science and Security

Feb 19, 2026J/K move · Enter open · S save

Towards Cross-lingual Values Judgment: A Consensus-Pluralism Perspective

Yukun Chen, Xinyu Zhang, Boyi Deng, Jialong Tang+5

State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Tongyi Lab, Alibaba Group+1

Cross-Cultural Language Model Evaluation Multilingual Language Model Evaluation

Open paper AskPDF arXiv

As large language models (LLMs) are employed worldwide, existing evaluation paradigms for their multilingual capabilities primarily focus on factual task performance, neglecting the ability to judge content's deep-level values across multiple languages. To bridge this gap, we first reveal two primary challenges in constructing values judgment benchmarks, cultural diversity and disciplinary complexity, and propose a novel two-stage human-AI collaborative annotation framework to alleviate them. This framework identifies the issue scope and nature, establishes specific annotation criteria, and utilizes multiple LLMs for final review. Building upon this framework, we introduce X-Value, the first Cross-lingual Values Judgment Benchmark designed to evaluate the capability of LLMs in judging deep-level values of content. X-Value comprises 4,750 Question-Answer pairs across 14 languages, covering 7 major global issue categories, and provides 12 granular annotation metadata to facilitate a rigorous evaluation of model performance. Systematic evaluations of X-Value are conducted across 17 LLMs using distinct prompting strategies. Multi-dimensional analysis of accuracy and F1-scores reveals their limitations in cross-lingual values judgment and indicates performance disparities across categories and languages. This work highlights the urgent need to improve the underlying, values-aware content judgment capability of LLMs.\footnote{Samples of X-Value are available at https://huggingface.co/datasets/Whitolf/X-Value.}