Multilingual Language Model Evaluation

Follow this topic Ask about this topic

The field studies whether language models understand and respond reliably across languages, rather than assuming results in English transfer. Researchers build native-speaker-authored and translated benchmarks, use controlled prompt comparisons, and probe linguistic features such as morphology, pragmatics, and tokenization. Evaluations span reasoning, translation, tool use, safety, bias, and calibration, combining automatic metrics with human judgments and internal-model analyses. A central tension is that translation artifacts, evaluator bias, and aggregate scores can hide language-specific failures; performance remains uneven, especially for lower-resource languages and cultural contexts.

Latest papers

488 papers69 papers in the last four weeks, up 41% on the four weeks before. 0.4% of all new papers.

OasisSimp: An Open-source Asian-English Sentence Simplification Dataset
Mar 14, 2026Hannah Liu, Muxin Tian, Iqra Ali +8Text Simplification Low-Resource Language Processing
University of Toronto · Queen Mary University of London · Yale University +5
Benchmarking Large Language Models on Reference Extraction and Parsing in the Social Sciences and Humanities
Mar 13, 2026Yurui Zhu, Giovanni Colavizza, Matteo RomanelloDocument Information Extraction LLM Information Extraction
Odoma LLC
Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Study
Mar 13, 2026Liel Binyamin, Elior SulemLanguage Model Pretraining Language Modeling
Faculty of Computer and Information Science, Institute for Applied AI Research Data Science Research Center Ben-Gurion University of the Negev
SectEval: Evaluating the Latent Sectarian Preferences of Large Language Models
Mar 13, 2026Aditya Maheshwari, Amit Gajkeshwar, Kaushal Sharma +1Social Bias in Language Models Language Model Bias Evaluation
Indian Institute of Management Indore, India
SommBench: Assessing Sommelier Expertise of Language Models
Mar 12, 2026William Brach, Tomas Bedej, Jacob Nielsen +10Multilingual Language Model Evaluation LLM Grounding
Slovak University of Technology, Bratislava, Slovakia · sommify, Helsinki, Finland · University of Southern Denmark, Odense, Denmark
SemBench: A Universal Semantic Framework for LLM Evaluation
Mar 12, 2026Mikel Zubillaga, Naiara Perez, Oscar Sainz +1LLM Evaluation Multilingual Language Model Evaluation
HiTZ Center - Ixa, University of the Basque Country UPV/EHU
Tiny Aya: Bridging Scale and Multilingual Depth
Mar 12, 2026Alejandro R. Salamanca, Diana Abagyan, Daniel D'souza +23Multilingual Language Models Machine Translation
Cohere Labs · Cohere
Multilingual Reasoning Gym: Multilingual Scaling of Procedural Reasoning Environments
Mar 11, 2026Konstantin Dobler, Simon Lehnerer, Federico Scozzafava +2Synthetic Data Generation for Language Models Cross-Lingual Reasoning in Language Models
Hasso Plattner Institute & ELLIS Unit Potsdam · Apple
Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck
Mar 11, 2026Hongbin Zhang, Kehai Chen, Xuefen Bai +4Bias in LLM-as-a-Judge Linguistic Bias in Language Models
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Keeta AI, Meituan, Beijing, China
The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluation
Mar 11, 2026Pavel Braslavski, Dmitrii Iarosh, Nikita Sushko +4Language Model Generation Evaluation Factual Knowledge in Language Models
HSE University · Ural Federal University · ITMO University +3
Do What I Say: A Spoken Prompt Dataset for Instruction-Following
Mar 10, 2026Maike Züfle, Sara Papi, Fabian Retkowski +5Audio-Language Model Evaluation Speech Language Models
Karlsruhe Institute of Technology, Germany · Fondazione Bruno Kessler, Italy · ACC Cyfronet AGH, Poland +2
LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
Mar 10, 2026Lukáš Eigler, Jindřich Libovický, David HurychSynthetic Data Evaluation Synthetic Benchmark Generation
Faculty of Mathematics and Physics, Charles University, Czech Republic
A Dataset for Probing Translationese Preferences in English-to-Swedish Translation
Mar 9, 2026Jenny Kunz, Anja Jarochenko, Marcel BollmannMachine Translation MT Evaluation
Department of Computer and Information Science, Linköping University
Is continuous CoT better suited for multi-lingual reasoning?
Mar 9, 2026Ali Hamza Bashir, Behzad Shomali, Markus Frey +3Cross-Lingual Reasoning in Language Models CoT Reasoning
Lamarr Institute · Fraunhofer IAIS · University of Bonn
Language Shapes Mental Health Evaluations in Large Language Models
Mar 6, 2026Jiayi Xu, Xiyang HuCross-Lingual Consistency in Language Models Multilingual Language Model Evaluation
University of North Carolina at Chapel Hill · Arizona State University
Mind the Gap: Pitfalls of LLM Alignment with Asian Public Opinion
Mar 6, 2026Hari Shankar, Vedanta S P, Sriharini Margapuri +3Cultural Bias in Language Models Cross-Cultural Language Model Evaluation
IIIT Hyderabad · IIIT Kottayam · IIT Kharagpur
ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts
Mar 5, 2026Trapoom Ukarapol, Nut Chukamphaeng, Kunat Pipatanakul +1Language Model Safety Evaluation LLM Safety
SCB DataX · Department of Computer Science and Technology, Tsinghua University · SCBX R&D +1
When Do Language Models Endorse Limitations on Human Rights Principles?
Mar 4, 2026Keenan Samway, Nicole Miu Takagi, Rada Mihalcea +4LLM Alignment Language Model Bias Evaluation
Max Planck Institute for Intelligent Systems, Tübingen, Germany · Jinesis AI Lab, University of Toronto & Vector Institute · University of Michigan +2
Evaluating Large Language Models' Responses to Sexual and Reproductive Health Queries in Nepali
Mar 4, 2026Medha Sharma, Supriya Khadka, Udit Chandra Aryal +15Language Model Safety Evaluation Medical QA
Visible Impact (Visim), Kathmandu, 44600, Nepal · Diyo.AI, Lalitpur, 44600, Nepal · Nepal Applied Mathematics and Informatics Institute for research (NAAMII), Lalitpur, 44600, Nepal +1
JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attacks
Mar 1, 2026Masahiro Kaneko, Ayana Niwa, Timothy BaldwinLLM Safety Benchmarks LLM Jailbreak Attacks
MBZUAI
A Typologically Grounded Evaluation Framework for Word Order and Morphology Sensitivity in Multilingual Masked LMs
Feb 28, 2026Anna Feldman, Libby Barak, Jing PengMultilingual Language Model Evaluation Language Modeling
Montclair State University New Jersey, USA
LSR: Linguistic Safety Robustness Benchmark for Low-Resource West African Languages
Feb 27, 2026Godwin Abuh FarunaLLM Safety Benchmarks Language Model Safety Evaluation
Fagmart Lab
Benchmarking BERT-based Models for Sentence-level Topic Classification in Nepali Language
Feb 27, 2026Nischal Karki, Bipesh Subedi, Prakash Poudyal +2Bidirectional Encoder Representations from Transformers Text Classification
Information and Language Processing Research Lab, Kathmandu University, Kavre, Nepal
TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models
Feb 26, 2026Reihaneh Iranmanesh, Saeedeh Davoudi, Pasha Abrishamchian +2Cultural Knowledge in Language Models Cross-Cultural Language Model Evaluation
IR Lab, Computer Science Department Georgetown University, Washington D.C.
Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets
Feb 25, 2026Hanna Yukhymenko, Anton Alexandrov, Martin VechevMultilingual Language Model Evaluation MT Evaluation
INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich
IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages
Feb 25, 2026Thanmay Jayakumar, Mohammed Safi Ur Rahman Khan, Raj Dabre +2Instruction Following Multilingual Language Model Evaluation
Nilekani Centre at AI4Bharat · Indian Institute of Technology Madras, India · IT University of Copenhagen +1
Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models
Feb 24, 2026Paola Merlo, Chunyang Jiang, Giuseppe Samo +1LLM Evaluation LLM Interpretability
Idiap Research Institute · University of Geneva
Multilingual Large Language Models do not comprehend all natural languages to equal degrees
Feb 23, 2026Natalia Moskvina, Raquel Montero, Masaya Yoshida +8Multilingual Language Models Multilingual Language Model Evaluation
Universitat Autònoma de Barcelona · Institució Catalana de Recerca i Estudis Avançats (ICREA) · Radboud University Nijmegen +1
BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Models
Feb 21, 2026Thura Aung, Jann Railey Montalan, Jian Gang Ngui +1LLM Evaluation Low-Resource Language Processing
King Mongkut’s Institute of Technology Ladkrabang · AI Singapore · National University of Singapore
Cross-Lingual Sentiment Misalignment: Auditing Multilingual Language Models for Inversion Risk, Dialectal Representation, and Affective Stability
Feb 19, 2026Nusrat Jahan Lia, Shubhashis Roy DiptaDialect Bias in Language Models Cross-Lingual Consistency in Language Models
Institute of Information Technology, University of Dhaka, Dhaka, Bangladesh · University of Maryland, Baltimore County, Baltimore, Maryland, USA

Mar 14, 2026J/K move · Enter open · S save

OasisSimp: An Open-source Asian-English Sentence Simplification Dataset

Hannah Liu, Muxin Tian, Iqra Ali, Haonan Gao+7

University of Toronto · Queen Mary University of London · Yale University+5

Text Simplification Low-Resource Language Processing Multilingual Language Models Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Sentence simplification aims to make complex text more accessible by reducing linguistic complexity while preserving the original meaning. However, progress in this area remains limited for mid-resource and low-resource languages due to the scarcity of high-quality data. To address this gap, we introduce the OasisSimp dataset, a multilingual dataset for sentence-level simplification covering five languages: English, Sinhala, Tamil, Pashto, and Thai. Among these, no prior sentence simplification datasets exist for Thai, Pashto, and Tamil, while limited data is available for Sinhala. Each language simplification dataset was created by trained annotators who followed detailed guidelines to simplify sentences while maintaining meaning, fluency, and grammatical correctness. We evaluate eight open-weight multilingual Large Language Models (LLMs) on the OasisSimp dataset and observe substantial performance disparities between high-resource and low-resource languages, highlighting the simplification challenges in multilingual settings. The OasisSimp dataset thus provides both a valuable multilingual resource and a challenging benchmark, revealing the limitations of current LLM-based simplification methods and paving the way for future research in low-resource sentence simplification. The dataset is available at https://OasisSimpDataset.github.io/.