Multilingual Language Model Evaluation

Follow this topic Ask about this topic

The field studies whether language models understand and respond reliably across languages, rather than assuming results in English transfer. Researchers build native-speaker-authored and translated benchmarks, use controlled prompt comparisons, and probe linguistic features such as morphology, pragmatics, and tokenization. Evaluations span reasoning, translation, tool use, safety, bias, and calibration, combining automatic metrics with human judgments and internal-model analyses. A central tension is that translation artifacts, evaluator bias, and aggregate scores can hide language-specific failures; performance remains uneven, especially for lower-resource languages and cultural contexts.

Latest papers

488 papers69 papers in the last four weeks, up 41% on the four weeks before. 0.4% of all new papers.

Red-Teaming Large Language Models in Low-Resource African Languages: A Case Study of Yoruba and Nigerian Pidgin
Oct 29, 2026Paula Chikaodinaka Anyanwu, Valdini Douglace Lemofouet, Blessing Ngozi Uzor +7Language Model Safety Evaluation LLM Red Teaming
African Institute for Mathematical Sciences (AIMS), Cameroon · Independent Researcher · Brown University +5
Gold vs.~Translation: Are We Measuring Language, Knowledge, or Locality in Translated Benchmarks?
Oct 28, 2026Klaudia-Doris Thellmann, Bernhard Stadler, Dan Saattrup Nielsen +2Benchmark Validity Multilingual Language Model Evaluation
TUD Dresden University of Technology and ScaDS.AI · Alexandra Institute
Component Translation and Judge-Labelled Refusal: A Multilingual Safety Evaluation
Oct 28, 2026Yongxi Zhou, Junwei Yao, Wenbo Ye +1Language Model Safety Evaluation LLM Safety Evaluation
Independent Researcher
The State and Fate of Multilingual, Contextual Evaluation in the NLP World
Oct 27, 2026Manan Uppadhyay, Himanshu Beniwal, Prashant Kodali +1Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
Microsoft Research India · IIT Gandhinagar
Japanese Discourse Structure Analysis Based on Segmented Discourse Representation Theory Using LLMs
Oct 27, 2026Satomu Tomisaki, Yosuke Mikami, Amane Watahiki +1Multilingual Language Model Evaluation
The University of Tokyo · RIKEN · Tohoku University
INVAR-Agri: Detecting Discrepancies in Multilingual Operator Manuals of Agricultural Implements with a Fine-Tuned LLM
Oct 26, 2026Timo OksanenLanguage Model Error Detection Document Understanding
Technical University of Munich, Germany; Chair of Agrimechatronics; Munich Institute of Robotics and Machine Intelligence (MIRMI)
Do Language Models Understand Igbo Culture? Beyond Multiple Choice
Oct 26, 2026Blessing Ngozi Uzor, Valdini Douglace Lemofouet, Paula Chikaodinaka Anyanwu +7Cultural Knowledge in Language Models Cross-Cultural Language Model Evaluation
African Institute for Mathematical Sciences (AIMS), Cameroon · University of Nigeria, Nsukka · Brown University +5
Quality over Quantity: Rethinking Data and Model Assumptions for Bengali Coreference Resolution
Oct 26, 2026Zenith Biswas, Andrew Thomas DyerCoreference Resolution Low-Resource Language Processing
Data Science and Artificial Intelligence, Saarland University, Saarbrücken, Germany · Language Science and Technology, Saarland University, Saarbrücken, Germany
When the Weakest Language Keeps Moving: ARMOR-DP for Multilingual BabyLM
Oct 24, 2026Hanyu Liao, Yijing Chen, Tao JiMultilingual Language Models Adaptive Sampling
East China Normal University · Fudan University
Disentangling Question, Response, and Verifier Languages in Multilingual P(True) Self-Evaluation
Oct 23, 2026Jewon Yeom, Jaewon Sok, Seonghyeon Park +3Language Model Self-Assessment Multilingual Language Model Evaluation
Graduate School of Data Science, Seoul National University
Bilingual Language Models Capture Cross-Linguistic Influence in Second-Language Reading
Oct 22, 2026Abderrahmane Issam, Irene E. Winther, Stefan L. FrankCross-Lingual Transfer Multilingual Language Model Evaluation
Maastricht University · Radboud University
Same Entity, Different Script: Diagnosing Cross-Script Entity Matching Failures in Language Models
Oct 22, 2026Athira Gopal, Patil Prathamesh Pravin, Ankush Kadu +2Cross-Lingual Consistency in Language Models Entity Resolution
QpiAI India
Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness
Oct 22, 2026Asal Mehrabi, Mohammad Aliannejadi, Clemencia SiroLLM Evaluation LLM-as-a-Judge
University of Amsterdam, Amsterdam, The Netherlands · Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands
Disentangling Robustness and Sycophancy in Multilingual LLMs
Oct 22, 2026Rhitabrat Pokharel, Ameeta AgrawalMultilingual Language Model Evaluation Language Model Robustness
Department of Computer Science, Portland State University, USA
SIM-UT: A User Simulation based Evaluation Framework for Medical Triage Chatbots
Oct 21, 2026Deepali Mittal, Dheeraj Mundhra, Yulia Sidorova +3Clinical Language Model Evaluation User Simulation
UserTrace · Cliniva · hearX Foundation +2
CulturalRiddles: A Multicultural Benchmark of Riddles
Oct 21, 2026My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan +161Cross-Cultural Language Model Evaluation Cultural Knowledge in Language Models
Benchmarking Large Language Models for Detecting Satirical Code-Mixed Sentences
Oct 21, 2026Răzvan-Alexandru Smădu, Ioana Buhnila, Daniela Tanase +3Code-Switching Text Classification
National University of Science and Technology POLITEHNICA Bucharest, Bucharest, Romania · Academy of Romanian Scientists, Bucharest, Romania · Center for Data Science in Humanities, Chosun University, South Korea +3
Same Answer, Different Reasons: Trace-Level Divergence in Multilingual Large Language Models
Oct 21, 2026Rui Sun, Peizhao Mei, Xiewei Cheng +4Multilingual Language Model Evaluation Reasoning Consistency in Language Models
University of Illinois Urbana-Champaign · Texas A&M University
Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models
Oct 20, 2026Haonan Huang, Yuwei Lu, Chengkun CaiMultilingual Language Model Evaluation Cross-Lingual Consistency in Language Models
Princeton University · Independent Researcher · University College London
Global Security Blind Spots: Translation as a Safety Bypass in Low-Resource Languages
Oct 19, 2026Jalil Alva, David Harper, Abigail E. Thornton +8LLM Safety Language Model Safety Evaluation
Welo Data
Cross-Lingual Reusability of LLM Pipelines: Financial Extraction, Multilingual Reasoning, and Semantic Screening
Oct 19, 2026Joong Ho, Choi, Sherine George +5Cross-Lingual Transfer Cross-Lingual Consistency in Language Models
Joshua · BNY AI Hub · BNY Responsible AI Office
DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia
Oct 19, 2026Jiuzhou Han, Abhishek Bijay Mishra, Trevor Cohn +1Multilingual Language Model Evaluation Multilingual Language Models
Google Research
Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs
Oct 19, 2026Abhinav Khattar, Aishwarya Chhabra, Ameya Velingker +24Language Modeling Language Model Pretraining
NVIDIA · AI4Bharat
Towards Figurative Language Understanding in Moroccan Darija: An Evaluation of Large Language Models
Oct 18, 2026Jihad R'baiti, Youssef Hmamouche, Emmanuele Chersoni +2Figurative Language Understanding Machine Translation
International Artificial Intelligence Center of Morocco, Mohammed VI Polytechnic University, Morocco. · Department of Language Science and Technology, The Hong Kong Polytechnic University, Hong Kong. · Division of Artificial Intelligence and the Humanities, The Hong Kong Polytechnic University, Hong Kong. +1
One Head Away from Language Confusion: Single Attention Heads Across Model Families
Oct 18, 2026Jiwon Jeong, Anjelo Jann Laroza, Chaewoon Yoon +4Multilingual Language Models Attention Head Analysis
Lotte Innovate · Mapúa University · University of Wisconsin–Madison +4
Efficiency Measured in English: Tokeniser Coverage Bounds Small Language Models on Indic Scripts
Oct 17, 2026Anna PramodTokenization in Language Models Small Language Models
School of Computer Science, University of Leeds, United Kingdom
When Opposite Premises Translate Identically: A Source-Conditioned Diagnostic for Multilingual Commonsense Items
Oct 17, 2026Shuai Wang, Rui HuMultilingual Language Model Evaluation MT Evaluation
Zhejiang University
Steering beyond Language: Improving Multilingual Training-Free Text Embeddings via Paired-Residual Subspace Removal
Oct 16, 2026Ailiang Lin, Zhuoyun Li, Yusong Wang +3Text Embeddings Cross-Lingual Representation Alignment
Institute of Science Tokyo · Tencent
Who Pays for Efficient Inference? Confidence-Based Small-to-Large LLM Routing Across Tasks and Languages
Oct 16, 2026Md Sabbir Hossen, Anichur Rahman, Pabon Shaha +3LLM Routing LLM Inference Acceleration
Bangladesh University, Bangladesh · Georgia Southern University, USA · University of Southern Mississippi, USA
Disentangling Language Effects on Tool Admission and Post-Retrieval Safety
Oct 16, 2026Aditya Nawal, Manit Baser, Mohan GurusamyLanguage Model Safety Evaluation LLM Agent Safety
National University of Singapore

Oct 29, 2026J/K move · Enter open · S save

Red-Teaming Large Language Models in Low-Resource African Languages: A Case Study of Yoruba and Nigerian Pidgin

Paula Chikaodinaka Anyanwu, Valdini Douglace Lemofouet, Blessing Ngozi Uzor, Ifeoluwa Dorcas Awotunde+6

African Institute for Mathematical Sciences (AIMS), Cameroon · Independent Researcher · Brown University+5

Language Model Safety Evaluation LLM Red Teaming Multilingual Language Model Evaluation Cross-Lingual Consistency in Language Models African Language Processing

Open paper AskPDF arXiv

As large language models (LLMs) become more widespread in Africa, their safety in low-resource languages remains largely untested since most red-teaming work focuses on English. This study evaluates six state-of-the-art LLMs using 400 explicit adversarial prompts in Yoruba and Nigerian Pidgin, categorised into four harm categories. The responses were scored by native-speaker annotators and reliability was assessed using Cohen's

κ

. While models that rejected up to 69% of English prompts rejected as few as 0% in Yoruba, this was not because they were safer, but because they failed to understand the prompts. Attack success rates increased for most models in the Nigerian languages, with GPT-4o Mini demonstrating the most significant cross-lingual safety discrepancy (+0.23 in Yoruba). When models did engage with adversarial prompts, many of the responses were harmful, with rates of harmful content reaching 0.63 in English and 0.50 in Nigerian Pidgin. The most difficult harm category to detect was cultural stereotypes and discrimination (ASR = 0.44 in Nigerian Pidgin). Comprehension in Yoruba was low across most models, and this directly suppressed apparent attack success without indicating actual safety. Gemini 2.0 Flash was the most consistent model across languages. Despite its explicit training in African languages, Aya 101 had the highest attack success rate in English (0.60) and one of the lowest Yoruba comprehension rates (0.28). These results demonstrate that English-centric safety alignment does not transfer to Nigerian languages, highlighting the urgent need for language-specific safety evaluations.