Multilingual Language Model Evaluation

Ask about this topic

Overview

The field studies whether language models understand and respond reliably across languages, rather than assuming results in English transfer. Researchers build native-speaker-authored and translated benchmarks, use controlled prompt comparisons, and probe linguistic features such as morphology, pragmatics, and tokenization. Evaluations span reasoning, translation, tool use, safety, bias, and calibration, combining automatic metrics with human judgments and internal-model analyses. A central tension is that translation artifacts, evaluator bias, and aggregate scores can hide language-specific failures; performance remains uneven, especially for lower-resource languages and cultural contexts.

Latest papers

CardsList

488 papers69 papers in the last four weeks, up 41% on the four weeks before. 0.4% of all new papers.

  1. What Makes Good Multilingual Reasoning? Disentangling Traces with Measurable Features

    Apr 6, 2026Dayeon Ki, Kevin Duh, Marine CarpuatMultilingual Language Model EvaluationCross-Lingual Reasoning in Language Models

    University of Maryland · Johns Hopkins University

  2. Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluation

    Apr 6, 2026Alhasan Mahmood, Samir Abdaljalil, Hasan KurbanMultilingual Language Model EvaluationLLM-as-a-Judge

    Hamad Bin Khalifa University, Doha, Qatar · Texas A&M University, College Station, TX USA

  3. Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation

    Apr 3, 2026Leen AlQadi, Ahmed Alzubaidi, Mohammed Alyafeai +6LLM EvaluationBenchmark Validity

    Technology Innovation Institute, Abu Dhabi, UAE

  4. GaelEval: Benchmarking LLM Performance for Scottish Gaelic

    Apr 2, 2026Peter Devine, William Lamb, Beatrice Alex +5Multilingual Language Model EvaluationLLM Evaluation

    University of Edinburgh · Lancaster University · University of Cardiff +2

  5. Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite

    Apr 2, 2026Klaudia Thellmann, Bernhard Stadler, Michael FärberBenchmark AuditingMT Evaluation

    TU Dresden and ScaDS.AI · InfAI e.V.

  6. MATH-PT: A Math Reasoning Benchmark for European and Brazilian Portuguese

    Apr 1, 2026Tiago Teixeira, Ana Carolina Erthal, Juan Belieni +5Mathematical Reasoning BenchmarksMultilingual Language Model Evaluation

    Instituto Superior Técnico, Universidade de Lisboa · Fundação Getulio Vargas · Instituto de Telecomunicações +2

  7. AfrIFact: Cultural Information Retrieval, Evidence Extraction and Fact Checking for African Languages

    Apr 1, 2026Israel Abebe Azime, Jesujoba Oluwadara Alabi, Crystina Zhang +16Automated Fact-CheckingEvidence Retrieval

    Saarland University, Saarland Informatic Campus, Germany · Masakhane NLP · University of Waterloo, Canada +11

  8. Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models

    Mar 31, 2026Linda Zeng, Steven Y. Feng, Michael C. FrankMultilingual Language ModelsMultilingual Language Model Evaluation

    Stanford University

  9. LLM Probe: Evaluating LLMs for Low-Resource Languages

    Mar 31, 2026Hailay Kidu Teklehaymanot, Gebrearegawi Gebremariam, Wolfgang NejdlMultilingual Language Model EvaluationLow-Resource Language Processing

    L3S Research Center, Leibniz University Hannover, Germany · Aksum University, Ethiopia

  10. MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios

    Mar 30, 2026Zhang Li, Zhibo Lin, Qiang Liu +7Document Parsing BenchmarksDocument Image Understanding

    Huazhong University of Science and Technology · Kingsoft Office

  11. TailNLG: A Multilingual Benchmark Addressing Verbalization of Long-Tail Entities

    Mar 29, 2026Lia Draetta, Michael Oliverio, Virginia Ramón-Ferrer +6Text GenerationLanguage Model Bias Evaluation

    University of Turin, Italy · Universidad Politécnica de Madrid, Spain

  12. Investigating the Influence of Language on Sycophantic Behavior of Multilingual LLMs

    Mar 29, 2026Bayan Abdullah Aldahlawi, A. B. M. Ashikur Rahman, Irfan AhmadLLM SycophancyMultilingual Language Model Evaluation

    King Fahd University of Petroleum and Minerals, Dhahran, KSA · SDAIA-KFUPM Joint Research Center for AI (JRCAI), Dhahran, KSA

  13. ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs

    Mar 27, 2026Inês Vieira, Inês Calvo, Iago Paulo +6LLM EvaluationLanguage Model Generation Evaluation

    1NOVA University of Lisbon, Portugal · 2NOVA LINCS

  14. Does Structured Intent Representation Generalize? A Cross-Language, Cross-Model Empirical Study of 5W3H Prompting

    Mar 26, 2026Peng GangStructured PromptingLLM Prompting

    Huizhou Lateni AI Technology Co., Ltd., Huizhou, China · Huizhou University, Huizhou, China

  15. Comparing Natural and Synthetic Structured Data: A Study of the Passive Verb Alternation in French and Italian

    Mar 26, 2026Giuseppe Samo, Paola MerloSynthetic Data EvaluationSynthetic Data Pretraining

    IDIAP Research Institute · University of Geneva

  16. Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks

    Mar 24, 2026Fatih UenalLLM EvaluationLegal NLP

    University of Colorado Boulder

  17. Is AI Catching Up to Human Expression? Exploring Emotion, Personality, Authorship, and Linguistic Style in English and Arabic with Six Large Language Models

    Mar 24, 2026Nasser A AlsadhanEmotion Representation in Language ModelsAI-Generated Text Detection

    College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia

  18. From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Service

    Mar 24, 2026Haoyu He, Jinyu Zhuang, Haoran Chu +5Hierarchical ClassificationIntent Classification

    J&T Express · School of Information Science and Technology, ShanghaiTech University

  19. Multi-Method Validation of Large Language Model Medical Translation Across High- and Low-Resource Languages

    Mar 23, 2026Chukwuebuka Anyaegbuna, Eduardo Juan Perez Guerrero, Jerry Liu +6Machine TranslationMultilingual Language Model Evaluation

  20. CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context

    Mar 23, 2026Giovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau +6Instruction FollowingMultilingual Language Model Evaluation

    Maritaca AI · Jusbrasil

  21. Rashid: A Cipher-Based Framework for Exploring In-Context Language Learning

    Mar 23, 2026Niyati Bafna, Ryan Soh-Eun Shim, Barbara Plank +2In-Context LearningLow-Resource Language Processing

    Johns Hopkins University, Center for Language and Speech Processing · MaiNLP, Center for Information and Language Processing, LMU Munich

  22. Do Multilingual VLMs Reason Equally? A Cross-Lingual Visual Reasoning Audit for Indian Languages

    Mar 23, 2026Swastik RMultilingual VLM EvaluationVLM Reasoning

    Indian Institute of Information Technology, Raichur

  23. Left Behind: Cross-Lingual Transfer as a Bridge for Low-Resource Languages in Large Language Models

    Mar 22, 2026Abdul-Salem BeibitkhanLow-Resource Language ProcessingMultilingual Language Model Evaluation

    North American University

  24. What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?

    Mar 19, 2026Gagan Bhatia, Ahmad Muhammad Isa, Maxime Peyrard +1Tokenization in Language ModelsTemporal Reasoning in Language Models

    University of Aberdeen · Université Grenoble Alpes & CNRS

  25. IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia

    Mar 18, 2026Priyaranjan Pattnayak, Sanchari ChowdhuriLanguage Model Safety EvaluationLLM Safety Benchmarks

    Oracle America Inc.

  26. Harm or Humor: A Multimodal, Multilingual Benchmark for Overt and Covert Harmful Humor

    Mar 18, 2026Ahmed Sharshar, Hosam Elgendy, Saad El Dine Ahmed +2LLM Safety BenchmarksLanguage Model Safety Evaluation

    Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria

  27. SEAHateCheck: Functional Tests for Detecting Hate Speech in Low-Resource Languages of Southeast Asia

    Mar 17, 2026Ri Chi Ng, Aditi Kumaresan, Yujia Hu +1Hate Speech DetectionLow-Resource Language Processing

    Singapore University of Technology and Design, Singapore

  28. Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs

    Mar 16, 2026Yara Alakeel, Chatrine Qwaider, Hanan Aldarmaki +1Arabic NLPArabic Language Models

    Saudi Data & AI Authority (SDAIA) · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Princess Nourah Bint Abdulrahman University (PNU)

  29. Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies

    Mar 16, 2026Giuseppe Samo, Paola MerloMultilingual Language Model Evaluation

    IDIAP Research Institute · University of Geneva

  30. TelcoAgent-Bench: A Multilingual Benchmark for Telecom AI Agents

    Mar 16, 2026Lina Bariah, Brahim Mefgouda, Farbod Tavakkoli +3AI Agent EvaluationAI Agent Benchmarks

    Research Institute for Digital Future, Khalifa University · AT&T CDO · GSMA