Multilingual Language Model Evaluation

Ask about this topic

Overview

The field studies whether language models understand and respond reliably across languages, rather than assuming results in English transfer. Researchers build native-speaker-authored and translated benchmarks, use controlled prompt comparisons, and probe linguistic features such as morphology, pragmatics, and tokenization. Evaluations span reasoning, translation, tool use, safety, bias, and calibration, combining automatic metrics with human judgments and internal-model analyses. A central tension is that translation artifacts, evaluator bias, and aggregate scores can hide language-specific failures; performance remains uneven, especially for lower-resource languages and cultural contexts.

Latest papers

CardsList

488 papers69 papers in the last four weeks, up 41% on the four weeks before. 0.4% of all new papers.

  1. IdiomX A Multilingual Benchmark for Idiom Understanding, Retrieval, and Interpretation

    Apr 25, 2026Ayman Ali ShararaFigurative Language UnderstandingMultilingual Language Model Evaluation

    MSc Data Science & Machine Learning (SPOC S21) Data ScienceTech Institute (DSTI)

  2. Multilingual Refusal Alignment for Safer Large Language Models

    Apr 24, 2026Aleksandra Krasnodębska, Wojciech Kusa, Aldo LipaniMultilingual Language Model EvaluationLLM Alignment

    NASK National Research Institute, Warsaw, Poland · University College London, London, UK

  3. Why are all LLMs Obsessed with Japanese Culture? On the Hidden Cultural and Regional Biases of LLMs

    Apr 23, 2026Joseba Fernandez de Landa, Carla Perez-Almendros, Jose Camacho-ColladosMultilingual Language Model EvaluationCultural Bias in Language Models

    HiTZ Center - Ixa, University of the Basque Country EHU · Cardiff University

  4. MKJ at SemEval-2026 Task 9: A Comparative Study of Generalist, Specialist, and Ensemble Strategies for Multilingual Polarization

    Apr 23, 2026Maziar Kianimoghadam JouneghaniMultilingual Language Model EvaluationLanguage Model Ensembles

    University of Turin

  5. RespondeoQA: a Benchmark for Bilingual Latin-English Question Answering

    Apr 22, 2026Marisa Hudspeth, Patrick J. Burns, Brendan O'ConnorMultilingual Language Model EvaluationMultilingual QA

    Manning College of Information & Computer Sciences, University of Massachusetts Amherst · Institute for the Study of the Ancient World, New York University

  6. The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models

    Apr 22, 2026Yilun Liu, Chunguang Zhao, Mengyao Piao +14Multilingual Language Model EvaluationLLM Evaluation

    Huawei, China · Fudan University, China

  7. Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs

    Apr 21, 2026Guy Mor-Lan, Omer Goldman, Matan Eyal +6Multilingual Language ModelsMultilingual Language Model Evaluation

    GGoogle Research · BBar-Ilan University · CUniversity of Cambridge +1

  8. CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks

    Apr 21, 2026Peiqin Lin, Chenyang Lyu, Wenjiang Luo +22Multilingual Language Model EvaluationLLM Evaluation

    Alibaba Group · Beijing Language and Culture University · LMU Munich +6

  9. MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation

    Apr 20, 2026Mehul Agarwal, Aditya Aggarwal, Arnav Goel +2Multilingual Language Model EvaluationText Generation

    SBILab, Indraprastha Institute of Information Technology Delhi

  10. MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval

    Apr 20, 2026Shaden Alshammari, Kevin Wen, Abrar Zainal +5Mathematical Reasoning BenchmarksMultilingual Language Model Evaluation

    MIT · HUMAIN · Individual Researcher +1

  11. Do LLMs Use Cultural Knowledge Without Being Told? A Multilingual Evaluation of Implicit Pragmatic Adaptation

    Apr 20, 2026Mehwish Nasim, Sanjeevan Selvaganapathy, Neel Ganapathi Sabhahit +6Multilingual Language ModelsMultilingual Language Model Evaluation

    1Network Analysis and Social Influence Modelling (NASIM) Lab · 3Computational Social Science & Systems Analysis University of Münster · School of Computing Macquarie University

  12. BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Stories

    Apr 18, 2026Yuxuan Ouyang, yingfeng luo, JingBo Zhu +1Multilingual Language Model EvaluationSocial Bias in Language Models

    School of Computer Science and Engineering, Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China

  13. No One Fits All: From Fixed Prompting to Learned Routing in Multilingual LLMs

    Apr 18, 2026Wei-Chi Wu, Sheng-Lun Wei, Hen-Hsen Huang +1Multilingual Language Model EvaluationLLM Prompting

    Department of Computer Science and Information Engineering, National Taiwan University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · AI Research Center (AINTU), National Taiwan University, Taiwan

  14. No Universal Courtesy: A Cross-Linguistic, Multi-Model Study of Politeness Effects on LLMs Using the PLUM Corpus

    Apr 17, 2026Hitesh Mehta, Arjit Saxena, Garima Chhikara +1Multilingual Language Model EvaluationLLM Evaluation

    Department of Computer Science Engineering, Delhi Technological University, New Delhi, India · Department of Electronics and Communication Engineering, Delhi Technological University, New Delhi, India

  15. Optimizing Korean-Centric LLMs via Token Pruning

    Apr 17, 2026Hoyeol Kim, Hyeonwoo KimMultilingual Language Model EvaluationLLM Pruning

    College of Computing · Independent Researcher

  16. Brain Score Tracks Shared Properties of Languages: Evidence from Many Natural Languages and Structured Sequences

    Apr 16, 2026Jingnong Qu, Ashvin Ranjan, Shane Steinert-ThrelkeldMultilingual Language Model EvaluationLanguage Modeling

    University of Washington

  17. MARCA: A Checklist-Based Benchmark for Multilingual Web Search

    Apr 15, 2026Thales Sales Almeida, Giovana Kerche Bonás, Ramon Pires +6LLM EvaluationMultilingual Language Model Evaluation

    Maritaca AI · Jusbrasil

  18. EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation

    Apr 15, 2026Francesco Andrea Causio, Vittorio De Vita, Olivia Riccomi +17Multilingual Language Model EvaluationMultilingual VLM Evaluation

    Section of Hygiene, University Department of Life Sciences and Public Health, Università Cattolica del Sacro Cuore, Rome, Italy · Società Italiana Intelligenza Artificiale in Medicina (SIIAM), Rome, Italy · NSBproject, Mantova, Italy +8

  19. IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages

    Apr 15, 2026Aviral Dawar, Roshan Karanth, Vikram Goyal +1Text-to-SQLMultilingual Language Model Evaluation

    Independent Researcher · BITS Pilani, Pilani, India · IIIT Delhi, New Delhi, India

  20. Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss

    Apr 14, 2026Ronald Skorobogat, Ameya Prabhu, Matthias BethgeMultilingual Language Model EvaluationLanguage Model Generation Evaluation

    Tübingen AI Center, University of Tübingen

  21. Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark

    Apr 14, 2026Terra Blevins, Stephen Mayhew, Marek Šuppa +11Named Entity RecognitionMultilingual Language Model Evaluation

    Northeastern University, USA · Duolingo, USA · Comenius University Bratislava, Slovakia +10

  22. MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts

    Apr 13, 2026Chen Hu, Yintao Tai, Antonio Vergari +2Multilingual Language ModelsAutoregressive Language Modeling

    School of Informatics, University of Edinburgh

  23. Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation

    Apr 13, 2026Lester James V. Miranda, Ivan Vulić, Anna KorhonenSupervised Fine-TuningMultilingual Language Models

    Language Technology Lab, University of Cambridge

  24. Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models

    Apr 10, 2026Avni Mittal, Shanu Kumar, Sandipan Dandapat +1Multilingual Language Model EvaluationBenchmark Design

    Microsoft Corporation, India · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Indian Institute of Technology Hyderabad

  25. Language Bias under Conflicting Information in Multilingual LLMs

    Apr 8, 2026Robert Östling, Murathan KurfalıLinguistic Bias in Language ModelsLanguage Model Bias Evaluation

    Stockholm University Sweden · RISE Research Institutes of Sweden Sweden

  26. To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models

    Apr 8, 2026Ane G. Domingo-Aldama, Iker De La Iglesia, Maitane Urruela +2Medical LLMsDomain Adaptation for LLMs

  27. Multilingual Cognitive Impairment Detection in the Era of Foundation Models

    Apr 8, 2026Damar Hoogland, Boshko Koloski, Jaya Caporusso +5Speech-Based Dementia DetectionClinical Language Model Evaluation

    Jožef Stefan Institute, Ljubljana, Slovenia · Jožef Stefan International Postgraduate School, Ljubljana, Slovenia · Institute of Synergetics and Psychotherapy Research, Paracelsus Medical University, Salzburg, Austria +2

  28. Swiss-Bench 003: Evaluating LLM Reliability and Adversarial Security for Swiss Regulatory Contexts

    Apr 7, 2026Fatih UenalLLM EvaluationLLM Reliability

    Department of Computer Science, University of Colorado Boulder

  29. Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation

    Apr 6, 2026Firoj Alam, Gagan Bhatia, Sahinur Rahman Laskar +1Language Model Generation EvaluationMultilingual Language Model Evaluation

    Qatar Computing Research Institute, HBKU, Qatar · †UPES, India

  30. Chinese Language Is Not More Efficient Than English in Vibe Coding: A Preliminary Study on Token Cost and Problem-Solving Rate

    Apr 6, 2026Simiao Ren, Xingyu Shen, Yuchen Zhou +3LLM Inference EfficiencyMultilingual Code Generation

    Tsang · ∗,†Scam.ai