Multilingual Language Model Evaluation

Overview

The field studies whether language models understand and respond reliably across languages, rather than assuming results in English transfer. Researchers build native-speaker-authored and translated benchmarks, use controlled prompt comparisons, and probe linguistic features such as morphology, pragmatics, and tokenization. Evaluations span reasoning, translation, tool use, safety, bias, and calibration, combining automatic metrics with human judgments and internal-model analyses. A central tension is that translation artifacts, evaluator bias, and aggregate scores can hide language-specific failures; performance remains uneven, especially for lower-resource languages and cultural contexts.

Latest papers

CardsList

530 papers56 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

  1. Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

    Jan 8, 2026Zheng Luo, T Pranav Kutralingam, Ogochukwu N Okoani +3Multilingual Language Model EvaluationLLM Tool Use

  2. MultiZebraLogic: A Multilingual Logical Reasoning Benchmark

    Nov 5, 2025Sofie Helene Bruun, Dan Saattrup SmartLogical ReasoningMultilingual Language Model Evaluation

  3. MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts

    Nov 1, 2025Naoto Iwase, Hiroki Okuyama, Junichiro IwasawaMultilingual Language Model EvaluationHealthcare

  4. KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs

    Oct 21, 2025Donghyeon Ko, Kyubyung Chae, Yeguk Jin +6Multilingual Language Model EvaluationLLM Evaluation

  5. Zoom In Disparities in Healthcare LLM Q&A

    Oct 20, 2025Ipek Baris Schlicht, Burcu Sayin, Zhixue Zhao +5Multilingual Language Model EvaluationKnowledge Augmentation for Language Models

  6. Evaluating Large Language Models on Urdu Idioms

    Oct 20, 2025Muhammad Farmal Khan, Mousumi AkterMultilingual Language Model EvaluationLow-Resource Language Processing

  7. Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages

    Oct 6, 2025Tarek Naous, Anagha Savit, Carlos Rafael Catalan +17Multilingual Language Model EvaluationCultural Bias in Language Models

  8. How Well Does Mathematical Reasoning Transfer Across Languages in English-Centric LLMs?

    Oct 2, 2025Wen Yang, Junhong Wu, Chong Li +2Cross-Lingual Reasoning in Language ModelsCross-Lingual Transfer

  9. Simulating Student-Tutor Interactions with LLMs for Multilingual Math Feedback

    Jun 5, 2025Junior Cedric Tonga, KV Aditya Srivatsa, Kaushal Kumar Maurya +2Educational Feedback GenerationGenerative AI in Education

  10. In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models

    Mar 3, 2025David Ponce, Thierry EtchegoyhenMultilingual Language ModelsMultilingual Language Model Evaluation

  11. Multilingual Hallucination Gaps in Large Language Models

    Oct 23, 2024Cléa Chataigner, Afaf Taïk, Golnoosh FarnadiHallucination in Language ModelsMultilingual Language Model Evaluation

  12. Leveraging LLMs for Translating and Classifying Mental Health Data

    Oct 16, 2024Konstantinos Skianis, A. Seza Doğruöz, John PavlopoulosDepression Severity EstimationClinical NLP

  13. Building Multilingual Datasets for Predicting Mental Health Severity through LLMs: Prospects and Challenges

    Sep 25, 2024Konstantinos Skianis, John Pavlopoulos, A. Seza DoğruözMultilingual Language Model EvaluationClinical Language Model Evaluation