Multilingual Language Model Evaluation

Overview

The field studies whether language models understand and respond reliably across languages, rather than assuming results in English transfer. Researchers build native-speaker-authored and translated benchmarks, use controlled prompt comparisons, and probe linguistic features such as morphology, pragmatics, and tokenization. Evaluations span reasoning, translation, tool use, safety, bias, and calibration, combining automatic metrics with human judgments and internal-model analyses. A central tension is that translation artifacts, evaluator bias, and aggregate scores can hide language-specific failures; performance remains uneven, especially for lower-resource languages and cultural contexts.

Latest papers

CardsList

530 papers56 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

  1. One Framework to Diagnose Them All: A Stratified Benchmark for Bengali LLM Hallucination

    Nov 19, 2026Asif Shahriyar Sushmit, Shefayat E Shams Adib, Abdullah Al Jubaer Gem +16Hallucination DetectionLLM Hallucination Detection

  2. PILAPS: A Bilingual Benchmark for Evaluating Pragmatic Inference across Phenomena and Settings

    Nov 19, 2026Jiani Cao, Anna Korhonen, Chen Cecilia LiuPragmatic Reasoning in Language ModelsMultilingual Language Model Evaluation

  3. Compression Effects on Low-Resource Multilingual Translation

    Nov 18, 2026Abbas SaniLow-Resource MTLLM Quantization

  4. MoE--LoRA Methods for Multilingual Adaptation: A Systematic Empirical Study

    Nov 17, 2026Nikit Srivastava, Joel Dag, René Speck +1Low-Rank AdaptationLLM Fine-Tuning

  5. AfriNORM: An Evaluation Benchmark for Normative Reasoning of LLMs Across African Cultures

    Nov 17, 2026Ahmad Samuel Gali, Tadesse Destaw Belay, Idris Abdulmumin +2Cross-Cultural Language Model EvaluationCultural Knowledge in Language Models

  6. A Comparative Evaluation of BERT-Based and LLM-Based Models for Indic Sentence Similarity

    Nov 14, 2026Aishwarya Gunjal, Kanchan Khemnar, Pawan Poojary +2Semantic Textual SimilaritySentence Embeddings

  7. Outside the Purview of the Reward: Capability Drift in Multilingual Post-Training

    Nov 11, 2026Sutanshu Saksena Raj, Purusharth Malik, Pasunuti Prasanjith +3RL Post-TrainingReward Hacking

  8. Is Jev a Good Multilingual Judge?

    Nov 11, 2026Wei Liu, Yi Fan, Xiyan FuLLM-as-a-JudgeMultilingual Language Model Evaluation

  9. The Narrative Gap: Can LLMs Help Us Navigate Diverse Narratives Across Languages?

    Nov 9, 2026Nikhil Sharma, Kelly Marchisio, Kenton Murray +1Multilingual IRMultilingual Language Model Evaluation

  10. Saidzoda-CA-EmbeddingGemma-300M: One Small Embedder for Tajik, Kazakh, Uzbek and Kyrgyz

    Nov 8, 2026Tohir Saidzoda, Nekruz Maxudov, Bedil Karimov +5Text Embedding ModelsMultilingual IR

  11. What Travels Across Languages? Reading, Transfer and Erasure of Cross-Lingual Concept Subspaces in Bangla Idioms

    Nov 8, 2026Syed Rifat Raiyan, Yousuf Ibne Kamal NiloyCross-Lingual TransferCross-Lingual Representation Alignment

  12. WarBias: A Bilingual LLM Benchmark of Stereotypes About Internally Displaced People and Veterans

    Nov 8, 2026Viktoriia Makovska, Tetiana Zakharchenko, Oleksandra OstafiichukSocial Bias in Language ModelsLanguage Model Bias Evaluation

  13. Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects

    Nov 7, 2026Hossam Elsafty, Tobias Deußer, Lorenz Sparrenberg +2Arabic Language Model EvaluationLLM Evaluation

  14. I play of the cello: Evaluating Constructional Bleeding in Multilingual LLMs with CALQUE

    Nov 6, 2026Toshiki Nakai, Yu-Yin Hsu, Marie-Catherine de Marneffe +6Multilingual Language ModelsMultilingual Language Model Evaluation

  15. The Invisible Gap: Romanized Indic Generation Fails Where Automatic Metrics Succeed

    Nov 6, 2026Madhu Sudhan Rao Beesetty, Surbhi Sharma, Devesh Maheshwari +1Language Model Generation EvaluationGenerative AI Evaluation

  16. Reasoning over Non-Parametric Memory in Multilingual LLMs: A Cognitive-Based Analysis of Relational Knowledge

    Nov 4, 2026Zohreh Abbasi Liasi, Batoul Hammoud, Anas Madkoor +1Language Model Reasoning EvaluationMultilingual Language Model Evaluation

  17. ALTO: African and Levantine Tech-Facilitated Gender-Based Violence Corpus

    Nov 4, 2026Tianwei Zhang, Ivy Njoki Nyagwencha, Zahraa Dirani +2African Language ProcessingMultilingual Language Model Evaluation