Multilingual Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 68% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 309

All topics

MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights
Jun 5, 2026Yilun Liu, Miao Zhang, Shimin Tao +9Multilingual Language Model Evaluation LLM Evaluation
Huawei, China
Improving Cross-Lingual Factual Recall via Consistency-Driven Reinforcement Learning
Jun 4, 2026Jonathan von Rad, Louis Arts, George Burgess +6Multilingual Language Model Evaluation RL for Language Models
University College London, Centre for Artificial Intelligence
Large Language Models are Perplexed by some Political Parties
Jun 4, 2026Paul Lerner, François YvonMultilingual Language Model Evaluation Political Bias in Language Models
Sorbonne Université, CNRS, ISIR 75005, Paris, France
When Surface Form Changes Moderation Decisions: A Paired Study of Code-Mixed Workflow Instability
Jun 4, 2026Suraj Babu Thimma Krishnaram, Yibo Hu, Karthikeyan SaravananMultilingual Language Model Evaluation Hate Speech Detection
Illinois Institute of Technology Chicago, IL, USA
Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models
Jun 2, 2026Hashmat Shadab Malik, Muzammal Naseer, Salman KhanMultilingual Language Model Evaluation Adversarial Attacks on VLMs
Mohamed Bin Zayed University of AI, UAE · Khalifa University, UAE · Australian National University, Australia
Auditing LLM-Governed Social Robots with Culture-Specific Moral Gradients
Jun 2, 2026Carmen Ng, Gjergji KasneciMultilingual Language Model Evaluation Moral Reasoning in Language Models
Technical University of Munich, Munich, Germany
Evaluating LLMs' Effectiveness on Real-World Consumer Device Repair Questions
Jun 2, 2026Atm Mizanur Rahman, Md Arid Hasan, Syed Ishtiaque Ahmed +1Multilingual Language Model Evaluation LLM Evaluation
University of Illinois Urbana-Champaign · University of Toronto
From Script to Semantics: Prompting Strategies for African NLI
Jun 2, 2026Anuj Tiwari, Terry Oko-odion, Hannah NwokochaMultilingual Language Model Evaluation LLM Prompting
Noida Institute of Engineering and Technology1, India1 · ML Collective1,2,3, Nigeria2,3
SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understanding
Jun 2, 2026Peerawat Chomphooyod, Jian Gang Ngui, Yosephine Susanto +5Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
1Chulalongkorn University · 2AI Singapore · 3MBZUAI +1
Sample-Size Scaling of the African Languages NLI Evaluation
Jun 2, 2026Anuj Tiwari, Oluwapelumi Ogunremu, Terry Oko-odion +2Multilingual Language Model Evaluation Low-Resource Language Processing
Noida Institute of Engineering and Technology1 · ML Collective1,2,3,4,5
The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment
Jun 2, 2026Sourabrata Mukherjee, Hamna Hamna, Kalika Bali +1Inter-Rater Reliability Multilingual Language Model Evaluation
Microsoft Research
Multilingual Idioms in Sentences and Conversations Across High-, Medium-, and Low-Resource Languages
Jun 1, 2026Saeed Almheiri, Bilal Elbouardi, Salsabila Zahirah Pranida +16Figurative Language Understanding Multilingual Language Models
1Mohamed bin Zayed University of Artificial Intelligence · University of Hamburg · 3Manipal University +4
TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages
May 31, 2026Victor Akinode, Senyu Li, Wassim Hamidouche +3LLM Safety Benchmarks Multilingual Language Model Evaluation
Mila - Quebec AI Institute · McGill University · Microsoft AI for Good Research Lab +1
IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages
May 31, 2026Ikhlasul Akmal Hanif, Muhammad Falensi Azmi, Filbert Aurelian Tjiaranata +2Multilingual Language Model Evaluation Cultural Bias in Language Models
Mohamed bin Zayed University of Artificial Intelligence · Independent Researcher · Universitas Indonesia
Understanding LLM Behavior in Multi-Target Cross-Lingual Summarization
May 31, 2026Sangwon Ryu, Yihong Liu, Mingyang Wang +4Multilingual Language Model Evaluation Text Summarization
GSAI, POSTECH · LMU Munich · MCML +2
Implicit Geographic Inference in LLM Medical Triage: Language-Driven Disparities in Emergency Recommendations
May 31, 2026Qi Han WongMultilingual Language Model Evaluation Clinical Triage
Toward Responsible and Epistemically Grounded Multilingual LLMs for Computational Social Science and Humanities
May 30, 2026Wajdi ZaghouaniMultilingual Language Model Evaluation Political Discourse Analysis
Northwestern University in Qatar
Which Institutional Frameworks Do Chatbots Assume? Auditing Jurisdictional Defaults in Multilingual LLMs
May 29, 2026Zhizhi Wang, Harini SureshMultilingual Language Model Evaluation LLM Auditing
Brown University
BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali
May 29, 2026Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +3Multilingual Language Model Evaluation Language Model Error Detection
Department of Computer Science and Engineering, Islamic University of Technology · Department of Computer Science and Engineering, University of California
Translation Analytics for Freelancers II: Benchmarking Local LLMs for Confidential Translation Workflows
May 29, 2026Yuri Balashov, Rex VanHorn, Mingxi Xu +1Multilingual Language Model Evaluation On-Device Language Model Inference
University of Georgia, Athens, Georgia, USA
LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories
May 29, 2026Krishnapriya Vishnubhotla, Sowmya Vajjala, Akriti Vij +1Multilingual Language Model Evaluation LLM-as-a-Judge
National Research Council, Canada · IMDA, Singapore
On the Robustness of Multilingual Text Embedding Rankings Across Learning Tasks, Languages, and Benchmark Datasets
May 29, 2026Ana Gjorgjevikj, Barbara Koroušić Seljak, Tome EftimovMultilingual Language Model Evaluation Text Embedding Evaluation
Computer Systems Department Jožef Stefan Institute Ljubljana, Slovenia
Multilingual and Cross-Lingual Citation Needed Detection on Wikipedia for Lower-Resource Languages
May 29, 2026Gerrit Quaremba, Amy Rechkemmer, Elizabeth Black +2Multilingual Language Model Evaluation Small Language Models
King’s College London · Wikimedia Foundation
Do Large Language Models Encode Institutional Experience? Evidence from Cross-Linguistic Moral Reasoning Under Ambiguity
May 29, 2026Nattavudh PowdthaveeMultilingual Language Model Evaluation Moral Reasoning in Language Models
School of Social Sciences, Nanyang Technological University, Singapore
Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Audit
May 29, 2026Jiwoo Choi, Seonwoo Ahn, Tongxin Zhang +1Gender Bias in Language Models Multilingual Language Model Evaluation
School of Digital Humanities and Computational Social Sciences, KAIST, South Korea
XLGoBench: Detecting cross-lingual skill gaps with algorithmic tasks
May 29, 2026Purvam Jain, Preethi Jyothi, Vihari Piratla +1Multilingual Language Model Evaluation Cross-Lingual Reasoning in Language Models
Google DeepMind · Indian Institute of Technology Bombay · International Centre for Theoretical Sciences, Tata Institute of Fundamental Research
When English Rewrites Local Knowledge: Global Narrative Dominance in Large Language Models
May 28, 2026Md Arid Hasan, Ruwad Naswan, Farhan Samir +2Multilingual Language Model Evaluation LLM Grounding
University of Toronto · BUET · University of Illinois Urbana-Champaign
CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild
May 28, 2026Sahajpreet Singh, Insyirah Mujtahid, Min-Yen Kan +1Multilingual Language Model Evaluation LLM Evaluation
National University of Singapore
Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions
May 28, 2026Volodymyr OvcharovMultilingual Language Model Evaluation Legal NLP
SecondLayer
Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese
May 28, 2026Wajdi Zaghouani, Kholoud K. Aldous, Yicheng GaoMultilingual Language Model Evaluation LLM Evaluation
Northwestern University in Qatar

Jun 5, 2026, cs.CLJ/K move · Enter open · S save

MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights

Yilun Liu, Miao Zhang, Shimin Tao, Minggui He+8

Huawei, China

Multilingual Language Model Evaluation LLM Evaluation Cross-Cultural Language Model Evaluation LLM Agent Evaluation

Open paper AskPDF arXiv

Multilingual and multicultural benchmarks now cover dozens of languages and model families, but the resulting score landscapes remain metric-rich and insight-poor, necessitating fine-grained multilingual post-evaluation diagnosis. However, single LLMs and open-ended agents are easily swamped by the long, noisy diagnostic input, and no reusable taxonomy exists for it. To address this, we propose MADE, a Multilingual Agentic Diagnosing Engine that decomposes post-evaluation analysis into planning, aggregate analysis, instance-level case inspection, multilingual and cultural reflection, and grounded report synthesis. MADE is paired with an expert-led 54-query and 15-language diagnostic set, evaluated on top of a large-scale multilingual evaluation substrate (33 model families, 11 benchmarks, 26 languages, 34 cultures, 8.66M evaluation records). Experiments show that MADE outperforms the strongest shared baseline by 47% in diagnosis report quality and is preferred by human multilingual experts in 87.9% of pairwise comparisons. Applied with multilingual experts, MADE further surfaces four actionable findings on deployment, iteration, and cross-cultural pitfalls, turning benchmark score tables into model-selection and remediation guidance.

Momentum

Related topics

Latest papers 309