Multilingual Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 68% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 309

All topics

MET: Theory-Grounded and Culture-Aware Multilingual Moral Reasoning
Jul 13, 2026Ayoung Lee, Ryan Kwon, Yunxiang Zhang +3Multilingual Language Model Evaluation Moral Reasoning in Language Models
Department of Computer Science and Engineering · Department of Philosophy
Cost of Reasoning in non-English Languages: A Case Study on Japanese
Jul 11, 2026Yuu JinnaiMultilingual Language Models Multilingual Language Model Evaluation
CyberAgent / Tokyo, Japan
Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety
Jul 11, 2026Chigozirim Ifebi, Brent Kong, Ayushi MehrotraLLM Safety Benchmarks Multilingual Language Model Evaluation
California Institute of Technology.
Validity of LLMs as data annotators: AMALIA on authority
Jul 9, 2026Manuel PitaMultilingual Language Model Evaluation Moral Reasoning in Language Models
Artificial Intelligence, Social Interaction and Complexity Laboratory · CICANT, Universidade Lusófona
Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nung
Jul 9, 2026Anh Trac Duc Dinh, Khang Nhat Hoang Vo, Vinh Cong Doan +2Multilingual Language Model Evaluation Multilingual IR
Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), VNU-HCM, Ho Chi Minh City, Vietnam · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration
Jul 8, 2026Weicheng Ma, John Guerrerio, Soroush VosoughiMultilingual Language Model Evaluation Social Bias in Language Models
College of Computing, Georgia Institute of Technology · Computer Science Department, Dartmouth College
Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
Jul 7, 2026Andrea Bacciu, Andrea Alfarano, Saab Mansour +2Multilingual Language Model Evaluation Selective Prediction
Amazon · INSAIT, Sofia
PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents
Jul 7, 2026Hongliang Li, Yijin Liu, Zhiwei Zhang +5Multilingual Language Model Evaluation Long-Horizon Agent Evaluation
Beijing Jiaotong University · Weixin AI, Tencent Inc
PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages
Jul 7, 2026Daryna Dementieva, Nikolay Babakov, Kathy Hämmerl +14Mathematical Reasoning Benchmarks Multilingual Language Model Evaluation
Technical University of Munich (TUM) · Munich Center for Machine Learning (MCML) · Independent Researcher +7
CrossHallu: Do Hallucination Signals Generalize Across Languages and Domains in Large Language Model's Internals?
Jul 4, 2026Aisha Alansari, Malak Alkhorasani, Hamzah LuqmanMultilingual Language Model Evaluation LLM Hallucination Detection
King Fahd University of Petroleum and Minerals · Imam Abdulrahman bin Faisal University
Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages
Jul 2, 2026A. Seza Doğruöz, Xixian Liao, Verena Blaschke +3Multilingual Language Model Evaluation LLM-as-a-Judge
LT3, IDLab, Universiteit Gent · Barcelona Supercomputing Center · LMU Munich & Munich Center for Machine Learning +5
SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses
Jul 2, 2026Anna ChornaMultilingual Language Model Evaluation LLM-as-a-Judge
Safety Targeted Embedding Exploit via Refinement
Jul 2, 2026Joshua Adrian CahyonoMultilingual Language Model Evaluation Adversarial Attacks on LLMs
Nanyang Technological University, Singapore.
Evaluating Large Language Model Raters for German Open-Response Clinical Questions: A Physician-Annotated Benchmark Study of Agreement, Evaluator Bias, and Abstention
Jul 1, 2026William Philipp, Finn Fassbender, Daniel Fister +12Multilingual Language Model Evaluation LLM-as-a-Judge
Section for Clinical Research-IT, Institute of Medical Biometry and Statistics, University of Luebeck, Luebeck, Germany · Medical Faculty, University of Tübingen, Tübingen, Germany · German Medical Students’ Association (bvmd), Berlin, Germany +9
MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages
Jul 1, 2026Maximilian Idahl, Jörg Tiedemann, Sampo Pyysalo +19Language Model Pretraining Synthetic Data Pretraining
ellamind · Leibniz University Hannover · University of Helsinki +14
MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark
Jul 1, 2026Xianru Chen, Yukai Huang, Mingxiang Chen +6Multilingual Language Model Evaluation LLM Evaluation
ByteDance Seed · Nanjing University · M-A-P
Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth
Jun 30, 2026Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmad ElShiekh +2Multilingual Language Model Evaluation LLM-as-a-Judge
1Perle AI
Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions
Jun 29, 2026Avisha Das, Mihir Parmar, Mohana Ramnath +1Multilingual Language Model Evaluation Instruction Following
Shiv Nadar University Chennai · Google Cloud AI Research · IIT Madras
EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbots
Jun 29, 2026Camilo Chacón SartoriLLM Safety Benchmarks Multilingual Language Model Evaluation
MindSurf
The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning
Jun 27, 2026Will Hawkins, Kaivalya Rawal, Jonathan Rystrøm +8LLM Safety Benchmarks Multilingual Language Model Evaluation
Oxford Internet Institute, University of Oxford · Hasso Plattner Institute, University of Potsdam · Chinese University of Hong Kong +2
A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation
Jun 24, 2026Abrar Alotaibi, Raed Mughus, Moataz AhmedMultilingual Language Model Evaluation LLM Evaluation
Information and Computer Science Department, King Fahd University of Petroleum & Minerals, Dhahran, 31261, Saudi Arabia. · College of Computer Science and Information Technology, Imam Abdulrahman Bin Faisal University, Dammam, 31441, Saudi Arabia. · SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum & Minerals, Dhahran, 31261, Saudi Arabia.
Same Lesson, Different Story: Cross-Lingual Reconstruction of Cultural Narratives in Large Language Models
Jun 23, 2026Jory Alshaalan, Haya Albaker, Abeer Aldayel +2Multilingual Language Model Evaluation LLM Grounding
College of Computer and Information Sciences King Saud University, Saudi Arabia
The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs
Jun 23, 2026Olaoye Anthony SomideMultilingual Language Models Multilingual Language Model Evaluation
DataLens Africa Research · CipherSense AI Technologies Ltd.
A Pāninian Foundation for Indic Language Processing
Jun 23, 2026Ritwik Banerjee, Lav R. VarshneyMultilingual Language Model Evaluation Low-Resource Language Processing
Department of Computer Science, Stony Brook University, USA · AI Innovation Institute, Stony Brook University · AI Innovation Institute, Stony Brook University, USA +1
Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts
Jun 22, 2026Arthur Wuhrmann, Gaetan Stein, Daniel Brunner +1Multilingual Language Model Evaluation Legal NLP
Swiss Federal Supreme Court · HES-SO Valais-Wallis
Language-Specific Sentiment Polarity Biases in Encoder and Large Language Model Classification of Product Reviews
Jun 22, 2026Advita Rajiv, Kavitha Kothur, Gautham ReddyMultilingual Language Model Evaluation LLM Evaluation
John P. Stevens High School, Edison, NJ 08820 · Ford Foundation, New York, NY 10017 · NC State University, Raleigh, NC 27606
BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams
Jun 21, 2026João Guilherme Alves Santos, Giovana Kerche Bonás, Thiago Laitz +2Multilingual Language Model Evaluation LLM Evaluation
University of Campinas (UNICAMP), Campinas-SP, Brazil · Tropic AI, Campinas-SP, Brazil · Maritaca AI, Campinas-SP, Brazil
BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCMultilingual Language Model Evaluation LLM-as-a-Judge
Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfer
Jun 20, 2026Prasoon Bajpai, Eleftheria Briakou, Colin Cherry +2Multilingual Language Models Multilingual Language Model Evaluation
Google DeepMind · Indian Institute of Technology Bombay
The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inference
Jun 20, 2026Naihao Deng, Alissa Shen, Yiming Feng +5LLM Inference Efficiency Multilingual Language Models
University of Michigan · University of Cambridge · University of Aberdeen

Jul 13, 2026, cs.CLJ/K move · Enter open · S save

MET: Theory-Grounded and Culture-Aware Multilingual Moral Reasoning

Ayoung Lee, Ryan Kwon, Yunxiang Zhang, Yuxuan Liu+2

Department of Computer Science and Engineering · Department of Philosophy

Multilingual Language Model Evaluation Moral Reasoning in Language Models

Open paper AskPDF arXiv

Language models are increasingly used for moral decision-making across diverse linguistic and cultural contexts, yet existing work overlooks multilinguality on three aspects: 1) multilingual evaluation benchmarks use direct translation, failing to adapt culture-specific items; 2) inference-time methods for moral reasoning rely on static, English-centric scaffolds and lack grounding in moral theory; 3) training methods for moral decision-making typically require expensive supervision from stronger models or human annotators. We address these gaps with three contributions. First, we introduce MCLASH, a multilingual moral decision-making benchmark to capture culturally situated moral intuitions and social norms across languages. Second, we propose MET (Multilingual Ethics with Theory-grounded reasoning), a two-step prompting method built on expert-curated, theory-based grounds drawn from psychology and philosophy: the model first selects situation- and culture-specific grounds, then reasons over them in the native language of the user. Third, we introduce MET-D (MET-Distillation), which enhances the second step through a self-distillation training stage that requires no external supervision. MET-D improves macro-F1 over the base model on all three models of different sizes and families (Qwen3-4B, Qwen3-8B, Gemma3-4B), by an average of 3.71 points on MCLASH and 4.23 on MMoralExceptQA, with a peak MCLASH gain of 12.94 points for Malay on Qwen3-8B. We further reveal that MET-D increases native-language reasoning by 62.13 points on average, and that beneficial grounds differ systematically across cultures. Together, these contributions open the path for culture-aligned, theory-grounded multilingual moral reasoning.

Momentum

Related topics

Latest papers 309