Multilingual Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 68% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 309

All topics

Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety
Jul 11, 2026Chigozirim Ifebi, Brent Kong, Ayushi MehrotraLLM Safety Benchmarks Multilingual Language Model Evaluation
California Institute of Technology.
Validity of LLMs as data annotators: AMALIA on authority
Jul 9, 2026Manuel PitaMultilingual Language Model Evaluation Moral Reasoning in Language Models
Artificial Intelligence, Social Interaction and Complexity Laboratory · CICANT, Universidade Lusófona
Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nung
Jul 9, 2026Anh Trac Duc Dinh, Khang Nhat Hoang Vo, Vinh Cong Doan +2Multilingual Language Model Evaluation Multilingual IR
Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), VNU-HCM, Ho Chi Minh City, Vietnam · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration
Jul 8, 2026Weicheng Ma, John Guerrerio, Soroush VosoughiMultilingual Language Model Evaluation Social Bias in Language Models
College of Computing, Georgia Institute of Technology · Computer Science Department, Dartmouth College
Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
Jul 7, 2026Andrea Bacciu, Andrea Alfarano, Saab Mansour +2Multilingual Language Model Evaluation Selective Prediction
Amazon · INSAIT, Sofia
PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents
Jul 7, 2026Hongliang Li, Yijin Liu, Zhiwei Zhang +5Multilingual Language Model Evaluation Long-Horizon Agent Evaluation
Beijing Jiaotong University · Weixin AI, Tencent Inc
PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages
Jul 7, 2026Daryna Dementieva, Nikolay Babakov, Kathy Hämmerl +14Mathematical Reasoning Benchmarks Multilingual Language Model Evaluation
Technical University of Munich (TUM) · Munich Center for Machine Learning (MCML) · Independent Researcher +7
CrossHallu: Do Hallucination Signals Generalize Across Languages and Domains in Large Language Model's Internals?
Jul 4, 2026Aisha Alansari, Malak Alkhorasani, Hamzah LuqmanMultilingual Language Model Evaluation LLM Hallucination Detection
King Fahd University of Petroleum and Minerals · Imam Abdulrahman bin Faisal University
Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages
Jul 2, 2026A. Seza Doğruöz, Xixian Liao, Verena Blaschke +3Multilingual Language Model Evaluation LLM-as-a-Judge
LT3, IDLab, Universiteit Gent · Barcelona Supercomputing Center · LMU Munich & Munich Center for Machine Learning +5
SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses
Jul 2, 2026Anna ChornaMultilingual Language Model Evaluation LLM-as-a-Judge
Safety Targeted Embedding Exploit via Refinement
Jul 2, 2026Joshua Adrian CahyonoMultilingual Language Model Evaluation Adversarial Attacks on LLMs
Nanyang Technological University, Singapore.
Evaluating Large Language Model Raters for German Open-Response Clinical Questions: A Physician-Annotated Benchmark Study of Agreement, Evaluator Bias, and Abstention
Jul 1, 2026William Philipp, Finn Fassbender, Daniel Fister +12Multilingual Language Model Evaluation LLM-as-a-Judge
Section for Clinical Research-IT, Institute of Medical Biometry and Statistics, University of Luebeck, Luebeck, Germany · Medical Faculty, University of Tübingen, Tübingen, Germany · German Medical Students’ Association (bvmd), Berlin, Germany +9
MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages
Jul 1, 2026Maximilian Idahl, Jörg Tiedemann, Sampo Pyysalo +19Language Model Pretraining Synthetic Data Pretraining
ellamind · Leibniz University Hannover · University of Helsinki +14
MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark
Jul 1, 2026Xianru Chen, Yukai Huang, Mingxiang Chen +6Multilingual Language Model Evaluation LLM Evaluation
ByteDance Seed · Nanjing University · M-A-P
Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth
Jun 30, 2026Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmad ElShiekh +2Multilingual Language Model Evaluation LLM-as-a-Judge
1Perle AI
Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions
Jun 29, 2026Avisha Das, Mihir Parmar, Mohana Ramnath +1Multilingual Language Model Evaluation Instruction Following
Shiv Nadar University Chennai · Google Cloud AI Research · IIT Madras
EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbots
Jun 29, 2026Camilo Chacón SartoriLLM Safety Benchmarks Multilingual Language Model Evaluation
MindSurf
The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning
Jun 27, 2026Will Hawkins, Kaivalya Rawal, Jonathan Rystrøm +8LLM Safety Benchmarks Multilingual Language Model Evaluation
Oxford Internet Institute, University of Oxford · Hasso Plattner Institute, University of Potsdam · Chinese University of Hong Kong +2
A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation
Jun 24, 2026Abrar Alotaibi, Raed Mughus, Moataz AhmedMultilingual Language Model Evaluation LLM Evaluation
Information and Computer Science Department, King Fahd University of Petroleum & Minerals, Dhahran, 31261, Saudi Arabia. · College of Computer Science and Information Technology, Imam Abdulrahman Bin Faisal University, Dammam, 31441, Saudi Arabia. · SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum & Minerals, Dhahran, 31261, Saudi Arabia.
Same Lesson, Different Story: Cross-Lingual Reconstruction of Cultural Narratives in Large Language Models
Jun 23, 2026Jory Alshaalan, Haya Albaker, Abeer Aldayel +2Multilingual Language Model Evaluation LLM Grounding
College of Computer and Information Sciences King Saud University, Saudi Arabia
The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs
Jun 23, 2026Olaoye Anthony SomideMultilingual Language Models Multilingual Language Model Evaluation
DataLens Africa Research · CipherSense AI Technologies Ltd.
A Pāninian Foundation for Indic Language Processing
Jun 23, 2026Ritwik Banerjee, Lav R. VarshneyMultilingual Language Model Evaluation Low-Resource Language Processing
Department of Computer Science, Stony Brook University, USA · AI Innovation Institute, Stony Brook University · AI Innovation Institute, Stony Brook University, USA +1
Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts
Jun 22, 2026Arthur Wuhrmann, Gaetan Stein, Daniel Brunner +1Multilingual Language Model Evaluation Legal NLP
Swiss Federal Supreme Court · HES-SO Valais-Wallis
Language-Specific Sentiment Polarity Biases in Encoder and Large Language Model Classification of Product Reviews
Jun 22, 2026Advita Rajiv, Kavitha Kothur, Gautham ReddyMultilingual Language Model Evaluation LLM Evaluation
John P. Stevens High School, Edison, NJ 08820 · Ford Foundation, New York, NY 10017 · NC State University, Raleigh, NC 27606
BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams
Jun 21, 2026João Guilherme Alves Santos, Giovana Kerche Bonás, Thiago Laitz +2Multilingual Language Model Evaluation LLM Evaluation
University of Campinas (UNICAMP), Campinas-SP, Brazil · Tropic AI, Campinas-SP, Brazil · Maritaca AI, Campinas-SP, Brazil
BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCMultilingual Language Model Evaluation LLM-as-a-Judge
Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfer
Jun 20, 2026Prasoon Bajpai, Eleftheria Briakou, Colin Cherry +2Multilingual Language Models Multilingual Language Model Evaluation
Google DeepMind · Indian Institute of Technology Bombay
The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inference
Jun 20, 2026Naihao Deng, Alissa Shen, Yiming Feng +5LLM Inference Efficiency Multilingual Language Models
University of Michigan · University of Cambridge · University of Aberdeen
Behavioral and Representational Evidence of Binomial Ordering Preferences in Large Language Models
Jun 19, 2026Zhiqing Yang, Yilun Liu, Yunpu Ma +2Multilingual Language Model Evaluation LLM Interpretability
Ludwig Maximilian University of Munich · Munich Center for Machine Learning
The Register Gap: A Meaning Intelligence Framework for Nigerian Public Discourse
Jun 18, 2026Celestine AchiMultilingual Language Model Evaluation LLM Evaluation
AGENTPR™ / AI-Powered PR / Cihan Digital Academy

Jul 11, 2026, cs.CRJ/K move · Enter open · S save

Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety

Chigozirim Ifebi, Brent Kong, Ayushi Mehrotra

California Institute of Technology.

LLM Safety Benchmarks Multilingual Language Model Evaluation Language Model Safety Evaluation Jailbreak Attacks Mechanistic Interpretability

Open paper AskPDF arXiv

Safety alignment in large language models remains brittle across languages: prompts reliably refused in English can elicit harmful compliance in non-English and low-resource settings. We introduce \textsc{Minionese}, a multilingual jailbreak benchmark spanning 18 languages, 4 resource tiers, and 4 perturbation types (standard translation, code-switching, transliteration, and translationese), paired with a geometric mechanistic analysis of refusal failure across language tiers. We show that each attack type produces a distinct vulnerability profile: transliteration vulnerability is mediated by script identity, code-switching maintains effectiveness through the lowest-resource tier, and a sharp safety regime transition between Tiers 2 and 3 is consistent across all models. Mechanistically, low-resource jailbreaks succeed by routing harmful content through a geometrically misaligned subspace that projects insufficiently onto the refusal directions, leaving the refusal mechanism intact but untriggered. These findings show that English-only safety evaluations are insufficient; they require accounting for script family, perturbation type, and per-language alignment coverage. The benchmark and analysis code is at https://github.com/Brentkong/Minionese-Comprehensive-Benchmark-and-Mechanistic-Study-of-Multilingual-LLM-Safety.git.

Momentum

Related topics

Latest papers 309