Multilingual Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 68% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 309

All topics

Behavioral and Representational Evidence of Binomial Ordering Preferences in Large Language Models
Jun 19, 2026Zhiqing Yang, Yilun Liu, Yunpu Ma +2Multilingual Language Model Evaluation LLM Interpretability
Ludwig Maximilian University of Munich · Munich Center for Machine Learning
The Register Gap: A Meaning Intelligence Framework for Nigerian Public Discourse
Jun 18, 2026Celestine AchiMultilingual Language Model Evaluation LLM Evaluation
AGENTPR™ / AI-Powered PR / Cihan Digital Academy
NRITYAM: Language Models Meet Art and Heritage of Dance
Jun 18, 2026Punit Kumar Singh, Niladri Ghosh, Advait Joshiınst +3Multilingual Language Model Evaluation LLM Evaluation
Shenzhen Technology University, 518118 Shenzhen, China · Indian Institute of Technology, India · Ramakrishna Mission Vivekananda Educational and Research Institute, India +4
Creating Multilingual Mental Health Dialogue Datasets: Limits of Persona-Based Localization via Nationality and Language
Jun 17, 2026Yunkai Xu, Saeed AbdullahMultilingual Language Model Evaluation Depression Detection
Pennsylvania State University, United States
G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment
Jun 17, 2026Fengying Ye, Yanming Sun, Runzhe Zhan +3Figurative Language Understanding Multilingual Language Model Evaluation
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Faculty of Arts and Humanities, University of Macau
When English Isn't the Best Teacher: Source Language Effects in Cross-Lingual In-Context Learning
Jun 16, 2026Fred Philippy, Siwen Guo, Jacques Klein +1Multilingual Language Model Evaluation In-Context Learning
Snt, University of Luxembourg, Luxembourg · Luxembourg Institute of Science and Technology, Luxembourg
ChLogic: Evaluating Robustness of Logical Reasoning in Chinese Expressions
Jun 16, 2026Peixian Zhou, Yuxu Chen, Chaorui Zhang +3Multilingual Language Model Evaluation Cross-Lingual Reasoning in Language Models
College of Mathematics, Sichuan University, China · Theory Lab, 2012 Labs, Huawei Technologies Co., Ltd.
NarrativeWorldBench: A Frontier-Saturated Benchmark and a Latent World Model for Long-Horizon Co-Creative Audio Drama
Jun 16, 2026Logan Mann, Abdur Rahman, Mohammad Saifullah +2LLM World Models Multilingual Language Model Evaluation
University of California, Santa Barbara · Pocket FM
P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs
Jun 15, 2026Rafael Ferreira, Inês Vieira, Inês Calvo +6Multilingual Language Model Evaluation Social Bias in Language Models
NOVA University of Lisbon, Portugal · NOVA LINCS
Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation
Jun 14, 2026Gili Lior, Tzviel Frostig, Gabriel Stanovsky +1Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
Google Research · The Hebrew University of Jerusalem · PhaseV Trials
Beyond Monolingual Deep Research: Evaluating Agents and Retrievers with Cross-Lingual BrowseComp-Plus
Jun 13, 2026Yuheng Lu, Qingcheng Zeng, Heli Qi +6Multilingual Language Model Evaluation Deep Research Agents
Waseda University · Northwestern University · RIKEN AIP +4
AmchiBias: Measuring Stereotypical Bias in Goan Identity Groups with a Minimal Pair Dataset in English and Konkani
Jun 13, 2026Michelle Barbosa, Sebastian Padó, Franziska WeeberMultilingual Language Model Evaluation Cultural Bias in Language Models
Institute for Natural Language Processing, University of Stuttgart
Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge
Jun 12, 2026Shaojie YinMultilingual Language Model Evaluation LLM Evaluation
Shanghai Artificial Intelligence Laboratory Shanghai, China
When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognates
Jun 11, 2026Junhong Liang, Noor Abo Mokh, Bashar AlhafniMultilingual Language Model Evaluation Arabic NLP
Mohamed bin Zayed University of Artificial Intelligence
MÖVE: A Holistic LLM Benchmark for the German Public Sector
Jun 11, 2026Camilla Dalerci, Thilo Michael, Robin Schaefer +1Multilingual Language Model Evaluation LLM Evaluation
Innovations Department, Bundesdruckerei GmbH, Berlin, Germany.
Polar: A Benchmark for Evaluating Political Bias in LLMs
Jun 11, 2026Sangho Kim, Heejin Kim, Yoonhee Park +2Multilingual Language Model Evaluation Political Bias in Language Models
Graduate School of Data Science, Seoul National University · Dept. of Computer Science and Engineering, Seoul National University
AfriSUD: A Dependency Treebank Collection for Evaluating Models on African Languages
Jun 10, 2026Happy Buzaaba, Cheikh Mouhamadou Bamba Dione, David Ifeoluwa Adelani +15Multilingual Language Model Evaluation Morphosyntactic Tagging
Princeton University · Laboratory for Artificial Intelligence, Princeton University · Gaston Berger University +20
Schützen: Evaluating LLM Safety in Bulgarian and German Contexts
Jun 9, 2026Kiril Georgiev, Yuxia Wang, Dimitar Iliyanov Dimitrov +2LLM Safety Benchmarks Multilingual Language Model Evaluation
Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski” · MBZUAI
The Shibboleth Effect: Auditing the Cross-Lingual Distributional Skew of Large Language Models
Jun 9, 2026Hakan MehmetcikMultilingual Language Model Evaluation LLM Auditing
Visiting Researcher, Kellogg Institute for International Studies, University of Notre Dame, IN, USA · Prof of International Relations, at the Faculty of Political Science, Marmara University, Istanbul, Türkiye
Who Brought Easter Eggs to Eid? Auditing LLM-Generated Cultural Translation of Math Word Problems Across Languages and Regions
Jun 9, 2026Parisa Suchdev, Juniper LovatoMultilingual Language Model Evaluation Cultural Bias in Language Models
Computational Ethics Lab · Department of Computer Science, University of Vermont · Vermont Complex Systems Center, University of Vermont +1
BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali Contexts
Jun 8, 2026Kazi Noshin, Sajib Acharjee Dip, Ranat Das Prangon +4Multilingual Language Model Evaluation LLM Evaluation
University of Illinois Urbana-Champaign, USA · Virginia Tech, USA · Bangladesh University of Engineering and Technology, Bangladesh +2
Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis
Jun 8, 2026Hyeji Choi, Yongtaek Lim, Minwoo KimMultilingual Language Model Evaluation Language Model Safety Evaluation
AI Safety Team, DATUMO.INC, Seoul, South Korea.
Friend or Foe? Language as an ideological switch in open-weight LLMs under Russian disinformation stress
Jun 7, 2026Anna Małgorzata Kamińska, Tetiana KlyninaMultilingual Language Model Evaluation Political Bias in Language Models
Institute of Culture Studies, University of Silesia in Katowice, ul. Uniwersytecka 4, 40-007 Katowice, Poland · University of Texas at Austin 2515, TX 78712 Speedway Str., Texas, USA · National Aviation University, Liubomyr Huzar Avenue 1, Kyiv 03058, Ukraine
Sycophancy as a Multilingual Alignment Failure: How Safety Degrades Across Languages, Topics, and Models
Jun 7, 2026Arya Shah, Himanshu Beniwal, Mayank Singh +1Multilingual Language Models Multilingual Language Model Evaluation
IIT Gandhinagar · Asian Institute of Technology
When Languages Disagree: Self-Evolving Multilingual LLM Judges
Jun 6, 2026Xiyan Fu, Wei LuMultilingual Language Model Evaluation LLM-as-a-Judge
Nanyang Technological University
Sci-Rho: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problems
Jun 6, 2026Muhammad Falensi Azmi, Ikhlasul Akmal Hanif, Vallerie Alexandra Putra +3VLM Evaluation VLM Robustness
†Independent Researcher · ‡MBZUAI · §Binus University +1
Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese
Jun 5, 2026Giordano de Pinho Souza, Glaucia Melo, Josefino Cabral Melo Lima +1Multilingual Language Model Evaluation Clinical Decision-Making
Federal University of Rio de Janeiro · Toronto Metropolitan University Toronto, CA
The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs
Jun 5, 2026Yang Zhang, Xiao Fei, Amr Mohamed +6Multilingual Language Model Evaluation LLM Evaluation
Ecole Polytechnique · MBZUAI · ENS-PSL +1
MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models
Jun 5, 2026Rishabh Makwana, Mamta, Deeksha Varshney +1VLM Evaluation VLM Robustness
Dwarkadas Jivanlal Sanghvi College of Engineering, Mumbai, India · King’s College London, London, UK · Indian Institute of Technology Jodhpur, Jodhpur, India
mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?
Jun 5, 2026Yerzhan Sapenov, Jaromir SavelkaMultilingual Language Models Multilingual Language Model Evaluation
School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA

Jun 19, 2026, cs.CLJ/K move · Enter open · S save

Behavioral and Representational Evidence of Binomial Ordering Preferences in Large Language Models

Zhiqing Yang, Yilun Liu, Yunpu Ma, Volker Tresp+1

Ludwig Maximilian University of Munich · Munich Center for Machine Learning

Multilingual Language Model Evaluation LLM Interpretability

Open paper AskPDF arXiv

Large language models (LLMs) can readily reproduce conventional expressions, yet their ability to model gradient frequency distributions remains underexplored. We investigate this using linguistic binomials, such as men and women, where both word permutations are grammatically valid but exhibit distinct, cross-linguistic variations in conventionality. We formalize binomial ordering as a distributional alignment problem, and construct a multilingual dataset of 600 binomial pairs across 8 languages. With categorical and distributional metrics, we measure and compare the corpus-derived preferences with model-induced ordering probabilities of 6 open-weight LLMs. While models often behaviorally recover the dominant corpus-preferred order, particularly for strongly conventionalized pairs, they align less well with the exact corpus preference distributions. This suggests that apparent directional order overstates how faithfully LLMs capture the statistical nuances of language use. Sparse probing verifies that the concept of preference strength is partially encoded among middle-to-late layers, and steering along probe-derived directions alters model-induced ordering distributions, demonstrating that the statistical behavioral preference of LLMs can be mechanistically measured and manipulated via internal representations.

Momentum

Related topics

Latest papers 309