Multilingual Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 68% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 309

All topics

Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLAR
May 28, 2026Debajyoti Mazumder, Divyansh Pathak, Prashant Kodali +3Multilingual Language Model Evaluation Cross-Lingual Reasoning in Language Models
Indian Institute of Science Education and Research, Bhopal, India · Microsoft Corporation · UNSW Sydney, Australia
UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning
May 27, 2026Volodymyr OvcharovMultilingual Language Model Evaluation Legal NLP
SecondLayer
Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial Malay
May 27, 2026Mariah Al Giptiah Binte Yusoff, Jakin Tan, Bocheng Chen +2Multilingual Language Model Evaluation Pragmatic Reasoning in Language Models
Nanyang Technological University · University of Mississippi · Indiana University
Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study
May 27, 2026Irune Zubiaga, Aitor Soroa, Rodrigo AgerriMultilingual Language Model Evaluation LLM-as-a-Judge
HiTZ Center - Ixa, University of the Basque Country EHU
Do LLMs Build World Models From Text? A Multilingual Diagnostic of Spatial Reasoning
May 27, 2026Zhikai Pan, Chih-Ting Liao, Chunrui Liu +5LLM World Models Multilingual Language Model Evaluation
University of New South Wales · Essential Energy · University of Alabama at Birmingham +3
IFMTBench: A Comprehensive Benchmark for Multilingual Translation Instruction Following
May 27, 2026Mingrui Sun, Mao Zheng, Zheng Li +1Multilingual Language Model Evaluation Machine Translation
Large Language Model Department, Tencent
DEPART: DEcomposing PARiTy across Multilingual LLMs
May 27, 2026Manan Uppadhyay, Prashant Kodali, Pranjal Chitale +3Multilingual Language Model Evaluation Cross-Lingual Reasoning in Language Models
Microsoft Research India · IIT Gandhinagar
MIRA: A Bilingual Benchmark for Medical Information Response Audit
May 27, 2026Mengyu Xu, Qiaoxin Yang, Qianqian Wang +3Multilingual Language Model Evaluation Healthcare
The University of Chicago · The Chinese University of Hong Kong, Shenzhen · Jinzhou Medical University +3
The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages
May 27, 2026Eric Onyame, Runtao Zhou, Kowshik Thopalli +2CoT Faithfulness Multilingual Language Model Evaluation
University of Virginia · Lawrence Livermore National Laboratory
Disentangling Language Roles in Multilingual LLM Task Execution
May 26, 2026Qishi Zhan, Minxuan Hu, Seoyeon Jang +7Multilingual Language Models Multilingual Language Model Evaluation
Marquette · Cornell · UC San Diego +7
JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors
May 26, 2026Jiho Jin, Junho Myung, Juhyun Oh +5Multilingual Language Model Evaluation LLM-as-a-Judge
1KAIST · 3Universitas Gadjah Mada · 2Google
Rethinking the Multilingual Reasoning Gap with Layer Swap
May 26, 2026Maxence Lasbordes, Amélie Chatelain, Djamé SeddahMultilingual Language Models Multilingual Language Model Evaluation
1LightOn, Paris · 2Inria, Paris
An In-Vitro Study on Cross-Lingual Generalization in Language Models
May 26, 2026Adrian CosmaMultilingual Language Model Evaluation Cross-Lingual Transfer
Dalle Molle Institute for Artificial Intelligence (IDSIA)
WhoSaidIt: Human-LLM Collaborative Annotation for Text-Based Multilingual Speaker-Attribute Classification
May 25, 2026Lingyu Gao, Will Monroe, David Smith +2Multilingual Language Model Evaluation LLM-Assisted Annotation
Duolingo
Forgotten Words: Benchmarking NeoBERT for Dementia Detection in Low-Resource Conversational Filipino and English Speech
May 25, 2026Rez Samantha Z. Floresca, Edric Castel C. Hao, Hannah Grachiella Buñales +3Multilingual Language Model Evaluation Speech-Based Dementia Detection
Ateneo de Manila Senior High School · Analog Devices, Inc.
On the Limits of Model Merging for Multilinguality in Pre-Training
May 25, 2026Seth Aycock, Fedor Vitiugin, Aleksandr Umnov +2Language Model Pretraining Multilingual Language Models
University of Amsterdam
SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models
May 25, 2026Khalid Yusuf DahirLLM Safety Benchmarks Multilingual Language Model Evaluation
Independent researcher
Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
May 24, 2026Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber +1Multilingual Language Model Evaluation MT Evaluation
TUD Dresden University of Technology · 2InfAI e.V. · 3Amazon +1
The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty
May 23, 2026Volodymyr OvcharovMultilingual Language Model Evaluation Low-Resource Language Processing
LEX AI Platform, legal.org.ua · Kyiv, Ukraine
Cultural Adaptation in Large Language Models for Political Discourse
May 22, 2026Wajdi ZaghouaniMultilingual Language Model Evaluation Political Discourse Analysis
Northwestern University in Qatar
Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs
May 22, 2026Casey Ford, Madison Van Doren, Sicheng Jin +1Multilingual Language Model Evaluation Adversarial Attacks on VLMs
Appen
DFKI-MLT at SemEval-2026 TASK 7: Steering Multilingual Models Towards Cultural Knowledge
May 21, 2026Yusser Al Ghussin, Daniil Gurgurov, Yasser Hamidullah +3Multilingual Language Models Multilingual Language Model Evaluation
German Research Center for Artificial Intelligence (DFKI GmbH) · Saarland Informatics Campus, Saarbrücken, Germany · Barcelona Supercomputing Center (BSC-CNS), Barcelona, Catalonia, Spain
Polite on the Surface, Broken in Practice: A Curated Dataset for Fixing Generation and Register Failures in Low-Resource Bangla Text Generation
May 21, 2026Md. Asaduzzaman Shuvo, Mahedi Hasan, Md. Tashin Parvez +2Multilingual Language Model Evaluation LLM Fine-Tuning
United International University, Bangladesh
Do LLMs Know What Luxembourgish Borrows? Probing Lexical Neology in Low-Resource Multilingual Models
May 20, 2026Nina Hosseini-KivananiMultilingual Language Model Evaluation Knowledge Augmentation for Language Models
University of Luxembourg, Luxembourg · Radio Télévision Luxembourg (RTL), Luxembourg
Lost in Interpretation: The Plausibility-Faithfulness Trade-off in Cross-Lingual Explanations
May 19, 2026Somnath Banerjee, Pranav Jha, Rima Hazra +1Multilingual Language Model Evaluation Faithfulness of Language Model Explanations
Indian Institute of Technology Kharagpur · TCG Crest · National University of Singapore
Prompting language influences diagnostic reasoning and accuracy of large language models
May 18, 2026Adrien Bazoge, Josselin Corvellec, Sofiane Djillali Sid-Ahmed +1Multilingual Language Model Evaluation LLM Prompting
Data Clinic, Nantes University Hospital, France
How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarking
May 18, 2026Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain +3VLM Evaluation Multilingual Language Model Evaluation
Penta Global Limited, Bangladesh · Center for Computational & Data Sciences, Independent University, Bangladesh
LLMs for automatic annotation of Mandarin narrative transcripts
May 17, 2026Qingwen Zhao, Hongao Zhu, Yunqi He +3Multilingual Language Model Evaluation LLM-Assisted Annotation
Shanghai Jiao Tong University, China · University of California San Diego, USA · The Hong Kong Polytechnic University, China +2
PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media
May 16, 2026Zoher Kachwala, Bao Tran Truong, Rasika Muralidharan +3Multilingual Language Model Evaluation Content Moderation
Observatory on Social Media, Indiana University, USA · Center Synergy of Systems, TUD Dresden University of Technology, Germany
Why Do Safety Guardrails Degrade Across Languages?
May 16, 2026Max Zhang, Ameen Patel, Sang T. Truong +1Multilingual Language Model Evaluation Language Model Safety Evaluation
Stanford University

May 28, 2026, cs.CLJ/K move · Enter open · S save

Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLAR

Debajyoti Mazumder, Divyansh Pathak, Prashant Kodali, Aditya Joshi+2

Indian Institute of Science Education and Research, Bhopal, India · Microsoft Corporation · UNSW Sydney, Australia

Multilingual Language Model Evaluation Cross-Lingual Reasoning in Language Models Factual Knowledge in Language Models Low-Resource Language Processing Cross-Lingual Consistency in Language Models

Open paper AskPDF arXiv

Large language models often exhibit a substantial gap between their performance in English and in lower-resourced languages on equivalent knowledge queries---a cross-lingual consistency issue that remains underexplored for Indian languages and their code-mixed counterparts. To study this gap, we introduce IndicKLAR, an Indic extension of the KLAR-CLC benchmark covering 18 of the 22 scheduled Indian languages. For 11 widely used language pairs, we additionally provide code-mixed variants. Both monolingual and code-mixed inputs verified by native speakers. This three-way alignment enables us to examine how knowledge recall consistency varies across English, code-mixed, and native Indian language inputs. Across nine open-weight models, we find that the accuracy gap between native-language and English inputs can reach

\sim

0.50, while code-mixed inputs substantially reduce this gap, bringing performance within

\sim

0.05 of English without any model-level intervention. Motivated by this finding, we evaluate several prompting strategies that differ in how explicitly language conversion is exposed: a two-stage translate-then-answer setup, a one-stage joint translation-and-answer prompt, and Translate-in-Thought (TinT)---a single-step strategy in which the model internally converts the input and outputs only the final answer. Across the native

\rightarrow

code-mixed

\rightarrow

English performance trajectory, we observe a consistent flip point---the transition from incorrect to correct prediction---between the native and code-mixed settings. Notably, this pattern holds both when the code-mixed representation is explicitly provided as input or when the model is prompted to convert internally using TinT.

Momentum

Related topics

Latest papers 309