Multilingual Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 68% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 309

All topics

Do LLM Debates Repeat Arguments Differently Across Languages?
Jul 26, 2026Huiqian LaiMultilingual Language Model Evaluation LLM Evaluation
Syracuse University
ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluation
Jul 25, 2026Ahmed Haj Ahmed, Alvin GrissomMultilingual Language Model Evaluation Reasoning Evaluation
Haverford College
Language-Routed RAG and Direct Option Scoring for Multilingual Financial QA: DS@GT at FinMMEval
Jul 24, 2026Justice Ayela, Kabir SahniMultilingual Language Model Evaluation Retrieval-Augmented Generation
Georgia Institute of Technology, North Ave NW, Atlanta, GA 30332
When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs
Jul 23, 2026Anna Mosolova, Djamé SeddahMultilingual Language Model Evaluation LLM Evaluation
INRIA Paris
CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages
Jul 23, 2026Muhammad Dehan Al Kautsar, Salsabila Pranida, Bilal Elbouardi +1Multilingual Language Model Evaluation Multi-Turn Dialogue Evaluation
Mohamed bin Zayed University of Artificial Intelligence
From a Word-Level Dictionary to Sentence-Level Semantics: Multilingual Grievance Labelling with Contextual Models
Jul 23, 2026Lin Tian, Marian-Andrei RizoiuMultilingual Language Model Evaluation Text Classification
Behavioral Data Science University of Technology Sydney Sydney, Australia
LKValues: Aligning Large Language Models with Sri Lankan Societal Values
Jul 22, 2026Nethmi Muthugala, Supryadi, Surangika Ranathunga +7Multilingual Language Model Evaluation LLM Alignment
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, Tianjin, China · School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand · Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka +2
On the Systematic Challenges of Culturally Loaded Machine Translation: Dream of the Red Chamber as the Cultural Lens
Jul 22, 2026Yiming Wang, Jiayuan DiMultilingual Language Model Evaluation Machine Translation
School of Foreign Languages, East China University of Science and Technology
Overview of FinMMEval 2026 Task 2: Multilingual Financial Short-Answer Question Answering
Jul 22, 2026Zhuohan Xie, Xueqing Peng, Georgi Georgiev +18Multilingual Language Model Evaluation Financial QA
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · The Fin AI, United States · FMI, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria +7
Overview of FinMMEval 2026 Task 1: Multilingual Financial Multiple-Choice Question Answering
Jul 22, 2026Zhuohan Xie, Yuyang Dai, Rania Elbadry +18Multilingual Language Model Evaluation Multiple-Choice Question Answering
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria · University of Arizona, Tucson, United States +6
Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs
Jul 21, 2026Alexander ManevMultilingual Language Model Evaluation LLM Alignment
Technical University of Munich
MIRA-Ev:A Benchmark for Granular Evidence Detection and Relational Reasoning in Clinical Exams
Jul 21, 2026Iker De la Iglesia, Johanna Ramirez-Romero, Jose Maria Villa-Gonzalez +3Multilingual Language Model Evaluation Argument Mining
HiTZ Center, University of the Basque Country (EHU) · Hospital Universitario de Cruces
Building a European Multilingual Evaluation Dataset: The MMLU Localisation Project within the EMT Network
Jul 20, 2026Pilar Sánchez-Gijón, Susana Valdez, Sofía Calvo Del Barrio +3Multilingual Language Model Evaluation
Universitat Autònoma de Barcelona / Spain · Leiden University/ the Netherlands · Directorate-General for Translation of the European Commission / Belgium
Enabling Multilingual Privacy Policy Audits: Large-Scale Analysis of Spanish Mobile Apps
Jul 20, 2026Marcos Moran, David Rodriguez, Luka Nenadic +2Privacy Auditing Multilingual Language Model Evaluation
1, 2, 5Universidad Politécnica de Madrid · 2, 5Information Processing and Telecommunications Center · 3ETH Zurich +1
Safety That Does Not Transfer: Cross-Lingual Clinical Correctness Drift in Deployable Medical Language Models
Jul 19, 2026Anthonio Oladimeji Gabriel, Dimeji Olawuyi, Toba Ajayi +1Multilingual Language Model Evaluation Language Model Safety Evaluation
AI SafetyX
KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding
Jul 19, 2026Timur Turatali, Aida Turdubaeva, Rustem Izmailov +2Multilingual Language Model Evaluation LLM Reliability
1The Cramer Project · Institute of IT, Kyrgyz State Technical University named after I. Razzakov · School of Computer Science, University of Windsor +2
Similarity All The Way Up: Multilingual Generalization in LLMs Relies on Language-Level Similarity Structures
Jul 18, 2026Supantho Rakshit, Adele Goldberg, Henry ConklinMultilingual Language Model Evaluation Cross-Lingual Reasoning in Language Models
Department of Electrical and Computer Engineering Princeton University · Department of Psychology Princeton University · Laboratory for Artificial Intelligence Princeton University
JOR-Bench: Japanese Operations Research Benchmarks for Large Language Models
Jul 18, 2026Yuu JinnaiMultilingual Language Model Evaluation LLM Evaluation
CyberAgent Tokyo, Japan
Conditional Reliability of Toxicity Signals for Multilingual and Code-Mixed Abuse Detection
Jul 17, 2026Indraveni Chebolu, Rohan Singh, Arnab Mallick +1Multilingual Language Model Evaluation Hate Speech Detection
Centre for Development of Advanced Computing, Hyderabad, India
Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
Jul 16, 2026Saima Afrin, Alessandro Midolo, Camilo Escobar-Velásquez +5Multilingual Language Model Evaluation Code Generation
William & Mary, VA, USA · University of Catania, Italy · Universidad de los Andes, Colombia +2
LLM Evaluators are Biased across Languages
Jul 16, 2026Ej Zhou, Lucas Resck, Zheng Hui +1Multilingual Language Model Evaluation Language Model Safety Evaluation
Language Technology Lab, University of Cambridge
HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization
Jul 15, 2026Abdullah Shaikh, Zain Naqi, Taha Zahid +2Multilingual Language Model Evaluation Language Model Bias Evaluation
Dhanani School of Science & Engineering Habib University, Pakistan
High-Order Question Generation in a Multilingual Educational Context
Jul 15, 2026Suna-Şeyma Uçar, Itziar Aldabe, Nora Aranberri +1Multilingual Language Model Evaluation LLM Prompting
HiTZ Basque Center for Language Technology Ixa NLP Group, University of the Basque Country UPV/EHU · LT3, Language and Translation Technology Team, Ghent University, Ghent, Belgium
The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol
Jul 15, 2026Serkan BallıMultilingual Language Model Evaluation LLM-as-a-Judge
Department of Software Engineering, Mehmet Akif Ersoy University, Burdur, Türkiye · Department of Software Engineering, Mehmet Akif Ersoy University, Türkiye
Graded Entity-Familiarity Readouts in Language Models: Polish Adaptation, Cross-Language Robustness, and Refusal Steering
Jul 15, 2026Grzegorz BrzezinkaMultilingual Language Model Evaluation Language Model Introspection
Prosit AS
Where Does the Noise Come From? A Variance-Components Decomposition of Non-Determinism in LLM Brand Answers
Jul 14, 2026Dmitrij ŻatuchinMultilingual Language Model Evaluation LLM Reliability
Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia. · Rankfor.AI, Wroclaw, Poland; Tallinn, Estonia.
GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models
Jul 14, 2026Bidyarthi Paul, Nahida Jannat Mayouree, Md. Asif Karim +2Mathematical Reasoning Benchmarks Multilingual Language Model Evaluation
Computer Science and Engineering Southeast University Dhaka, Bangladesh · Computer Science and Engineering Ahsanullah University of Science and Technology Dhaka, Bangladesh
A Durability and Cross-Language Transfer Benchmark for a Validated Teaching-Feedback Classification Protocol
Jul 13, 2026Esteban U. Vega BarajasMultilingual Language Model Evaluation Sentiment Analysis
Universidad de Guadalajara
MET: Theory-Grounded and Culture-Aware Multilingual Moral Reasoning
Jul 13, 2026Ayoung Lee, Ryan Kwon, Yunxiang Zhang +3Multilingual Language Model Evaluation Moral Reasoning in Language Models
Department of Computer Science and Engineering · Department of Philosophy
Cost of Reasoning in non-English Languages: A Case Study on Japanese
Jul 11, 2026Yuu JinnaiMultilingual Language Models Multilingual Language Model Evaluation
CyberAgent / Tokyo, Japan

Jul 26, 2026, cs.CLJ/K move · Enter open · S save

Do LLM Debates Repeat Arguments Differently Across Languages?

Huiqian Lai

Syracuse University

Multilingual Language Model Evaluation LLM Evaluation Multi-Agent Debate

Open paper AskPDF arXiv

LLM debate is usually evaluated by final answers, yet transcripts reveal whether later turns develop new arguments or return to earlier claims in new wording. We study this process with \textit{prior-argument similarity}, which compares extracted argument units with earlier units in the same debate. In controlled eight-turn debates over 71 motions, six languages, and four model agents, Chinese is the only tested language with a consistently positive gap relative to English across three multilingual embedding models. The gap persists across agents, turn positions, regression adjustment, metric variants, extraction-length controls, a second-extractor subset, and cross-encoder tail rescoring. Manual calibration shows weak item-level alignment but a high-similarity tail enriched for substantive repetition. A diversity-aware prompt lowers \textit{prior-argument similarity} across languages, yet does not significantly narrow the Chinese--English gap. Multilingual debate evaluation should therefore measure argumentative development over time and report both average and gap terms.

Momentum

Related topics

Latest papers 309