Multilingual Language Model Evaluation

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 68% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 309

All topics

Generative vs. Encoder Models for Multilingual NER: A Comprehensive Empirical Study on Naamapadam
Aug 30, 2026Jakkala Mahesh, Jatavath Shravan Kumar, Komalla Shivani +1Named Entity Recognition Multilingual Language Model Evaluation
Rajiv Gandhi University of Knowledge Technologies, Basar, Telangana, India
Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation
Aug 25, 2026Xiulin Yang, Ethan Gotlieb Wilcox, Catherine ArnettMultilingual Language Model Evaluation LLM Evaluation
Georgetown University · EleutherAI
BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian
Aug 13, 2026Jophin John, Michael Hoffmann, Jan Fillies +2Multilingual Language Model Evaluation LLM Grounding
Leibniz Supercomputing Centre (LRZ), Garching, Germany · Stanford University, Stanford, California, USA · Freie Universität Berlin, Germany +2
When the API Speaks the Wrong Language: Revisiting Post-Training for Multilingual Tool Use
Aug 12, 2026Siddharth Chauhan, Thomas Butler, Abhishek Singhania +2Supervised Fine-Tuning Multilingual Language Model Evaluation
Amazon.
Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents
Aug 11, 2026Sourabrata Mukherjee, Kalika Bali, Sunayana SitaramMultilingual Language Model Evaluation Tool-Augmented Language Model Agents
Microsoft Research India
From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch
Aug 10, 2026Laurens Samson, Iva Gornishka, Gossa Lô +2Multilingual Language Model Evaluation LLM Evaluation
City of Amsterdam · Socially Intelligent Artificial Systems Group, University of Amsterdam · Fundamental AI Lab, University of Technology Nuremberg
Hidden Language Consistency Phenomena in Reasoning LLMs
Aug 9, 2026Muhammad Ali Shafique, Kelly MarchisioMultilingual Language Models Multilingual Language Model Evaluation
Kansas State University · Cohere
Wisdom in Unity: The Role of Multilingual Training in Figurative Language Identification in Proverbs
Aug 8, 2026Rama Alomair, Remas Alsubaie, Walaa Saifalislam +5Figurative Language Understanding Multilingual Language Models
College of Computer and Information Sciences, King Saud University
SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs
Aug 8, 2026Debopriyo Banerjee, Kapil Rajesh Kavitha, Angana Borah +11LLM Safety Benchmarks Multilingual Language Model Evaluation
Inception · Mohamed bin Zayed University of Artificial Intelligence · University of Michigan +4
Evaluating Dedicated Monolingual and Joint Multilingual Causal Models for Dravidian Languages
Aug 7, 2026Venkata Naga Sai Vishnu Rohit PulipakaMultilingual Language Models Multilingual Language Model Evaluation
Independent Researcher
Grammar Engineering Meets LLMs: Development of Cantonese and Irish ParGram Treebanks
Aug 7, 2026Chit-Fung Lam, Elaine Uí DhonnchadhaMultilingual Language Model Evaluation Structured Output Generation
University of Oxford · University of Manchester · Trinity College Dublin
Measuring the Cross-Lingual Comprehension Gap: How the language of the evidence shapes what language models understand
Aug 6, 2026Rafael da Silva, Jeff EicherMultilingual Language Model Evaluation Multilingual QA
Eastern University
MameLoshnLM: Yiddish Language Model and Evaluation Benchmark
Aug 6, 2026Uri Katz, Omer Goldman, Tomasz Limisiewicz +2Language Model Pretraining Multilingual Language Model Evaluation
Bar-Ilan University · University of Cambridge · University of Washington +1
NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gap
Aug 5, 2026Dasol Choi, Joonyong Park, Daegon Yu +3Multilingual Language Model Evaluation Benchmark Validity
AIM Intelligence · HAERAE LAB · KT Corp. +3
Mind the Cap: Output-Budget Regimes Change the Measured Multilingual Reasoning Gap
Aug 4, 2026Ankit Goyal, Jaideep RayMultilingual Language Model Evaluation LLM Evaluation
MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning
Aug 4, 2026Martin Böckling, Elizaveta Nosova, Heiko Paulheim +1Multilingual Language Model Evaluation Spatial Reasoning Benchmarks
Data and Web Science Group, University of Mannheim, Germany
M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models
Aug 4, 2026Tomáš Burkert, Angelika Peljak-Łapińska, David ZelenýMultilingual Language Model Evaluation Low-Resource Language Processing
RWS TrainAI
Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili
Aug 4, 2026Ruolei Zhang, Teddy Njuguna, Yue FengMultilingual Language Model Evaluation Language Model Safety Evaluation
University of Birmingham
ConlangBench: Exploring Language Knowledge and Learning in LLMs through Diverse Constructed Languages
Aug 4, 2026Jinhong Jeong, Seungyeop Yi, Sangah Lee +1Multilingual Language Model Evaluation Low-Resource Language Processing
1Yonsei University · 2Seoul National University
Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment $\unicode{x2013}$ Is English Enough?
Aug 4, 2026Adnan Al Ali, Kathy Hämmerl, Jindřich Libovický +1Multilingual Language Model Evaluation Cross-Lingual Representation Alignment
Charles University, Faculty of Mathematics and Physics, Czech Republic · Technical University of Munich, Germany · Munich Center for Machine Learning
VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP
Aug 4, 2026Tu Tran Do, Nhat Ngoc Nguyen, Khanh-Tung Tran +3Figurative Language Understanding Multilingual Language Model Evaluation
Posts and Telecommunications Institute of Technology (PTIT), Vietnam · School of Computer Science and Information Technology, University College Cork, Ireland
Semantic Alignment of AI Models: Concept Collapse, Checkpoint Dynamics, and Cross-Lingual Transfer
Aug 3, 2026Tyler Ashoff, Jordan RoduMultilingual Language Model Evaluation Representation Geometry in Language Models
Department of Statistics, University of Virginia. · Department of Statistics, University of Virginia
Same violence, different answer: how AI responds to coercive control against women across languages
Aug 2, 2026Lyu Chang, Sònia Estradé Albiol, Núria Vergés BoschMultilingual Language Model Evaluation Language Model Safety Evaluation
Department of Electronic and Biomedical Engineering, Faculty of Physics, Universitat de Barcelona, Martí i Franquès 1, 08028 Barcelona, Spain · Department of Sociology, Universitat de Barcelona, Av. Diagonal 690–696, 08034 Barcelona, Spain
Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+
Aug 2, 2026Sherzod Hakimov, Karl Osswald, Jelle Psurek +3Multilingual Language Model Evaluation LLM Evaluation
Computational Linguistics, Department of Linguistics University of Potsdam, Germany · 1German Research Center for Artificial Intelligence (DFKI), Berlin, Germany
MORFES: A Benchmark for Productive Inflectional Competence in Modern Greek
Jul 30, 2026Ioakeim Perros, Cleopatra Papadopoulou, Ayoub Kirouane +1Multilingual Language Model Evaluation LLM Evaluation
Sophea AI, KIEFER SA, Athens, Greece
Language Models are not Equally Robust to Non-Canonical Tokenization across Languages
Jul 29, 2026Poulami Ghosh, Preethi JyothiMultilingual Language Model Evaluation Language Model Robustness
*IIT Bombay, India
Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation
Jul 28, 2026Mihael ArcanIn-Context Example Selection Multilingual Language Model Evaluation
Home Lab, Galway, Ireland
Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context
Jul 28, 2026Abhishek Kumar Singh, Shrey Nag, Sachita +2LLM Safety Benchmarks Multilingual Language Model Evaluation
GTBIT¹ · TIET² · IGDTUW³ +2
Language Shapes Instruction Hierarchy Compliance in Multilingual LLMs
Jul 26, 2026Jiwon Moon, Yerin Hwang, Kyomin JungMultilingual Language Model Evaluation LLM Security
IPAI, Seoul National University · Dept. of ECE, Seoul National University
Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh
Jul 26, 2026Moniruzzaman Mahadi, Abrar Mohammed Tanzim Alam, Sayma Siddika Monalisa +3Multilingual Language Model Evaluation LLM Grounding
Department of Electrical and Computer Engineering, North South University Dhaka, Bangladesh · Department of Computer Science and Engineering, BRAC University Dhaka, Bangladesh

Aug 30, 2026, cs.CLJ/K move · Enter open · S save

Generative vs. Encoder Models for Multilingual NER: A Comprehensive Empirical Study on Naamapadam

Jakkala Mahesh, Jatavath Shravan Kumar, Komalla Shivani, Sujoy Sarkar

Rajiv Gandhi University of Knowledge Technologies, Basar, Telangana, India

Named Entity Recognition Multilingual Language Model Evaluation Decoder-Only Language Models Low-Resource Language Processing

Open paper AskPDF arXiv

Language is humanity's most consequential technology, yet for over a billion speakers across India's twenty-two constitutionally recognised languages, its digital layer remains structurally incomplete. Named Entity Recognition (NER), the foundational step in transforming raw text into machine-interpretable knowledge, has been studied exhaustively for English but remains largely unsolved across most Indic languages. This paper presents a rigorous comparative study of generative and encoder-based neural architectures for NER on all eleven languages of the Naamapadam benchmark. We evaluate five classic model families spanning sequence-to-sequence transformers and multilingual encoders; four decoder-only large language models (LLMs) fine-tuned with LoRA and 4-bit NF4 quantisation; and nine generative models in zero-to-5-shot inference. Under strict CoNLL span-level evaluation, encoder-based models (mBERT and XLM-R, both F1=0.675 on Hindi) substantially outperform every generative architecture in ten of eleven languages, with gaps of 7.5-40 percentage points against the strongest competitor (Gemma-2-2B: avg F1=0.427). The best few-shot result reaches only 28% of the encoder baseline. We identify three language clusters--encoder-dominant, partial-coverage, and failure-zone; and provide actionable deployment guidelines grounded in transfer learning and low-resource NLP principles.

Momentum

Related topics

Latest papers 309