Multilingual Language Models

Follow this topic Ask about this topic

Multilingual language models study how systems can understand, reason, translate, and generate text across languages, including those with limited training data or distinct scripts. Common approaches include multilingual pretraining and data balancing, continued pretraining and instruction tuning, language-aware tokenization, and transfer methods such as shared representations, adapters, and mixture-of-experts models. Researchers evaluate performance with translation, comprehension, reasoning, and safety benchmarks, while probing internal representations and using inference-time steering to improve language control. A central tension is how to expand language coverage and transfer shared knowledge without degrading performance, reinforcing English or high-resource-language biases, or overlooking differences between languages and scripts.

Latest papers

266 papers39 papers in the last four weeks, up 95% on the four weeks before. 0.2% of all new papers.

On the Limits of Model Merging for Multilinguality in Pre-Training
May 25, 2026Seth Aycock, Fedor Vitiugin, Aleksandr Umnov +2Language Model Pretraining Multilingual Language Models
University of Amsterdam
Learning to Route Languages for Multilingual Policy Optimization
May 25, 2026Geyang Guo, Hiromi Wakaki, Yuki Mitsufuji +2Multilingual Language Models RL for Language Models
Georgia Institute of Technology · Sony Group Corporation · Sony AI
Mimir: Large-scale Multilingual Concept Modeling
May 24, 2026Elio Musacchio, Lucia Siciliani, Pierpaolo BasileLanguage Model Pretraining Multilingual Language Models
Department of Computer Science University of Bari Aldo Moro Bari, Italy
Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
May 23, 2026Bo Li, Tianyu Dong, Shaolin Zhu +1Multilingual Language Models Mixture-of-Experts Language Models
School of Software, Tsinghua University · College of Intelligence and Computing, Tianjin University
HiMed: Incentivizing Hindi Reasoning in Medical LLMs
May 23, 2026Dingfeng Jiang, Han Yan, Chenze Ma +12Multilingual Language Models Healthcare
1The Chinese University of Hong Kong, Shenzhen · 2Indian Institute of Technology (Banaras Hindu University) Varanasi · 3Tongji University +4
Multilingual Knowledge Transfer under Data Constraints via Lexical Interventions
May 22, 2026Anastasiia Sedova, Natalie Schluter, Skyler Seto +1Language Model Pretraining Multilingual Language Models
1Apple
DFKI-MLT at SemEval-2026 TASK 7: Steering Multilingual Models Towards Cultural Knowledge
May 21, 2026Yusser Al Ghussin, Daniil Gurgurov, Yasser Hamidullah +3Multilingual Language Models Multilingual Language Model Evaluation
German Research Center for Artificial Intelligence (DFKI GmbH) · Saarland Informatics Campus, Saarbrücken, Germany · Barcelona Supercomputing Center (BSC-CNS), Barcelona, Catalonia, Spain
Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection
May 21, 2026Yusser Al Ghussin, Daniil Gurgurov, Tanja Baeumel +3Multilingual Language Models Language Model Steering
Saarland University · German Research Center for Artificial Intelligence (DFKI) · Centre for European Research in Trusted AI (CERTAIN) +1
Brain-LLM Alignment Tracks Training Data, Not Typology
May 21, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuMultilingual Language Models Cross-Lingual Representation Alignment
The University of Hong Kong Hong Kong, China · Stellaris AI Limited Hong Kong, China
LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance
May 21, 2026Yuchun Fan, Bei Li, Peiguang Li +9Multilingual Language Models Reinforcement Learning
NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · Meituan Inc. 3NiuTrans Research, Shenyang, China
Hy-MT2: A Family of Fast, Efficient and Powerful Multilingual Translation Models in the Wild
May 21, 2026Mao Zheng, Zheng Li, Tao Chen +10Multilingual Language Models Machine Translation
Tencent Hunyuan Team
Language models struggle with compartmentalization
May 19, 2026Thomas Vincent Howe, David WingateMultilingual Language Models LLM Interpretability
Department of Computer Science · Brigham Young University · Provo, UT 84602
A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM $Δ$ Integration into Upcycled MoE
May 18, 2026Hao Zhou, Tianhao Li, Zhijun Wang +6Multilingual Language Models Mixture-of-Experts Language Models
National Key Laboratory for Novel Software Technology, Nanjing University · Tongyi Lab, Alibaba Group · Zhejiang University
Closing the Gap at CRAC 2026: Two-Stage Adaptation for LLM-Based Multilingual Coreference Resolution
May 16, 2026Antoine Bourgois, Olga Seminck, Thierry PoibeauMultilingual Language Models Coreference Resolution
Lattice (CNRS UMR 8094 & ENS-PSL & Universit´e Sorbonne Nouvelle), Montrouge, France
Toward LLMs Beyond English-Centric Development
May 15, 2026Sho Takase, Ukyo HondaMultilingual Language Models Continual Learning for LLMs
CyberAgent
ML-Embed: Inclusive and Efficient Embeddings for a Multilingual World
May 14, 2026Ziyin Zhang, Zihan Liao, Hang Yu +2Multilingual Language Models Text Embeddings
School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Ant Group, Hangzhou, China.
Merging Methods for Multilingual Knowledge Editing for Large Language Models: An Empirical Odyssey
May 13, 2026Kunil Lee, Ki-Young Shin, Jong-Hyeok Lee +1Knowledge Editing Multilingual Language Models
Department of Computer Science and Engineering, POSTECH · Designovel Co., Ltd. · LLSOLLU +1
TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
May 13, 2026Chong Li, Yingzhuo Deng, Wen Yang +2Multilingual Language Models Tokenizer Adaptation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
IndicMedDialog: A Parallel Multi-Turn Medical Dialogue Dataset for Accessible Healthcare in Indic Languages
May 13, 2026Shubham Kumar Nigam, Suparnojit Sarkar, Piyush PatelMultilingual Language Models Healthcare
University of Birmingham, Dubai, United Arab Emirates · Heritage Institute of Technology, Kolkata, India · Madan Mohan Malaviya University of Technology, India
Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings
May 13, 2026Paul Jeha, Anastasiia Sedova, Louis Béthune +4Language Model Pretraining Multilingual Language Models
1Apple · 2DTU
DiM\textsuperscript{3}: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging
May 13, 2026Zijing Wang, Mingyang Wang, Ercong Nie +6Multilingual Language Models Multimodal Large Language Models
Northeastern University, China · CIS, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Germany +1
Macro: Enhancing Multilingual Counterfactual Explanations through Alignment-as-Preference Optimization
May 12, 2026Yilong Wang, Qianli Wang, Bohao Chu +3Multilingual Language Models Counterfactual Explanations
1Technische Universität Berlin · 2German Research Center for Artificial Intelligence (DFKI) · University of Duisburg-Essen +5
Phoenix-VL 1.5 Medium Technical Report
May 11, 2026Team Phoenix, :, Arka Ray +30Multilingual Language Models Domain Adaptation
HTX × Mistral AI ∗
A multilingual hallucination benchmark: MultiWikiQHalluA
May 4, 2026Freja Thoresen, Dan Saattrup SmartMultilingual Language Models Multilingual Language Model Evaluation
Alexandra Institute · Rued Langgaards Vej 7, 2300 København
A Systematic Benchmark of Machine Transliteration Models for the Tajik-Farsi Language Pair: A Comparative Study from Rule-Based to Transformer Architectures
May 4, 2026Mullosharaf K. ArabovMultilingual Language Models Transformer
Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia
Multilingual Safety Alignment via Self-Distillation
May 3, 2026Ruiyang Qin, Qingzhuo Wang, Dongrui Liu +3Multilingual Language Models LLM Safety Alignment
Tongji University · Shanghai AI Laboratory
Attention Sinks in Massively Multilingual Neural Machine Translation:Discovery, Analysis, and Mitigation
May 2, 2026Hillary Mutisya, John MuganeMultilingual Language Models Cross-Attention
Thiomi NLP · Harvard University
Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMs
May 2, 2026Anjishnu Mukherjee, Chutong Meng, Antonios AnastasopoulosMultilingual Language Models Multilingual Language Model Evaluation
Department of Computer Science George Mason University
BabelSafe: A Policy-Grounded Multilingual Safety Benchmark for LLMs
May 1, 2026Yunhan Zhao, Zhaorun Chen, Xingjun Ma +1LLM Safety Benchmarks Multilingual Language Models
University of Illinois Urbana-Champaign · Fudan University · University of Chicago
The Digital Afterlife of Empires: Four Language Models Converge on the Same Imperial Cartography of Writing
Apr 30, 2026Hiroki FukuiMultilingual Language Models Cultural Bias in Language Models
Research Institute of Criminal Psychiatry / Sex Offender Medical Center · Department of Neuropsychiatry, Kyoto University

May 25, 2026J/K move · Enter open · S save

On the Limits of Model Merging for Multilinguality in Pre-Training

Seth Aycock, Fedor Vitiugin, Aleksandr Umnov, Christof Monz+1

University of Amsterdam

Language Model Pretraining Multilingual Language Models Multilingual Language Model Evaluation Language Model Merging Model Merging

Open paper AskPDF arXiv

Endowing models with consistent multilingual performance can be achieved by mixing pre-training data, or post-training approaches such as language-specific model merging. In this work, we test whether merging can be applied to monolingually pre-trained models. We conduct a controlled study on the efficacy of mixed, merged, and monolingual pre-training setups. We find that while monolingual pre-training results in strong in-language performance, merging any combination of monolingual models leads to performance collapse due to interference. Our analysis suggests representational similarity is a prerequisite for model merging. We therefore conclude that the flexibility of merging in fine-tuning does not extend trivially to language-specific pre-training.