Multilingual Language Models

Follow this topic Ask about this topic

Multilingual language models study how systems can understand, reason, translate, and generate text across languages, including those with limited training data or distinct scripts. Common approaches include multilingual pretraining and data balancing, continued pretraining and instruction tuning, language-aware tokenization, and transfer methods such as shared representations, adapters, and mixture-of-experts models. Researchers evaluate performance with translation, comprehension, reasoning, and safety benchmarks, while probing internal representations and using inference-time steering to improve language control. A central tension is how to expand language coverage and transfer shared knowledge without degrading performance, reinforcing English or high-resource-language biases, or overlooking differences between languages and scripts.

Latest papers

265 papers39 papers in the last four weeks, up 95% on the four weeks before. 0.2% of all new papers.

Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters
Oct 26, 2026Hafsteinn EinarssonLanguage Model Pretraining Low-Resource Language Processing
University of Iceland
Quality over Quantity: Rethinking Data and Model Assumptions for Bengali Coreference Resolution
Oct 26, 2026Zenith Biswas, Andrew Thomas DyerCoreference Resolution Low-Resource Language Processing
Data Science and Artificial Intelligence, Saarland University, Saarbrücken, Germany · Language Science and Technology, Saarland University, Saarbrücken, Germany
When the Weakest Language Keeps Moving: ARMOR-DP for Multilingual BabyLM
Oct 24, 2026Hanyu Liao, Yijing Chen, Tao JiMultilingual Language Models Adaptive Sampling
East China Normal University · Fudan University
"Not" is Not Lost in Translation: A Causal Analysis of Negation Representations in Multilingual LLMs
Oct 23, 2026Giang Son Nguyen, Wray Buntine, Dung D. Le +1Cross-Lingual Consistency in Language Models Multilingual Language Models
Nanyang Technological University · VinUniversity · Monash University
CrisiLangs: A Multilingual Resource for Crisis Warning Generation
Oct 22, 2026Giacomo Gonella, Stefano Menini, Marco GueriniMultilingual Language Models Early Warning Detection
Fondazione Bruno Kessler, Italy · University of Trento, Italy
Scaling Multilingual Pretraining of Pixel Language Models
Oct 21, 2026Ilker Kesen, Jenny Kunz, Kasra Malihi +1Multilingual Language Models Language Model Scaling
Humboldt-Universität zu Berlin · Linköping University · University of Copenhagen
Same Answer, Different Reasons: Trace-Level Divergence in Multilingual Large Language Models
Oct 21, 2026Rui Sun, Peizhao Mei, Xiewei Cheng +4Multilingual Language Model Evaluation Reasoning Consistency in Language Models
University of Illinois Urbana-Champaign · Texas A&M University
How to Use Parallel Data for Multilingual Pretraining
Oct 20, 2026Zhihong Huang, Yao Lu, Derek F. Wong +2Language Model Pretraining Multilingual Language Models
University of Macau · University College London · UNSW Sydney +1
Cross-Lingual Reusability of LLM Pipelines: Financial Extraction, Multilingual Reasoning, and Semantic Screening
Oct 19, 2026Joong Ho, Choi, Sherine George +5Cross-Lingual Transfer Cross-Lingual Consistency in Language Models
Joshua · BNY AI Hub · BNY Responsible AI Office
DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia
Oct 19, 2026Jiuzhou Han, Abhishek Bijay Mishra, Trevor Cohn +1Multilingual Language Model Evaluation Multilingual Language Models
Google Research
Where Are Language-Specific Features Represented in Multilingual LLMs? Everywhere
Oct 18, 2026Mikaela Martins, Jonathan DunnMultilingual Language Models Concept Localization in Language Models
Department of Linguistics, University of Illinois Urbana-Champaign
One Head Away from Language Confusion: Single Attention Heads Across Model Families
Oct 18, 2026Jiwon Jeong, Anjelo Jann Laroza, Chaewoon Yoon +4Multilingual Language Models Attention Head Analysis
Lotte Innovate · Mapúa University · University of Wisconsin–Madison +4
Steering beyond Language: Improving Multilingual Training-Free Text Embeddings via Paired-Residual Subspace Removal
Oct 16, 2026Ailiang Lin, Zhuoyun Li, Yusong Wang +3Text Embeddings Cross-Lingual Representation Alignment
Institute of Science Tokyo · Tencent
Unfragmenting Nawatl NMT: Single-Variant Specialization for Masewalkopa
Oct 15, 2026Luis A. Mercado-Campos, Alexis PalmerBack-Translation Machine Translation
Department of Linguistics, University of Colorado Boulder
Multi-SFT: Permissively-Licensed Multilingual Instruction-Following
Oct 15, 2026Daniela Baiamonte, Kamyar Zeinalipour, Elena Fano +2Instruction Tuning Multilingual Language Models
Yukai! · University of Siena
The Right Byte First: How UTF-16 Helps Multilingual Boundary Prediction
Oct 15, 2026Hyunsik KimTokenization in Language Models Byte-Level Language Model
Samsung Research
Translation Circuit Discovery and Targeted Optimization for Low-Resource Multilingual Translation
Oct 15, 2026Atai Waris Khan, Abdullah Mazhar, Raghu Kumar +1Multilingual Language Models Low-Resource MT
Indraprastha Institute of Information Technology Delhi
Calibration Language Shapes Block-Influence Pruning in Multilingual Language Models
Oct 15, 2026Vedant Pandhare, Piyush DhokaLLM Pruning Neural Network Pruning
Independent Researcher
Wolof-Adapted Industry-Ready Open-Weight Language Models
Oct 14, 2026Gwénolé Lecorvé, Delphine Charlet, Pierre Adam +15Language Modeling Open-Weight Language Models
Orange Research, France
Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation
Oct 14, 2026Joel Dag, Nikit Srivastava, Hamada M. Zahera +1Multilingual Language Models Mixture-of-Experts Language Models
Data Science Group, Heinz Nixdorf Institute, Paderborn University, Germany
Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training
Oct 14, 2026JiWoo Hwang, Lukas Edman, Alexander FraserLanguage Model Pretraining Multilingual Language Models
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
Qwenfant: Train Big Character-level BabyLMs on Concise Parallel Text
Oct 13, 2026Lukas Edman, Alexander FraserLanguage Modeling Language Model Pretraining
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
Probing the Geometry of Color Representations in Multilingual Language Models
Oct 13, 2026Pablo Loyola, Edison Marrese-Taylor, Andrés Hoyos-IdroboRepresentation Geometry in Language Models Multilingual Language Models
Rakuten Group, Inc. · Graduate School of Engineering, The University of Tokyo, Japan
Cross-Lingual Transfer of Evil Vectors Is Directionally Asymmetric
Oct 13, 2026Juann Kim, Minku KangCross-Lingual Transfer Activation Steering
University of Southern California · KAIST
The Multilingual Artificial Hivemind: Do All LLMs Think Alike in Every Language?
Oct 12, 2026Xaver Raphael Davey, Shubhra Mishra, Anton Adelöw +1Output Diversity in Language Models Homogenization in Language Models
KTH Royal Institute of Technology
HexaBee-4B: Six-Language Translation with Multilingual Answering and Bidirectional Supervision
Oct 12, 2026Iman Barati, Mohammadmostafa Rostamkhani, Mohammad BadzohrehMachine Translation Multilingual Language Models
Iran University of Science and Technology, Tehran, Iran · Tarbiat Modares University, Tehran, Iran
In-Place Indic Tokenizer Expansion and Continual Pretraining of Nemotron-3-Nano-30B-A3B
Oct 11, 2026Avinash Benki, Prasanjith Pasunuti, Sutanshu Saksena Raj +3Language Model Pretraining Tokenizer Adaptation
Latent Core Tokenizer: Compress, but Meaningfully
Oct 8, 2026Felermino D. M. A. Ali, Millicent Ochieng, Ogbemi Ekwejunor-Etchie +3Morphology-Aware Tokenization Tokenization in Language Models
Microsoft Research Africa · Microsoft Research Accelerator · Microsoft Research India
Language-Specific Effects of Tokenizer Choice in Multilingual Language Models
Oct 8, 2026Clara Meister, Gül Sena Altıntaş, Antoine BosselutMultilingual Language Models Multilingual Language Model Evaluation
EPFL · University of Toronto, Vector Institute
Does Modern Standard Arabic (MSA) Dominate Arabic Dialects in LLMs? A Representation-Level Analysis
Oct 8, 2026Abdu Sallouh, Nicholas Popovič, Michael FärberArabic Language Models Arabic NLP
Technical University of Dresden · Mohamed bin Zayed University of Artificial Intelligence · ScaDS.AI, TU Dresden

Oct 26, 2026J/K move · Enter open · S save

Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters

Hafsteinn Einarsson

University of Iceland

Language Model Pretraining Low-Resource Language Processing Cross-Lingual Transfer Multilingual Language Models

Open paper AskPDF arXiv

IceBERT, the reference encoder for Icelandic, found that a cleaned web crawl is enough to train a strong model and that starting from a multilingual model can help on some tasks. We revisit both findings with the ModernBERT architecture. We train Icelandic encoders from 12M to 1B parameters on 9B tokens of open data, and rerun IceBERT's benchmarks and five newer tasks for 30 encoders under one protocol with confidence intervals. Our 1B model trained from scratch, to our knowledge the first monolingual Icelandic encoder of that size, is significantly better than IceBERT on all eleven tasks; warm-started from the English Ettin encoder of the same shape, it has the best score of all encoders on every task we rank. Because our model shapes match the English Ettin encoders and the multilingual mmBERT, we can also test the warm start under controlled conditions, at six model sizes and four token budgets. With our vocabulary transplanted, a warm start was never significantly worse than scratch training on the same Icelandic text, and from base size up it was not significantly worse on four tasks than scratch training on 4 times the text (8 times at base and 1B). As models grow, most of the transfer comes from the source model's transformer layers, so even an English source helps. Keeping the source vocabulary instead cost 5 F1 on question answering and nearly three times the compute.