Multilingual Language Models

Ask about this topic

Overview

Multilingual language models study how systems can understand, reason, translate, and generate text across languages, including those with limited training data or distinct scripts. Common approaches include multilingual pretraining and data balancing, continued pretraining and instruction tuning, language-aware tokenization, and transfer methods such as shared representations, adapters, and mixture-of-experts models. Researchers evaluate performance with translation, comprehension, reasoning, and safety benchmarks, while probing internal representations and using inference-time steering to improve language control. A central tension is how to expand language coverage and transfer shared knowledge without degrading performance, reinforcing English or high-resource-language biases, or overlooking differences between languages and scripts.

Latest papers

CardsList

266 papers39 papers in the last four weeks, up 95% on the four weeks before. 0.2% of all new papers.

Mar 2, 2026

EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training

Large language models (LLMs) are predominantly trained on English-centric data, resulting in uneven performance for smaller languages. We study whether continued pretraining (CPT) can improve Estonian capabilities in multilingual LLMs while preserving English and general reasoning performance. Using Llama 3.1 8B and Apertus 8B as base models, we apply CPT with Estonian-enriched multilingual replay, followed by mostly English supervised fine-tuning, preference optimization, and chat vector merging. Evaluation on Estonian benchmarks, targeted pairwise human evaluation, and an Estonian Chatbot Arena-style setup shows consistent improvements in Estonian language competence, reasoning, translation, and instruction-following. Although Apertus exhibits stronger Estonian capabilities before adaptation, the more English-centric Llama model achieves substantially larger gains after adaptation. While some English capabilities regress relative to the original instruction-tuned models, chat vector merging substantially restores English instruction-following and reasoning performance. These findings suggest that CPT with balanced multilingual replay and lightweight post-training alignment can substantially improve single-language capabilities in multilingual LLMs.

Aleksei Dorkin, Taido Purason, Emil Kalbaliyev +7

Institute of Computer Science, University of Tartu, Tartu, Estonia · Department of Software Science, Tallinn University of Technology, Tallinn, Estonia · Institute of the Estonian Language, Tallinn, Estonia +1

Multilingual Language ModelsLanguage Model PretrainingContinued PretrainingLLM Post-TrainingLow-Resource Language Processing
AskPDF
Feb 27, 2026

Universal Conceptual Structure in Neural Translation: Probing NLLB-200's Multilingual Geometry

Do neural machine translation models learn language-universal conceptual representations, or do they merely cluster languages by surface similarity? We investigate this question by probing the representation geometry of Meta's NLLB-200, a 200-language encoder-decoder Transformer, through six experiments that bridge NLP interpretability with cognitive science theories of multilingual lexical organization. Using the Swadesh core vocabulary list embedded across 135 languages, we find that the model's embedding distances significantly correlate with phylogenetic distances from the Automated Similarity Judgment Program (ρ=0.13ρ= 0.13, p=0.020p = 0.020), demonstrating that NLLB-200 has implicitly learned the genealogical structure of human languages. We show that frequently colexified concept pairs from the CLICS database exhibit significantly higher embedding similarity than non-colexified pairs (U=42656U = 42656, p=1.33×10−11p = 1.33 \times 10^{-11}, d=0.96d = 0.96), indicating that the model has internalized universal conceptual associations. Per-language mean-centering of embeddings improves the between-concept to within-concept distance ratio by a factor of 1.19, providing geometric evidence for a language-neutral conceptual store analogous to the anterior temporal lobe hub identified in bilingual neuroimaging. Semantic offset vectors between fundamental concept pairs (e.g., man to woman, big to small) show high cross-lingual consistency (mean cosine = 0.84), suggesting that second-order relational structure is preserved across typologically diverse languages. We release InterpretCognates, an open-source interactive toolkit for exploring these phenomena, alongside a fully reproducible analysis pipeline.

Kyle Elliott Mathewson

University of Alberta

Multilingual Language ModelsRepresentation Geometry in Language ModelsCross-Lingual Consistency in Language ModelsCross-Lingual Representation Alignment
AskPDF
Feb 25, 2026

Identifying Crucial Attention Heads for Multilingual Language Models: Retrieval and Retrieval-Transition Heads

Retrieval heads, a subset of attention heads in Transformers, were studied in English, showing its crucial role in retrieving information from the context. We expand the study of retrieval heads to multilingual context and find that while nearly half of all retrieval heads are often shared across multiple languages, language-specific retrieval heads also emerge. We further design a cross-lingual needle-in-the-haystack task, to identify Retrieval Transition Heads (RTH)\textit{Retrieval Transition Heads (RTH)} that retrieve key information and further map it to target-language output. Our experiments reveal that RTHs do not always overlap with RH, and can be more vital for Chain-of-Thought reasoning in multilingual LLMs. Across four multilingual benchmarks (MMLU-ProX, MGSM, MLQA, and XQuAD) and two model families (Qwen2.5-7B Instruct and Llama3.1-8B Instruct), we demonstrate that masking RTH induces bigger performance drop than masking Retrieval Heads (RH). Specifically, for Llama3.1-8B Instruct, masking the top-25 RTHs results in an average 37.3-point drop (against 19.5-point drop from RH masking) in reasoning accuracy (MMLU-ProX, MGSM) and 9.0 F1-score reduction (against 5.1 drop for RH) in extractive QA (MLQA, XQuAD). Our work advances the understanding of multilingual LMs by isolating the attention heads responsible for mapping to target languages.

Shaswat Patel, Vishvesh Trivedi, Yue Han +2

New York University

Multilingual Language ModelsCross-Lingual Reasoning in Language ModelsAttention MechanismsMulti-Head AttentionCross-Lingual TransferAttention Head Specialization
AskPDF
Feb 25, 2026

Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment

Multilingual pretraining typically lacks explicit alignment signals, leading to suboptimal cross-lingual alignment in the representation space. In this work, we show that training standard pretrained models for cross-lingual alignment with a multi-way parallel corpus in a diverse pool of languages can substantially improve multilingual and cross-lingual representations for NLU tasks. We construct a multi-way parallel dataset using translations of English text from an off-the-shelf NMT model for a pool of six target languages and achieve strong cross-lingual alignment through contrastive learning. This leads to substantial performance gains across both seen and unseen languages for multiple tasks from the MTEB benchmark evaluated for XLM-Roberta and multilingual BERT base models. Using a multi-way parallel corpus for contrastive training yields substantial gains on bitext mining (21.3%), semantic similarity (5.3%), and classification (28.4%) compared to English-centric (En-X) bilingually parallel data, where X is sampled from a pool of multiple target languages. Furthermore, finetuning mE5 model on a small dataset with multi-way parallelism significantly improves bitext mining compared to one without, underscoring the importance of multi-way cross-lingual supervision even for models already pretrained for high-quality sentence embeddings.

Barah Fazili, Koustava Goswami

IIT Bombay, Mumbai, India · Adobe Research, Bangalore, India

Cross-Lingual Representation AlignmentMultilingual Language ModelsSentence Embeddings
AskPDF
Feb 24, 2026

MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation

We introduce MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code. Through targeted adaptation, this model family achieves state-of-the-art results on Catalan- and Spanish-specific tasks, while establishing robust performance across specialized biomedical and legal domains. To bridge the gap between research and production, we incorporate Matryoshka Representation Learning (MRL), enabling flexible vector sizing that significantly reduces inference and storage costs. Ultimately, the MrBERT family demonstrates that modern encoder architectures can be optimized for both localized linguistic excellence and efficient, high-stakes domain specialization. We open source the complete model family on Huggingface.

Daniel Tamayo, Iñaki Lacunza, Paula Rivera-Hidalgo +4

Barcelona Supercomputing Center

Multilingual Language ModelsLanguage Model PretrainingDomain Adaptation for LLMsEfficient Language Model Inference
AskPDF
Feb 23, 2026

BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop

The goal of the BabyLM is to stimulate new research connections between cognitive modeling and language model pretraining. We invite contributions in this vein to the BabyLM Workshop, which will also include the 4th iteration of the BabyLM Challenge. As in previous years, the challenge features two ``standard'' tracks (Strict and Strict-Small), in which participants must train language models on under 100M or 10M words of data, respectively. This year, we move beyond our previous English-only pretraining datasets with a new Multilingual track, focusing on English, Dutch, and Chinese. For the workshop, we call for papers related to the overall theme of BabyLM, which includes training efficiency, small-scale training datasets, cognitive modeling, model evaluation, and architecture innovation.

Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul +7

IBM Research, MIT · ETH Zürich · Cornell University +7

Language Model PretrainingSmall Language ModelsMultilingual Language Models
AskPDF
Feb 23, 2026

Multilingual Large Language Models do not comprehend all natural languages to equal degrees

Large Language Models (LLMs) play a critical role in how humans access information. While their core use relies on comprehending written requests, our understanding of this ability is currently limited, because most benchmarks evaluate LLMs in high-resource languages predominantly spoken by Western, Educated, Industrialised, Rich, and Democratic (WEIRD) communities. The default assumption is that English is the best-performing language for LLMs, while smaller, low-resource languages are linked to less reliable outputs, even in multilingual, state-of-the-art models. To track variation in the comprehension abilities of LLMs, we prompt 3 popular models on a language comprehension task across 12 languages, representing the Indo-European, Afro-Asiatic, Turkic, Sino-Tibetan, and Japonic language families. Our results suggest that the models exhibit remarkable linguistic accuracy across typologically diverse languages, yet they fall behind human baselines in all of them, albeit to different degrees. Contrary to what was expected, English is not the best-performing language, as it was systematically outperformed by several Romance languages, even lower-resource ones. We frame the results by discussing the role of several factors that drive LLM performance, such as tokenization, language distance from Spanish and English, size of training data, and data origin in high- vs. low-resource languages and WEIRD vs. non-WEIRD communities.

Natalia Moskvina, Raquel Montero, Masaya Yoshida +8

Universitat Autònoma de Barcelona · Institució Catalana de Recerca i Estudis Avançats (ICREA) · Radboud University Nijmegen +1

Multilingual Language ModelsMultilingual Language Model Evaluation
AskPDF
Feb 18, 2026

Aladdin-FTI @ AMIYA Three Wishes for Arabic NLP: Fidelity, Diglossia, and Multidialectal Generation

Arabic dialects have long been under-represented in Natural Language Processing (NLP) research due to their non-standardization and high variability, which pose challenges for computational modeling. Recent advances in the field, such as Large Language Models (LLMs), offer promising avenues to address this gap by enabling Arabic to be modeled as a pluricentric language rather than a monolithic system. This paper presents Aladdin-FTI, our submission to the AMIYA shared task. The proposed system is designed to both generate and translate dialectal Arabic (DA). Specifically, the model supports text generation in Moroccan, Egyptian, Palestinian, Syrian, and Saudi dialects, as well as bidirectional translation between these dialects, Modern Standard Arabic (MSA), and English. The code and trained model are publicly available.

Jonathan Mutal, Perla Al Almaoui, Simon Hengchen +1

Faculté de traduction et d’interprétation, Université de Genève

Arabic NLPArabic DialectsArabic Language ModelsMachine TranslationMultilingual Language ModelsLow-Resource Language Processing
AskPDF
Feb 17, 2026

Adaptive Engram Memory System for Indonesian Language Model: Generative AI Based on TOBA LM for Batak and Minang Language

This study presents TOBA-LM, a trilingual language model based on GPT-2 architecture with 1.2 billion parameters, trained on a corpus encompassing Indonesian, Batak, and Minangkabau using syllabic-agglutinative tokenization. The architecture integrates an Engram Memory mechanism, an adaptive n-gram-based memory system with a 500,000 x 768 embedding table that captures morphological dependencies through bigram and trigram pathways. Empirical results demonstrate a training efficiency of 80%, with the loss value dropping from 6.4 to 1.7996 in only 12,973 steps -- significantly faster than the conventional transformer architecture, which required over 70,000 steps to achieve comparable convergence. These findings confirm that the integration of external statistical memory substantially reduces computational requirements for developing regional language models under limited resources.

Hokky Situngkir, Kevin Siringoringo, Andhika Bernard Lumbantobing

AI Research Center IT Del & Bandung Fe Institute · Bandung Fe Institute · Bandung Fe Institute & Adjunct Science Fellow in InaAI

Language Model PretrainingMultilingual Language ModelsMemory-Augmented Language ModelsLow-Resource Language Processing
AskPDF
Feb 16, 2026

ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset

Multilinguality is a core capability for modern foundation models, yet training high-quality multilingual models remains challenging due to uneven data availability across languages. A further challenge is the performance interference that can arise from joint multilingual training, commonly referred to as the "curse of multilinguality". We study multilingual data curation across thirteen languages and find that many reported regressions are not inherent to multilingual scaling but instead stem from correctable deficiencies in data quality and composition rather than fundamental capacity limits. In controlled bilingual experiments, improving data quality for any single language benefits others: curating English improves non-English performance in 12 of 13 languages, while curating non-English yields reciprocal improvements in English. Bespoke per-language curation produces substantially larger within-language improvements. Extending these findings to large-scale general-purpose training mixtures, we show that curated multilingual allocations comprising under 8% of total tokens remain remarkably effective. We operationalize this approach within an effort that produced a 20T-token pretraining corpus derived entirely from public sources. Models with 3B and 8B parameters trained on a 1T-token random subset achieve competitive multilingual accuracy with 4-10x fewer training FLOPs than strong public baselines, establishing a new Pareto frontier in multilingual performance versus compute. Moreover, these benefits extend to frontier model scale: the 20T-token corpus served as part of the pretraining dataset for Trinity Large (400B/A13B), which exhibits strong multilingual performance relative to its training FLOPs. These results show that targeted, per-language data curation mitigates multilingual interference and enables compute-efficient multilingual scaling.

DatologyAI, :, Aldo Gael Carranza +32

DatologyAI

Multilingual Language ModelsTraining Data CurationLanguage Model PretrainingEfficient Language Model Training
AskPDF
Feb 16, 2026

Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography

We present a crowdsourced dataset for Piedmontese, an endangered Romance language of northwestern Italy. The dataset comprises 145 Italian-Piedmontese parallel sentences derived from Flores+, with translations produced by speakers writing in their natural orthographic style rather than adhering to standardized conventions, along with manual word alignment. We use this resource to benchmark several large language models on tokenization parity, topic classification, and machine translation. Our analysis reveals that Piedmontese incurs a tokenization penalty relative to higher-resource Romance languages, yet LLMs achieve classification performance approaching that of Italian, French, and English. Machine translation results are asymmetric: models translate adequately from Piedmontese into high-resource languages, but generation into Piedmontese remains challenging. The dataset and code are publicly released.

Gianluca Vico, Jindřich Libovický

Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Malostranské náměstí 25, 118 00 Praha, Czech Republic

Tokenization in Language ModelsLow-Resource Language ProcessingLow-Resource MTMultilingual Language Model EvaluationMultilingual Language Models
AskPDF
Feb 2, 2026

Language Steering for Multilingual In-Context Learning

If multilingual language model representations contain separable language-associated structure, then shifting between languages may require only simple activation-space interventions. To test this hypothesis, we study multilingual in-context learning (ICL) as a controlled setting: few-shot demonstrations are provided in English while the test query is in a target language, creating a clear cross-lingual transfer challenge. We propose language vectors, which are computed as mean activation differences between parallel source and target language examples at a particular layer and applied as inference-time offsets to the model's hidden states. Across three tasks spanning 19 languages and three models, our method improves over English few-shot prompting without any parameter updates. Beyond performance, the vectors encode interpretable linguistic structure: closely related languages form tight clusters, and vectors transfer effectively across tasks and languages, with closely related languages' vectors recovering more of the gain than distant ones. Together, these findings suggest that language occupies structured directions in multilingual activation space, and that inference-time interventions on English demonstrations can narrow the cross-lingual ICL gap.

Neeraja Kirtane, Kuan-Hao Huang

Texas A&M University

Multilingual Language ModelsCross-Lingual TransferIn-Context LearningActivation SteeringRepresentation Geometry in Language Models
AskPDF
Jan 23, 2026

Cross-Lingual Activation Steering for Multilingual Language Models

Large language models exhibit strong multilingual capabilities, yet significant performance gaps persist between dominant and non-dominant languages. Prior work attributes this gap to imbalances between shared and language-specific neurons in multilingual representations. We propose Cross-Lingual Activation Steering (CLAS), a training-free inference-time intervention that selectively modulates neuron activations. We evaluate CLAS on classification and generation benchmarks, achieving average improvements of 2.3% (Acc.) and 3.4% (F1) respectively, while maintaining high-resource language performance. We discover that effective transfer operates through functional divergence rather than strict alignment; performance gains correlate with increased language cluster separation. Our results demonstrate that targeted activation steering can unlock latent multilingual capacity in existing models without modification to model weights.

Rhitabrat Pokharel, Ameeta Agrawal, Tanay Nagar

Department of Computer Science, Portland State University, USA · Independent Researcher

Multilingual Language ModelsLanguage Model SteeringCausal Interventions in Language ModelsCross-Lingual Transfer
AskPDF
Jan 15, 2026

HOMURA: Taming the Sand-Glass for Time-Constrained LLM Translation via Reinforcement Learning

Large Language Models (LLMs) have achieved remarkable strides in multilingual translation but are hindered by a systemic cross-lingual verbosity bias, rendering them unsuitable for strict time-constrained tasks like subtitling and dubbing. Current prompt-engineering approaches struggle to resolve this conflict between semantic fidelity and rigid temporal feasibility. To bridge this gap, we first introduce Sand-Glass, a benchmark specifically designed to evaluate translation under syllable-level duration constraints. Furthermore, we propose Homura, a reinforcement learning framework that explicitly optimizes the trade-off between semantic preservation and temporal compliance. By employing a constrained reinforcement learning objective featuring a novel dynamic syllable-ratio reward, Homura effectively "tames" the output length. Experimental results demonstrate that Homura significantly outperforms strong baselines, achieving precise length control that respects linguistic density hierarchies without compromising semantic adequacy.

Ziang Cui, Mengran Yu, Chenyu Shi +2

Bilibili Inc., Shanghai, China

Multilingual Language ModelsRL for Language ModelsMachine TranslationMT Evaluation
AskPDF
Jan 10, 2026

AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages

Large language models (LLMs) are increasingly multilingual, yet open models continue to underperform relative to proprietary systems, with the gap most pronounced for African languages. Continued pre-training (CPT) offers a practical route to language adaptation, but improvements on demanding capabilities such as mathematical reasoning often remain limited. This limitation is driven in part by the uneven domain coverage and missing task-relevant knowledge that characterize many low-resource language corpora. We present AfriqueLLM, a suite of open LLMs adapted to 20 African languages through CPT on 26B tokens. We perform a comprehensive empirical study across five base models spanning sizes and architectures, including Llama 3.1, Gemma 3, and Qwen 3, and systematically analyze how CPT data composition shapes downstream performance. In particular, we vary mixtures that include math, code, and synthetic translated data, and evaluate the resulting models on a range of multilingual benchmarks. Our results identify data composition as the primary driver of CPT gains. Adding math, code, and synthetic translated data yields consistent improvements, including on reasoning-oriented evaluations. Within a fixed architecture, larger models typically improve performance, but architectural choices dominate scale when comparing across model families. Moreover, strong multilingual performance in the base model does not reliably predict post-CPT outcomes; robust architectures coupled with task-aligned data provide a more dependable recipe. Finally, our best models improve long-context performance, including document-level translation. Models and code have been released on Huggingface and Github.

Hao Yu, Tianyi Xu, Michael A. Hedderich +3

McGill University, Canada · Mila-Quebec AI Institute, Canada · LMU Munich & Munich Center for Machine Learning, Germany +2

Language Model PretrainingMultilingual Language ModelsContinued PretrainingLow-Resource Language ProcessingAfrican Language Processing
AskPDF
Jan 9, 2026

What Matters When Building Universal Multilingual Named Entity Recognition Models?

Recent progress in universal multilingual named entity recognition (NER) has been driven by multilingual transformer models, task-specific architectures, custom loss functions, and large-scale training datasets. However, despite substantial prior work, we find that many design decisions for such models are made without systematic justification, with individual components evaluated only in combination rather than in isolation. We argue that this impedes progress in the field by making it difficult to identify which choices improve multilingual generalization. In this work, we conduct an extensive empirical evaluation on transformer backbones, architectures, training objectives, data composition, and threshold selection for zero-shot, multilingual NER models. Building on these findings, we present Otter, a small encoder model that achieves consistent improvements over strong multilingual NER baselines, outperforming similarly sized models by 5.7 percentage points in F1. Further, it remains within 1.4 points of Qwen3-32B and 5.4 points below Gemma3-27B, while being roughly 90x smaller and one to two orders of magnitude faster at inference.

Jonas Golde, Patrick Haller, Alan Akbik

Humboldt Universität zu Berlin

Named Entity RecognitionMultilingual Language ModelsMultilingual Language Model EvaluationCross-Lingual Transfer
AskPDF
Dec 5, 2025

LMSpell: Spell Correction with Pre-Trained Language Models

Spell correction is still a challenging problem for many languages, especially low-resource languages (LRLs). While pre-trained language models (PLMs) have been employed for spell correction, there has been no proper comparison across PLMs. We present the first empirical study on the effectiveness of the three types of PLMs for spell correction across multiple languages, including low-resource languages. We show that even relatively small PLMs such as the 270M-parameter Gemma 3 and mBART50, when fine-tuned on a dataset of only 5k sentences, can outperform rule-based spell correctors, highlighting a practical pathway for building effective spell correction systems with limited data. We also present a case study with Sinhala to shed light on the plight of spell correction for LRLs.

Akesh Gunathilake, Nadil Karunarathna, Tharusha Bandaranayake +3

Department of Computer Science and Engineering, University of Moratuwa, Katubedda, 10400, Sri Lanka · School of Mathematical and Computational Sciences, Massey University, Auckland, 102904, New Zealand

Low-Resource Language ProcessingLanguage Model Post-TrainingMultilingual Language Models
AskPDF
Nov 4, 2025

AyurParam: A State-of-the-Art Bilingual Language Model for Ayurveda

Current large language models excel at broad, general-purpose tasks, but consistently underperform when exposed to highly specialized domains that require deep cultural, linguistic, and subject-matter expertise. In particular, traditional medical systems such as Ayurveda embody centuries of nuanced textual and clinical knowledge that mainstream LLMs fail to accurately interpret or apply. We introduce AyurParam-2.9B, a domain-specialized, bilingual language model fine-tuned from Param-1-2.9B using an extensive, expertly curated Ayurveda dataset spanning classical texts and clinical guidance. AyurParam's dataset incorporates context-aware, reasoning, and objective-style Q&A in both English and Hindi, with rigorous annotation protocols for factual precision and instructional clarity. Benchmarked on BhashaBench-Ayur, AyurParam not only surpasses all open-source instruction-tuned models in its size class (1.5--3B parameters), but also demonstrates competitive or superior performance compared to much larger models. The results from AyurParam highlight the necessity for authentic domain adaptation and high-quality supervision in delivering reliable, culturally congruent AI for specialized medical knowledge.

Sravan Gorugantu, Mohd Nauman, Vijay Devane +7

BharatGen Team

Medical LLMsHealthcareDomain Adaptation for LLMsLLM Fine-TuningMultilingual Language Models
AskPDF
Oct 11, 2025

Language steering in latent space to mitigate unintended code-switching

Multilingual Large Language Models (LLMs) often exhibit hallucinations such as unintended code-switching, reducing reliability in downstream tasks. We propose latent-space language steering, a lightweight inference-time method that identifies language directions via Principal Component Analysis (PCA) on parallel translations and steers token embeddings along these axes to control language identity. Our approach mitigates code-switching (reduced next-token distributional divergence by up to 55%, reduced Code-Switching Index by 63--99%) while preserving broad semantic direction with negligible computational overhead and requires only minimal parallel data for calibration. Additionally, we evaluate language classification using a single principal component (96-100% accuracy) and further analyze the layer-wise evolution of language representations.

Andrey Goncharov, Nikolai Kondusov, Alexey Zaytsev

Language Model SteeringMultilingual Language ModelsCode-SwitchingRepresentation Steering
AskPDF
Oct 3, 2025

Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models

Confidence calibration, the alignment of a model's predicted confidence with its actual accuracy, is crucial for the reliable deployment of Large Language Models (LLMs). However, this critical property remains largely under-explored in multilingual contexts. In this work, we conduct the first large-scale, systematic studies of multilingual calibration across six model families and over 100 languages, revealing that non-English languages suffer from systematically worse calibration. To diagnose this, we investigate the model's internal representations and find that the final layer, biased by English-centric training, provides a poor signal for multilingual confidence. In contrast, our layer-wise analysis uncovers a key insight that late-intermediate layers consistently offer a more reliable and better-calibrated signal. Building on this, we introduce a suite of training-free methods, including Language-Aware Confidence Ensemble (LACE), which adaptively selects an optimal ensemble of layers for each specific language. Our study highlights the hidden costs of English-centric alignment and offer a new path toward building more globally equitable and trustworthy LLMs by looking beyond the final layer.

Ej Zhou, Caiqi Zhang, Tiancheng Hu +4

Language Technology Lab, University of Cambridge

Multilingual Language ModelsMultilingual Language Model EvaluationConfidence Estimation in Language ModelsLanguage Model Calibration
AskPDF
Aug 12, 2025

SinLlama -- A Large Language Model for Sinhala

Low-resource languages such as Sinhala are often overlooked by open-source Large Language Models (LLMs). In this research, we extend an existing multilingual LLM (Llama-3-8B) to better serve Sinhala. We enhance the LLM tokenizer with Sinhala specific vocabulary and perform continual pre-training on a cleaned 10 million Sinhala corpus, resulting in the SinLlama model. This is the very first decoder-based open-source LLM with explicit Sinhala support. When SinLlama was instruction fine-tuned for three text classification tasks, it outperformed base and instruct variants of Llama-3-8B by a significant margin.

H. W. K. Aravinda, Rashad Sirajudeen, Samith Karunathilake +3

Dept. of Computer Science & Engineering, University of Moratuwa, Sri Lanka. · School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand. · Central Scientific Instruments Organisation, Academy of Scientific and Innovative Research, Chandigarh, India. +1

Language Model PretrainingMultilingual Language ModelsLanguage ModelingTokenizer Adaptation
AskPDF
Jul 15, 2025

Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages

Understanding the multilingual mechanisms of large language models (LLMs) provides insight into how they process different languages, yet this remains challenging. Existing studies often focus on individual neurons, but their polysemantic nature makes it difficult to isolate language-specific units from cross-lingual representations. To address this, we explore sparse autoencoders (SAEs) for their ability to learn monosemantic features that represent concrete and abstract concepts across languages in LLMs. While some of these features are language-independent, the presence of language-specific features remains underexplored. In this work, we introduce SAE-LAPE\textit{SAE-LAPE}, a method based on feature activation probability, to identify language-specific features within the feed-forward network. We find that many such features predominantly appear in the middle to late layers of the model and are interpretable. These features influence the model's multilingual performance and language output, and can be used for language identification with performance comparable to fastText, along with more interpretability. Our code and complete figures are available at https://github.com/LyzanderAndrylie/language-specific-features.

Lyzander Marciano Andrylie, Inaya Rahmanisa, Mahardika Krisna Ihsani +3

Faculty of Computer Science, Universitas Indonesia · Department of Natural Language Processing, MBZUAI

Multilingual Language ModelsRepresentation LearningLanguage IdentificationLLM InterpretabilitySparse Autoencoders
AskPDF
Jul 13, 2025

NMIXX: Domain-Adapted Neural Embeddings for Cross-Lingual eXploration of Finance

Financial text embeddings must distinguish changes in event status, perspective, and obligations even when passages share similar wording. NMIXX adapts existing encoders through 18.8k source-linked triplets: paraphrases and Korean-English translations preserve meaning, while targeted financial rewrites introduce semantic contrasts. We examine this recipe across seven backbones on English and Korean financial and general-domain semantic textual similarity (STS), and analyze the composition and passage lengths of KorFinSTS. BGE-M3 attains the highest adapted financial correlations in this comparison, improving from 0.1969 to 0.2967 on FinSTS and from 0.0512 to 0.2732 on KorFinSTS. Its general English and Korean correlations decrease by 0.0391 and 0.0463. Across the seven models, five improve their mean financial correlation, but all reduce their mean general-domain correlation. Per-language comparisons and benchmark-weight sensitivity analysis reveal differences obscured by a single aggregate score. The study contributes a finance-specific supervision design and evidence for evaluating adaptation jointly with retained general semantic capability; direct cross-language retrieval remains outside its evaluation scope.

Hanwool Lee, Sara Yu, Yewon Hwang +4

FinancialNLPLab, MODULABS Shinhan Securities Seoul, Republic of Korea · FinancialNLPLab, MODULABS KT Seoul, Republic of Korea · FinancialNLPLab, MODULABS EMRO Seoul, Republic of Korea +4

Multilingual Language ModelsText EmbeddingsSemantic Textual SimilarityDomain AdaptationText Embedding ModelsFinance
AskPDF
May 21, 2025

Language Specific Knowledge: Do Models Know Better in X than in English?

Often, multilingual language models are trained with the objective to map semantically similar content (in different languages) in the same latent space. In this paper, we show a nuance in this training objective, and find that by changing the language of the input query, we can improve the question answering ability of language models. We make two main contributions. First, we introduce the term Language Specific Knowledge (LSK) to denote queries that are best answered in an ``expert language'' for a given LLM, thereby enhancing its question-answering ability. We introduce the problem of language selection -- for some queries, language models can perform better when queried in languages other than English, sometimes even better in low-resource languages -- and the goal is to select the optimal language for the query. Second, we introduce a variety of simple to strong baselines to empirically motivate the language selection problem (including one of our own methods called LSKExtractor). During our evaluation, we employ three datasets that contain knowledge about both cultural and social behavioral norms. Overall, the results show that principled language selection can improve the performance of a language model, and that the expected question-to-language map is not always intuitive: Gemma models know most about China and Middle East in Spanish; Qwen models know most about authority and responsibility in Arabic and Chinese. Broadly, our research contributes to the open-source development of language models that are inclusive and more aligned with the cultural and linguistic contexts in which they are deployed.

Ishika Agarwal, Nimet Beyza Bozdag, Dilek Hakkani-Tür

Department of Computer Science University of Illinois, Urbana-Champaign

Multilingual Language ModelsMultilingual QAFactual Knowledge in Language ModelsCultural Knowledge in Language Models
AskPDF
Mar 3, 2025

In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models

Instruction following is a critical ability for Large Language Models to be used directly by humans. This often requires supervised fine-tuning on curated instruction datasets, sometimes complemented with an alignment step. However, in multilingual scenarios, obtaining high-quality data for these stages remains challenging, motivating the exploration of In-Context Learning (ICL) as a possible alternative. In this work, we study whether ICL can serve as a substitute for Instruction Tuning in multilingual language models, while also examining how the comparison changes with model scale. Our results indicate that a gap remains between ICL and Instruction Tuning, motivating further research to reduce it.

David Ponce, Thierry Etchegoyhen

Vicomtech Foundation, Basque Research and Technology Alliance (BRTA) · University of the Basque Country - EHU

Multilingual Language ModelsMultilingual Language Model EvaluationIn-Context LearningInstruction Tuning
AskPDF
May 22, 2024

Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian

There has been a surge in the development of various Large Language Models (LLMs). However, text generation for languages other than English often faces significant challenges, including poor generation quality and reduced computational performance due to the disproportionate representation of tokens in the model's vocabulary. In this work, we address these issues by developing a pipeline for the adaptation of English-oriented pre-trained models to other languages and constructing efficient bilingual LLMs. Using this pipeline, we construct Vikhr, a series of bilingual open-source instruction-following LLMs designed specifically for the Russian language. Vikhr'' refers to the name of the Mistral LLM series and means a strong gust of wind.'' Unlike previous Russian-language models that typically rely on LoRA adapters on top of English-oriented models, sacrificing performance for lower training costs, Vikhr features an adapted tokenizer vocabulary and undergoes the continued pre-training and instruction tuning of all weights. This not only enhances the model's performance but also significantly improves its computational and contextual efficiency. We also expanded the instruction datasets and corpora for continued pre-training. The model weights, instruction sets, and code are publicly available.

Aleksandr Nikolich, Konstantin Korolev, Sergei Bratchikov +2

ITMO University · HSE University · Vikhr +2

Language Model PretrainingMultilingual Language ModelsLLM Fine-TuningLanguage ModelingInstruction TuningTokenizer Adaptation
AskPDF