cs.CLApr 1, 2026

Positional Cognitive Specialization: Where Do LLMs Learn To Comprehend and Speak Your Language?

Authors: Luis Frentzen Salim, Lun-Wei Ku, Hsing-Kuo Kenneth Pao

Organizations: Institute of Information Science, Academia Sinica · Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Adapting large language models (LLMs) to new languages is an expensive and opaque process. Understanding how language models acquire new languages and multilingual abilities is key to achieve efficient adaptation. Prior work on multilingual interpretability research focuses primarily on how trained models process multilingual instructions, leaving unexplored the mechanisms through which they acquire new languages during training. We investigate these training dynamics on decoder-only transformers through the lens of two functional cognitive specializations: language perception (input comprehension) and production (output generation). Through experiments on low-resource languages, we demonstrate how perceptual and productive specialization emerges in different regions of a language model by running layer ablation sweeps from the model's input and output directions. Based on the observed specialization patterns, we propose CogSym, a layer-wise heuristic that enables effective adaptation by exclusively fine-tuning a few early and late layers. We show that tuning only the 25% outermost layers achieves downstream task performance within 2-3% deviation from the full fine-tuning baseline. CogSym yields consistent performance with adapter methods such as LoRA, showcasing generalization beyond full fine-tuning. These findings provide insights to better understand how LLMs learn new languages and push toward accessible and inclusive language modeling.

Figures & tables

Figure 1: Word-level translation task performance of each ablation sweep, namely front sweep (begs), rear sweep (ends), and both-ends or Cognitive-Symmetric (CogSym) sweep. Plots visualize emerging patterns as each region is expanded.
Figure 2: Downstream task performance of each ablation sweep
Figure 3: Comparison between each sweep strategy with regard to singular region budget kk
Figure 4: 3D view of word-translation task plot with training steps as an additional axis for German–Javanese pair
Figure 5: Word translation task performance of 4-position variant with k=8k=8
Figure 6: Word-level translation task performance comparison between CogSym (green), FIM (purple), LSN count (magenta), and AlphaLoRA (yellow)
Figure 7: Full distribution of each layer budget allocation method, higher bars indicate greater importance
Figure 8: Word-level translation task performance of LoRA, which shows consistent patterns with full finetuning
Figure 9: Word-level translation task performance comparison between CogSym (green), front-first training (dark purple), and rear-first training (pink)
More in this topic

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 265 papers

Explore similar work

CardsList
Apr 14, 2026

ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs

Despite advances in multilingual capabilities, most large language models (LLMs) remain English-centric in their training and, crucially, in the reasoning traces they produce: even for non-English problems, they predominantly reason in English, leaving non-English users with intermediate reasoning they cannot follow. We address this disparity with two contributions. (a) We introduce ReasonXL\texttt{ReasonXL}, the first large-scale parallel corpus of cross-domain reasoning traces in five European languages (English, German, French, Italian, and Spanish): over 2.5M aligned samples (11B tokens) per language, each comprising prompt, reasoning trace, and final output, enabling direct supervision of language-specific reasoning. (b) Using ReasonXL\texttt{ReasonXL}, we demonstrate that LLMs can be adapted to reason entirely in a target language through a simple two-stage pipeline of supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR), matching or exceeding the base models' English-reasoning performance in two of four languages and recovering most of it in the remaining two.

Daniil Gurgurov, Tom Röhr, Sebastian von Rohrscheidt +3

German Research Center for Artificial Intelligence (DFKI), Saarland University · Berliner Hochschule für Technik

Cross-Lingual Reasoning in Language ModelsMultilingual Language Models
AskPDF
Oct 18, 2026

Where Are Language-Specific Features Represented in Multilingual LLMs? Everywhere

Previous work has argued that multilingual LLMs encode language-specific features most strongly in their initial and final layers, implying that intermediate layers support language-agnostic representations. This paper reconsiders this finding by performing four different experiments: three that undertake a fine-grained analysis of representations of Portuguese and Spanish, two closely related languages, and one large-scale language identification experiment that covers over 200 languages. The results of these experiments challenge the strict separation between language-specific and language-agnostic layers in multilingual LLMs, using Llama 3.2 1B as a test case. By analyzing and manipulating contextual sentence embeddings, we demonstrate that language identity remains strongly recoverable throughout the model rather than being restricted to initial and final layers. Moreover, we find that the model tends to classify ambiguous strings as closer to Spanish than to Portuguese, revealing a skew in model behavior that differs from human judgments.

Mikaela Martins, Jonathan Dunn

Department of Linguistics, University of Illinois Urbana-Champaign

Multilingual Language ModelsConcept Localization in Language ModelsRepresentation Geometry in Language ModelsLanguage IdentificationLinguistic Bias in Language Models
AskPDF
May 18, 2026

A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMΔΔ Integration into Upcycled MoE

Expanding Large Language Models~(LLMs) to new languages is a costly endeavor, demanding extensive Continued Pre-Training~(CPT) and data-intensive alignment. While recent data-free merging techniques attempt to bypass alignment by fusing a multilingual CPT-enhanced model with its instruct counterpart, they are plagued by a critical trade-off: mitigating parameter conflicts to preserve original abilities inevitably dilutes new language acquisition, and vice-versa. To resolve this conflict, we introduce \method, which upcycles a dense model into a Mixture-of-Experts~(MoE) architecture, allocating different experts to different languages. Alignment ability is then transferred by grafting a MoE-expanded parameter delta~(ΔpostΔ_{\text{post}}) to the CPT-enhanced base model, bypassing the complex alignment phase. Experiments demonstrate \method's superiority even against baselines with similar FLOPs or number of parameters; it improves performance on expanded languages while effectively preserving original capabilities. We further show our approach is highly applicable across different models and Post-training deltas.

Hao Zhou, Tianhao Li, Zhijun Wang +6

National Key Laboratory for Novel Software Technology, Nanjing University · Tongyi Lab, Alibaba Group · Zhejiang University

Multilingual Language ModelsMixture-of-Experts Language ModelsLLM AlignmentLanguage Model MergingLanguage Model Post-Training
AskPDF