cs.CLAug 31, 2026

Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs

Authors: Deniz Bayazit, Badr AlKhamissi, Antoine Bosselut

Organizations: EPFL

Abstract

Latent language identification is often used to argue that multilingual language models route computation through language-specific states, such as English pivots. However, existing probes infer latent language from different signals, such as the geometry of hidden states or what can be decoded from intermediate representations. Since such claims shape conclusions about how models share and route information across languages, we ask whether these probes measure the same phenomenon or expose distinct aspects of multilingual computation. We study this question across model families, training regimes, domains, tasks, checkpoints, and up to 27 languages. We find that identification probes systematically disagree: the GMM-based representation probe, which draws evidence from hidden state geometry, shows earlier cross-lingual mixing, whereas decoding-based probes, which rely on output-space decodability, retain sharper language-specific and more English-biased signals. These differences track model multilinguality and training progression, but are comparatively stable across domains. Our results suggest a more cautious interpretation of latent language identification, where current probes expose different aspects of multilingual processing, rather than directly revealing a single internal lingua franca.

Explore similar work

CardsList
May 27, 2026cs.CL

DEPART: DEcomposing PARiTy across Multilingual LLMs

Multilingual Large Language Models (mLLMs) leaderboards report per-language accuracy but rarely explain why disparities emerge, leaving systemic biases unattributed and offering practitioners no actionable levers. We first establish that these gaps are systematic rather than artifacts of sampling noise via distribution-free Friedman and Kruskal--Wallis tests, then introduce a two-step Bayesian hierarchical framework that decomposes multilingual performance variance into interpretable components. First, isolating the variance attributable to language identity, we show that observable language features (script, family, typological distance) explain Rling2=79%R^2_{\text{ling}} = 79\% of this variance on understanding tasks and 92%92\% on reasoning, with a model's internal representational similarity to English emerging as the dominant predictor across both task buckets. Second, decomposing the full (model×\timesbenchmark×\timeslanguage) cube, we find that NLU and reasoning have fundamentally divergent variance profiles: model identity dominates understanding (66.7%66.7\% of variance), whereas the benchmark×\timesmodel interaction dominates reasoning (46.3%46.3\%). Together these results recast multilingual evaluation from passive performance mapping into an explainable, diagnostic framework with concrete levers for targeting the root drivers of language disparity.
Manan Uppadhyay, Prashant Kodali, Pranjal Chitale +3
MultilingualIdentity
AskPDF
Jun 1, 2026cs.CL

Multilinguality of Large Language Models From a Structural Perspective

Large language models (LLMs) have excelled in processing multiple languages through pre- and post-training on multilingual data, even though English dominates the training data. Prior work focusing on token representations has revealed how those LLMs process non-English text. Although these analyses have provided insightful findings, they fail to capture a structural view, which is an inherent property of language. In this study, we explore the multilinguality of LLMs through representational structural analysis. Our findings reveal that low-resource languages are structurally more different from English than high- and mid-resource languages, and that language-specific post-training alters their structures while preserving inter-language relationships.
Haruki Sakajo, Yusuke Sakai, Hidetaka Kamigaito +1
MultilingualLow-Resource LanguagesRepresentational Capacity
AskPDF
Aug 31, 2026cs.CL

Latent Mechanisms of Language Control in Multilingual Language Models

Multilingual large language models can exhibit unintended code-switching -- unnecessarily alternating between languages during generation. We present a comparative study of three methods that identify language-controlling latents in cross-layer transcoders: activation value-based selection (ValSel), activation frequency-based selection (FreqSel), and LLM-generated latent annotation-based selection (AnnSel). To evaluate the efficacy of these methods in identifying language-controlling latents, we introduce two multilingual benchmarks that exhibit code-switching for fine-grained analysis of language steering across seven languages. Through targeted intervention experiments on Gemma-2-2B and Qwen3-4B, we find that all three methods effectively manipulate generation language, with FreqSel achieving the strongest overall performance, while AnnSel offering interpretable latent selection through explicit language annotations. A knock-out analysis suggests the methods select non-overlapping but each-functional latent subsets, indicating redundancy rather than a single canonical language direction. Code and data can be found at https://github.com/rm-3284/Latent-Mechanism-Multilingual.
Ryo Mitsuhashi, Sabri Boughorbel, Majd Hawasly
Multilingual Language ModelsCode-SwitchingProtein Language ModelMultilingual BenchmarkVideo Latents
AskPDF