Multilingual generation requires large language models (LLMs) to determine both what to say and in what language to say it, yet these two commitments are difficult to disentangle from model outputs alone. We use translation as a controlled probe: for multi-parallel segments that a model translates well in one language direction but poorly in another, content is held fixed while language varies. Using activation patching, we write residual-stream states from successful runs into failing ones and compare content-matched states with content-independent controls, allowing us to trace content-specific and language-related signals across layers. Across three LLMs of different scales and architectures, we find a consistent three-stage pattern of multilingual generation. Intermediate layers first express a default English signal; content information becomes causally available one layer later; and the final target language is resolved only at later layers. The ordering of the English default before content also appears in a reading-comprehension task. Our sparse autoencoder analyses reveal a further dissociation: the intermediate language signal is only weakly represented by sparse features, whereas a later language distinction is clearly sparse, but resists intervention at the patched position. These results provide a mechanistic account of how language selection and content information emerge and interact during multilingual generation.
Figures & tables
Figure 1: Per-layer effect of last-token patching in gemma-3-4b-pt , pooled over instances and read per donor condition. Left : Δ\textschigh (chrF++ against the HIGH target’s gold translation) for the six different-target pairs whose HIGH direction targets English ( n=980 ). All four donors rise together at L15–16; only at L17 does matched separate. Middle : Δ\textsclow for the seven same-target pairs ( n=1403 ), where no language flip is possible; the matched – shuffled gap opens at L17. Right : Belebele accuracy for nine language contrasts ( n=943 ); the clean baseline is 0.10 and the HIGH pass scores 0.87 . xCOMET-XL versions of the two MT panels are in Appendix F .
Figure 2: Target-flip onset atlas over all 15 different-target pairs under matched patching. The horizontal axis is the median onset layer , computed from the target-exclusive-character discriminator, and the vertical axis is the peak fraction of instances whose output fastText identifies as the HIGH target language. Pairs whose HIGH target is high-resource Latin form one cluster (onset L15–17, .68 – .94 flipped); sister targets sit at L24 and near zero; other non-English targets fall between. One pair, eng → kas vs. eng → hin , is omitted because its discriminator is saturated at baseline: 91.8% of clean eng → kas outputs are already identified as Hindi.
Figure 3: Δ\textschigh when patching the denoised vectors and their two controls ( n=1055 ), pooled over the six different-target pairs with English as HIGH target.
Gemma-3
Qwen3.5
Llama-3.1
4B
9B
8B
Layers / attention
34 / full
32 / hybrid
32 / full
English flip onset
16 ( .47 )
18 ( .56 )
13 ( .41 )
Content onset
17 ( .50 )
19 ( .59 )
14 ( .44 )
Target onset
24 ( .71 )
28 ( .88 )
24 ( .75 )
mat. − shuf. @ flip
−0.02†
+0.95†
+0.74‡
Table 1: Replication across three models. Onsets are layer indices with relative depth in parentheses. Target onset is the late-layer resolution of the specific target language from the two sister pairs ( eng → nno vs. eng → swe , eng → ukr vs. eng → rus ). chrF++ differences are paired means over pooled instances, all p<10−3 except † , which are null ( p=0.96 and p=0.12 ). ‡ is significant ( p<10−3 ) but is 1.6% of the 46.6 chrF++ effect at that layer, against 13% one layer later. Belebele rows are accuracies at the final layer. SAE reconstruction fidelity is 1−∥h^−h∥/∥h∥ over the swept layers. “n/a” marks a sparse-flip test which we could not run because fidelity is too low.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
LOW
HIGH
Count
nob_Latn
eng_Latn
140
nob_Latn
zho_Hans
125
hin_Deva
zho_Hans
133
hin_Deva
eng_Latn
145
npi_Deva
hin_Deva
61
ukr_Cyrl
rus_Cyrl
60
Appendix
Table A.1: Contrastive LOW/HIGH languages for gemma-3-4b-pt on Belebele. Count is the number of documents answered incorrectly in the LOW language and correctly in the HIGH language. nob_Latn and npi_Deva are closely related languages to nno_Latn and kas_Deva, which are not available in Belebele.
LOW
HIGH
Count
τlo – τhi
Same target (different source)
cmn_Hans-nno_Latn
eng_Latn-nno_Latn
262
0.60–0.80
kas_Deva-eng_Latn
hin_Deva-eng_Latn
550
0.50–0.90
khm_Khmr-eng_Latn
cmn_Hans-eng_Latn
99
0.60–0.80
ukr_Cyrl-eng_Latn
rus_Cyrl-eng_Latn
85
0.85–0.95
nno_Latn-eng_Latn
swe_Latn-eng_Latn
60
0.90–1.00
Appendix
Table A.2: Contrastive LOW/HIGH translation directions for gemma-3-4b-pt , grouped by the relation between the two directions. Count is the number of segments surviving the selection rule of Eq. 1 under the per-pair xCOMET-XL thresholds τlo (LOW maximum) and τhi (HIGH minimum).
Figure B.1: Logit lens decodes the coarse script (left), fine sister family (middle) and the English pivot (right).
LOW
HIGH
Count
Same target (different source)
cmn_Hans-nno_Latn
eng_Latn-nno_Latn
174
kas_Deva-cmn_Hans
ukr_Cyrl-cmn_Hans
247
kas_Deva-eng_Latn
hin_Deva-eng_Latn
498
khm_Khmr-deu_Latn
rus_Cyrl-deu_Latn
227
khm_Khmr-eng_Latn
cmn_Hans-eng_Latn
201
Appendix
Table E.1: Contrastive LOW/HIGH translation directions for Qwen3.5-9B-Base ( 22 pairs). Thresholds are identical to Gemma’s in Table A.2 .
LOW
HIGH
Count
τlo – τhi
Same target (different source)
cmn_Hans-nno_Latn
eng_Latn-nno_Latn
65
0.70–0.90
kas_Deva-cmn_Hans
ukr_Cyrl-cmn_Hans
552
0.40–0.70
kas_Deva-eng_Latn
hin_Deva-eng_Latn
793
0.60–0.80
khm_Khmr-deu_Latn
rus_Cyrl-deu_Latn
280
0.70–0.90
khm_Khmr-eng_Latn
cmn_Hans-eng_Latn
262
0.70–0.90
Appendix
Table E.2: Contrastive LOW/HIGH translation directions for Llama-3.1-8B ( 22 pairs), with the re-derived per-pair xCOMET-XL thresholds.
LOW
HIGH
Qwen
Llama
nob_Latn
eng_Latn
98
241
nob_Latn
zho_Hans
89
218
hin_Deva
zho_Hans
165
157
hin_Deva
eng_Latn
177
180
npi_Deva
hin_Deva
74
79
ukr_Cyrl
rus_Cyrl
30
78
Appendix
Table E.3: Contrastive LOW/HIGH languages on Belebele for Qwen3.5-9B-Base and Llama-3.1-8B , over the same nine language contrasts used for Gemma (Table A.1 ). Count is the number of documents answered incorrectly in the LOW language and correctly in the HIGH one. The selection criterion is binary and identical across models, so no thresholds apply.
Figure F.1: xCOMET-XL counterparts of Figure 1 . Left : the six different-target pairs with English as HIGH target. Right : the seven same-target pairs. Both reproduce the onsets read off chrF++; the late-layer mean -over- matched ordering of the left panel of Figure 1 does not reproduce here.
Figure F.2: Cumulative least-squares R2 of the residual mean_diff direction on its top- 50 decoder vectors (Eq. 6 ). Left : pooled across 6 pairs whose HIGH-target is English with the Standard Error of the Mean (SEM). The dip at L15–16 is where the English flip is causal. Middle : English-Nynorsk vs English-Swedish (n=148). Right : English-Ukrainian vs English-Russian (n=151).
Figure F.3: The dashboard to inspect active SAE feature IDs when generating tokens in a specific language.
Figure F.4: Patching by add or ablate the SAE feature IDs (or do both) to test English flip. fraction reported by using fastText lid.176.
Figure F.5: Patching by add or ablate the SAE feature IDs (or do both) to test sister language ( ukr → rus ) flip. fraction reported by using fastText lid.176.
Figure F.6: Patching by add or ablate the SAE feature IDs (or do both) for disjoint pairs with the source as sister languages ( ukr → rus ). fraction reported by using fastText lid.176.
Figure F.7: Shared vs. instance-specific SAE feature mass in the HIGH vectors. The crossover begins at the content-onset layer L17.
Previous work has argued that multilingual LLMs encode language-specific features most strongly in their initial and final layers, implying that intermediate layers support language-agnostic representations. This paper reconsiders this finding by performing four different experiments: three that undertake a fine-grained analysis of representations of Portuguese and Spanish, two closely related languages, and one large-scale language identification experiment that covers over 200 languages. The results of these experiments challenge the strict separation between language-specific and language-agnostic layers in multilingual LLMs, using Llama 3.2 1B as a test case. By analyzing and manipulating contextual sentence embeddings, we demonstrate that language identity remains strongly recoverable throughout the model rather than being restricted to initial and final layers. Moreover, we find that the model tends to classify ambiguous strings as closer to Spanish than to Portuguese, revealing a skew in model behavior that differs from human judgments.
Mikaela Martins, Jonathan Dunn
Department of Linguistics, University of Illinois Urbana-Champaign
Multilingual large language models can exhibit unintended code-switching -- unnecessarily alternating between languages during generation. We present a comparative study of three methods that identify language-controlling latents in cross-layer transcoders: activation value-based selection (ValSel), activation frequency-based selection (FreqSel), and LLM-generated latent annotation-based selection (AnnSel). To evaluate the efficacy of these methods in identifying language-controlling latents, we introduce two multilingual benchmarks that exhibit code-switching for fine-grained analysis of language steering across seven languages. Through targeted intervention experiments on Gemma-2-2B and Qwen3-4B, we find that all three methods effectively manipulate generation language, with FreqSel achieving the strongest overall performance, while AnnSel offering interpretable latent selection through explicit language annotations. A knock-out analysis suggests the methods select non-overlapping but each-functional latent subsets, indicating redundancy rather than a single canonical language direction. Code and data can be found at https://github.com/rm-3284/Latent-Mechanism-Multilingual.
Ryo Mitsuhashi, Sabri Boughorbel, Majd Hawasly
Princeton University New Jersey, USA · Prince Sattam bin Abdulaziz University Riyadh, Saudi Arabia · QCRI, Hamad Bin Khalifa University, Doha, Qatar
Latent language identification is often used to argue that multilingual language models route computation through language-specific states, such as English pivots. However, existing probes infer latent language from different signals, such as the geometry of hidden states or what can be decoded from intermediate representations. Since such claims shape conclusions about how models share and route information across languages, we ask whether these probes measure the same phenomenon or expose distinct aspects of multilingual computation. We study this question across model families, training regimes, domains, tasks, checkpoints, and up to 27 languages. We find that identification probes systematically disagree: the GMM-based representation probe, which draws evidence from hidden state geometry, shows earlier cross-lingual mixing, whereas decoding-based probes, which rely on output-space decodability, retain sharper language-specific and more English-biased signals. These differences track model multilinguality and training progression, but are comparatively stable across domains. Our results suggest a more cautious interpretation of latent language identification, where current probes expose different aspects of multilingual processing, rather than directly revealing a single internal lingua franca.