Multilingual generation requires large language models (LLMs) to determine both what to say and in what language to say it, yet these two commitments are difficult to disentangle from model outputs alone. We use translation as a controlled probe: for multi-parallel segments that a model translates well in one language direction but poorly in another, content is held fixed while language varies. Using activation patching, we write residual-stream states from successful runs into failing ones and compare content-matched states with content-independent controls, allowing us to trace content-specific and language-related signals across layers. Across three LLMs of different scales and architectures, we find a consistent three-stage pattern of multilingual generation. Intermediate layers first express a default English signal; content information becomes causally available one layer later; and the final target language is resolved only at later layers. The ordering of the English default before content also appears in a reading-comprehension task. Our sparse autoencoder analyses reveal a further dissociation: the intermediate language signal is only weakly represented by sparse features, whereas a later language distinction is clearly sparse, but resists intervention at the patched position. These results provide a mechanistic account of how language selection and content information emerge and interact during multilingual generation.
Figures & tables
Figure 1: Per-layer effect of last-token patching in gemma-3-4b-pt , pooled over instances and read per donor condition. Left : Δ\textschigh (chrF++ against the HIGH target’s gold translation) for the six different-target pairs whose HIGH direction targets English ( n=980 ). All four donors rise together at L15–16; only at L17 does matched separate. Middle : Δ\textsclow for the seven same-target pairs ( n=1403 ), where no language flip is possible; the matched – shuffled gap opens at L17. Right : Belebele accuracy for nine language contrasts ( n=943 ); the clean baseline is 0.10 and the HIGH pass scores 0.87 . xCOMET-XL versions of the two MT panels are in Appendix F .
Figure 2: Target-flip onset atlas over all 15 different-target pairs under matched patching. The horizontal axis is the median onset layer , computed from the target-exclusive-character discriminator, and the vertical axis is the peak fraction of instances whose output fastText identifies as the HIGH target language. Pairs whose HIGH target is high-resource Latin form one cluster (onset L15–17, .68 – .94 flipped); sister targets sit at L24 and near zero; other non-English targets fall between. One pair, eng → kas vs. eng → hin , is omitted because its discriminator is saturated at baseline: 91.8% of clean eng → kas outputs are already identified as Hindi.
Figure 3: Δ\textschigh when patching the denoised vectors and their two controls ( n=1055 ), pooled over the six different-target pairs with English as HIGH target.
Gemma-3
Qwen3.5
Llama-3.1
4B
9B
8B
Layers / attention
34 / full
32 / hybrid
32 / full
English flip onset
16 ( .47 )
18 ( .56 )
13 ( .41 )
Content onset
17 ( .50 )
19 ( .59 )
14 ( .44 )
Target onset
24 ( .71 )
28 ( .88 )
24 ( .75 )
mat. − shuf. @ flip
−0.02†
+0.95†
+0.74‡
Table 1: Replication across three models. Onsets are layer indices with relative depth in parentheses. Target onset is the late-layer resolution of the specific target language from the two sister pairs ( eng → nno vs. eng → swe , eng → ukr vs. eng → rus ). chrF++ differences are paired means over pooled instances, all p<10−3 except † , which are null ( p=0.96 and p=0.12 ). ‡ is significant ( p<10−3 ) but is 1.6% of the 46.6 chrF++ effect at that layer, against 13% one layer later. Belebele rows are accuracies at the final layer. SAE reconstruction fidelity is 1−∥h^−h∥/∥h∥ over the swept layers. “n/a” marks a sparse-flip test which we could not run because fidelity is too low.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
LOW
HIGH
Count
nob_Latn
eng_Latn
140
nob_Latn
zho_Hans
125
hin_Deva
zho_Hans
133
hin_Deva
eng_Latn
145
npi_Deva
hin_Deva
61
ukr_Cyrl
rus_Cyrl
60
Appendix
Table A.1: Contrastive LOW/HIGH languages for gemma-3-4b-pt on Belebele. Count is the number of documents answered incorrectly in the LOW language and correctly in the HIGH language. nob_Latn and npi_Deva are closely related languages to nno_Latn and kas_Deva, which are not available in Belebele.
LOW
HIGH
Count
τlo – τhi
Same target (different source)
cmn_Hans-nno_Latn
eng_Latn-nno_Latn
262
0.60–0.80
kas_Deva-eng_Latn
hin_Deva-eng_Latn
550
0.50–0.90
khm_Khmr-eng_Latn
cmn_Hans-eng_Latn
99
0.60–0.80
ukr_Cyrl-eng_Latn
rus_Cyrl-eng_Latn
85
0.85–0.95
nno_Latn-eng_Latn
swe_Latn-eng_Latn
60
0.90–1.00
Appendix
Table A.2: Contrastive LOW/HIGH translation directions for gemma-3-4b-pt , grouped by the relation between the two directions. Count is the number of segments surviving the selection rule of Eq. 1 under the per-pair xCOMET-XL thresholds τlo (LOW maximum) and τhi (HIGH minimum).
Figure B.1: Logit lens decodes the coarse script (left), fine sister family (middle) and the English pivot (right).
LOW
HIGH
Count
Same target (different source)
cmn_Hans-nno_Latn
eng_Latn-nno_Latn
174
kas_Deva-cmn_Hans
ukr_Cyrl-cmn_Hans
247
kas_Deva-eng_Latn
hin_Deva-eng_Latn
498
khm_Khmr-deu_Latn
rus_Cyrl-deu_Latn
227
khm_Khmr-eng_Latn
cmn_Hans-eng_Latn
201
Appendix
Table E.1: Contrastive LOW/HIGH translation directions for Qwen3.5-9B-Base ( 22 pairs). Thresholds are identical to Gemma’s in Table A.2 .
LOW
HIGH
Count
τlo – τhi
Same target (different source)
cmn_Hans-nno_Latn
eng_Latn-nno_Latn
65
0.70–0.90
kas_Deva-cmn_Hans
ukr_Cyrl-cmn_Hans
552
0.40–0.70
kas_Deva-eng_Latn
hin_Deva-eng_Latn
793
0.60–0.80
khm_Khmr-deu_Latn
rus_Cyrl-deu_Latn
280
0.70–0.90
khm_Khmr-eng_Latn
cmn_Hans-eng_Latn
262
0.70–0.90
Appendix
Table E.2: Contrastive LOW/HIGH translation directions for Llama-3.1-8B ( 22 pairs), with the re-derived per-pair xCOMET-XL thresholds.
LOW
HIGH
Qwen
Llama
nob_Latn
eng_Latn
98
241
nob_Latn
zho_Hans
89
218
hin_Deva
zho_Hans
165
157
hin_Deva
eng_Latn
177
180
npi_Deva
hin_Deva
74
79
ukr_Cyrl
rus_Cyrl
30
78
Appendix
Table E.3: Contrastive LOW/HIGH languages on Belebele for Qwen3.5-9B-Base and Llama-3.1-8B , over the same nine language contrasts used for Gemma (Table A.1 ). Count is the number of documents answered incorrectly in the LOW language and correctly in the HIGH one. The selection criterion is binary and identical across models, so no thresholds apply.
Figure F.1: xCOMET-XL counterparts of Figure 1 . Left : the six different-target pairs with English as HIGH target. Right : the seven same-target pairs. Both reproduce the onsets read off chrF++; the late-layer mean -over- matched ordering of the left panel of Figure 1 does not reproduce here.
Figure F.2: Cumulative least-squares R2 of the residual mean_diff direction on its top- 50 decoder vectors (Eq. 6 ). Left : pooled across 6 pairs whose HIGH-target is English with the Standard Error of the Mean (SEM). The dip at L15–16 is where the English flip is causal. Middle : English-Nynorsk vs English-Swedish (n=148). Right : English-Ukrainian vs English-Russian (n=151).
Figure F.3: The dashboard to inspect active SAE feature IDs when generating tokens in a specific language.
Figure F.4: Patching by add or ablate the SAE feature IDs (or do both) to test English flip. fraction reported by using fastText lid.176.
Figure F.5: Patching by add or ablate the SAE feature IDs (or do both) to test sister language ( ukr → rus ) flip. fraction reported by using fastText lid.176.
Figure F.6: Patching by add or ablate the SAE feature IDs (or do both) for disjoint pairs with the source as sister languages ( ukr → rus ). fraction reported by using fastText lid.176.
Figure F.7: Shared vs. instance-specific SAE feature mass in the HIGH vectors. The crossover begins at the content-onset layer L17.