One Head Away from Language Confusion: Single Attention Heads Across Model Families
Authors: Jiwon Jeong, Anjelo Jann Laroza, Chaewoon Yoon, Arjun Pillai, Pranav Kokati, Christian Hoang, Seunghyeok Hong
Organizations: Lotte Innovate · Mapúa University · University of Wisconsin–Madison · Irvington High School · Interlake High School · FPT University · Division of Social Science & AI, Hankuk University of Foreign Studies 107, Imun-ro, Dongdaemun-gu, Seoul, South Korea
Large language models sometimes reply in an unexpected language, a failure known as language confusion. We ablate individual attention heads in 12 model checkpoints spanning seven model series, using FLORES-200 continuations in five languages and Language Confusion Benchmark (LCB) chat replies. In ten models, zeroing one head causes substantially more departures from the expected language than zeroing any other head in its layer; its LCB pass-rate drop persists, though reduced, under mean ablation. Seven heads move 8.0 to 51.5 percent of all FLORES continuations out of the prompt's language, versus at most 1.4 percent for other same-layer heads; the other three mainly affect explicitly requested languages. These seven heads matter less in base models given plain text. In two models, replacing the values a head reads at the requested-language name with another language's mean makes replies fail and shifts some into that language, unlike a random change of equal size. Replacing a head's output with another language's mean redirects 84 to 99 percent of scored non-English monolingual LCB replies into that language in five models. Yet the requested language's mean improves crosslingual replies beyond every control head only in the two Gemma models, whose baselines are low. Code and results are available in the public repository at https://github.com/christian-hoang-04/anonymous-release.
Figures & tables
Figure 1: Methodology for identifying and analyzing at most one head per instruction-tuned model, accompanied by a real-world demonstration of the correct-to-wrong language confusion metric.
FLORES C→W
LCB monolingual
LCB crosslingual
Model
H
Head
Zero
Mean
Others
Base
Δ NLL
LPR
Δ LPR
LPR
Δ LPR
Heads found on FLORES
Qwen2.5-1.5B
12
L22H6
0.500
0.450
0.007
0.150
+ 0.228
0.982
− 0.273
0.704
− 0.276
Qwen2.5-3B
16
L27H13
0.515
0.338
0.014
0.202
+ 0.234
0.982
− 0.537
0.888
− 0.449
Qwen2.5-7B
28
L19H1
0.200
0.018
0.006
0.025
+ 0.026
0.984
− 0.009
0.950
− 0.074
Qwen3-1.7B
16
L18H12
0.324
0.028
0.002
0.067
+ 0.128
0.985
− 0.149
0.823
− 0.508
Table 1: Head-ablation effects across the 12 evaluated models. H: heads per layer. FLORES: Zero and Mean are C→W (Eq. ( 1 )) under zero and mean ablation; Others is the largest same-layer Zero, and Base is Zero in the base checkpoint. Δ NLL: loss change under zero ablation. LCB: baseline pass rate (LPR) on the non-English prompts and its paired change under zero ablation ( Δ LPR; Section 3.3 ). s : 125-prompt FLORES screen; gray: 95% CI includes zero; –: not evaluated. Heads found on LCB are compared on LCB (Section 4.2 ); all screens are in Appendix B . For Qwen3-4B, Table 1 reports the all-prompt estimate ( −0.076 ; 1,196 prompts, 1,187 pairs), while Section 4.2 reports the held-out estimate ( −0.073 ; 1,096 prompts, 1,089 pairs), excluding the 100-prompt selection screen.
Figure 2: FLORES C→W on 2,500 prompts when the selected head is zeroed. Left: Qwen2.5-1.5B L22H6 in the Base and Instruct checkpoints, on raw text and with the Instruct chat template. Right: OLMo-2-1B L12H8 in the released checkpoints (Base on raw text, the others with the chat template). The y-axis ranges differ.
Qwen2.5-1.5B
Gemma-3-1B
Attention from the last prompt token
To the name
0.773
0.132
To as many nearby tokens
0.003
0.009
Control head, to the name
0.039
0.093
Fail
Other
Fail
Other
Name masked
34
0
32
0
Table 2: Mechanism checks for Qwen2.5-1.5B L22H6 and Gemma-3-1B L11H3 on 480 crosslingual LCB prompts that name the requested language. The control head attends most to the name among the other heads of the layer (L22H7, L11H1). Values: the values the head reads at the name, replaced by a same-, other- or second-language mean, or moved by the other-language distance in a random direction (three seeds). All interventions act in the decoding steps after the first token. Fail: baseline-passing replies (323 and 51) that fail; Other: replies entirely in German, or French for German prompts (in the second-language row, in that language), of the 471 to 476 scored in each condition; –: Gemma-3-1B only. Per-sample results in Appendix F .
Figure 3: Share of outputs in the swapped-in language when the head’s output is replaced by its FLORES mean for another language (German, or French on German prompts). FLORES: the 2,000 non-English continuations, LCB monolingual: replies to the non-English monolingual prompts that are entirely in that language. Exact values in Table 9 .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Model
FLORES head ( C→W )
Mean
Δ NLL (layer mean)
Status
Note
Qwen2.5-1.5B
L22H6 (0.500)
0.450
+0.228 (+0.004)
Follow-up
Base counterpart: 0.150
Qwen2.5-3B
L27H13 (0.515)
0.338
+0.234 (+0.006)
Follow-up
Base counterpart: 0.202
Qwen2.5-7B
L19H1 (0.200)
0.018
+0.026 (+0.001)
Follow-up
L0H25 (0.629) and L0H22 (0.466) fail English retention (0.512 and 0.518)
Qwen3-1.7B
L18H12 (0.324)
0.028
+0.128 (+0.003)
Follow-up
L0H3 (0.466) fails the loss condition ( Δ NLL >1 )
Qwen3-4B
None (screen 0.024)
–
–
Screen only
LCB screened directly
Gemma-3-1B
L11H3 (0.413)
0.089
+0.720 ( − 0.012)
Follow-up
Same head has little effect in Base
Appendix
Table 3: FLORES head screens and full-set follow-ups. Status: Follow-up, the model went on to the 2,500 prompts; Screen only, no head met the screening rule, so only the 125-prompt screen was run (screen values are the strongest result there, not full-set estimates). Mean: C→W under pooled mean ablation. OLMo-3-7B ran the follow-up but is not counted (see text).
Selected head
Same-layer controls
Model
Head
Δ mono LPR
Δ cross LPR
Δ mono
Δ cross
Qwen2.5-1.5B
L22H6
− 0.273 [ − 0.305, − 0.240]
− 0.276 [ − 0.303, − 0.250]
− 0.001 to + 0.004
− 0.007 to + 0.000
Qwen2.5-3B
L27H13
− 0.537 [ − 0.575, − 0.503]
− 0.449 [ − 0.478, − 0.421]
+ 0.000 to + 0.003
− 0.025 to + 0.004
Qwen2.5-7B
L19H1
− 0.009 [ − 0.020, + 0.003]
− 0.074 [ − 0.089, − 0.059]
+ 0.001 to + 0.003
− 0.008 to + 0.001
Qwen3-1.7B
L18H12
− 0.149 [ − 0.174, − 0.123]
− 0.508 [ − 0.535, − 0.478]
− 0.004 to − 0.003
+ 0.000 to + 0.021
Qwen3-4B
L24H27
+ 0.000 [ − 0.010, + 0.010]
− 0.076 [ − 0.093, − 0.061]
− 0.005 to + 0.010
− 0.001 to + 0.013
Appendix
Table 4: LCB pass-rate change under zero ablation, with 95% paired-bootstrap intervals. Same-layer controls: range over three other heads of the layer (one, L8H12, for Llama-3.2-1B). The Llama-3.2-1B, Llama-3.2-3B and Qwen3-4B heads were found by screening LCB directly, the others on FLORES.
Model
Head
Δ mono LPR
Δ cross LPR
Qwen2.5-1.5B
L22H6
− 0.246 [ − 0.276, − 0.215]
− 0.247 [ − 0.273, − 0.220]
Qwen2.5-3B
L27H13
− 0.409 [ − 0.444, − 0.373]
− 0.331 [ − 0.358, − 0.303]
Qwen2.5-7B
L19H1
+ 0.003 [ − 0.005, + 0.010]
− 0.035 [ − 0.047, − 0.024]
Qwen3-1.7B
L18H12
− 0.067 [ − 0.086, − 0.048]
− 0.230 [ − 0.257, − 0.205]
Qwen3-4B
L24H27
− 0.004 [ − 0.014, + 0.006]
− 0.040 [ − 0.052, − 0.027]
Gemma-3-1B
L11H3
− 0.244 [ − 0.275, − 0.213]
− 0.092 [ − 0.110, − 0.075]
Appendix
Table 5: LCB pass-rate change under mean ablation, with 95% paired-bootstrap intervals. Below the line: the uncounted OLMo-3-7B head and the two heads the LCB screen selects instead of the FLORES head; under zero ablation, L16H9 changes the pass rates by −0.009 and −0.459 , and L0H1 by −0.116 and −0.271 .
Model
Switched prompts
Before
After
Before/after reply
Qwen2.5-1.5B
918
0.475
0.558
–
Qwen2.5-3B
514
0.590
0.657
0.582
Qwen3-1.7B
739
0.635
0.683
0.600
Gemma-3-1B
873
0.639
0.559
0.586
OLMo-2-1B
101
0.702
0.668
0.607
Appendix
Table 6: Content of the continuations that switch to English when the head is zeroed. Switched prompts: their number, Before and After: cosine similarity between the prompt and the continuation without and with the ablation, Before/after reply: similarity between the two continuations, –: not measured. Same-layer control heads switch at most two prompts each.
Masked
Values replaced
Model
Prompts
Pass
Name
Near
Ctrl
Zeroed
Same
Other
In other
Qwen2.5-1.5B
96
62
6
0
0
20
0
17
8/95
384 new
261
28
0
0
85
1
73
36/378
all 480
323
34
0
0
105
1
90
44/473
Gemma-3-1B
96
12
7
0
0
8
0
10
7/95
384 new
39
25
0
2
28
0
33
17/376
Appendix
Table 7: Mechanism checks on the 96 earlier and 384 new crosslingual LCB prompts. Pass: replies that pass at baseline, of which the columns from Name to Other count the ones that fail, Name, Near and Ctrl: the head’s attention to the language name or to as many nearby tokens masked, or the control head’s attention to the name, Zeroed: the head zeroed, Same and Other: the values the head reads at the name replaced by their mean over prompts that name the same or another language, In other: scored replies entirely in that other language, out of all scored replies, all in the decoding steps after the first generated token. The masks on the 96 prompts come from the earlier run, whose clean replies these runs reproduce exactly.
Model
Condition
Fail
First
Second
Change
Qwen2.5-1.5B
Same language
1
0
0
0.391
(323 pass)
Other language
90
44
2
2.007
Random, seed 1
5
0
0
2.034
Random, seed 2
8
0
0
2.028
Random, seed 3
4
0
0
2.043
Second language
217
0
208
1.976
Appendix
Table 8: Value patch controls on the 480 crosslingual prompts. Fail: baseline-passing replies that fail; First and Second: scored replies (471 to 476) entirely in the first donor language (German, or French for German prompts) or the second (Spanish, Italian or French); Change: mean norm of the change to the head’s output per decoding step, not recorded for the mask (–).
Base model
Our instruct head
Directions topped
Mean-score rank
Qwen3-1.7B
L18H12
20/20
1/448
Llama-3.2-1B
L8H25
7/20
1/512
Gemma-3-1B
L11H3
0/20
3/104
Appendix
Table 10: Instruction-tuned heads under the Translation Heads ranking on released base-model scores. In Llama-3.2-1B, L9H8 ties L8H25 in the number of directions topped.
Large language models (LLMs) demonstrate strong multilingual capabilities, yet often fail to consistently generate responses in the intended language, exhibiting a phenomenon known as language confusion. Prior mitigation approaches based on sequence-level fine-tuning, such as DPO, ORPO, and GRPO, operate at the level of entire responses and can lead to unintended degradation of general model capabilities, motivating the need for more fine-grained alternatives. To address this, we introduce Token-Level Policy Optimization (TLPO), a fine-tuning framework designed to mitigate language confusion through localized, token-level updates. TLPO identifies error-prone positions, explores alternative candidate tokens, and updates the policy using a tailored objective to suppress error-inducing outputs at a granular level. This selective intervention enables effective mitigation of language confusion without compromising the model's general abilities. Experiments on multiple multilingual LLMs across diverse languages demonstrate that TLPO significantly outperforms baselines in improving language consistency while preserving downstream task accuracy.
Large language models (LLMs) sometimes exhibit language confusion when generating non-English text. Existing approaches typically rely on fine-tuning to mitigate this issue. In contrast, we propose a tuning-free paradigm for reducing language confusion. Within this paradigm, we introduce two methods: Language-Aware Token Boosting (LATB), which applies targeted perturbations to tokens associated with the desired language, and Adaptive Language-Aware Token Boosting (Adaptive-LATB), which dynamically adjusts these perturbations based on the model's confidence in the intended language. Experiments demonstrate that our methods effectively improve multilingual alignment by reducing language confusion, while maintain the summarization quality without requiring any additional fine-tuning. Our code is publicly available. https://github.com/scbdatax/genai-datax-language-aware-token-boosting.
Multilingual LLMs answer the same multi-hop reasoning question across languages, but we lack a mechanistic account of whether they share an internal circuit. We identify Bridge Routing Heads (BRH) in two large multilingual LLMs through a three-stage pipeline. The resulting language-specific head sets exhibit near-complete mutual exclusivity across the five languages, with a mean Jaccard similarity of only 0.017 for Llama 3.1 70B and 0.057 for Qwen 2.5 72B, revealing language-idiosyncratic circuits. Ablating general BRH increases two-hop Negative Log-Likelihood (NLL) by 39-89x the random-head baseline, providing direct causal evidence of their role. Amplifying these heads in a failing target-language pass rescues up to 51.7% of cross-lingual failures, with no training. The two models share this dual-circuit pattern but allocate heads differently: Llama concentrates chaining in a large general pool, while Qwen leans on larger language-specific pools. Together these results show that activation-level intervention alone can recover correct answers from cross-lingual reasoning failures.