One Head Away from Language Confusion: Single Attention Heads Across Model Families

Jiwon Jeong; Anjelo Jann Laroza; Chaewoon Yoon; Arjun Pillai; Pranav Kokati; Christian Hoang; Seunghyeok Hong

cs.CLOct 18, 2026

One Head Away from Language Confusion: Single Attention Heads Across Model Families

Authors: Jiwon Jeong, Anjelo Jann Laroza, Chaewoon Yoon, Arjun Pillai, Pranav Kokati, Christian Hoang, Seunghyeok Hong

Organizations: Lotte Innovate · Mapúa University · University of Wisconsin–Madison · Irvington High School ·

Abstract

Large language models sometimes reply in an unexpected language, a failure known as language confusion. We ablate individual attention heads in 12 model checkpoints spanning seven model series, using FLORES-200 continuations in five languages and Language Confusion Benchmark (LCB) chat replies. In ten models, zeroing one head causes substantially more departures from the expected language than zeroing any other head in its layer; its LCB pass-rate drop persists, though reduced, under mean ablation. Seven heads move 8.0 to 51.5 percent of all FLORES continuations out of the prompt's language, versus at most 1.4 percent for other same-layer heads; the other three mainly affect explicitly requested languages. These seven heads matter less in base models given plain text. In two models, replacing the values a head reads at the requested-language name with another language's mean makes replies fail and shifts some into that language, unlike a random change of equal size. Replacing a head's output with another language's mean redirects 84 to 99 percent of scored non-English monolingual LCB replies into that language in five models. Yet the requested language's mean improves crosslingual replies beyond every control head only in the two Gemma models, whose baselines are low. Code and results are available in the public repository at https://github.com/christian-hoang-04/anonymous-release.

Figures & tables

Panel of Figure 1 — Figure 1: Methodology for identifying and analyzing at most one head per instruction-tuned model, accompanied by a real-world demonstration of the correct-to-wrong language confusion metric.

Heads found on FLORES
			FLORES $C\to W$					LCB monolingual		LCB crosslingual
Model	H	Head	Zero	Mean	Others	Base	$\Delta$ NLL	LPR	$\Delta$ LPR	LPR	$\Delta$ LPR
Qwen2.5-1.5B	12	L22H6	0.500	0.450	0.007	0.150	$+$ 0.228	0.982	$-$ 0.273	0.704	$-$ 0.276
Qwen2.5-3B	16	L27H13	0.515	0.338	0.014	0.202	$+$ 0.234	0.982	$-$ 0.537	0.888	$-$ 0.449
Qwen2.5-7B	28	L19H1	0.200	0.018	0.006	0.025	$+$ 0.026	0.984	$-$ 0.009	0.950	$-$ 0.074
Qwen3-1.7B	16	L18H12	0.324	0.028	0.002	0.067	$+$ 0.128	0.985	$-$ 0.149	0.823	$-$ 0.508

Table 1: Head-ablation effects across the 12 evaluated models. H: heads per layer. FLORES: Zero and Mean are

C\to W

(Eq. ( 1 )) under zero and mean ablation; Others is the largest same-layer Zero, and Base is Zero in the base checkpoint.

\Delta

NLL: loss change under zero ablation. LCB: baseline pass rate (LPR) on the non-English prompts and its paired change under zero ablation (

\Delta

LPR; Section 3.3 ). s : 125-prompt FLORES screen; gray: 95% CI includes zero; –: not evaluated. Heads found on LCB are compared on LCB (Section 4.2 ); all screens are in Appendix B . For Qwen3-4B, Table 1 reports the all-prompt estimate (

-0.076

; 1,196 prompts, 1,187 pairs), while Section 4.2 reports the held-out estimate (

-0.073

; 1,096 prompts, 1,089 pairs), excluding the 100-prompt selection screen.

Figure 2: FLORES

C\to W

on 2,500 prompts when the selected head is zeroed. Left: Qwen2.5-1.5B L22H6 in the Base and Instruct checkpoints, on raw text and with the Instruct chat template. Right: OLMo-2-1B L12H8 in the released checkpoints (Base on raw text, the others with the chat template). The y-axis ranges differ.

Attention from the last prompt token
	Qwen2.5-1.5B		Gemma-3-1B
To the name	0.773		0.132
To as many nearby tokens	0.003		0.009
Control head, to the name	0.039		0.093
	Fail	Other	Fail	Other
Name masked	34	0	32	0

Table 2: Mechanism checks for Qwen2.5-1.5B L22H6 and Gemma-3-1B L11H3 on 480 crosslingual LCB prompts that name the requested language. The control head attends most to the name among the other heads of the layer (L22H7, L11H1). Values: the values the head reads at the name, replaced by a same-, other- or second-language mean, or moved by the other-language distance in a random direction (three seeds). All interventions act in the decoding steps after the first token. Fail: baseline-passing replies (323 and 51) that fail; Other: replies entirely in German, or French for German prompts (in the second-language row, in that language), of the 471 to 476 scored in each condition; –: Gemma-3-1B only. Per-sample results in Appendix F .

Figure 3: Share of outputs in the swapped-in language when the head’s output is replaced by its FLORES mean for another language (German, or French on German prompts). FLORES: the 2,000 non-English continuations, LCB monolingual: replies to the non-English monolingual prompts that are entirely in that language. Exact values in Table 9 .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	FLORES head ( $C\to W$ )	Mean	$\Delta$ NLL (layer mean)	Status	Note
Qwen2.5-1.5B	L22H6 (0.500)	0.450	+0.228 (+0.004)	Follow-up	Base counterpart: 0.150
Qwen2.5-3B	L27H13 (0.515)	0.338	+0.234 (+0.006)	Follow-up	Base counterpart: 0.202
Qwen2.5-7B	L19H1 (0.200)	0.018	+0.026 (+0.001)	Follow-up	L0H25 (0.629) and L0H22 (0.466) fail English retention (0.512 and 0.518)
Qwen3-1.7B	L18H12 (0.324)	0.028	+0.128 (+0.003)	Follow-up	L0H3 (0.466) fails the loss condition ( $\Delta$ NLL $>1$ )
Qwen3-4B	None (screen 0.024)	–	–	Screen only	LCB screened directly
Gemma-3-1B	L11H3 (0.413)	0.089	+0.720 ( $-$ 0.012)	Follow-up	Same head has little effect in Base

Appendix

Table 3: FLORES head screens and full-set follow-ups. Status: Follow-up, the model went on to the 2,500 prompts; Screen only, no head met the screening rule, so only the 125-prompt screen was run (screen values are the strongest result there, not full-set estimates). Mean:

C\to W

under pooled mean ablation. OLMo-3-7B ran the follow-up but is not counted (see text).

		Selected head		Same-layer controls
Model	Head	$\Delta$ mono LPR	$\Delta$ cross LPR	$\Delta$ mono	$\Delta$ cross
Qwen2.5-1.5B	L22H6	$-$ 0.273 [ $-$ 0.305, $-$ 0.240]	$-$ 0.276 [ $-$ 0.303, $-$ 0.250]	$-$ 0.001 to $+$ 0.004	$-$ 0.007 to $+$ 0.000
Qwen2.5-3B	L27H13	$-$ 0.537 [ $-$ 0.575, $-$ 0.503]	$-$ 0.449 [ $-$ 0.478, $-$ 0.421]	$+$ 0.000 to $+$ 0.003	$-$ 0.025 to $+$ 0.004
Qwen2.5-7B	L19H1	$-$ 0.009 [ $-$ 0.020, $+$ 0.003]	$-$ 0.074 [ $-$ 0.089, $-$ 0.059]	$+$ 0.001 to $+$ 0.003	$-$ 0.008 to $+$ 0.001
Qwen3-1.7B	L18H12	$-$ 0.149 [ $-$ 0.174, $-$ 0.123]	$-$ 0.508 [ $-$ 0.535, $-$ 0.478]	$-$ 0.004 to $-$ 0.003	$+$ 0.000 to $+$ 0.021
Qwen3-4B	L24H27	$+$ 0.000 [ $-$ 0.010, $+$ 0.010]	$-$ 0.076 [ $-$ 0.093, $-$ 0.061]	$-$ 0.005 to $+$ 0.010	$-$ 0.001 to $+$ 0.013

Appendix

Table 4: LCB pass-rate change under zero ablation, with 95% paired-bootstrap intervals. Same-layer controls: range over three other heads of the layer (one, L8H12, for Llama-3.2-1B). The Llama-3.2-1B, Llama-3.2-3B and Qwen3-4B heads were found by screening LCB directly, the others on FLORES.

Model	Head	$\Delta$ mono LPR	$\Delta$ cross LPR
Qwen2.5-1.5B	L22H6	$-$ 0.246 [ $-$ 0.276, $-$ 0.215]	$-$ 0.247 [ $-$ 0.273, $-$ 0.220]
Qwen2.5-3B	L27H13	$-$ 0.409 [ $-$ 0.444, $-$ 0.373]	$-$ 0.331 [ $-$ 0.358, $-$ 0.303]
Qwen2.5-7B	L19H1	$+$ 0.003 [ $-$ 0.005, $+$ 0.010]	$-$ 0.035 [ $-$ 0.047, $-$ 0.024]
Qwen3-1.7B	L18H12	$-$ 0.067 [ $-$ 0.086, $-$ 0.048]	$-$ 0.230 [ $-$ 0.257, $-$ 0.205]
Qwen3-4B	L24H27	$-$ 0.004 [ $-$ 0.014, $+$ 0.006]	$-$ 0.040 [ $-$ 0.052, $-$ 0.027]
Gemma-3-1B	L11H3	$-$ 0.244 [ $-$ 0.275, $-$ 0.213]	$-$ 0.092 [ $-$ 0.110, $-$ 0.075]

Appendix

Table 5: LCB pass-rate change under mean ablation, with 95% paired-bootstrap intervals. Below the line: the uncounted OLMo-3-7B head and the two heads the LCB screen selects instead of the FLORES head; under zero ablation, L16H9 changes the pass rates by

-0.009

and

-0.459

, and L0H1 by

-0.116

and

-0.271

Model	Switched prompts	Before	After	Before/after reply
Qwen2.5-1.5B	918	0.475	0.558	–
Qwen2.5-3B	514	0.590	0.657	0.582
Qwen3-1.7B	739	0.635	0.683	0.600
Gemma-3-1B	873	0.639	0.559	0.586
OLMo-2-1B	101	0.702	0.668	0.607

Appendix

Table 6: Content of the continuations that switch to English when the head is zeroed. Switched prompts: their number, Before and After: cosine similarity between the prompt and the continuation without and with the ablation, Before/after reply: similarity between the two continuations, –: not measured. Same-layer control heads switch at most two prompts each.

Model	Prompts	Pass	Name	Near	Ctrl	Zeroed	Same	Other	In other
			Masked				Values replaced
Qwen2.5-1.5B	96	62	6	0	0	20	0	17	8/95
	384 new	261	28	0	0	85	1	73	36/378
	all 480	323	34	0	0	105	1	90	44/473
Gemma-3-1B	96	12	7	0	0	8	0	10	7/95
	384 new	39	25	0	2	28	0	33	17/376

Appendix

Table 7: Mechanism checks on the 96 earlier and 384 new crosslingual LCB prompts. Pass: replies that pass at baseline, of which the columns from Name to Other count the ones that fail, Name, Near and Ctrl: the head’s attention to the language name or to as many nearby tokens masked, or the control head’s attention to the name, Zeroed: the head zeroed, Same and Other: the values the head reads at the name replaced by their mean over prompts that name the same or another language, In other: scored replies entirely in that other language, out of all scored replies, all in the decoding steps after the first generated token. The masks on the 96 prompts come from the earlier run, whose clean replies these runs reproduce exactly.

Model	Condition	Fail	First	Second	Change
Qwen2.5-1.5B	Same language	1	0	0	0.391
(323 pass)	Other language	90	44	2	2.007
	Random, seed 1	5	0	0	2.034
	Random, seed 2	8	0	0	2.028
	Random, seed 3	4	0	0	2.043
	Second language	217	0	208	1.976

Appendix

Table 8: Value patch controls on the 480 crosslingual prompts. Fail: baseline-passing replies that fail; First and Second: scored replies (471 to 476) entirely in the first donor language (German, or French for German prompts) or the second (Spanish, Italian or French); Change: mean norm of the change to the head’s output per decoding step, not recorded for the mask (–).

Base model	Our instruct head	Directions topped	Mean-score rank
Qwen3-1.7B	L18H12	20/20	1/448
Llama-3.2-1B	L8H25	7/20	1/512
Gemma-3-1B	L11H3	0/20	3/104

Appendix

Table 10: Instruction-tuned heads under the Translation Heads ranking on released base-model scores. In Llama-3.2-1B, L9H8 ties L8H25 in the number of directions topped.

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 265 papers

TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models
Apr 29, 2026Jinho Choo, JunSeung Lee, Jimyeong Kim +3Multilingual Language Models RL for Language Models
Samsung SDS
Language-Aware Token Boosting: LLM Language Confusion Reduction Without Tuning
Jun 8, 2026Trapoom Ukarapol, Pakhapoom Sarapat, Nut ChukamphaengMultilingual Language Models LLM Alignment
§SCB DataX · ‡Tsinghua University · †SCBX
Bridge Routing Heads: Where Multilingual Multi-hop Reasoning Lives in LLMs
Oct 7, 2026Seunghan Kim, Minyeong Choe, Hyunil Kim +1Cross-Lingual Reasoning in Language Models Multilingual Language Models
Chosun University · Soongsil University

Apr 29, 2026J/K move · Enter open · S save

TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models

Jinho Choo, JunSeung Lee, Jimyeong Kim, Yeeho Song+2

Samsung SDS

Multilingual Language Models RL for Language Models LLM Fine-Tuning

Open paper AskPDF arXiv

Large language models (LLMs) demonstrate strong multilingual capabilities, yet often fail to consistently generate responses in the intended language, exhibiting a phenomenon known as language confusion. Prior mitigation approaches based on sequence-level fine-tuning, such as DPO, ORPO, and GRPO, operate at the level of entire responses and can lead to unintended degradation of general model capabilities, motivating the need for more fine-grained alternatives. To address this, we introduce Token-Level Policy Optimization (TLPO), a fine-tuning framework designed to mitigate language confusion through localized, token-level updates. TLPO identifies error-prone positions, explores alternative candidate tokens, and updates the policy using a tailored objective to suppress error-inducing outputs at a granular level. This selective intervention enables effective mitigation of language confusion without compromising the model's general abilities. Experiments on multiple multilingual LLMs across diverse languages demonstrate that TLPO significantly outperforms baselines in improving language consistency while preserving downstream task accuracy.

Abstract

Multilingual Language Models

Explore similar work

TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models

Language-Aware Token Boosting: LLM Language Confusion Reduction Without Tuning

Bridge Routing Heads: Where Multilingual Multi-hop Reasoning Lives in LLMs