Bridge Routing Heads: Where Multilingual Multi-hop Reasoning Lives in LLMs

Seunghan Kim; Minyeong Choe; Hyunil Kim; Haehyun Cho

cs.CLOct 7, 2026

Bridge Routing Heads: Where Multilingual Multi-hop Reasoning Lives in LLMs

Authors: Seunghan Kim, Minyeong Choe, Hyunil Kim, Haehyun Cho

Organizations: Chosun University · Soongsil University

Abstract

Multilingual LLMs answer the same multi-hop reasoning question across languages, but we lack a mechanistic account of whether they share an internal circuit. We identify Bridge Routing Heads (BRH) in two large multilingual LLMs through a three-stage pipeline. The resulting language-specific head sets exhibit near-complete mutual exclusivity across the five languages, with a mean Jaccard similarity of only 0.017 for Llama 3.1 70B and 0.057 for Qwen 2.5 72B, revealing language-idiosyncratic circuits. Ablating general BRH increases two-hop Negative Log-Likelihood (NLL) by 39-89x the random-head baseline, providing direct causal evidence of their role. Amplifying these heads in a failing target-language pass rescues up to 51.7% of cross-lingual failures, with no training. The two models share this dual-circuit pattern but allocate heads differently: Llama concentrates chaining in a large general pool, while Qwen leans on larger language-specific pools. Together these results show that activation-level intervention alone can recover correct answers from cross-lingual reasoning failures.

Figures & tables

Figure 1: Overview of Bridge Routing Heads (BRH). (a) The first hop of a two-hop query resolves the bridge entity

B

(here, Japan ) within the model’s hidden states. (b) When the BRH routing contribution is under-weighted in the target language, the bridge signal is lost and the model produces an incorrect answer. (c) Amplifying BRH outputs by

1{+}\alpha

restores routing of the bridge signal, recovering the correct answer—without any training.

Language	Order	Family	Script
English (EN)	SVO	IE (Germanic)	Latin
Korean (KO)	SOV	Koreanic	Hangul
Chinese (ZH)	SVO	Sino-Tibetan	Han
Japanese (JA)	SOV	Japonic	Kana + Han
Spanish (ES)	SVO	IE (Romance)	Latin

Table 1: Typological coverage of the five languages in our dataset. IE = Indo-European; Han = Han / Chinese characters.

Panel of Figure 2 — Figure 2: One Patchscopes example for a Llama 3.1 70B Korean-specific BRH. The patched hidden state at x: causes the model to generate the gold bridge “Australia” despite the question being in Korean—direct evidence that this BRH encodes the bridge entity. Full set: Table 14 .

Model	Pool	EN	KO	ZH	JA	ES
Llama	Knowledge-verified ( $\mathcal{K}_{X}$ )	10,059	7,106	6,672	6,982	7,967
	Two-hop correct	3,538	2,467	1,981	2,290	2,653
	Two-hop incorrect	6,521	4,639	4,691	4,692	5,314
	EN correct / $X$ incorrect	—	519	489	420	582
Qwen	Knowledge-verified ( $\mathcal{K}_{X}$ )	4,101	7,395	2,654	5,994	8,960
	Two-hop correct	1,700	1,977	795	1,952	2,242

Table 2: Per-language sizes of the knowledge-verified pool and experiment-specific subsets.

Model	Set	After scoring	After ablation	After Patchscopes	Final (% of total)
Llama 3.1 70B	general	191	70	—	70 (1.37%)
(5,120 Heads)	ko-spec	204	114	8	8 (0.16%)
	zh-spec	204	115	11	11 (0.21%)
	ja-spec	204	114	31	31 (0.61%)
	es-spec	204	120	9	9 (0.18%)
Qwen 2.5 72B	general	175	42	—	42 (0.82%)

Table 3: Head counts at each stage of the BRH filtering pipeline. After scoring uses the top-

N

% threshold from Step 1 (§ 3.3 ): top 10% for Llama, top 13% for Qwen.

Figure 3: Layer-wise activation patching on the five-language dataset, KL divergence at answer_colon .

r

means the mean Pearson correlation coefficient with range (min, max) over the 380 off-diagonal pairs. Each panel overlays the 20 directional language-pair curves (grey) and their mean (red).

Model	Lang	Base NLL	$\Delta$ NLL (95% CI)	Rand $\Delta$	Scale	$\Delta$ NLL (95% CI)	Rand $\Delta$	Scale
			General BRH			Language-Specific BRH
Llama 3.1 70B	ko	0.72	2.07 (1.85–2.30)	0.053	39 $\times$	0.12 (0.09–0.15)	0.008	16 $\times$
	zh	0.78	1.98 (1.73–2.23)	0.048	41 $\times$	0.08 (0.05–0.10)	0.010	8 $\times$
	ja	0.70	1.68 (1.51–1.87)	0.040	42 $\times$	0.33 (0.28–0.38)	0.031	11 $\times$
	es	0.85	1.87 (1.66–2.10)	0.042	45 $\times$	0.13 (0.11–0.16)	0.011	12 $\times$
Qwen 2.5 72B	ko	0.95	1.15 (1.02–1.31)	0.014	83 $\times$	0.65 (0.58–0.73)	0.017	37 $\times$

Table 4: Mean-ablation results with 95% bootstrap CIs. The Scale column reports how many times larger the BRH effect is than the random baseline; dashes indicate cases where the random baseline is non-positive, making the ratio undefined.

Figure 4: Layer position of every final BRH, stacked by set type.

		Intra-Lingual Amplification			Cross-Lingual Transfer
Model	Lang	$\alpha{=}0.25$	$\alpha{=}0.5$	$\alpha{=}1.0$	$\alpha{=}0.25$	$\alpha{=}0.5$	$\alpha{=}1.0$
Llama 3.1 70B	ko	2.0 (1.6–2.4)	3.5 (3.0–4.1)	6.1 (5.5–6.8)	22.2 (18.8–25.9)	30.4 (26.6–34.5)	41.0 (36.9–45.3)
	zh	2.6 (2.1–3.1)	4.8 (4.2–5.5)	8.6 (7.9–9.5)	25.0 (21.3–29.0)	35.6 (31.5–39.9)	51.5 (47.1–55.9)
	ja	3.5 (3.0–4.0)	5.9 (5.3–6.6)	10.1 (9.3–11.0)	22.9 (19.1–27.1)	37.9 (33.4–42.6)	51.7 (46.9–56.4)
	es	1.4 (1.1–1.8)	2.9 (2.5–3.4)	4.8 (4.3–5.5)	29.2 (25.7–33.0)	36.8 (33.0–40.8)	36.9 (33.1–40.9)
Qwen 2.5 72B	ko	7.8 (7.1–8.5)	12.3 (11.4–13.2)	17.2 (16.3–18.3)	15.6 (11.6–20.7)	20.1 (15.5–25.6)	23.0 (18.1–28.6)

Table 5: Amplification accuracies (%) with 95% Wilson CIs. Intra-Lingual Amplification uses target-language failures; Cross-Lingual Transfer further requires a correct English answer. Subset sizes are given in Table 2 .

Figure 5: Accuracy as a function of the amplification scale

\alpha

. Top : Intra-lingual amplification (rescue rate). Bottom : Cross-lingual transfer.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Property ID	Relation	$r_{1}$	$r_{2}$
P27	country of citizenship	10,468	8,908
P1441	present in work	9,148	1,445
P495	country of origin	8,272	3,615
P264	record label	6,438	2,416
P175	performer	6,057	1,109
P108	employer	5,600	532

Appendix

Table 6: All 29 Wikidata relation types used in the dataset, with their property IDs and per-hop occurrence counts. A dash (–) indicates a relation that does not appear in that hop position.

Entity Type	Subject ( $A$ )	Bridge ( $B$ )	Answer ( $C$ )
Human / Person	26,906	23,804	33,963
Country	313	18,893	13,213
Film	12,176	7,309	1,415
Film Character	8,699	760	72
Video Game	7,859	1,680	1,012
Literary Character	7,699	1,458	360

Appendix

Table 7: Entity-type distribution across the three chain positions. Counts indicate the number of instances in which an entity of that type occupies the given position.

Query Type	Input	Example (English)
Two-Hop	$A,r_{1},r_{2}$	What is the capital of the country of citizenship of Stephen Harper?
First-Hop	$A,r_{1}$	What is the country of citizenship of Stephen Harper?
Second-Hop	$B,r_{2}$	What is the capital of Canada?
Shortcut (no $A$ )	$r_{1},r_{2}$	What is the capital of the country of citizenship of?
Shortcut (no $r_{1}$ )	$A,r_{2}$	What is the capital of Stephen Harper?

Appendix

Table 8: Five query variants derived from each two-hop triple. The running example uses

A\!=\!\text{Stephen Harper}

r_{1}\!=\!\text{country of citizenship}

B\!=\!\text{Canada}

r_{2}\!=\!\text{capital}

C\!=\!\text{Ottawa}

Lang.	Type	Example
en	Two-Hop	What is the capital of the country of citizenship of Stephen Harper?
	First-Hop	What is the country of citizenship of Stephen Harper?
	Second-Hop	What is the capital of Canada?
	SC-no- $A$	What is the capital of the country of citizenship of?
	SC-no- $r_{1}$	What is the capital of Stephen Harper?
ko	Two-Hop	Stephen Harper의 국적의 수도는 무엇입니까?

Appendix

Table 9: One representative chain ( Stephen Harper

\xrightarrow{\text{country}}

Canada

\xrightarrow{\text{capital}}

Ottawa ) shown in all five query variants for each language. Entity labels are retrieved from Wikidata; interrogative framing is generated via per-language relation templates. SC-no-

A

: shortcut with subject removed; SC-no-

r_{1}

: shortcut with first-hop relation removed.

Figure 6: Layer-wise activation patching, logit difference at answer_colon . Layout mirrors Figure 3 ; each panel overlays all 20 directional language-pair curves (grey) and their mean (red). The inset reports the mean and range of off-diagonal Pearson

r

across the 380 language-pair cross-correlations.

Variant	Spearman $\rho$	Pool size (L / Q)	Recall
$\mathrm{FH{+}TH{-}SH}^{\dagger}$	1.000	191 / 175	—
$\mathrm{TH}$	0.962	305 / 291	84–89%
$\mathrm{TH{-}SH}$	0.328	149 / 98	26–35%
$\mathrm{FH{+}TH{-}2SH}$	0.388	102 / 85	27–28%

Appendix

Table 10: Sensitivity of the BRS pipeline to alternative contrasts. † denotes the original contrast used in the main paper. Spearman

\rho

is the mean per-head rank correlation against the original BRS over the 10 {general, KO/ZH/JA/ES-specific}

\times

2-model sets. Pool size reports the Llama and Qwen general candidate pool sizes (L / Q). Recall is the fraction of the original general pool retained inside each variant’s general pool, given as the Llama–Qwen range.

		Correct-only			Balanced
Model	Lang	.25	.5	1.0	.25	.5	1.0
Llama	ko	95	92	90	65	65	71
	zh	96	92	93	60	68	75
	ja	96	90	87	55	62	65
	es	100	94	88	66	63	57
	all	96.8	92.0	89.5	61.5	64.5	67.0

Appendix

Table 11: Two-hop accuracy (%) under general-BRH amplification at

\alpha\in\{0.25,0.5,1.0\}

(column headers). Correct-only : 100 items per language the unmodified model answers correctly, so the

\alpha=0

baseline is 100% by construction. Balanced : 100 items per language, 50 originally correct and 50 originally incorrect, so the baseline is 50%. all pools the four languages (400 items).

Model	Set	$k$	.25	.5	1.0
Llama	general	70	$-0.5$	$-1.6$	$-1.6$
	ko-spec	8	$-0.5$	$+0.5$	$-0.5$
	zh-spec	11	$-1.0$	$-0.5$	$-0.5$
	ja-spec	31	$-2.1$	$-1.6$	$-2.1$
	es-spec	9	$0.0$	$0.0$	$0.0$
Qwen	general	42	$0.0$	$-0.5$	$-0.5$

Appendix

Table 12: Change in English MMLU accuracy (percentage points) with amplification active, by amplified head set and

\alpha

(column headers). MMLU is 5-shot over 194 items; the unmodified baselines are 81.96% (Llama) and 88.66% (Qwen).

k

is the number of heads in the set.

Model	Lang	$k=0$ (main)	$k=1$	$k=2$	$k=3$
Llama	ko	8	1	1	0
	zh	11	0	0	0
	ja	31	13	13	1
	es	9	0	0	0
Qwen	ko	87	49	49	10
	zh	38	8	8	0

Appendix

Table 13: Final language-specific BRH counts under progressively stricter Patchscopes thresholds.

k

indexes the success-count requirement:

k=0

is the main paper default (

\geq 1

successful trial);

k=1

2

3

require

\geq 3

\geq 4

\geq 5

successful trials respectively.

k=1

and

k=2

yield identical counts because no candidate head has exactly

3

successful trials.

Model	Lang	Question (target language / English gloss)	Gold $B$	Generated continuation
Llama 3.1 70B	ko	Rod Laver의 국적의 수도는 무엇입니까? What is the capital of Rod Laver’s nationality?	Australia	“Australia: Australia is a country in the southern hemisphere, …”
	zh	Mark Thatcher的母亲的配偶是谁？ Who is the spouse of Mark Thatcher’s mother?	M. Thatcher	“Margaret Thatcher: Margaret Thatcher was the Prime Minister of the UK …”
	ja	Psychoのプロデューサーの配偶者は誰ですか？ Who is the spouse of Psycho’s producer?	A. Hitchcock	“Hitchcock: Hitchcock is a film directed by Alfred Hitchcock …”
	es	¿Cuál es la capital del país de ciudadanía de Alexander Arekeev? What is the capital of Alexander Arekeev’s country of citizenship?	Russia	“Russia: Russia is a country located in northern Eurasia …”
Qwen 2.5 72B	ko	James Sirius Potter의 등장 작품의 출판사는 누구입니까? Who is the publisher of the work in which James Sirius Potter appears?	Harry Potter	“Harry Potter is the protagonist of the Harry Potter series written by J.K. Rowling …”
	zh	Elsa Einstein的配偶的配偶是谁？ Who is the spouse of Elsa Einstein’s spouse?	A. Einstein	“spouse, Albert Einstein: Albert Einstein was a German-born theoretical physicist …”

Appendix

Table 14: Patchscopes examples for language-specific BRH. Each row shows the target-language question (top) with an English gloss (bottom, italics). The gold bridge entity

B

appears explicitly in every continuation.

Figure 7: Bridge Routing Score (BRS) heatmaps for Llama 3.1 70B (left) and Qwen 2.5 72B (right). Color encodes the per-head BRS averaged across all five languages, computed before any filtering step (yellow = high, dark purple = low). The $x$ -axis is the head index (0–63) and the $y$ -axis is the layer index (0–79), with layer 0 at the bottom.

Explore similar work

Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads
Feb 25, 2026Shaswat Patel, Vishvesh Trivedi, Yue Han +2Multilingual Language Models Efficient Latent Reasoning
New York University
Do Language Models Reason Across Languages?
Jan 10, 2026Yan Meng, Wafaa Mohammed, Christof MonzMultilingual Language Models LLM Reasoning Strategies
Language Technology Lab · University of Amsterdam
DEPART: DEcomposing PARiTy across Multilingual LLMs
May 27, 2026Manan Uppadhyay, Prashant Kodali, Pranjal Chitale +3
Microsoft Research India · IIT Gandhinagar

Feb 25, 2026, cs.CLJ/K move · Enter open · S save

Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads

Shaswat Patel, Vishvesh Trivedi, Yue Han, Yihuai Hong+1

New York University

Multilingual Language Models Efficient Latent Reasoning Transformer Architectures Chain-of-Thought Reasoning

Open paper AskPDF arXiv

Recent work has identified a subset of attention heads in Transformer as retrieval heads, which are responsible for retrieving information from the context. In this work, we first investigate retrieval heads in multilingual contexts. In multilingual language models, we find that retrieval heads are often shared across multiple languages. Expanding the study to cross-lingual setting, we identify Retrieval-Transition heads(RTH), which govern the transition to specific target-language output. Our experiments reveal that RTHs are distinct from retrieval heads and more vital for Chain-of-Thought reasoning in multilingual LLMs. Across four multilingual benchmarks (MMLU-ProX, MGSM, MLQA, and XQuaD) and two model families (Qwen-2.5 and Llama-3.1), we demonstrate that masking RTH induces bigger performance drop than masking Retrieval Heads (RH). Our work advances understanding of multilingual LMs by isolating the attention heads responsible for mapping to target languages.

Abstract

Explore similar work

Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads

Do Language Models Reason Across Languages?

DEPART: DEcomposing PARiTy across Multilingual LLMs