Cross-Loop Query Handoff: How Later Loops Redirect Two-Hop Retrieval in the Ouro Looped Language Models

Ziyu Mu; Zehua Wang

cs.LGOct 25, 2026

Cross-Loop Query Handoff: How Later Loops Redirect Two-Hop Retrieval in the Ouro Looped Language Models

Authors: Ziyu Mu, Zehua Wang

Organizations: Faculty of Science The University of British Columbia · Department of Electrical and Computer Engineering The University of British Columbia

Abstract

How do later loops in a pretrained looped language model repair two-hop retrieval? We study Ouro-1.4B and Ouro-2.6B by transplanting attention queries between loops at the same layer, head and final prompt position. On synthetic in-context questions, injecting the layer-indexed collection of loop-3 queries throughout loop 1 rescues 90% and 97% of questions answered incorrectly at loop 1 and correctly at loop 3, versus 24% and 21% for shuffled donors. The rescue also holds under greedy generation in Ouro-2.6B; in Ouro-1.4B it is limited to four-candidate ranking. Crossed donor-recipient swaps favor retrieval through the donor's bridge entity over copying its answer, although a substantial positional component remains in Ouro-2.6B. Late keys and values rescue far fewer targets. Freezing queries at their loop-1 values also damages two-hop accuracy more than a matched question that supplies the bridge. On natural questions, rescue is smaller and replicates only in Ouro-2.6B, and crossed swaps do not establish bridge redirection. These results identify a transferable collection of late-loop queries that supports bridge-conditioned retrieval on synthetic scenes; they do not locate where the bridge is computed or establish a single-vector handoff.

Figures & tables

Panel of Figure 1 — Figure 1: Rescue rates on two-hop targets (wrong at loop 1, right at loop 3), read at the loop-1 exit. (a) Rescue round: the same example’s loop-3 query at $q$ , another target’s loop-3 query (shuffled donor), and another example’s loop-1 query (same-loop donor). (b) Converse round (Section 5.1 ; different scenes): the late query and the late keys and values of $\mathcal{F}$ , each from the same example or from a shuffled donor, and the same example’s late keys and values at every position except $q$ ; the key–value arms leave queries unpatched. Dashed line: chance (25%). Intervals are in Tables 2 and 4 ; the same-loop donor and same-example late $K,V$ of $\mathcal{F}$ rates are point estimates.

$\mathcal{F}$	the two fact lines the question uses (person–box, box–color)
$q$	the final prompt position (“Answer:”), where the answer is read out
$\mathcal{S}^{\star}$	scenes whose two one-hop questions are right at every loop

Table 1: Notation.

Model	$s\to r$	$n$	Same	Shuffled	$\Delta_{\mathrm{rescue}}$ (pts)
1.4B	$3\to 1$	245	89.8%	23.7%	66.1 [60.0, 72.2]
1.4B	$3\to 2$	91	100%	19.8%	80.2 [71.4, 87.9]
2.6B	$3\to 1$	223	96.9%	20.6%	76.2 [70.4, 81.6]
2.6B	$3\to 2$	12	100%	25.0%	75.0 [50.0, 100.0]

Table 2: Query rescue at

q

, all layers and heads (600 two-hop scenes per model). Targets are wrong at loop

r

and right at loop

s

; rates are the share of targets correct at the loop-

r

exit after the patch. Same: the same example’s loop-

s

query. Shuffled: another target’s loop-

s

query. 95% paired bootstrap intervals.

Model, patch	$r$	Rec.	Bridge	Ans.	Line
1.4B, none	1	56.3	10.5	15.3	17.8
1.4B, swap	1	15.5	45.0 [41.0, 49.0]	13.3	26.2
1.4B, swap	3	14.7	49.2 [45.2, 53.2]	9.0	27.2
2.6B, none	1	68.0	10.3	11.5	10.2
2.6B, swap	1	9.0	48.0 [44.0, 52.0]	8.2	34.8
2.6B, swap	3	10.8	45.8 [41.8, 49.8]	7.2	36.2

Table 3: Two-hop query swaps (600 new scenes per model): the donor’s loop-3 query at

q

, all layers and heads, patched into the recipient at loop

r

. Shares of predictions (%) equal to each of four pairwise-different colors: the recipient’s answer (Rec.), the recipient’s color for the donor’s bridge box (Bridge), the donor’s answer (Ans.), and the recipient’s color at the line position of the donor’s answer (Line); chance 25%. 95% intervals for the bridge share.

Patched into loop 1	1.4B	2.6B
$Q$ at $q$	86.7% [82.5, 90.5]	98.2% [96.4, 99.6]
$Q$ at $q$ , shuffled donor	27.0% [21.7, 32.3]	27.4% [21.5, 33.2]
$K$ / $V$ / $K,V$ at $\mathcal{F}$	24.3 / 22.4 / 26.2%	4.9 / 22.4 / 33.2%
$K,V$ at $\mathcal{F}$ , shuffled donor	20.9% [16.0, 25.9]	9.4% [5.8, 13.5]
$K,V$ at all positions but $q$	13.7% [9.5, 17.9]	11.2% [7.2, 15.2]
Primary: $Q$ $-$ $K,V$ at $\mathcal{F}$ (pts)	60.5 [52.9, 68.1]	65.0 [57.8, 72.2]

Table 4: Converse test (600 new scenes per model; targets wrong at loop 1 and right at loop 3,

n=263

and 223). Each row patches the same example’s loop-3 state into loop 1, except the shuffled-donor rows; every

K,V

row leaves queries unpatched. 95% intervals where reported (Bonferroni 2 for the primary contrast).

	Ouro-1.4B			Ouro-2.6B
Held at loop 1	$\rho$	SCD (pts)	One-hop loss	$\rho$	SCD (pts)	One-hop loss
$Q$ at $q$ ( $I_{Q}$ )	$-0.12$ [ $-0.23$ , $-0.02$ ]	27.5 [22.2, 32.8]	12.2 [8.8, 15.8]	$-0.14$ [ $-0.26$ , $-0.03$ ]	36.0 [31.2, 40.8]	3.0 [1.5, 4.8]
$K$ at $\mathcal{F}$ ( $I_{K}$ )	0.72 [0.62, 0.81]	5.8 [2.2, 9.2]	4.2 [2.5, 6.2]	0.54 [0.42, 0.64]	12.5 [9.0, 16.0]	3.2 [1.8, 5.0]
$V$ at $\mathcal{F}$ ( $I_{V}$ )	0.01 [ $-0.16$ , 0.15]	30.8 [25.8, 35.8]	4.5 [2.5, 6.8]	$-0.62$ [ $-0.88$ , $-0.41$ ]	15.8 [11.5, 20.0]	39.8 [35.0, 44.5]
$C_{Q}$ / $C_{K}$ / $C_{V}$	1.04 / 0.96 / 0.98	$-1.2$ / 1.2 / 0.8	0 / 0 / 0	0.99 / 0.98 / 0.99	0.2 / 0.8 / 0.2	0 / 0 / 0
$V$ at $\mathcal{F}$	0.03 [ $-0.10$ , 0.15]			$-0.65$ [ $-0.87$ , $-0.46$ ]

Table 5: Freezes: in loops 2–4, one component is held at its loop-1 value.

\rho

is the retained two-hop loop gain at loop 3 (1 = no cost, 0 = the whole gain removed); SCD is the two-hop loss minus the one-hop loss, which is the larger of the two one-hop accuracy losses (both in points). Matched-size controls freeze the same component outside the question and outside

\mathcal{F}

. Top: 400 scenes per model. Bottom: co-freezes on 600 further scenes (native gain 34.5 and 33.2 points). 95% paired bootstrap intervals.

Model	Set	Layers	$n$	Rescue	Shuffled	$\Delta_{\mathrm{rescue}}$ [95% CI]
1.4B	Initial	All	176	7.4	6.3	1.1 [ $-1.1$ , 3.4]
2.6B	Initial	All	134	33.6	19.4	14.2 [7.5, 20.9]
2.6B	Replication	All	110	31.8	23.6	8.2 [1.8, 15.5]
2.6B	Initial	Last quarter	134	62.7	32.8	29.9 [20.9, 38.8]
2.6B	Replication	Last quarter	110	63.6	38.2	25.5 [16.4, 35.5]

Table 6: Natural-QA query rescue at loop 1 using loop-3 queries. Initial set: 1,105 items from 2WikiMultiHopQA and MuSiQue; replication: 646 unused 2WikiMultiHopQA items.

n

counts targets wrong at loop 1 and right at loop 3. Rescue and shuffled-donor rates are percentages;

\Delta_{\mathrm{rescue}}

is in points. Intervals are 95%; both 2.6B replication intervals (all layers and last quarter) are Bonferroni-corrected over two primary tests. Last-quarter patches are follow-up analyses.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Question	loop 1	loop 2	loop 3	loop 4
1.4B	two-hop	58.25	76.25	93.75	87.50
	person $\to$ box	97.50	100.00	100.00	100.00
	box $\to$ color	95.50	98.00	99.00	98.25
	two-hop (rescue round)	53.0	77.5	92.5	–
2.6B	two-hop	65.25	97.00	99.50	99.25
	person $\to$ box	99.50	100.00	100.00	100.00

Appendix

Table 7: Native forced-choice accuracy (%) at the loop-

t

exit. Freeze round: 400 scenes per model, each with one two-hop and two one-hop questions. Rescue round: 600 two-hop scenes per model, screened at loops 1–3. In Ouro-1.4B, two-hop accuracy falls from loop 3 to loop 4.

Multi-Hop Retrieval

See all 16 papers

The Key Handoff: Retrieval in Hybrid Language Models
Sep 26, 2026Kaan Kale, Oguzhan Baser, Sriram VishwanathHybrid Language Models Hybrid Transformer-State Space Model Architectures
Georgia Institute of Technology, USA · University of Texas at Austin, USA
Closing the Loop: Practical Training Recipes for Looped Language Models
Sep 30, 2026Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5Efficient Language Model Training LLM Training
RND NLP, DAIMLD, Russian Federation
DiscoLoop: Looping Discrete Embeddings and Continuous Hidden States for Multi-hop Reasoning
Jul 1, 2026Hengyu Fu, Tianyu Guo, Zixuan Wang +5Recurrent Transformers Multi-Hop Reasoning
University of California, Berkeley

Sep 26, 2026J/K move · Enter open · S save

The Key Handoff: Retrieval in Hybrid Language Models

Kaan Kale, Oguzhan Baser, Sriram Vishwanath

Georgia Institute of Technology, USA · University of Texas at Austin, USA

Hybrid Language Models Hybrid Transformer-State Space Model Architectures Multi-Hop QA Hybrid Attention Mechanistic Interpretability

Open paper AskPDF arXiv

A two-hop question makes a language model retrieve twice: once to produce a bridge entity, and once to retrieve with it. Transformers resolve that entity in their early layers. Hybrid models replace most of the attention with a recurrent state, so where the key becomes usable, and where it is spent on an answer, is not known. Answering "Where is the ball?" from "the ball belongs to Alice" and "Alice is in the garden" turns on Alice, a name the question does not mention. A model could reach garden through Alice, the key it computed, or through where the fact sits in the prompt. Across twelve models, dense and hybrid, we move a hidden state from one story into another where the two routes lead to different places, and read off which place the model gives. An attention layer converts the key in every model we tested, and crossing that layer removes its usable effect, all of it where no attention follows. In sequential hybrids this makes the answer a handoff: recurrent layers carry the key forward, and attention spends it. Retrieval does not always stop there. Writing a different fact into the memory of the recurrent layers that follow the last attention layer can move the answer toward that fact, multiplying its odds by 1.3 to 2.7, and which hybrids do this is not settled by their architecture or training. That read is addressed by the key, not by position: recurrent state in a hybrid is not only a carrier, but a memory that later layers can query.

Abstract

Multi-Hop Retrieval

Explore similar work

The Key Handoff: Retrieval in Hybrid Language Models

Closing the Loop: Practical Training Recipes for Looped Language Models

DiscoLoop: Looping Discrete Embeddings and Continuous Hidden States for Multi-hop Reasoning