The Key Handoff: Retrieval in Hybrid Language Models

Kaan Kale; Oguzhan Baser; Sriram Vishwanath

cs.CLSep 26, 2026

The Key Handoff: Retrieval in Hybrid Language Models

Authors: Kaan Kale, Oguzhan Baser, Sriram Vishwanath

Organizations: Georgia Institute of Technology, USA · University of Texas at Austin, USA

Abstract

A two-hop question makes a language model retrieve twice: once to produce a bridge entity, and once to retrieve with it. Transformers resolve that entity in their early layers. Hybrid models replace most of the attention with a recurrent state, so where the key becomes usable, and where it is spent on an answer, is not known. Answering "Where is the ball?" from "the ball belongs to Alice" and "Alice is in the garden" turns on Alice, a name the question does not mention. A model could reach garden through Alice, the key it computed, or through where the fact sits in the prompt. Across twelve models, dense and hybrid, we move a hidden state from one story into another where the two routes lead to different places, and read off which place the model gives. An attention layer converts the key in every model we tested, and crossing that layer removes its usable effect, all of it where no attention follows. In sequential hybrids this makes the answer a handoff: recurrent layers carry the key forward, and attention spends it. Retrieval does not always stop there. Writing a different fact into the memory of the recurrent layers that follow the last attention layer can move the answer toward that fact, multiplying its odds by 1.3 to 2.7, and which hybrids do this is not settled by their architecture or training. That read is addressed by the key, not by position: recurrent state in a hybrid is not only a carrier, but a memory that later layers can query.

Figures & tables

Panel of Figure 1 — Figure 1: The handoff, and a pair that measures it. (a) In a sequential hybrid the final token’s stream is hatched while it carries the key to an attention layer that reads it, and solid once it carries the answer; after the last attention layer a recurrent layer can retrieve a binding from its own memory. (b) The two prompts share a row, so the donor’s carried answer, that row, and the key it computes point at three different places; hall is predicted by none and serves as the reference. Only the donor’s final-token state moves, into the same layer input of the recipient. Names and places stand in for the arrow task’s keys and numbers; the literal prompts are in Appendix A .

Figure 2: Key availability rises before the main answer-conversion step. Transplants at every layer input in four hybrids and two dense transformers; 200 identities with 97.5% identity-bootstrap intervals. The probe chooses among the donor identity’s three keys, so its chance level is one in three. Dense panels carry no attention markers because every one of their layers has attention. The resolving layer’s step makes the key effect negative in the three hybrids with four attention layers (Section 3.2 ; Appendix Table 16 ).

Top-1 among seven (%), chance 14
	Nemotron-H	Granite-H	Nano
rebinding	86	71	67
row mismatch	39	35	28
Causal scrubbing: donor shares only … ( $\times$ )
the key	3.42	3.18	3.29
its own answer	0.62	0.62	0.61

Table 1: The transferred state carries a key, and a row pointer with it. The three tests of this section at the resolving layer’s input (Section 3.2 ), 200 identities each. Top-1 is how often the key’s place wins among seven candidates; the scrubbing rows give the factor by which the key’s answer gains over the recipient’s other candidates,

\times 1

being no change. Constructions, intervals, and the preregistered form of the scrub test are in Appendix B .

Figure 3: The conversion uses the attention available in a late band of depth. Each row is a model, filled circles the arrow task and hollow squares the binding task, with 95% identity-bootstrap intervals. Left: the effective attention count carrying the rise,

e^{H}

. Middle: the share of it carried by the single largest attention layer,

C_{\rm attn}

. Right: the share falling on attention layers at all, which only the sequential hybrids make informative. Appendix Table 34 gives the first two against a uniform allocation.

Figure 4: Recurrent outputs carry the key to the reader. Both lookups after the swap, each as a fraction of the lookup its own transplant produces, with 95% identity-bootstrap intervals;

g

and

h

of Equation 4 . Left: Bob’s lookup created, all nominated layers against all matched controls. Middle: Alice’s lookup left standing, the same two sets, which is where they separate. Right: each nominated layer alone (points) against the joint swap (bar), which can fall below zero when one layer moves the lookup toward Alice. Appendix Table 17 reports both beside the redirection.

	8-head	All heads	Random sets
	injection	clamped	beaten
Nemotron-H (L40)	$\approx$ 100%	$\approx$ 0%	100%
Granite-H (L35)	69–75%	$\approx$ 0%	100%
Nano (L33)	93–96%	36–39%	100%

Table 2: A few heads read the key. Columns give the share of the transplant’s key effect reproduced by injecting the eight heads’ changes, the share surviving a clamp of all the layer’s heads, and the fraction of random eight-head sets beaten; ranges span rebinding and row mismatch (Appendix Tables 20 – 22 ).

	Test	Control	Transposed
Alice	Garden	Garden	Kitchen
Bob	Kitchen	Kitchen	Garden
Carol	Hall	Hall	Hall
Question	Where is the ball?	Where is the cup?	Where is the cup?
Own answer	Garden	Hall	Hall

Table 7

Figure 5: Changing a stored location moves the answer in four hybrids. Left and middle: geometric-mean factors for kitchen’s probability against hall (

e^{\sigma_{\rm own}}

) and against garden (

e^{\sigma_{\rm test}}

), on logarithmic axes starting at

\times 1

. Both are read under ordinary execution; shade gives the task version, word or paragraph. Right: the change in how often the memory’s place ranks first, on the word task, with color giving the incoming state, ordinary or skipped. Intervals are 95% over 200 identities; Appendix Table 29 lists every model and version tested.

	Nemotron-H	Granite-Small
Key over row	$\times$ 1.9	$\times$ 1.4
Key over test prompt’s	$\times$ 2.4	$\times$ 1.5
Row over test prompt’s	$\times$ 1.3	$\times$ 1.1

Table 3: The answer follows the key. Geometric-mean factors after rotating the key rows, in ordinary execution; 200 identities per model. Intervals, the share of identities, and the construction are in Appendix Table 27 .

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Intervention	Data	Role
Flow localization	Residual patch	20 paired	Exploratory
Head ranking	Head-output patch	30 discovery + 300 held-out	Confirmatory
Fixed-donor crossing	Whole-state transplant	200 graphs / 600 crosses	Confirmatory
Fixed-recipient crossing	Whole-state transplant	200 graphs / 600 crosses	Confirmatory
Common assay: factorial	Whole-state transplant	200 identities / 1,800 cells	Confirmatory
Common assay: path patching	Head-delta injection, clamps	Same 200 identities	Confirmatory

Appendix

Table 4: Experiments, data, and inferential roles.

Symbol	Meaning
$x_{d}$ , $x_{r}$ ; $h_{l}(x)$	Donor and recipient prompts; the answer-position state entering layer $l$ (“pre-L $l$ ”)
$h_{l}$ , $o_{l}$ , $m_{l}$	At the final token: the residual state entering layer $l$ ; the output a recurrent layer adds to it ( $h_{l+1}=h_{l}+o_{l}$ ); the recurrent layer’s memory, accumulated over the preceding tokens. Transplants replace $h_{l}$ , writer swaps $o_{l}$ , memory swaps $m_{l}$
$\Delta\ell(c)$	Change in the logit of candidate $c$ when the donor’s state replaces the recipient’s (Eq. 1 )
$A$ ; $B_{1},B_{2},B_{3}$ ; $A_{r}$	The donor’s answer; the values the recipient binds to the three keys; the recipient’s own answer
$\mathcal{A}$ , $\mathcal{P}$ , $\mathcal{K}$	The fixed-answer, position, and key causal models
$S(B_{j})$ ; $S^{\rm map}$ , $S^{\rm con}$	Selectivity of a candidate over the other two bindings; mapping (fixed-donor) and key (fixed-recipient) selectivity (Eq. 8 )

Appendix

Table 5: Notation.

Checkpoint	Revision	Class	Layers	Task	Site	Outcome
nvidia/Nemotron-H-8B-Base-8K	94ea861e	H	52	Pass	pre-L40	Primary model
ibm-granite/granite-4.0-h-micro-base	372ede0b	H	40	Pass	pre-L35	Both crossings pass
ibm-granite/granite-4.0-h-small-base	4b4faa80	H	40	Pass	—	Memory swap only
Qwen/Qwen2.5-7B	d1497293	D	28	Pass	pre-L22/26 ‡	Balanced / reference
mistralai/Mistral-7B-v0.3	caa1feb0	D	32	Pass	pre-L19	Profile panel
Zyphra/Zamba2-2.7B	31afeeac	H	54	Fail §	—	Below accuracy gate

Appendix

Table 6: Model panel. H: sequential hybrid; P: parallel hybrid (attention in every layer); D: dense Transformer. “Task: Fail” means unmodified accuracy fell below the fixed threshold; § evaluated with the model’s beginning-of-sequence token after failing without it (Bamba-9B-v2 and both Zamba2 checkpoints fail both ways). For the profile panel the site column gives the layer with the largest key effect, and Table 14 gives each model’s concentration and effective attention count. Crossed-assay sites were chosen by the prespecified screen, and every prespecified site and threshold is in the frozen configuration files whose SHA-256 the supplement lists; ‡ Qwen2.5’s two sites were taken from the screen’s descriptive audit and fixed before its factorial sample, and the depth sweep covers every layer.

Quantity (logits)	Crossed assay	Common assay	Normalized
Recipient margin, $A_{r}$ vs. runner-up	1.45	1.71 $\pm$ 0.21	1
Mapping selectivity, $F$	2.37	1.48 $\pm$ 0.29	0.88 $\pm$ 0.21
Key selectivity, $F$	1.02	1.14 $\pm$ 0.10	0.66 $\pm$ 0.09
Mapping selectivity, pre-L39	—	1.12 $\pm$ 0.20	0.67 $\pm$ 0.15
Key selectivity, pre-L39	—	0.85 $\pm$ 0.08	0.50 $\pm$ 0.07
$F$ minus pre-L39, mapping	—	0.36 $\pm$ 0.09	—

Appendix

Table 7: Scale across assays. Normalized values divide selectivity by the recipient’s own

A_{r}

-versus-runner-up margin at the identity level. The pre-L39 whole-residual transplant is the matched-layer control.

	Donor’s decisive row at position
Donor’s key	1 (Alice’s fact)	2 (Bob’s fact)	3 (Carol’s fact)
Alice	Garden / Garden	Garden / Kitchen	Garden / Hall
Bob	Kitchen / Garden	Kitchen / Kitchen	Kitchen / Hall
Carol	Hall / Garden	Hall / Kitchen	Hall / Hall

Appendix

Table 8: The factorial separates the key from the position. The recipient binds Alice, Bob, and Carol to garden, kitchen, and hall, whose facts occupy rows 1, 2, and 3. Each cell lists the location key-based retrieval predicts / the location row-based retrieval predicts. On the diagonal (gray) the two coincide; on the six off-diagonal cells they differ, and the remaining third location is the reference candidate.

Panel of Figure 6 — Figure 6: Three crossed constructions. (a) One donor state enters three recipients that bind the same key $m$ to $B_{1},B_{2},B_{3}$ . (b) Three donors whose queries reach different keys $m_{j}$ enter one recipient; the donor row $m_{j}\to A$ is placed where the recipient holds $m_{j+1}\to B_{j+1}$ , and the recipient maps the donor’s query $q_{j}$ away from $B_{j}$ (to a dead end in most cells, Appendix A ), so neither the query token nor the row position predicts $B_{j}$ . (c) Donor key and donor row position vary independently; diagonal cells are excluded. Colors: orange donor path, teal recipient bindings under test, red adversarially aligned row, blue the recipient’s own path; the lower-right panel lists each account’s predicted answer.

		Nemotron-H, pre-L40	Granite-H, pre-L35
Fixed donor	Mapping selectivity (logits)	2.37 $\pm$ 0.21	2.48 $\pm$ 0.22
	Mapped vs. donor answer $A$ (logits)	1.57 $\pm$ 0.19	1.67 $\pm$ 0.24
	Mapped vs. recipient answer $A_{r}$ (logits)	4.17 $\pm$ 0.27	4.84 $\pm$ 0.31
	Tracking rate (%)	80.8 $\pm$ 4.5	86.0 $\pm$ 3.8
	Top-1 rate among seven (%)	85.8 $\pm$ 4.5	70.8 $\pm$ 5.5
Fixed recipient	Key selectivity (logits)	1.02 $\pm$ 0.12	0.99 $\pm$ 0.14

Appendix

Table 9: Crossed assay: 200 held-out graphs (600 crosses) per construction and model, with 95% intervals; chance tracking is 33.3%. The unmodified Nemotron-H recipient’s margin between

A_{r}

and its runner-up is 1.45 logits. ∗ Accuracy when the recipient is queried from

m_{j}

directly. Unmodified accuracies are in Appendix Table 10 .

		Nemotron-H	Granite-H
Fixed-donor crossing	Unmodified donor accuracy (%)	88.5	82.0
	Unmodified recipient accuracy (%)	71.7	84.7
	Donor prefers $A$ over $A_{r}$ (%)	95.5	—
	Recipient prefers $A_{r}$ over $A$ (%)	92.2	—
Fixed-recipient crossing	Unmodified donor accuracy (%)	85.5	82.0
	Unmodified recipient accuracy (%)	82.0	84.7

Appendix

Table 10: Unmodified accuracies (seven-candidate top-1 on the held-out graphs) and Granite-H layer-level mediation for the crossed assay. Granite-H used one held-out sample for both constructions; its mediation totals come from the mediation sample and differ slightly from Table 9 .

Nemotron-H	Attention	Recurrent, before L40	Recurrent, after L40
Arrow task	0.947	0.036	0.016 $\pm$ 0.005
Word task	0.862	0.046	0.092 $\pm$ 0.010
Numeric twin	0.936	0.036	0.028 $\pm$ 0.005

Appendix

Table 11: Where the rise in the donor-answer effect falls, by task. Shares of the whole rise, with 95% identity-bootstrap intervals on the post-attention share, the one that separates the tasks: it is higher on the word task than on the arrow task by 0.075

\pm

0.011 and than on the numeric twin by 0.064

\pm

0.012. The word task is the one on which Section 5 finds the memory read and the numeric twin the one on which it does not. Within the word task’s post-attention band the rise concentrates in L44 (0.044 of the whole, against 0.024, 0.017, 0.006 and 0.001 for L42, L46, L50 and L48), the layer whose output clamp costs the post-attention lookup most among all five clamped individually, on two identity pools (Appendix F ).

Site	Key $\kappa$	Position $\pi$	$\kappa-\pi$	Class
Nemotron-H pre-L40	1.13 $\pm$ 0.12	0.36 $\pm$ 0.10	0.77 $\pm$ 0.11	Key-dominant
Nemotron-H pre-L39	0.84 $\pm$ 0.09	0.28 $\pm$ 0.07	0.56 $\pm$ 0.09	Key-dominant
Qwen2.5 pre-L22	0.86 $\pm$ 0.16	0.87 $\pm$ 0.14	$-$ 0.02 $\pm$ 0.14	Balanced
Qwen2.5 pre-L26	$-$ 0.63 $\pm$ 0.12	0.02 $\pm$ 0.11	$-$ 0.65 $\pm$ 0.09	Neither †
Granite-H pre-L35	1.07 $\pm$ 0.13	0.25 $\pm$ 0.09	0.82 $\pm$ 0.10	Key-dominant
Nemotron-H pre-L40 (colon format)	0.81 $\pm$ 0.12	0.31 $\pm$ 0.08	0.49 $\pm$ 0.09	Key-dominant

Appendix

Table 12: Key

\times

position factorial, in logits, on 200 held-out graphs (1,200 off-diagonal cells) per site.

\kappa

and

\pi

are the key-vs-reference and position-vs-reference contrasts of Eq. 2 ; they carry 97.5% intervals; the difference carries the 90% interval used for the prespecified

\pm 0.30

balanced region. Every key-dominant row is also dual-supported. The Granite-H class and the second-format class were predicted in advance. † The key is suppressed below the reference candidate and position does not reliably exceed it.

Model	No class	Balanced	Key-dominant
Nemotron-H	pre-L0–L26; pre-L41–L51 (fixed answer)	pre-L27, L28, L31–L33	pre-L29, L30, L34–L40
Granite-H	pre-L0–L24; pre-L36, L39 (fixed answer)	—	pre-L25 § , L26–L35
Qwen2.5	pre-L0–L17; pre-L24, L26, L27 (fixed answer)	pre-L18, L19, L21–L23	pre-L20

Appendix

Table 13: Prespecified class at every layer input. Position-dominant labels occur only in the fixed-answer regime (Granite-H pre-L37, L38; Qwen2.5 pre-L25), where the key lies below the reference candidate and the positional contrast is about 0.1 logits. § Key-dominant without dual support. Source and recipient seven-way accuracies were 84/84% (Nemotron-H), 77/81% (Granite-H), and 89/91% (Qwen2.5). Consecutive layers separated only by a position-wise MLP return identical values because the MLP is a deterministic function of the transplanted final-token state.

Figure 7: Ridge-probe decodability from the donor’s answer-position residual at each layer input in Nemotron-H (site 52 is the final layer output), with 95% identity-clustered intervals; dotted lines mark chance and dashed lines the attention layers. Shuffled-label probes stay at chance at every site.

Model	Type	Layers	Key peak (class)	$C$ , arrow	$C$ , binding	$e^{H}$ , arrow / binding	Attn. share
Nemotron-H-8B	Sequential hybrid	52 (4 attn.)	pre-L40 (key)	0.68 $\pm$ 0.03	0.69 $\pm$ 0.02	1.9 / 1.9	0.95
Nemotron-H-4B	Sequential hybrid	52 (4 attn.)	pre-L40 (key)	0.63 $\pm$ 0.03	0.68 $\pm$ 0.02	1.9 / 1.8	0.94
Granite-H	Sequential hybrid	40 (4 attn.)	pre-L35 (key)	0.70 $\pm$ 0.02	0.55 $\pm$ 0.02	1.8 / 2.0	0.96
Falcon-H1-3B	Parallel hybrid	32 (all)	pre-L21 (key)	0.59 $\pm$ 0.02	0.56 $\pm$ 0.02	4.6 / 4.8	1
Falcon-H1-7B	Parallel hybrid	44 (all)	pre-L29 (balanced)	0.70 $\pm$ 0.02	0.73 $\pm$ 0.01	3.2 / 3.0	1
Qwen2.5-7B	Dense	28	pre-L22 (balanced)	0.61 $\pm$ 0.02	0.54 $\pm$ 0.02	2.9 / 3.2	1

Appendix

Table 14: The model panel behind Figure 3 : concentration

C

, the largest single layer’s share of the rise in the donor-answer effect over all layers (95% intervals), on both tasks; the effective attention count

e^{H}

carrying that rise; attention’s share of the rise (the summed positive increments at attention layers divided by the sum over all layers); and the site and class of the largest key effect

\kappa

on the arrow task.

Figure 8: Depth profiles of the twelve models in Table 14 , in the format of Figure 2 . Dashed lines mark attention layers in the sequential hybrids; the parallel hybrids and dense models have attention in every layer.

Figure 9: Depth profiles on the binding task of Section 2 for the models of Table 14 that passed the binding-task gate, in the format of Figure 8 .

Task	Model	Peak = resolver	Coincide (%)	Peak margin over runner-up
Arrow	Nemotron-H-8B	L40	100.0	0.29 $\pm$ 0.04 (L38)
Arrow	Nemotron-H-4B	L40	100.0	0.18 $\pm$ 0.02 (L39)
Arrow	Granite-H	L35	99.4	0.01 $\pm$ 0.01 (L33)
Arrow	RecurrentGemma-9B	L29	100.0	0.40 $\pm$ 0.07 (L28)
Arrow	Nemotron-3-Nano-30B	L33	100.0	0.14 $\pm$ 0.04 (L31)
Arrow	Falcon-H1-3B	L21	97.8	0.02 $\pm$ 0.01 (L20)

Appendix

Table 15: Stability of peak alignment under resampling of identities (2,000 replicates per profile). “Coincide” is the share of replicates in which the layer input with the largest key effect is the input of the layer with the largest donor-answer increment. The margin is the peak key effect minus the largest key effect at any other layer input (the runner-up, in parentheses), with its 95% interval.

			Arrow task						Binding task
Model	Attn. blocks	Late attn.	Rise span	Depth	Attn. in span	$e^{H}$	$\kappa$ in	Consumed	Rise span	Depth	Attn. in span	$e^{H}$	$\kappa$ in	Consumed
Nemotron-H-8B	4	2	L29–L40	0.57–0.78	2	1.9	1.13	1.55	L29–L40	0.57–0.78	2	1.9	0.70	1.30
Nemotron-H-4B	4	2	L29–L40	0.57–0.78	2	1.9	0.66	1.69	L29–L40	0.57–0.78	2	1.8	0.77	1.39
Granite-H	4	2	L25–L35	0.64–0.90	2	1.8	1.07	1.29	L25–L35	0.64–0.90	2	2.0	0.49	1.43
RecurrentGemma-9B	12	6	L26–L31	0.70–0.84	2	2.0	0.83	0.56	—	—	—	—	—	—
Nemotron-3-Nano-30B	6	3	L33–L42	0.65–0.82	2	1.7	1.06	0.85	L33–L42	0.65–0.82	2	1.9	0.83	0.77

Appendix

Table 16: Conversion band per profile: the layer span carrying the middle 80% of the cumulative positive rise in the donor-answer effect, its position as a fraction of depth, the attention layers inside it, and the effective attention count

e^{H}

; “

\kappa

in” is the key effect entering the resolving layer and “Consumed” its drop across that layer as a fraction of that value. “Late attn.” counts attention layers in the second half of the network. RecurrentGemma-9B’s binding run failed its gate.

Condition	Redirection	Guest lookup	Host lookup	Shared key kept	Host frame
Nemotron-H-8B : reader L40, 8 of 32 heads, gate 84/84%
Writer L33 swapped	0.04 $\pm$ 0.03	0.04 $\pm$ 0.04	0.95 $\pm$ 0.06	1.00 $\pm$ 0.02	$-$ 0.02 $\pm$ 0.04
Control L35 swapped	0.13 $\pm$ 0.02	0.14 $\pm$ 0.02	0.87 $\pm$ 0.05	1.00 $\pm$ 0.01	0.13 $\pm$ 0.04
All writers L28/L33/L39 swapped	0.58 $\pm$ 0.05	0.31 $\pm$ 0.09	0.10 $\pm$ 0.20	0.99 $\pm$ 0.03	0.32 $\pm$ 0.14
Control set L31/L35/L37 swapped	0.26 $\pm$ 0.03	0.26 $\pm$ 0.03	0.72 $\pm$ 0.09	1.00 $\pm$ 0.02	0.24 $\pm$ 0.05
Residual after L33 replaced	0.69 $\pm$ 0.02	0.74 $\pm$ 0.03	0.33 $\pm$ 0.17	0.99 $\pm$ 0.04	0.80 $\pm$ 0.13

Appendix

Table 17: Writer swap. Redirection of the recipient’s lookup from the host’s key to the guest’s, which normalizes the guest-minus-host contrast between the two transplants (0 = host transplant, 1 = guest transplant), the guest and host lookups

g

and

h

of Equation 4 , the shared-key lookup retained under answer-swap guests, and the host’s own logit shift toward the guest key’s value; 95% identity-bootstrap intervals over 200 identities.

Swap	Guest lookup	Host lookup	Paired guest-lookup difference
Nemotron-H-8B : writers L28/L33/L39, controls L31/L35/L37, reader L40; pattern: distributed
L28 alone	0.12 $\pm$ 0.04	0.78 $\pm$ 0.08	vs. L31: 0.16 $\pm$ 0.05
Control L31 alone	$-$ 0.04 $\pm$ 0.02	0.98 $\pm$ 0.04
L33 alone	0.04 $\pm$ 0.04	0.95 $\pm$ 0.06	vs. L35: $-$ 0.10 $\pm$ 0.05
Control L35 alone	0.14 $\pm$ 0.02	0.87 $\pm$ 0.05
L39 alone	0.14 $\pm$ 0.03	0.86 $\pm$ 0.04	vs. L37: $-$ 0.03 $\pm$ 0.03

Appendix

Table 18: Single-writer and leave-one-out swaps on the writer-swap pairs. Guest and host lookups as fractions of the guest and host transplants’ (95% identity-bootstrap intervals); the last column gives the paired guest-lookup difference used by the classification rules.

Figure 10: A recurrent supplier contributes to Nano’s later lookup. (a) Host-key-vs-reference lookup at the two attention-layer inputs; the paired later excess over the no-key control is positive. (b) Paired supplier interruption, specificity, and rescue effects. (c) Answer-logit loss under head clamps before and after disruption; the annotated interaction tests increased L42 dependence. Intervals resample identities (95%).

Paired contrast	Estimate (95%)
Later excess over no-key	0.242 $\pm$ 0.097
Q1: L37 interruption drop	0.154 $\pm$ 0.061
L39 comparison drop	$-$ 0.005 $\pm$ 0.017
Q2: L37 versus L39 drop	0.159 $\pm$ 0.057
Q3: rescue improvement	0.087 $\pm$ 0.050
Rescue shortfall from uninterrupted	0.067 $\pm$ 0.039

Appendix

Table 19: Nano late-supplier follow-up on the original delay pool. Recipient key lookup is the host-key-vs-reference logit-change contrast; host clamp loss is the change in the host answer’s logit. All entries are paired effects with 95% identity-bootstrap intervals. Q3 also has a separately frozen point-tolerance criterion, described in the text. The last three rows are the ratios the main text quotes, resampled as ratios, so their intervals are asymmetric.

Figure 11: Eight attention heads transmit most of the lookup effect. Nemotron-H, L40, 200 identities, 95% intervals. The two panels change recipient bindings or donor keys, respectively; selectivity is the predicted value’s score increase minus the mean increase of the other two recipient-bound values. Dashed lines show the whole-state transplant. Injecting the eight selected heads’ changes nearly reproduces it; holding their outputs fixed sharply reduces it. Detailed values and Granite-H’s comparison are in Appendix Tables 20 and 21 .

Condition	What carries the transplant	Mapping selectivity	Key selectivity	Top-1 (%)
$F$	Whole state	1.48 $\pm$ 0.29	1.14 $\pm$ 0.10	42
$P_{\rm T}$	Top-8 head outputs only	1.46 $\pm$ 0.28	1.18 $\pm$ 0.10	40
$P_{\rm all}$	All 32 head outputs	1.48 $\pm$ 0.27	1.20 $\pm$ 0.10	41
$P_{\bar{\rm T}}$	Other 24 head outputs	0.22 $\pm$ 0.06	0.17 $\pm$ 0.03	4
$P_{\rm B}$	Bottom-8 head outputs	$-$ 0.08 $\pm$ 0.03	$-$ 0.07 $\pm$ 0.01	3
$R_{\rm by}$	Residual; heads clamped to recipient	0.01 $\pm$ 0.04	$-$ 0.10 $\pm$ 0.02	3

Appendix

Table 20: Path patching at L40 in Nemotron-H on the common assay (200 identities), in logits with 95% intervals. Top-1 is the seven-way rate for the predicted candidate; the unmodified recipient scores 4%.

R_{\rm by}+P_{\rm all}

reproduces

F

to zero error.

Condition	What carries the transplant	Mapping selectivity	Key selectivity	Top-1 (%)
$F$	Whole state	1.44 $\pm$ 0.29	1.09 $\pm$ 0.10	38
$P_{\rm T}$	Top-8 head outputs only	1.00 $\pm$ 0.20	0.81 $\pm$ 0.08	25
$P_{\rm all}$	All 32 head outputs	1.42 $\pm$ 0.27	1.15 $\pm$ 0.10	41
$P_{\bar{\rm T}}$	Other 24 head outputs	0.52 $\pm$ 0.11	0.41 $\pm$ 0.05	8
$P_{\rm B}$	Bottom-8 head outputs	$-$ 0.14 $\pm$ 0.04	$-$ 0.12 $\pm$ 0.02	4
$R_{\rm by}$	Residual; heads clamped to recipient	0.04 $\pm$ 0.06	$-$ 0.07 $\pm$ 0.04	3

Appendix

Table 21: Path patching at L35 in Granite-H on the common assay, in the format of Table 20 . Head sets were ranked on a 30-identity discovery pool and frozen before any test identity; the random bank holds 255 eight-head sets. The top-eight route carries 69%

\pm

6 of

F

’s mapping and 75%

\pm

4 of its key selectivity, the rest spread across the other heads; it beats the bottom eight by 1.14 and 0.93 logits and every random set (

p=1/256

);

R_{\rm by}+P_{\rm all}

reproduces

F

to zero error.

Condition	What carries the transplant	Mapping selectivity	Key selectivity	Top-1 (%)
$F$	Whole state	1.18 $\pm$ 0.25	1.11 $\pm$ 0.13	28
$P_{\rm T}$	Top-8 head outputs only	1.14 $\pm$ 0.26	1.04 $\pm$ 0.13	26
$P_{\rm all}$	All 32 head outputs	1.01 $\pm$ 0.23	0.93 $\pm$ 0.12	18
$P_{\bar{\rm T}}$	Other 24 head outputs	$-$ 0.14 $\pm$ 0.04	$-$ 0.12 $\pm$ 0.02	3
$P_{\rm B}$	Bottom-8 head outputs	$-$ 0.17 $\pm$ 0.05	$-$ 0.14 $\pm$ 0.02	3
$R_{\rm by}$	Residual; heads clamped to recipient	0.43 $\pm$ 0.10	0.43 $\pm$ 0.06	4

Appendix

Table 22: Path patching at L33 in Nemotron-3 Nano on the common assay, in the format of Table 20 . Head sets were ranked on a 30-identity discovery pool and frozen before any test identity (L33 heads 7, 5, 11, 0, 4, 1, 15, 13). The top-eight route carries 96%

\pm

7 of

F

’s mapping and 93%

\pm

4 of its key selectivity, exceeds the bottom eight by 1.31 and 1.18 logits, and exceeds every one of 63 random eight-head sets (

p=1/64

). Because Nano has a second attention layer after L33 (L42), the residual bypass keeps part of the effect.

Model	shares the key	shares the answer	shares the row	over answer	over row
	Key’s answer against the recipient’s others ( $\times$ )			Preservation advantage
Nemotron-H	3.42 $\pm$ 0.35	0.62 $\pm$ 0.04	0.47 $\pm$ 0.02	0.72 $\pm$ 0.11	0.86 $\pm$ 0.11
Granite-H	3.18 $\pm$ 0.35	0.62 $\pm$ 0.04	0.51 $\pm$ 0.03	0.97 $\pm$ 0.12	1.08 $\pm$ 0.11
Nemotron-3 Nano	3.29 $\pm$ 0.42	0.61 $\pm$ 0.05	0.52 $\pm$ 0.04	1.14 $\pm$ 0.15	1.12 $\pm$ 0.14

Appendix

Table 23: Preserving the key better preserves the transplant’s effect. Left: after the donor state is resampled from donors that share the named factor, the factor by which the key’s answer gains over the recipient’s other candidates, the exponentiated contrast reported in Table 1 , with 95% intervals;

\times 1

is no change. Right: the preregistered test, the preservation advantages of key-preserving over answer- and row-preserving resampling (Eq. 10 ). Positive values indicate a reduction in centered mean-squared error relative to the original whole-state transplant.

Family	Unit	Learned d.o.f.	Held-out outcome on the common assay
Crossed transplant	Complete residual	0	Key-dominant, dual-supported; mapping 1.48, key 1.14
Path patching	Head-output route	0 (set prespecified)	Top-8 route carries 98–104% of $F$ in Nemotron-H, 93–96% in Nano and 69–75% in Granite-H
Causal scrubbing	Factor-preserving resample	0	Key-preserving scrub preserves, answer- and row-preserving do not, in all three hybrids

Appendix

Table 24: The three families on identical held-out identities. Method-specific questions differ, so the outcome column is not a ranking.

Study	Manipulation	What it identifies	What crossing adds
Feng & Steinhardt (2024)	Binding-ID vectors added to, or swapped between, entity–attribute pairs	A binding subspace shared across entities and its geometry	A reference baseline and a key $\times$ position factorial, so that “binding” splits into the bound value and the bound row, each with a preregistered bound
Gur-Arieh et al. (2026)	Counterfactual prompt pairs that vary one retrieval cue (positional, lexical, reflexive) with activation patching	Which retrieval mechanism a model uses at a site, and that the mechanisms mix	Computed-key assembly in hybrids compared with dense models; a reference baseline and prespecified factorial classification
Sharma et al. (2026)	Patching a predicate (filter) representation between list prompts	A general filter-head circuit that selects list items	The key and position of a bound variable rather than a predicate, and three intervention families compared on one assay

Appendix

Table 25: Relation to prior manipulations of in-context binding. Earlier binding studies also use counterfactual pairs and whole-state interventions. Our comparisons extend that approach to computed keys, hybrid architectures, and shared outcomes across intervention families.

Condition or paired contrast	Original (95%)	Fresh template (95%)
Shifted lookup	0.418 $\pm$ 0.063	0.126 $\pm$ 0.066
All five outputs clamped	$-$ 0.012 $\pm$ 0.023	0.004 $\pm$ 0.032
L42/L44 clamped	0.111 $\pm$ 0.045	0.010 $\pm$ 0.049
L46/L48 clamped	0.290 $\pm$ 0.053	0.104 $\pm$ 0.057
Same-key, different-answer rescue	0.406 $\pm$ 0.057	0.117 $\pm$ 0.060
Different-key, same-answer control	$-$ 0.041 $\pm$ 0.050	$-$ 0.053 $\pm$ 0.046

Appendix

Table 26: Nemotron-H suffix localization and cross-donor rescue, and their prospective replication on fresh graphs under a new template. Key-vs-reference logit changes and paired differences, with 95% identity-bootstrap intervals (200 identities per pool). Clamps use recipient baselines; rescue/control use different shifted donors. All six primary criteria pass in each pool.

Figure 12: Key-specific rescue of the post-attention lookup. Nemotron-H, key effect

\kappa

in logits with 95% identity-bootstrap intervals. (a) Shifted lookup and collective Mamba clamp on fresh word/numeric twins. (b) The nominated pair, comparison pair, and cross-donor rescue in the original word pool and a fresh pool under a new template, with all sites frozen. Both support all six primary contrasts.

Model	Incoming state	Key over row	Key over test prompt’s	Row over test prompt’s	Identities, key $>$ row
Nemotron-H	Ordinary execution	0.65 $\pm$ 0.04	0.89 $\pm$ 0.05	0.24 $\pm$ 0.03	98%
	Shifted	0.82 $\pm$ 0.05	1.00 $\pm$ 0.07	0.18 $\pm$ 0.03	100%
Granite-Small	Ordinary execution	0.31 $\pm$ 0.03	0.40 $\pm$ 0.02	0.09 $\pm$ 0.01	98%
	Shifted	0.38 $\pm$ 0.03	0.48 $\pm$ 0.03	0.10 $\pm$ 0.01	100%

Appendix

Table 27: The crossed memory, which writes the same bindings into key rows rotated by one. Contrasts are logit changes of the crossed memory against the control memory, differenced between two candidates, with 95% identity-bootstrap intervals over 200 identities; the last column is the share of identities in which the place the key is bound to gains more than the place now in the key’s row.

Template	Execution	All five memories	L44 memory only
Original	Shifted	$0.331\pm 0.036\;/\;0.674\pm 0.065$	$0.187\pm 0.028\;/\;0.369\pm 0.042$
Original	Ordinary	$0.704\pm 0.045\;/\;0.978\pm 0.057$	$0.385\pm 0.034\;/\;0.537\pm 0.041$
Fresh wording	Shifted	$\mathbf{0.111}\pm 0.022\;/\;\mathbf{0.233}\pm 0.036$	$\mathbf{0.064}\pm 0.013\;/\;\mathbf{0.118}\pm 0.020$
Fresh wording	Ordinary	$\mathbf{0.375}\pm 0.027\;/\;\mathbf{0.530}\pm 0.035$	$\mathbf{0.202}\pm 0.018\;/\;\mathbf{0.285}\pm 0.021$

Appendix

Table 28: Recurrent-memory effects for all five layers and L44 alone. Each cell gives the supplied value over the memory prompt’s own answer / over the recipient’s value, in logits with 95% intervals. Bold marks the fresh-wording replication. The all-five-layer results also appear with separate contrast columns in Table 29 .

Model (recurrent layers after last attention)	Task version	(S / R / M, %)	Shifted	Ordinary	Shifted	Ordinary	identities (%)	Read
		Accuracy	$\sigma_{\rm own}$ : over the memory prompt’s answer		$\sigma_{\rm test}$ : over the test prompt’s answer		Consistent
Nemotron-H-8B (5 Mamba-2)	Word	71 / 68 / 69	0.33 $\pm$ 0.04	0.70 $\pm$ 0.04	0.67 $\pm$ 0.07	0.98 $\pm$ 0.06	100	Both conditions
	Story	64 / 61 / 63	0.11 $\pm$ 0.02	0.37 $\pm$ 0.03	0.23 $\pm$ 0.04	0.53 $\pm$ 0.03	99	Both conditions
	Paragraph	71 / 66 / 70	0.44 $\pm$ 0.04	0.79 $\pm$ 0.05	0.89 $\pm$ 0.09	1.09 $\pm$ 0.06	100	Both conditions
	Short word	76 / 74 / 78	0.33 $\pm$ 0.04	0.78 $\pm$ 0.04	0.68 $\pm$ 0.08	1.05 $\pm$ 0.06	99	Both conditions
	Numeric twin	88 / 87 / 86	0.00 $\pm$ 0.01	0.00 $\pm$ 0.00	0.00 $\pm$ 0.01	0.00 $\pm$ 0.01	18	Not detected

Appendix

Table 29: Every matched-memory swap that was run. All recurrent memories after the last attention layer are replaced; 200 fresh identities per row; accuracy is unmodified seven-candidate top-1 for the source, recipient, and memory prompts (S / R / M). Scores are

\sigma_{\rm own}

and

\sigma_{\rm test}

(Equation 6 ) in logits;

e^{\sigma}

is the geometric-mean probability-ratio factor of Figure 5 . “Consistent identities” is the share of identities whose ordinary

\sigma_{\rm own}

and

\sigma_{\rm test}

are both positive. “Read” applies one rule to every row: both scores have a 95% lower bound above zero in that condition. The two conditions here are ordinary execution and the shifted state, which is the pair this panel was preregistered under; Section 5 reports the skipped state as its second condition, and the two meet in the ordinary column. † The shifted state failed its validity check, so that condition is not scored. Task versions: “Story” uses a second sentence template, “Paragraph” writes the facts as one paragraph, “Short word” is the word task without its three distractor facts, and “present tense” states its facts in the present tense. Checkpoints and revisions are in Table 6 ; RecurrentGemma-9B runs in float32 for this experiment.

	Increase in supplied location’s score relative to
Incoming state after the last attention layer	Memory’s own answer	Recipient’s answer
Nemotron-H : both factors contribute
Ordinary execution	0.735 $\pm$ 0.048	1.006 $\pm$ 0.057
Own pre-attention state, attention skipped	0.678 $\pm$ 0.046	1.241 $\pm$ 0.075
Foreign prompt’s pre-attention state	0.365 $\pm$ 0.045	0.740 $\pm$ 0.080
Ordinary minus own state (attention ran)	+0.058 $\pm$ 0.014	$-$ 0.236 $\pm$ 0.030

Appendix

Table 30: Bypass-source experiment: transposed-minus-control change in the supplied location’s score, relative to the memory prompt’s own answer and to the recipient’s answer, under three incoming states after the last attention layer, with the paired differences between them; 95% identity-bootstrap intervals over 200 identities per model.

Condition (final token only)	Baseline (%)	Intervened (%)	Paired drop (points, 95%)	Correct $\to$ wrong	Wrong $\to$ correct
Word wording : source 70.7%, recipient 62.0%, 100 identities
Post-attention outputs, calibration means (primary)	69.8	67.3	$2.5\pm 1.9$	9.0	12.6
Earlier five recurrent outputs, calibration means	69.8	62.2	$7.6\pm 2.8$	17.9	16.2
Post-attention outputs, zeros	69.8	50.7	$19.1\pm 3.5$	34.1	15.6
Earlier five recurrent outputs, zeros	69.8	32.3	$37.5\pm 4.0$	61.0	16.9
Post-attention minus earlier, means			$-5.1\pm 3.1$

Appendix

Table 31: Ordinary seven-candidate top-1 accuracy when final-token Mamba outputs are replaced, Nemotron-H, no residual transplant. Paired drops are baseline minus intervened accuracy, averaged within identity (nine source prompts and one recipient prompt, weighted 9:1); transition rates are conditional on baseline correctness. The memory-swap rows compare the transposed with the control memory on the same units.

Variant	Model	Gate (%)	Site	$\kappa$	$\pi$	Class	$C$
Filler context	Nemotron-H-8B	65/66	pre-L40	0.81 $\pm$ 0.12	0.36 $\pm$ 0.09	Key-dominant (dual)	0.90 $\pm$ 0.02
Filler context	Qwen2.5-7B	66/66	pre-L22	0.53 $\pm$ 0.13	0.53 $\pm$ 0.13	Balanced	0.52 $\pm$ 0.02
Filler context	Llama-3.1-8B	83/86	pre-L16	0.78 $\pm$ 0.10	0.37 $\pm$ 0.08	Key-dominant (dual)	0.35 $\pm$ 0.01
Word-valued attributes	Nemotron-H-8B	70/77	pre-L40	0.82 $\pm$ 0.12	0.20 $\pm$ 0.08	Key-dominant (dual)	0.89 $\pm$ 0.01
Word-valued attributes	Granite-H	66/62	pre-L35	0.66 $\pm$ 0.09	0.10 $\pm$ 0.06	Key-dominant (dual)	0.89 $\pm$ 0.04
Word-valued attributes	Qwen2.5-7B	66/68	pre-L21	0.39 $\pm$ 0.10	0.33 $\pm$ 0.08	Balanced	0.32 $\pm$ 0.01

Appendix

Table 32: Task-family variants of the binding task. Gate is source/recipient seven-way top-1; the site is the last attention layer for the sequential hybrids and the key peak otherwise;

\kappa

and

\pi

carry 97.5% intervals and

C

a 95% interval. Runs below the accuracy gate are absent.

Model	Task	Site	Preregistered class	Classes across the grid	Layers changed (max / mean)
Nemotron-H-8B	Arrow	pre-L40	Key-dominant (dual)	Key-dominant (dual)	4 / 1.3 of 52
Nemotron-H-8B (pre-L39)	Arrow	pre-L39	Key-dominant (dual)	Key-dominant (dual)	4 / 1.3 of 52
Nemotron-H-8B	Colon	pre-L40	Key-dominant (dual)	Key-dominant (dual)	0 / 0.0 of 2
Nemotron-H-8B	Binding	pre-L40	Key-dominant (dual)	Key-dominant (dual)	0 / 0.0 of 52
Nemotron-H-4B	Arrow	pre-L40	Key-dominant (dual)	Key-dominant (dual)	0 / 0.0 of 52
Nemotron-H-4B	Binding	pre-L40	Key-dominant (dual)	Key-dominant (dual)	4 / 1.3 of 52

Appendix

Table 33: Threshold sensitivity of the classification. “Site” is the last attention layer for sequential hybrids and the key peak otherwise; “classes across the grid” lists every class the site receives over the 18 settings; “layers changed” is the maximum and mean number of layers in the depth profile whose class differs from the preregistered setting. Binding-task rows are absent for models that failed the binding gate.

		Arrow task			Binding task
Model	Attn. blocks	$C_{\rm attn}$	$C_{\rm attn}/(1/n)$	$e^{H}$	$C_{\rm attn}$	$C_{\rm attn}/(1/n)$	$e^{H}$
Nemotron-H-8B	4	0.71	2.9	1.9	0.72	2.9	1.9
Nemotron-H-4B	4	0.66	2.7	1.9	0.71	2.8	1.8
Granite-H	4	0.73	2.9	1.8	0.57	2.3	2.0
RecurrentGemma-9B	12	0.80	9.6	2.0	—	—	—
Nemotron-3-Nano-30B	6	0.78	4.7	1.7	0.75	4.5	1.9

Appendix

Table 34: Resolution concentration restricted to attention-containing layers on the donor-answer effect.

n

counts attention-containing layers with a measured increment (the final layer has none);

C_{\rm attn}/(1/n)

is the ratio to the uniform null over those layers;

e^{H}

is the effective attention count carrying the positive increments.

Alternative	Evidence against it
Fixed donor answer	One donor tracks three recipient bindings on 81% of crosses; $S_{A}$ scrub destroys the effect
Recipient keeps its own answer $A_{r}$	$\Delta\ell(B_{j})-\Delta\ell(A_{r})=4.17$ logits
Query token, not key, is transferred	Recipient maps $q_{j}$ away from $B_{j}$ in 97% of cells; excluding the rest leaves $\kappa$ unchanged
State is only a row pointer	The key beats the position by 0.77 logits with a reference baseline; $S_{P}$ scrub destroys the effect
Any nearby site works	Pre-L39 whole residual carries three quarters of the effect, same class
Any eight heads carry it	Top-8 path exceeds all 255 random eight-head paths

Appendix

Table 35: Alternative accounts and the evidence against each.

Hybrid Language Models

See all 14 papers

What Attention Recalls and Recurrence Controls in Hybrid Language Models
Sep 3, 2026Kirill Afendulev, Alexey Dontsov, Elena Tutubalina +1Hybrid Attention Recurrent Transformers
AIRI, Moscow, Russia · YSDA, Moscow, Russia · HSE University, Moscow, Russia
LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replay
Sep 7, 2026Simon P. VillaniHybrid Language Models Language Modeling
Reasoning Primitives in Hybrid and Non-Hybrid LLMs: Do Architectural Differences Yield Advantages in State-Tracking and Recall?
Apr 23, 2026Shivam Rawat, Lucie Flek, Florian Mai +1CoT Reasoning LLM Memory
Lamarr Institute for Machine Learning and Artificial Intelligence · Rheinische Friedrich-Wilhelms-Universität Bonn

Sep 3, 2026J/K move · Enter open · S save

What Attention Recalls and Recurrence Controls in Hybrid Language Models

Kirill Afendulev, Alexey Dontsov, Elena Tutubalina, Anton Korznikov

AIRI, Moscow, Russia · YSDA, Moscow, Russia · HSE University, Moscow, Russia

Hybrid Attention Recurrent Transformers In-Context Retrieval Causal Interventions in Language Models

Open paper AskPDF arXiv

Hybrid language models combine attention with a fixed-size recurrent state, but the role of each channel remains unclear. We introduce two cache-level interventions. Split-prefill keeps only the KV cache or only the recurrent state from a prefilled context, then generates an answer. State-swap pairs the KV cache from one context with the recurrent state from another in a single forward pass. On Qwen3.5 and Falcon-H1, the two channels split sharply by function. Exact retrieval survives only through attention (64-98% of full accuracy) and collapses to zero through recurrence. Output language and persona reverse the pattern: both survive recurrence (70-80% and 3-5x) while KV-only drops to ~1% language accuracy. State-swap confirms this causally: the answer takes its value from the KV side and its language from the recurrent side. Recurrent-only generation also accepts words that were never in the context but share meaning or parts with seen items. Attention provides a lookup over what was said; the recurrent state shapes how the model says it next.

Abstract

Hybrid Language Models

Explore similar work

What Attention Recalls and Recurrence Controls in Hybrid Language Models

LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replay

Reasoning Primitives in Hybrid and Non-Hybrid LLMs: Do Architectural Differences Yield Advantages in State-Tracking and Recall?