What Attention Recalls and Recurrence Controls in Hybrid Language Models

Kirill Afendulev; Alexey Dontsov; Elena Tutubalina; Anton Korznikov

cs.CLSep 3, 2026

What Attention Recalls and Recurrence Controls in Hybrid Language Models

Authors: Kirill Afendulev, Alexey Dontsov, Elena Tutubalina, Anton Korznikov

Organizations: AIRI, Moscow, Russia · YSDA, Moscow, Russia · HSE University, Moscow, Russia

Abstract

Hybrid language models combine attention with a fixed-size recurrent state, but the role of each channel remains unclear. We introduce two cache-level interventions. Split-prefill keeps only the KV cache or only the recurrent state from a prefilled context, then generates an answer. State-swap pairs the KV cache from one context with the recurrent state from another in a single forward pass. On Qwen3.5 and Falcon-H1, the two channels split sharply by function. Exact retrieval survives only through attention (64-98% of full accuracy) and collapses to zero through recurrence. Output language and persona reverse the pattern: both survive recurrence (70-80% and 3-5x) while KV-only drops to ~1% language accuracy. State-swap confirms this causally: the answer takes its value from the KV side and its language from the recurrent side. Recurrent-only generation also accepts words that were never in the context but share meaning or parts with seen items. Attention provides a lookup over what was said; the recurrent state shapes how the model says it next.

Figures & tables

Figure 1: Split-prefill . A prompt is split into a context ( prompt 1 ) and a query ( prompt 2 ). We prefill the context, save its two internal channels (attention KV cache, recurrent state), then continue on the query while keeping only one channel from the context and resetting the other.

Figure 2: Main result. Split-prefill scores on four diagnostic tasks for Qwen3.5-4B (Gated DeltaNet hybrid, left) and Falcon-H1-3B-Instruct (Mamba hybrid, right), normalised to the full-model baseline. Token and position retrieval survive in the attention KV cache and collapse with recurrent state only. Language and persona reverse the pattern: they survive only the recurrent state and are nearly absent from KV alone. Numerical scores are in Table 1 .

	Qwen3.5-4B			Falcon-H1-3B
Task	full	rec	kv	full	rec	kv
KV retrieve	1.00	0.00	0.89	1.00	0.00	0.64
List indexing	1.00	0.00	0.98	0.96	0.00	0.83
Lang. follow	0.97	0.70	0.01	0.94	0.79	0.01
Persona	0.69	0.49	0.14	0.71	0.28	0.06

Table 1: Accuracy of the model under split-prefill on four diagnostic tasks. Top two rows: exact retrieval survives only the KV cache. Bottom two: language and persona transfer survive only the recurrent state. rec / kv = rec-only / kv-only . Details in Appendix D .

Panel of Figure 3 — Figure 3: Facts from KV, Style from Recurrence. Hybrid cache $\mathcal{K}(c_{A})+\mathcal{R}(c_{B})$ yields the value from $c_{A}$ and the language from $c_{B}$ .

Panel of Figure 4 — Figure 4: Recall $\neq$ Control. KV-only names the requested language correctly but answers in English; rec-only reverses the pattern.

Panel of Figure 5 — Figure 5: False Familiars. Rec-only falsely accepts an associated but absent target; KV-only correctly rejects.

Panel of Figure 6 — Figure 6: Broken Binding. “sun-” (from sunflower) and “-light” (from moonlight) both appear in the list; the compound “sunlight” does not.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Task	full	rec-only	kv-only
Qwen3.5-4B	KV retrieve	1.00 [0.99, 1.00]	0.00 [0.00, 0.00]	0.89 [0.86, 0.92]
	List index	1.00 [0.99, 1.00]	0.00 [0.00, 0.00]	0.98 [0.96, 0.99]
	Lang. foll.	0.97 [0.96, 0.98]	0.70 [0.67, 0.72]	0.01 [0.01, 0.02]
	Persona	0.69 [0.66, 0.72]	0.49 [0.46, 0.53]	0.14 [0.12, 0.16]
Qwen3.5-9B	KV retrieve	1.00 [0.99, 1.00]	0.00 [0.00, 0.00]	0.94 [0.92, 0.96]
	List index	1.00 [0.99, 1.00]	0.00 [0.00, 0.00]	0.99 [0.98, 1.00]

Appendix

Table 2: Main split-prefill results across all evaluated models. Rows without markers: means over 3 seeds with 95% hierarchical bootstrap CIs in brackets.

n{=}200

per seed for retrieval, 450 for language, 225 for persona. † Cells where kv-only exceeds rec-only on a behavioural task; see text. ‡ Single-seed extension (seed 42 only); intervals are within-seed bootstraps and do not quantify across-seed variability. ∗ Additional architectures; point estimates only (bootstrap CIs not computed for these runs).

Task	lower KV + full rec	full KV + lower rec
KV retrieve	0.02 [0.01, 0.03]	1.00 [0.99, 1.00]
List indexing	0.01 [0.00, 0.02]	1.00 [0.99, 1.00]
Lang. follow	0.95 [0.94, 0.96]	0.71 [0.68, 0.73]
Persona	0.68 [0.65, 0.71]	0.65 [0.62, 0.68]

Appendix

Table 3: Layer-wise sweep on Qwen3.5-4B. The two modes dissociate layer-wise as well as channel-wise: retrieval requires KV in full, behavioural transfer requires recurrent state in full.

Figure 7: State-swap construction. Two contexts $c_{A}$ and $c_{B}$ each pair a dictionary with a language instruction; prefilling each yields its own KV cache and recurrent state. We assemble a hybrid cache (KV from $c_{A}$ + recurrent state from $c_{B}$ ), feed a shared query, and read off which channel each property of the answer follows. In the illustrated cross, the answer language (German) follows the recurrent source $c_{B}$ while the dictionary value is retrieved from the KV source $c_{A}$ .

Model	Crossed state	value $\to$ KV	lang $\to$ rec	both
Qwen3.5-4B	$\mathcal{K}_{A}+\mathcal{R}_{B}$	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]
	$\mathcal{K}_{B}+\mathcal{R}_{A}$	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]
Qwen3.5-9B	$\mathcal{K}_{A}+\mathcal{R}_{B}$	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]
	$\mathcal{K}_{B}+\mathcal{R}_{A}$	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]	1.00 [1.00, 1.00]
Falcon-H1-3B	$\mathcal{K}_{A}+\mathcal{R}_{B}$	1.00 [0.99, 1.00]	1.00 [0.99, 1.00]	0.99 [0.98, 1.00]
	$\mathcal{K}_{B}+\mathcal{R}_{A}$	1.00 [0.99, 1.00]	0.99 [0.98, 1.00]	0.99 [0.98, 1.00]

Appendix

Table 4: State-swap (Demo 1). Each row is one crossing direction. Unmarked rows: pooled across 3 seeds,

n{=}100

examples per seed per direction (

n{=}300

total per row). ‡ Single-seed extension (seed 42 only); intervals are within-seed bootstraps. ∗ Additional architectures; point estimates only. For Jamba2-3B the same-source controls reach only

0.86

–

0.89

both-correct.

Model	Mode	Eng. label	Target-lang. label
Qwen3.5-4B	full	0.15 [0.07, 0.25]	0.90 [0.82, 0.97]
	rec-only	0.05 [0.00, 0.12]	0.95 [0.88, 1.00]
	kv-only	0.50 [0.38, 0.63]	0.05 [0.00, 0.12]
Qwen3.5-9B	full	0.05 [0.00, 0.12]	1.00 [1.00, 1.00]
	rec-only	0.10 [0.03, 0.18]	0.95 [0.88, 1.00]
	kv-only	0.90 [0.82, 0.97]	0.05 [0.00, 0.12]

Appendix

Table 5: Self-report probe (Demo 2). Fraction of responses that contain the English language name (col. 3) or the language name in the target language (col. 4). Unmarked rows:

n{=}20

per seed, 3 seeds, 95% bootstrap CIs. ‡ Single-seed extension (seed 42 only, one example per target language); point estimates shown without intervals.

Model	Mode	assoc-absent FPR	neutral-absent FPR	present TPR
Qwen3.5-4B	full	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	0.97 [0.94, 0.99]
	rec-only	0.87 [0.82, 0.91]	0.10 [0.07, 0.14]	0.98 [0.96, 1.00]
	kv-only	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	0.71 [0.65, 0.77]
Qwen3.5-9B	full	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	1.00 [1.00, 1.00]
	rec-only	0.89 [0.84, 0.93]	0.01 [0.00, 0.02]	0.93 [0.90, 0.96]
	kv-only	0.01 [0.00, 0.03]	0.00 [0.00, 0.00]	0.86 [0.81, 0.92]

Appendix

Table 6: DRM trap (Demo 3). Three rates per (model, mode): associated-absent false-positive (the trap), neutral-absent false-positive (yes-bias control), and present true-positive (verifies the model can affirm true membership). Unmarked rows:

n{=}100

per condition per seed, 3 seeds, 95% bootstrap CIs. ‡ Single-seed extension (seed 42 only); intervals are within-seed bootstraps.

Model	Mode	conj. FPR	neutral FPR	left-only FPR	right-only FPR	present TPR
Qwen3.5-4B	full	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	1.00 [1.00, 1.00]
	rec-only	0.32 [0.28, 0.36]	0.10 [0.07, 0.12]	0.18 [0.15, 0.22]	0.05 [0.04, 0.07]	0.53 [0.49, 0.57]
	kv-only	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	0.79 [0.75, 0.82]
Qwen3.5-9B	full	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	1.00 [1.00, 1.00]
	rec-only	0.17 [0.13, 0.20]	0.03 [0.02, 0.05]	0.08 [0.05, 0.10]	0.03 [0.01, 0.04]	0.27 [0.23, 0.30]
	kv-only	0.06 [0.04, 0.08]	0.00 [0.00, 0.01]	0.04 [0.03, 0.06]	0.02 [0.01, 0.03]	0.77 [0.73, 0.81]

Appendix

Table 7: Memory-conjunction trap (Demo 4). Five rates per (model, mode): conjunction-absent FPR (the trap), neutral-absent FPR (yes-bias control), left-only and right-only FPR (only one of the two morphemes present in the list), and present TPR (compound itself in the list). Unmarked rows:

n{=}200

per condition per seed, 3 seeds, 95% bootstrap CIs. ‡ Single-seed extension (seed 42 only); intervals are within-seed bootstraps.

Hybrid Attention

Hybrid attention combines full attention with efficient mechanisms such as linear, recurrent, local, or sparse attention to balance recall and computation.

See all 77 papers

The Key Handoff: Retrieval in Hybrid Language Models
Sep 26, 2026Kaan Kale, Oguzhan Baser, Sriram VishwanathHybrid Language Models Hybrid Transformer-State Space Model Architectures
Georgia Institute of Technology, USA · University of Texas at Austin, USA
LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replay
Sep 7, 2026Simon P. VillaniHybrid Language Models Language Modeling
Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs
Oct 5, 2026Hyunji Lee, Joykirat Singh, Zaid Khan +5Long-Context Language Modeling Recurrent Transformers
UNC Chapel Hill · The University of Texas at Austin · Mila +1

Sep 26, 2026J/K move · Enter open · S save

The Key Handoff: Retrieval in Hybrid Language Models

Kaan Kale, Oguzhan Baser, Sriram Vishwanath

Georgia Institute of Technology, USA · University of Texas at Austin, USA

Hybrid Language Models Hybrid Transformer-State Space Model Architectures Multi-Hop QA Hybrid Attention Mechanistic Interpretability

Open paper AskPDF arXiv

A two-hop question makes a language model retrieve twice: once to produce a bridge entity, and once to retrieve with it. Transformers resolve that entity in their early layers. Hybrid models replace most of the attention with a recurrent state, so where the key becomes usable, and where it is spent on an answer, is not known. Answering "Where is the ball?" from "the ball belongs to Alice" and "Alice is in the garden" turns on Alice, a name the question does not mention. A model could reach garden through Alice, the key it computed, or through where the fact sits in the prompt. Across twelve models, dense and hybrid, we move a hidden state from one story into another where the two routes lead to different places, and read off which place the model gives. An attention layer converts the key in every model we tested, and crossing that layer removes its usable effect, all of it where no attention follows. In sequential hybrids this makes the answer a handoff: recurrent layers carry the key forward, and attention spends it. Retrieval does not always stop there. Writing a different fact into the memory of the recurrent layers that follow the last attention layer can move the answer toward that fact, multiplying its odds by 1.3 to 2.7, and which hybrids do this is not settled by their architecture or training. That read is addressed by the key, not by position: recurrent state in a hybrid is not only a carrier, but a memory that later layers can query.

Abstract

Hybrid Attention

Explore similar work

The Key Handoff: Retrieval in Hybrid Language Models

LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replay

Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs