Who Asks, Who Answers: Linear Attention Builds the Query in Hybrid Language Models

Wenqi Zeng; Chen Feng; YuAng Chen

cs.AINov 1, 2026

Who Asks, Who Answers: Linear Attention Builds the Query in Hybrid Language Models

Authors: Wenqi Zeng, Chen Feng, YuAng Chen

Organizations: The Hong Kong University of Science and Technology · The Hong Kong Polytechnic University · The Chinese University of Hong Kong

Abstract

Hybrid language models interleave linear-attention layers with a small number of softmax-attention layers. Prior work largely attributes in-context retrieval to softmax attention, treating linear attention primarily as a mechanism for compressing past context. We revisit this division of labor and show that linear attention plays a direct role in retrieval by constructing the query used by softmax attention. Studying Qwen3.5 (Gated DeltaNet) and Granite4.0 (Mamba-2), we identify an Encode--Address circuit. An early Encode block preserves token identity, while an Address block constructs the final-position representation from which a softmax reader forms its attention query. Transplanting this representation between prompts with different queried keys redirects retrieval in 0.97--0.99 of cases. The Address block's recurrent state carries the retrieval target, and its computation transfers across architectures: replacing Granite's Address block with Qwen's restores retrieval accuracy to 0.92--0.94 without updating either model's weights. Across six Qwen3.5 models, Address computation becomes distributed at larger scales over a chain of blocks, each guiding a successive reader, as the query no longer passes from one reader to the next through attention. These findings reveal a division of labor in hybrid retrieval: linear attention constructs the query, while softmax attention performs the lookup. Code is available at https://github.com/yuang-chen/LinearQuery

Figures & tables

Figure 1: Overview. (a) The Encode block re-encodes token identity, the Address block constructs the retrieval query, and softmax attention retrieves the matching value. (b) Transplanting the address from prompt A into prompt B makes the model answer A’s question. (c) Addresses transfer between Qwen3.5 (Gated DeltaNet) and Granite4.0 (Mamba-2)

Figure 2: Performance of Qwen3.5-9B truncated after layer

L

. Retrieval appears abruptly when the softmax layer L19 is kept; knowledge grows gradually with depth.

Figure 3: Removal and transplant of each linear-attention block of Qwen3.5-9B ( chat8 );

G_{k}

comprises layers

4k

4k{+}2

. (a) Removing the Encode block

G_{0}

destroys retrieval. Removing the Address block

G_{4}

leaves accuracy at 1.00 but halves the logit gap and reduces the reader’s attention on the queried entry from 0.83 to 0.06. (b) Transplanting

G_{4}

’s output at the final position from a prompt about another key makes the model answer that key and moves the reader to the donor’s entry; no other block has an effect.

Patched at swapped tokens	Value swap	Key swap
Encode block output	0.98	0.98
Input token embedding	1.00	1.00
Any other block’s output	0.00	0.00
No Patching	0.00	0.00

Table 1: The Encode block of Qwen3.5-9B ( chat8 ). Left: success rate of token-level interventions in restoring the original key–value associations. Right: retrieval accuracy and WikiText-2 perplexity under modifications of

G_{0}

Figure 4: How the Address block constructs the retrieval target in Qwen3.5-9B. (a) Reader selectivity when its query or keys are computed without the indicated block; numbers show top-1 rates. Solid, light, and hatched bars denote the three prompt formats. (b) Transplanting the Address block’s final-position output, or its recurrent state, redirects retrieval; the states of earlier blocks and the block’s input at the final position do not. (c) Address block outputs computed without any single upstream block still redirect retrieval, but those computed without all three do not.

	Output transplant		Block replacement
Donor $\rightarrow$ receiver	Cross- model	Same- model	Removed	Linear proj.	Zeroed mixer	Donor Address
Qwen3.5-0.8B $\rightarrow$ Granite4.0-1B	0.98	1.00	0.01	0.24	0.28	0.92
Granite4.0-1B $\rightarrow$ Qwen3.5-0.8B	0.91	0.97	0.30	0.28	0.28	0.95
Qwen3.5-9B $\rightarrow$ Granite4.0-7B a	0.99	1.00	0.00	0.14	0.18	0.94
Granite4.0-7B $\rightarrow$ Qwen3.5-9B b	0.83	0.92	0.35	0.30	0.21	0.46

Table 2: Cross-architecture transfer on chat8 . Output transplantation : success rate of redirecting retrieval using the donor Address block’s output, compared with a same-model baseline using two prompts processed by the receiver model. Block replacement : receiver accuracy after replacing its Address block with the donor block or a control. a Both Address blocks of Granite4.0-7B are replaced. b Qwen3.5-9B is evaluated after L19 (intact accuracy: 0.75), using both Address blocks of Granite4.0-7B as the donor.

Model	Linear / softmax	Address block $\to$ guided readers	Removal	Transplant
Qwen3.5-0.8B	18 / 6	$G_{3}\to$ L15, L19, L23	0.06	0.99
Qwen3.5-2B	18 / 6	$G_{3}\to$ L15, L19, L23	0.41	1.00
Qwen3.5-4B	24 / 8	$G_{4}\to$ L19, L23, L27	0.94	0.70
Qwen3.5-9B	24 / 8	$G_{4}\to$ L19, L23, L27	0.92	0.98
Qwen3.5-27B	48 / 16	$G_{8}$ – $G_{10}\to$ L39, L43; $G_{11},G_{12}\to$ L51, L59	1.00	0.98
Qwen3.5-35B-A3B	30 / 10	$G_{6}\to$ L27; $G_{7}\to$ L31; $G_{8}\to$ L35	0.34	0.75

Table 3: Address blocks and the readers they guide in Qwen3.5 models. Removal : accuracy after removing the Address blocks. Transplant : fraction of prompts redirected to the donor’s target when the Address blocks are transplanted jointly.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

		Transplant		Removal		Layer 0
Model	Block	whole block	best single layer	whole block	worst single layer	removed
Qwen3.5-0.8B	L12–14	0.99 (0.84)	0.00 (0.03, L14)	0.13	0.90 (L14)	0.00
Qwen3.5-9B	L16–18	0.99 (0.78)	0.00 (0.10, L18)	1.00	0.99 (L17)	0.05
Granite4.0-1B	L16–24	1.00 (0.83)	0.93 (0.70, L23)	0.00	0.73 (L24)	0.00

Appendix

Table 4: Whole-block versus single-layer interventions on the block preceding the reader ( chat8 ). Transplant : fraction of donor–host pairs that return the donor’s value, with the reader’s attention on the donor’s entry in parentheses (host baseline

\leq 0.01

); for single layers, the best layer of the block is shown. Removal : accuracy with the block, or the most damaging single layer of the block, removed. The last column gives accuracy with layer 0 alone removed.

		Removal		Transplant
Model	Address	Encode	Address	Address	best other block
Qwen3.5-0.8B	L12–14	0.00 / 0.00 / 0.00	0.13 / 0.03 / 0.01	0.99 / 1.00 / 0.97	0.00 / 0.00 / 0.00
Qwen3.5-9B	L16–18	0.01 / 0.00 / 0.00	1.00 / 1.00 / 0.76	0.99 / 0.97 / 0.98	0.00 / 0.00 / 0.00
Granite4.0-1B	L16–24	0.00 / 0.00 / 0.00	0.00 / 0.01 / 0.00	1.00 / 0.98 / 0.92	0.00 / 0.00 / 0.00
Granite4.0-7B	L16–24, L26–34	0.00 / 0.00 / 0.00	0.01 / 0.06 / 0.00	1.00 / 1.00 / 0.90	0.00 / 0.00 / 0.00

Appendix

Table 5: The circuit across models ( chat8 / list8 / rev8 ). Removal: accuracy with the block removed (intact

\geq 0.90

). Transplant: fraction of pairs that return the donor’s value when the block’s output is transplanted at the final position. For Granite4.0-7B, the Address comprises two blocks,

G_{2}

and

G_{3}

, which are removed and transplanted jointly; transplanting

G_{2}

alone gives 0.03 / 0.06 / 0.05.

Figure 5: Attention of the top head of each reader layer on the donor’s entry after transplanting single core blocks (above the line) or their combinations (below) at the final position, in Qwen3.5-27B (a) and Qwen3.5-35B-A3B (b). Intact host $\leq 0.04$ .

Knowledge Retrieval in Language Models

See all 35 papers

Sep 26, 2026

The Key Handoff: Retrieval in Hybrid Language Models

A two-hop question makes a language model retrieve twice: once to produce a bridge entity, and once to retrieve with it. Transformers resolve that entity in their early layers. Hybrid models replace most of the attention with a recurrent state, so where the key becomes usable, and where it is spent on an answer, is not known. Answering "Where is the ball?" from "the ball belongs to Alice" and "Alice is in the garden" turns on Alice, a name the question does not mention. A model could reach garden through Alice, the key it computed, or through where the fact sits in the prompt. Across twelve models, dense and hybrid, we move a hidden state from one story into another where the two routes lead to different places, and read off which place the model gives. An attention layer converts the key in every model we tested, and crossing that layer removes its usable effect, all of it where no attention follows. In sequential hybrids this makes the answer a handoff: recurrent layers carry the key forward, and attention spends it. Retrieval does not always stop there. Writing a different fact into the memory of the recurrent layers that follow the last attention layer can move the answer toward that fact, multiplying its odds by 1.3 to 2.7, and which hybrids do this is not settled by their architecture or training. That read is addressed by the key, not by position: recurrent state in a hybrid is not only a carrier, but a memory that later layers can query.

Kaan Kale, Oguzhan Baser, Sriram Vishwanath

Georgia Institute of Technology, USA · University of Texas at Austin, USA

Knowledge Retrieval in Language Models Hybrid Language Models Hybrid Transformer-State Space Model Architectures Multi-Hop QA Hybrid Attention Mechanistic Interpretability

Ask PDF

Oct 19, 2026

Competing Learning in Hybrid Architectures

Many modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention. Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval. In this paper, we reveal a Competing Learning phenomenon where different kinds of mixers compete to learn the same function, and when any layer learns a function, it suppresses the corresponding learning signal for other layers. Layer attribution analysis reveals that under typical data distributions, modern recurrence learns retrieval functions faster than softmax attention, causing attention to fail to learn retrieval in most cases. Stopping weight updates in recurrence in the middle of training helps attention learn retrieval. Based on these results, we propose a staged training procedure for pre-training hybrid architectures on natural language. Empirical results on models up to 1B parameters show that our method results in substantial improvement in language modeling loss, common-sense reasoning, as well as a wide range of retrieval-related tasks across different context lengths.

Xingyu Shen, Yingfa Chen, Ziqing Qiao +3

Department of Science and Technology, Tsinghua University, Beijing, China

Hybrid Language Models Hybrid Attention In-Context Retrieval

Ask PDF

Jul 8, 2025

A Systematic Analysis of Hybrid Linear Attention

Transformers face quadratic complexity and memory issues with long sequences, prompting the adoption of linear attention mechanisms using fixed-size hidden states. However, linear models often suffer from limited recall performance, leading to hybrid architectures that combine linear and full attention layers. Despite extensive hybrid architecture research, the choice of linear attention component has not been deeply explored. We systematically evaluate various linear attention models across generations - vector recurrences to advanced gating mechanisms - both standalone and hybridized. To enable this comprehensive analysis, we trained and open-sourced 72 models: 36 at 340M parameters (20B tokens) and 36 at 1.3B parameters (100B tokens), covering six linear attention variants across five hybridization ratios. Benchmarking on standard language modeling and recall tasks reveals that superior standalone linear models do not necessarily excel in hybrids. While language modeling remains stable across linear-to-full attention ratios, recall significantly improves with increased full attention layers, particularly below a 3:1 ratio. Our study highlights selective gating, hierarchical recurrence, and controlled forgetting as critical for effective hybrid models. We recommend architectures such as HGRN-2 or GatedDeltaNet with a linear-to-full ratio between 3:1 and 6:1 to achieve Transformer-level recall efficiently. Our models are open-sourced at https://huggingface.co/collections/m-a-p/hybrid-linear-attention-research-686c488a63d609d2f20e2b1e.

Dustin Wang, Rui-Jie Zhu, Steven Abreu +9

1UC Santa Cruz · 2ByteDance Seed · University of Groningen +3

Linear Attention Hybrid Attention

Ask PDF

Abstract

Knowledge Retrieval in Language Models

Explore similar work

The Key Handoff: Retrieval in Hybrid Language Models

Competing Learning in Hybrid Architectures

A Systematic Analysis of Hybrid Linear Attention