How Linear Attention Remembers

Kichang Lee; JaeYeon Park; Songkuk Kim; JeongGil Ko

cs.LGSep 27, 2026

How Linear Attention Remembers

Authors: Kichang Lee, JaeYeon Park, Songkuk Kim, JeongGil Ko

Organizations: KAIST · Yonsei University · Dankook University

Abstract

Linear attention replaces the growing key--value (KV) cache of standard attention with a fixed-size recurrent state, substantially reducing memory growth with context length. This efficiency, however, changes how past information is stored: many tokens must share and repeatedly update the same memory. We study how this recurrent state functions as a memory system. Using an analytical decomposition together with controlled causal interventions in pretrained GLA and GDN models, we trace how recalled information is written, retained, and later accessed. We find that fact-specific information enters recurrent memory through concentrated, content-dependent writes and is later accessed through concentrated query-time read pathways. Multiple facts can remain selectively accessible within the same state, yet their internal representations exhibit cross-fact causal coupling rather than independent KV-like storage. As memory load increases, both recall and targeted editability degrade, whereas elapsed context alone has a substantially smaller effect within the tested regime. Causal interventions on subsequent writes further show that interference is shaped by their overlap with existing memory. Finally, in hybrid architectures that combine recurrent layers with full attention, the runtime memory directly supporting recall shifts predominantly to the full-attention KV state. Together, these results reveal how fixed-size recurrent memory supports selective recall despite shared storage, while exposing the interference and capacity limits that distinguish it from token-addressable KV memory.

Figures & tables

Figure 1: Causal intervention framework for tracing recurrent memory. All single-fact interventions are applied only to a development-selected recurrent head, which is then frozen for confirmation, rather than to all heads. (1) At fact time, recipient and donor runs share the same prefix but induce different content-specific writes; we block or replace the selected head ’s recipient write to test necessity and content specificity. (2) The resulting recurrent state evolves under subsequent token-dependent transformations and writes. (3) Immediately before the query, we replace the selected head ’s recipient state with its donor counterpart to test whether donor information is present in the accumulated state. (4) During query-time readout, we patch or restore the selected head ’s output to test whether this pathway mediates the effect of the stored information. (5) We extend the same setup to multiple facts sharing a recurrent state to examine selective recall and cross-fact coupling.

Figure 2: Development head screen. Donor-state swaps reveal concentrated causal effects; boxes mark the heads frozen for confirmation.

Figure 3: Persistent writes are content-specific and non-additive. (a) Recall under fact-time blocking. (b) Donor-margin shifts after donor and control updates. (c) Interaction between the selected GLA head (Hub; L21H3) and a development-selected secondary head (

B

; L14H3). Hub+

B

is the observed joint intervention; additive is the arithmetic sum of the separately measured Hub-only and

B

-only effects. Error bars show source-bootstrap 95% CIs.

Figure 4: Query-time output mediates the state-swap effect. Donor-state swap and donor-output patch produce comparable shifts toward the donor answer, whereas restoring the clean recipient output after a donor-state swap removes the effect.

Figure 5: Selective recall coexists with cross-fact causal sensitivity. (a) Target transfer. (b) Non-target break rate. (c) Effect on fact

B

after decomposing fact

A

’s query-boundary state direction into components parallel and orthogonal to fact

B

’s direction. Random controls are norm-matched to the corresponding components. Error bars show source-bootstrap 95% CIs.

Figure 6: Hybrid recall shifts to full-attention KV memory. Each effect is the donor-margin shift from replacing one memory substrate, normalized by the shift from replacing both recurrent/local and full-attention KV state. Bars show paired source-bootstrap 95% CIs.

Figure 7: Competition and overlapping writes degrade recurrent memory. (a) Recall under increasing memory load at fixed sequence length. (b) Recall with increasing fact-to-query distance at fixed

K=4

. (c) Target-margin change when a later write key is aligned with or orthogonalized against the target fact’s write-key direction. The

K=32

condition uses a separate high-load layout; bars show source-bootstrap 95% CIs.

Figure 8: Retention, editability, and collateral are distinct failure modes. (a) Clean recall, (b) target edit strength, and (c) non-target break rate as memory load increases. Dotted curves show random-update controls and the dashed line the 5% collateral criterion. Bars show source-cluster bootstrap 95% CIs; missing GLA editing points were not evaluated.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$d_{\mathrm{model}}$	Layers	Recurrent	Full attn.	Recurrent heads/dims	Max. context
GLA-340M	1024	24	24	0	4; $d_{k}=128,d_{v}=256$	4096
GDN-340M	1024	24	24	0	4; $d_{k}=d_{v}=256$	2048
GDN-1.3B	2048	24	24	0	8; $d_{k}=d_{v}=256$	2048
Hybrid GDN-340M	1024	24	16	8	4; $d_{k}=d_{v}=256$	2048
Qwen3.5-4B	2560	32	24	8	16 K/32 V; $d_{k}=d_{v}=128$	262,144
Qwen3.5-9B	4096	32	24	8	16 K/32 V; $d_{k}=d_{v}=128$	262,144

Appendix

Table 1: Architectural configurations of the evaluated language-model backbones. “Recurrent” denotes GLA or Gated DeltaNet blocks as appropriate. For Qwen3.5, key and value head counts are reported separately. Maximum context follows the released model configurations; each experiment uses the sequence length specified in its protocol.

Figure 9: Development-set selection of recurrent heads. (a) Mean signed donor-margin shift

J_{\ell,h}

for every recurrent head, evaluated on the same 16 development sources per model. The outlined cell denotes the selected head, with layer and head indices reported using zero-based indexing. (b) Per-source difference between the selected head’s intervention effect and the strongest competing head on that source. Positive values indicate that the selected head has the largest effect. Both panels are development-set diagnostics and are not used as independent confirmation results.

Figure 10: Selected components remain dominant at both writing and reading on held-out sources. Each point denotes one recurrent component and shows its mean donor-minus-recipient margin shift under the fact-time write and pre-query state interventions (

n=32

sources). Stars indicate the development-selected components. Dashed lines mark the prespecified 0.25-logit meaningful-effect threshold. Spearman correlations are computed over the complete 96-component rankings.

Figure 11: Answer-related information becomes sharply accessible at the independently identified hub layer. Four-choice logit-lens accuracy is measured at block input, after the recurrent sublayer, and after the MLP across depth. Curves show the 96-source localization split for retention (top) and three updates (bottom). Shading marks GLA layer 21 and GDN layer 20. Annotated values report the block-input to post-recurrent increase on a disjoint 128-source confirmation split.

Figure 12: A neutral-state counterfactual selectively suppresses the queried association. Accuracy before and after replacing the selected head’s pre-query state with its counterpart from a length-matched execution in which the target fact is absent (

n=128

sources). The left panel evaluates the queried target, and the right panel aggregates the remaining three facts. Error bars show paired source-bootstrap 95% CIs for the change from clean.

Figure 13: Head-level reuse does not imply a task-invariant semantic direction. (a) Qualified donor-answer rates after swapping the same frozen head across binding counts, templates, value sets, and a three-update condition (

n=128

sources, source-bootstrap 95% CIs). (b) Exploratory 32-source basis-transfer control. A rank-4 output basis learned from color examples transfers the color effect but not the shape effect, while most of the shape effect lies in the displayed rank-8 orthogonal component. C and S denote colors and shapes.

Task	$n$	Clean	KV donor	KV/full
Entity update, WikiText	128	93.0	85.2	97.2
Entity update, PG19	128	89.1	89.8	96.2
RULER single-NIAH, ${\sim}4$ K	64	100.0	100.0	97.2

Appendix

Table 2: Hybrid recall remains dominated by full-attention KV memory across additional tasks and contexts. Clean accuracy, donor-answer rates under recurrent-only and KV-only cache replacement, and the percentage of the full donor-margin effect recovered by KV replacement. All entries are percentages.

Figure 14: Read-path concentration varies across checkpoints and task formats. (a,b) Fraction of the full donor-margin effect recovered by jointly swapping the development-ranked top

k

recurrent components on 64 held-out sources. GLA checkpoints use the same compact construction, whereas RWKV7 checkpoints use identical naturalized inputs. Error bars are source-bootstrap 95% CIs. (c) Frozen top-1 effects for RWKV7-13.3B on a fresh format-matched cohort. Mixed-format conditions are omitted because their clean task accuracy is insufficient for reliable write/read attribution.

Figure 15: State-derived features recover a subset of native recall errors. Native and corrected four-choice accuracy for source-disjoint correction models. Horizontal intervals show 95% CIs for corrected accuracy. Labels beside the corrected points report the number of native errors recovered minus the number of originally correct cases disrupted. “GLA shapes” changes the value set, template, and source range.

Filler tokens	Recipient	Donor	Evidence removed
512	56.3	53.1	0.0
128	71.9	68.8	3.1

Appendix

Table 3: Baseline competence is insufficient for causal analysis in the prose-retrieval pilot. Joint full-vocabulary top-1 accuracy across both prespecified query paraphrases, clustered over the same 32 source families. Entries are percentages.

Explore similar work

Jul 2, 2026

A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

Linear-attention and state-space language models compress the prefix into a fixed-size recurrent state, yielding O(1) memory at the cost of a lossy exact memory: when many key--value associations compete, earlier facts are overwritten and needle recall degrades. Inspired by Complementary Learning Systems, we give linear attention a hippocampal complement. HOLA (Hippocampal Linear Attention) keeps the usual delta-rule state as a compressive memory and adds a bounded exact KV cache, forming a semiparametric test-time memory: the state models linearly compressible structure, while the cache stores associations that should not be forced through that state. The cache writes without a learned eviction module, keeping tokens with large beta * ||e||, the prediction residual actually committed to the state; a decoupled RMSNorm-gamma cache read then turns these exact KV pairs into sharp retrieval rather than soft averaging. At 340M parameters trained on 15B SlimPajama tokens, HOLA lowers Wikitext perplexity from 27.32 to 22.92 (-16.1%), below a full-attention Transformer++ (26.88), and improves LAMBADA perplexity from 30.95 to 30.26. It also achieves the best linear in-context retrieval and remains much more robust than GDN or a matched HOLA+recency cache on RULER needle-in-a-haystack recall out to 32k tokens (16x its training length).

Wanyun Cui

Shanghai University of Finance and Economics

Long-Context Retrieval Memory-Augmented Language Models KV Caching Long-Context Language Modeling Persistent Memory for Language Models Delta-Rule Linear Attention+1

Ask PDF

May 11, 2026

Variational Linear Attention: Stable Associative Memory for Long-Context Transformers

Linear attention reduces the quadratic cost of softmax attention to

\mathcal{O}(T)

, but its memory state grows as

\mathcal{O}(T)

in Frobenius norm, causing progressive interference between stored associations. We introduce Variational Linear Attention (VLA), which reframes the memory update as an online regularised least-squares problem with an adaptive penalty matrix maintained via the Sherman-Morrison rank-1 formula. We prove that normalising the write direction to unit length gives the recurrence Jacobian spectral norm exactly

1

for all sequence lengths and head dimensions (Proposition 2), and that the state norm is self-limiting under bounded inputs (Proposition 1). Empirically, VLA reduces

\|S_t\|_F

109\times

relative to standard linear attention at

T{=}1{,}000

, achieves near-perfect exact-match accuracy on multi-query associative recall within the effective per-head memory regime (

n_\text{pairs} < d_h

), maintaining substantially higher retrieval performance than DeltaNet and standard linear attention under increasing memory load, and maintains 62% accuracy at the per-head capacity boundary. A Triton-fused kernel achieves

14\times

speedup over sequential Python and

\mathcal{O}(T)

scaling, crossing below softmax attention latency at approximately 43,000 tokens.

Vishal Pandey, Gopal Singh

Independent Researcher London, UK · Metriqual Athens, GR

Efficient Attention Linear Attention Associative Memory

Ask PDF

Oct 11, 2026

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ongoing computation and preserve information from an ever-growing history. This coupling has reinforced a prevailing view that the state's memory capacity defines the model's long-context capability. We challenge this presumed ceiling: recurrent computation need not bear the full burden of long-term storage. We introduce NoteBank, a sparse episodic memory that archives selected text before future queries are known, uses the backbone's native features for writing and addressing, and reads retrieved evidence through its native forward computation via replay. This training-free, plug-and-play interface requires no external embedding model, separately trained retriever, or separate neural reader. At one million tokens, it raises Gated DeltaNet's mean accuracy across six RULER retrieval tasks from 1.7% to 70.3%, with no observed significant retrieval degradation from 4K to 1M tokens. This recovery adds

\leqslant

0.025% to analytical FLOPs, with persistent context storage approximately 1/2,600 that of a Transformer's full MHA KV cache at matched width, depth, and precision. Experiments extend from three 1.3B recurrent backbones to Mamba-2-8B and post-trained Qwen3.5-9B, demonstrating gains on LongBench v1/v2 and MemoryAgentBench. Together, these results show that sparse evidence storage and native replay can substantially expand the long-context capability of pretrained linear attention models while retaining the efficiency of recurrent computation.

Li Nanbo, Firas Laakom, Yilan Zhang +10

King Abdullah University of Science and Technology (KAUST)

Long-Context Modeling Memory-Augmented Language Models Linear Attention

Ask PDF