History Replays Itself: Unleashing Long-Context Capability in Linear Attention

Li Nanbo; Firas Laakom; Yilan Zhang; Piotr Piękos; Yimeng Chen; Liangyu Wang; Yifan Xu; Mateusz Ostaszewski

cs.AIOct 11, 2026

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

Authors: Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Piękos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, +5 more

Organizations: King Abdullah University of Science and Technology (KAUST)

Abstract

Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ongoing computation and preserve information from an ever-growing history. This coupling has reinforced a prevailing view that the state's memory capacity defines the model's long-context capability. We challenge this presumed ceiling: recurrent computation need not bear the full burden of long-term storage. We introduce NoteBank, a sparse episodic memory that archives selected text before future queries are known, uses the backbone's native features for writing and addressing, and reads retrieved evidence through its native forward computation via replay. This training-free, plug-and-play interface requires no external embedding model, separately trained retriever, or separate neural reader. At one million tokens, it raises Gated DeltaNet's mean accuracy across six RULER retrieval tasks from 1.7% to 70.3%, with no observed significant retrieval degradation from 4K to 1M tokens. This recovery adds $\leqslant$ 0.025% to analytical FLOPs, with persistent context storage approximately 1/2,600 that of a Transformer's full MHA KV cache at matched width, depth, and precision. Experiments extend from three 1.3B recurrent backbones to Mamba-2-8B and post-trained Qwen3.5-9B, demonstrating gains on LongBench v1/v2 and MemoryAgentBench. Together, these results show that sparse evidence storage and native replay can substantially expand the long-context capability of pretrained linear attention models while retaining the efficiency of recurrent computation.

Figures & tables

Figure 1: NoteBank (NB) recovers million-token retrieval with negligible compute overhead . a , Mean accuracy over six RULER-NIAH retrieval tasks for three pretrained 1.3B recurrent models, without NoteBank (dashed) and with training-free top-2 note replay (solid). b , At 1M tokens, NoteBank adds

\leqslant

0.025% layer-level analytical FLOPs across the three backbones (GDN shown). GDN + NoteBank requires 74.5 MiB of persistent context storage, including its recurrent state—approximately

1/2{,}600

of the 192 GiB MHA KV cache at matched width, depth, and BF16 precision. Square areas represent storage, with a magnified inset for NoteBank. FLOPs include all evaluated stages at the full-model level and matched output length; they do not measure runtime. Storage excludes model weights and transient workspace.

Figure 2: Overview of NoteBank. During context encoding, native features “keys” and “queries” (i.e., ${\bm{k}}$ and ${\bm{q}}$ ) guide span selection and indexing; each span $\mathbf{z}_{i}$ is stored with addresses $\mathcal{A}_{i}$ . Taking the retained context state ${\bm{S}}_{T}$ , the backbone encodes the question to obtain queries for retrieving relevant spans. The native forward pass starts from the same ${\bm{S}}_{T}$ and processes the retrieved evidence and then the question, and generates the answer. Backbone parameters remain frozen throughout. We refer to Algorithm 1 for more details.

Figure 4: Efficiency of training-free NoteBank. (a) Relative analytical FLOPs overhead, decreasing with context length as fixed costs are amortised. (b) Incremental persistent GPU storage grows linearly with context. (c) Measured inference latency on an H100, with batch size 1 and 32 generated tokens. NB uses top-2 replay; InfLLM uses the official default top-96.

Panel of Figure 5 — Figure 5: Evidence preservation, access, and utilisation in NoteBank. a , Gold-value coverage in the archive ( Archived ) and top-4 retrieved notes ( Retrieved@4 ); shading marks stored but unretrieved values. b , Native vs. oracle routing on RULER, with archive, reader, and replay budget fixed. c , The same intervention on four LongBench QA tasks (top-1; frozen/LoRA readers). Oracle notes contain reference answers, a proxy for relevance rather than guaranteed supporting evidence.

Table 6

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Training-free long-range retrieval across architectures and context lengths: full results on RULER 6 NIAH tasks.

Backbone	Method	Single-doc QA	Multi-doc QA	Summarisation	Few-shot	Overall
Transformer †	Plain	13.40	18.97	8.43	23.40	16.05
Titans †	SSC	12.80	18.07	7.63	27.50	16.50
GDN ‡	MARCH	8.87	7.90	13.23	29.47	14.87
	Plain	6.88	7.02	13.99	34.08	15.49
	JRT	7.72	6.63	14.79	33.56	15.67
	NB-TF	6.66	7.33	13.52	36.76	16.07

Appendix

Table 5: From retrieval to real-world long-context tasks. Scores are macro-averaged over evaluated LongBench-v1 tasks, within each category and overall. NB-TF requires no additional training; NB-LoRA adapts the read stage. † Reference results aggregated from Behrouz et al. (2026) ; ‡ Reference results aggregated from Zhang et al. (2026) . Both † and ‡ require pretraining on 16K-context length data; out GDN and Mamba-2 were pretrained on 2K and 4K-length sequences, respectively. Bold marks the best score within each recurrent backbone.

Backbone	Method	FactCon. Single-hop	FactCons. Multi-hop	EventQA	RULER QA	Overall
	Plain	0.0	0.0	0.0	0.0	0.0
	Clean start	14.0	2.0	17.7	16.0	13.0
	NB-TF (top-1)	19.0	3.0	17.9	16.5	14.5
Mamba-2-8B	NB-TF (top-4)	22.0	2.5	16.4	17.0	14.7
	Plain	42.5	5.5	90.2	67.5	55.7
	NB-TF (top-1)	42.5	5.5	90.2	67.5	55.7

Appendix

Table 6: MemoryAgentBench results by task family and replay budget. Accuracy (%) for Mamba-2-8B and Qwen3.5-9B, using matched examples within each backbone. RULER QA differs from the RULER-NIAH tasks reported in this paper. Overall reports the macro-average over evaluated tasks. For Mamba-2, Clean start resets the inference state without replay, while NB-TF additionally replays retrieved evidence. Bold marks the best score within each backbone, including ties.

Figure 7: Breakdown of NoteBank’s computation overhead. Added analytical FLOPs for NB-TF with top-2 replay relative to Plain on RULER, using matched output lengths. Stacked bars separate writing/indexing, question encoding, routing, and replay for GDN, GDN-2, and Mamba-2. Annotations report routing costs at 1M tokens in GFLOPs.

	Gated DeltaNet		Transformer
Metric	Plain	+ NB-TF	Full	+ SW4K	+ InfLLM
RULER @ 1M $\uparrow$	1.71	70.33	0	1.17	0
GPU peak (GiB) $\downarrow$	2.93	3.27	55.18	5.41	7.30
CPU archive (MiB) $\downarrow$	0	0.98	0	0	48930.00

Appendix

Table 7: Retrieval quality vs. measured storage at 1M tokens. RULER averages six NIAH tasks. GPU peak includes model weights. CPU archive reports compact payload accounting for NB’s token IDs and span-location records for NoteBank, and counted host KV for InfLLM.

Routing Signal	Auxiliary Retriever	Time @ 1M (ms)	RULER @ 1M		LongBench
Routing Signal	Auxiliary Retriever	Time @ 1M (ms)	GDN	Mamba-2	GDN	Mamba-2
Random	None	–	1.7	0.0	21.96	13.83
Input embedding	None	7,600	30.8	20.8	22.40	16.70
Native (NB)	None	26	59.8	37.6	23.03	14.25
BM25	Lexical index	87	64.0	52.7	23.50	18.66
BGE	External encoder	2,630	45.4	34.0	23.77	17.24

Appendix

Table 8: Routing quality and computational cost. Scores average six RULER tasks at 1M tokens and 12 LongBench tasks, with LoRA readers fixed across routers within each backbone and benchmark. Bold marks the best score within each auxiliary-retriever group. Native routing matches backbone features and aggregates channel votes; input embeddings are mean-pooled. Reported times use a 4092-note bank at 1M: Native includes question probing and voting, BM25 runs on CPU, and BGE reports full-bank encoding. Timing scopes differ and do not constitute a matched query-latency comparison; no Random timing is available. None indicates no auxiliary retriever, not zero computational cost.

Figure 8: Effect of the per-chunk write budget. Six-task mean RULER accuracy versus analytical added FLOPs for NB-TF with

W\in\{1,2\}

on GDN and Mamba-2, averaged over replay budgets

R\in\{1,2,4\}

. Labels indicate context lengths. Writing two notes consistently improves accuracy with little additional arithmetic cost.

Figure 9: Effect of the replay budget on retrieval and computation. Six-task mean RULER accuracy versus analytical added FLOPs for NB-TF with

R\in\{1,2,4\}

on GDN and Mamba-2. Labels indicate context lengths; connected markers compare replay budgets at the same length. Additional replay generally improves accuracy, with gains depending on the backbone and context length.

Figure 10: Effect of chunk size on retrieval and computation. Six-task mean RULER accuracy versus analytical added FLOPs for NB-TF with different chunk sizes

C

, averaged over replay budgets

R\in\{1,2,4\}

. Labels indicate context lengths. Smaller chunks particularly benefit Mamba-2 at long contexts, while very large chunks degrade retrieval for both backbones.

Figure 11: Joint effects of span and replay budgets. Six-task mean RULER accuracy (%) for NB-TF on GDN (top) and Mamba-2 (bottom), at context lengths from 4K to 1M. Each heatmap varies the span budget $B\in\{64,128,256\}$ across columns and replay budget $R\in\{1,2,4\}$ across rows. Longer spans do not consistently improve accuracy; their benefit depends on the replay budget and backbone.

Figure 12: Replay–span interaction with read adaptation. LongBench macro-average scores over twelve evaluated tasks for NB-LoRA on GDN (top) and Mamba-2 (bottom). Left: varying replay budget $R$ , with averages over span budgets. Middle: varying span budget $B$ , with averages over replay budgets. Right: scores for each $(B,R)$ configuration. The best replay and span budgets depend on the backbone.

Explore similar work

Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Jul 8, 2026Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6Long-Context Retrieval Memory-Augmented Language Models
Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen
MMLA: Memory-Mediated Learning Architecture for Predictive Dual-State Adaptation
Jun 27, 2026Junyi Zou, Avrova DonzContinual Learning for LLMs Test-Time Adaptation
1MMLA-org · 2Communication University of China (CUC), No. 1 Dingfuzhuang East Street, Chaoyang District, Beijing 100024, China
Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
Sep 29, 2026Oliver Sieberling, Bharat Runwal, David Jin +3Linear Attention Long-Context Language Modeling
Massachusetts Institute of Technology · MIT-IBM Computing Research Lab

Jul 8, 2026J/K move · Enter open · S save

Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy, Matthijs Douze+5

Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen

Long-Context Retrieval Memory-Augmented Language Models Recurrent Neural Networks Delta-Rule Linear Attention In-Context Learning Gated Delta Networks+1

Open paper AskPDF arXiv

Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.

Abstract

Explore similar work

Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

MMLA: Memory-Mediated Learning Architecture for Predictive Dual-State Adaptation

Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling