The Query Knows What to Forget: A Second Erase Direction for Linear Attention

Dhruman Gupta; Aritra Das; Debayan Gupta

cs.LGAug 13, 2026

The Query Knows What to Forget: A Second Erase Direction for Linear Attention

Authors: Dhruman Gupta, Aritra Das, Debayan Gupta

Organizations: Truth Audit Labs

Abstract

Linear attention keeps a state of fixed size. At long context, many stored items share this state, and interference between them degrades retrieval. Gated DeltaNet-2 (GDN-2), like every delta-rule model before it, derives its erase vector from the key of the current token. However, the interference in its reads is measured through the query, and the erase step cannot reach it. We introduce the Query-derived Erase Direction (QED). QED adds a second erase direction derived from the query and orthogonal to the key. In the fast-weight view, a key-directed delta edit cannot change the key-orthogonal part of a read. It uses the editable part to cancel old-state content measured along the query. It also improves retrieval at every length past the training window, and it about doubles the usable context length on S-NIAH-1.

Figures & tables

Arm	Gate	Project	Direction
GDN-2	—	—	no correction
QED-nogate-noproj	no	no	query
QED-nogate	no	yes	query
QED (full)	yes	yes	query
QED-rand (control)	yes	yes	fixed random unit vector for each head

Table 1: The arms. Gate is the

b_{t}

gate on the correction. Project is

P^{\perp}_{k_{t}}

. Direction is the source of the correction in equation ( 6 ).

Seed	GDN-2	QED (full)	QED-rand	QED-nogate	QED-nogate-noproj
0	2.3714	2.3719	2.3712	2.3718	2.3724
1	2.3746	2.3735	2.3735	2.3733	2.3741

Table 2: Final validation loss at 15B tokens (nats). The full spread across all ten runs is 0.0034 nats. Within each seed, the spread is at most 0.0013 nats. The best value for each seed is bold.

Figure 1: S-NIAH-1 accuracy against context length, for each arm and each seed. The dashed line marks the 2048-token training length. Error bars are 95% binomial intervals. QED is above GDN-2 at every length past the training length, in both seeds. The ablation arms do not agree between the seeds.

Seed	Arm	1K	2K	4K	8K	16K	32K	FDA	SQuAD	SWDE
0	GDN-2	100.0	99.0	93.4	64.4	21.8	9.4	4.45	29.39	19.08
0	QED	100.0	100.0	99.6	91.6	39.8	15.8	7.80	29.29	21.51
0	QED-rand	99.6	99.6	97.6	66.0	22.0	7.0	6.44	29.22	19.89
0	QED-nogate	100.0	99.2	89.4	51.2	17.0	6.8	7.35	28.82	19.80
0	QED-nogate-noproj	100.0	100.0	99.0	91.0	33.4	14.0	4.54	29.93	19.98
1	GDN-2	100.0	95.6	79.2	50.6	21.2	6.0	4.26	32.37	19.62

Table 3: S-NIAH-1 accuracy (%) at each context length, with the recall tasks FDA, SQuAD-completion and SWDE (accuracy, %). All rows are 15B tokens. The best value for each seed and each column is bold.

Figure 2: The four contrasts that the arms isolate, both seeds, with 95% intervals. The mechanism and the direction keep one sign at both seeds. The gate and the projection reverse between the seeds.

Figure 3: FDA and its comprehension control, with 95% intervals. Every QED arm is above GDN-2 on FDA, in both seeds. SQuAD-completion does not separate the arms.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

$s$	$\overline{\lambda}_{h}$	4K	8K	16K	32K	FDA
$-1$	$-0.101$	100.0	91.2	39.8	17.8	6.99
$0$	$0$	100.0	90.0	41.4	17.6	7.53
$0.5$	$0.051$	100.0	89.6	42.0	18.4	7.53
$1$	$0.101$	100.0	89.2	41.8	18.0	7.80
$1.5$	$0.152$	100.0	90.4	41.6	17.0	7.80
$2$	$0.203$	99.8	91.4	43.6	15.6	7.62

Appendix

Table 4: Inference-time strength sweep for both QED checkpoints. The weights are fixed, and

s=1

is the value used during training. Entries other than

\overline{\lambda}_{h}

are accuracies (%).

Delta-Rule Linear Attention

Delta-rule linear attention updates a compact recurrent memory to retrieve and revise sequence information with linear-time processing.

See all 35 papers

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzLong-Context Language Modeling Recurrent Transformers
NVIDIA
Don't Read Everything: A Curvature-Conditioned Query for Linear Attention
May 31, 2026Dong Le, Thong Nguyen, Cong-Duy Nguyen +1In-Context Retrieval Long-Context Language Modeling
Nanyang Technological University · National University of Singapore · VinUniversity
Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention
Jun 25, 2026Xiao Li, Chengruidong Zhang, Hao Luo +15Memory-Augmented Language Models Long-Context Language Modeling
1Qwen Team · 2Nanjing University · 3Zhejiang University

May 21, 2026J/K move · Enter open · S save

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Ali Hatamizadeh, Yejin Choi, Jan Kautz

NVIDIA

Long-Context Language Modeling Recurrent Transformers Linear Attention Delta-Rule Linear Attention Gated Delta Networks

Open paper AskPDF arXiv

Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory. The hard part is not just what to forget, but how to edit this compressed memory without scrambling existing associations. Delta-rule models subtract the current read before writing a new value, and Kimi Delta Attention (KDA) sharpens forgetting with channel-wise decay. But the active edit still uses a single scalar gate to control two different things: how much old content to erase on the key side and how much new content to commit on the value side. We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and KDA by inheriting adaptive forgetting and channel-wise decay while addressing their shared limitation, the scalar tie between erasing and writing. Gated Delta Rule-2 separates these roles with a channel-wise erase gate b_t and a channel-wise write gate w_t, reducing to KDA when both gates collapse to the same scalar and to Gated DeltaNet when the decay also collapses. We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training. At 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval. Its advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, where it improves the evaluated multi-key retrieval setting and remains strong in both recurrent and hybrid settings. Code is available at https://github.com/NVlabs/GatedDeltaNet-2.

Abstract

Delta-Rule Linear Attention

Explore similar work

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention