Linear attention keeps a state of fixed size. At long context, many stored items share this state, and interference between them degrades retrieval. Gated DeltaNet-2 (GDN-2), like every delta-rule model before it, derives its erase vector from the key of the current token. However, the interference in its reads is measured through the query, and the erase step cannot reach it. We introduce the Query-derived Erase Direction (QED). QED adds a second erase direction derived from the query and orthogonal to the key. In the fast-weight view, a key-directed delta edit cannot change the key-orthogonal part of a read. It uses the editable part to cancel old-state content measured along the query. It also improves retrieval at every length past the training window, and it about doubles the usable context length on S-NIAH-1.
Figures & tables
Arm
Gate
Project
Direction
GDN-2
—
—
no correction
QED-nogate-noproj
no
no
query
QED-nogate
no
yes
query
QED (full)
yes
yes
query
QED-rand (control)
yes
yes
fixed random unit vector for each head
Table 1: The arms. Gate is the bt gate on the correction. Project is Pkt⊥ . Direction is the source of the correction in equation ( 6 ).
Seed
GDN-2
QED (full)
QED-rand
QED-nogate
QED-nogate-noproj
0
2.3714
2.3719
2.3712
2.3718
2.3724
1
2.3746
2.3735
2.3735
2.3733
2.3741
Table 2: Final validation loss at 15B tokens (nats). The full spread across all ten runs is 0.0034 nats. Within each seed, the spread is at most 0.0013 nats. The best value for each seed is bold.
Figure 1: S-NIAH-1 accuracy against context length, for each arm and each seed. The dashed line marks the 2048-token training length. Error bars are 95% binomial intervals. QED is above GDN-2 at every length past the training length, in both seeds. The ablation arms do not agree between the seeds.
Seed
Arm
1K
2K
4K
8K
16K
32K
FDA
SQuAD
SWDE
0
GDN-2
100.0
99.0
93.4
64.4
21.8
9.4
4.45
29.39
19.08
0
QED
100.0
100.0
99.6
91.6
39.8
15.8
7.80
29.29
21.51
0
QED-rand
99.6
99.6
97.6
66.0
22.0
7.0
6.44
29.22
19.89
0
QED-nogate
100.0
99.2
89.4
51.2
17.0
6.8
7.35
28.82
19.80
0
QED-nogate-noproj
100.0
100.0
99.0
91.0
33.4
14.0
4.54
29.93
19.98
1
GDN-2
100.0
95.6
79.2
50.6
21.2
6.0
4.26
32.37
19.62
Table 3: S-NIAH-1 accuracy (%) at each context length, with the recall tasks FDA, SQuAD-completion and SWDE (accuracy, %). All rows are 15B tokens. The best value for each seed and each column is bold.
Figure 2: The four contrasts that the arms isolate, both seeds, with 95% intervals. The mechanism and the direction keep one sign at both seeds. The gate and the projection reverse between the seeds.
Figure 3: FDA and its comprehension control, with 95% intervals. Every QED arm is above GDN-2 on FDA, in both seeds. SQuAD-completion does not separate the arms.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Seed
s
λh
4K
8K
16K
32K
FDA
0
−1
−0.101
100.0
91.2
39.8
17.8
6.99
0
0
0
100.0
90.0
41.4
17.6
7.53
0
0.5
0.051
100.0
89.6
42.0
18.4
7.53
0
1
0.101
100.0
89.2
41.8
18.0
7.80
0
1.5
0.152
100.0
90.4
41.6
17.0
7.80
0
2
0.203
99.8
91.4
43.6
15.6
7.62
Appendix
Table 4: Inference-time strength sweep for both QED checkpoints. The weights are fixed, and s=1 is the value used during training. Entries other than λh are accuracies (%).
Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory. The hard part is not just what to forget, but how to edit this compressed memory without scrambling existing associations. Delta-rule models subtract the current read before writing a new value, and Kimi Delta Attention (KDA) sharpens forgetting with channel-wise decay. But the active edit still uses a single scalar gate to control two different things: how much old content to erase on the key side and how much new content to commit on the value side. We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and KDA by inheriting adaptive forgetting and channel-wise decay while addressing their shared limitation, the scalar tie between erasing and writing. Gated Delta Rule-2 separates these roles with a channel-wise erase gate b_t and a channel-wise write gate w_t, reducing to KDA when both gates collapse to the same scalar and to Gated DeltaNet when the decay also collapses. We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training. At 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval. Its advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, where it improves the evaluated multi-key retrieval setting and remains strong in both recurrent and hybrid settings. Code is available at https://github.com/NVlabs/GatedDeltaNet-2.
Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks. Existing remedies act on the write side of memory through gating, delta updates, or kernel feature maps, but the read step is left unchanged: every past key contributes additively to the output, so useful targets are diluted by the bulk of stored vectors. We borrow one specific piece of softmax's geometry to construct a cheap read-time contraction of the query. A second-order Taylor expansion of the softmax log-partition at the isotropic-attention point gives a local quadratic model whose curvature coincides with the running key covariance, a quantity that can be maintained with the same recurrent/chunkwise mechanism as the linear-attention state. The associated linear operator contracts the query along the high-variance directions of memory before it reads the state. We call this mechanism Curvature-Conditioned Query (CCQ). CCQ modifies only the read step and is composable with any linear-attention backbone. Attached to GLA and Gated DeltaNet, it improves perplexity, zero-shot downstream accuracy, S-NIAH retrieval at and beyond the training context, length-extrapolation perplexity from 4K to 20K, and LongBench accuracy.
Dong Le, Thong Nguyen, Cong-Duy Nguyen +1
Nanyang Technological University · National University of Singapore · VinUniversity
Delta-rule linear attention improves recurrent memory updates by correcting what is already stored at the current write address before writing new content. However, the active correction is still anchored to that same write address. As a result, stale information stored at a different address cannot be actively removed before new content is written elsewhere. We propose Erase-then-Delta Attention (EDA), a memory update rule that decouples where to erase from where to write. The key insight is that recurrent memory models should not only correct the current write, but also selectively suppress outdated memory at an independently chosen address. Concretely, our method first applies a targeted erase step along a learned erase direction, and then performs the standard delta-style corrective write along the current write direction. This preserves the corrective behavior of delta-rule updates while expanding their memory-management capacity. Language-model pretraining experiments across dense 2.5B and MoE 25B-A2.8B model families show that EDA performs best in both settings. The gain persists after 80B-token long-context midtraining of the MoE models, where EDA also performs best in long-context evaluations from 4k to 128k contexts. A compact update analysis and memory-state probes suggest why: EDA keeps the delta-rule corrective write intact while allocating an additional cleanup path most strongly when passive decay is weak. These results suggest that recurrent memory models should decide not only what to write, but also what stale information to erase and where.
Xiao Li, Chengruidong Zhang, Hao Luo +15
1Qwen Team · 2Nanjing University · 3Zhejiang University