Organizations: College of Control Science and Engineering, Zhejiang University · Polytechnic Institute, Zhejiang University · Department of Mathematics, National University of Singapore · School of Science, Huzhou Normal University
Linear attention and state space models (SSMs) offer efficient alternatives to softmax attention by maintaining a fixed-size recurrent state, but this design requires careful management to preserve useful historical associations and limit interference. The delta rule helps reduce interference through correction of the value associated with each key, but it does not explicitly account for how related value features are distributed across the state. We observe that a state S∈Rdk×dv can be viewed as a bank of dk addresses storing dv-dimensional values. While a key-side read S⊤k retrieves the value associated with k, a value-side read Se aggregates addresses whose stored value features match the selector e. Based on this observation, we propose Dual DeltaNet (DDN), which extends Kimi Delta Attention (KDA) with a two-sided delta rule. The key-side delta rule reads with the current key to correct its associated value, while the complementary value-side delta rule reads with selected features of the current value to reorganize the addresses it occupies. We derive a chunkwise parallel training algorithm and a rank-2 recurrence that increases arithmetic intensity during inference. Compared with KDA, GDN-2, and Mamba-3, DDN achieves the strongest overall performance across language modeling, zero-shot downstream tasks, and retrieval, while incurring modest training and inference overhead relative to KDA. Code: https://github.com/ReichtumQian/DualDeltaNet
Figures & tables
Figure 1: Illustration of the value-side read and delta rule. (a) The selector et extracts the address response Stet of selected value features, which occupy addresses 1 and 3 in this example. (b) Comparing this response with the target (vt⊤et)kt produces a negative correction at addresses 1 and 3 and a positive correction at address 5, reorganizing the selected features toward the target address.
v~t
←vt−ℓt,
k~t
←(τt−βtv~t⊤et)kt−rt.
Algorithm 1 Rank-2 factorized form
Input
Output
FLOPs
Arithmetic intensity
St−1:(dk,dv)αt,kt,qt:(dk)vt:(dv)βt:(1)
St:(dk,dv)ot:(dv)
7dkdv+O(dk+dv)
2(2dkdv+3dk+2dv+1)7dkdv+O(dk+dv)≈1.75
Table 1: Arithmetic intensity for (a) KDA and (b) DDN, measured in FLOPs/byte for 2-byte inputs and outputs. The batch and head dimensions cancel out. DDN’s higher arithmetic intensity can improve hardware utilization during memory-bound phases such as autoregressive decoding.
Table 2: Results on language modeling and zero-shot downstream tasks at the 130M, 340M, and 1B scales after approximately 5B, 15B, and 50B FineWeb-Edu training tokens, respectively. All accuracy values are percentages. Bold and underlined values denote the best and second-best distinct scores within each model-size group in each column, respectively.
Model
SWDE
SQuAD
FDA
TQA
NQ
DROP
Avg.
130M models; 5B tokens
Mamba-3 SISO
5.85
18.97
0.82
35.13
4.47
11.60
12.81
Mamba-3 MIMO
6.57
16.89
1.36
34.83
4.72
13.51
12.98
GDN-2
10.35
21.35
2.09
37.03
5.61
16.77
15.53
KDA
8.82
20.07
1.91
37.92
5.70
16.68
15.18
DDN (ours)
10.62
21.95
2.63
39.40
5.96
16.53
16.18
Table 4: Real-world retrieval at the 130M, 340M, and 1B scales after approximately 5B, 15B, and 50B training tokens. Bold and underlined values denote the best and second-best distinct scores within each model-size group in each column.
Figure 3: Training throughput and peak allocated memory of 1B models on an NVIDIA H200. The horizontal axis denotes sequence length × batch size. All baselines retain their default recomputation settings.
Model
1024
2048
4096
8192
Prefill
Mamba-3 SISO
0.34
0.68
1.35
2.69
Mamba-3 MIMO
0.40
0.81
1.64
3.35
GDN-2
0.40
0.80
1.58
3.16
KDA
0.35
0.70
1.39
2.81
DDN (ours)
0.39
0.79
1.57
3.15
Table 5: Prefill and prefill + decode latency in seconds for 1B models on an NVIDIA H200 with batch size 64. Generation length equals input length. Bold and underlined values denote the lowest and second-lowest distinct latencies in each column within each group, respectively.
Model
Downstream
Retrieval
NIAH
130M models; 5B tokens
KDA
37.38
15.18
38.10
DDN ( et=v^t )
37.41
15.87
39.45
DDN
38.03
16.18
40.57
340M models; 15B tokens
KDA
42.40
21.30
51.20
Table 6: Ablation results for 130M and 340M models trained on approximately 5B and 15B FineWeb-Edu tokens, respectively. Downstream, Retrieval, and NIAH report average accuracy in percent. Bold and underlined values denote the best and second-best scores at each scale.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Depth
Heads
dk
dv
dmodel
dMLP
State size
130M scale
Mamba-3 SISO
12
24
64
64
768
2,208
98,304
Mamba-3 MIMO
12
24
64
64
768
2,064
98,304
GDN-2
12
6
128
128
768
2,112
98,304
KDA
12
6
128
128
816
2,336
98,304
DDN (ours)
12
6
128
128
800
2,320
98,304
Appendix
Table 7: Architecture configurations. For the linear-attention models, dk and dv are the key and value dimensions. For Mamba-3, they denote the state and head dimensions, respectively.
Scale
130M
340M
1B
Training steps
9,500
30,000
95,000
Peak learning rate
6×10−4
4×10−4
4×10−4
Warmup steps
512
1,024
1,024
Appendix
Table 8: Training steps, peak learning rates, and warmup steps of pretraining.
Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory. The hard part is not just what to forget, but how to edit this compressed memory without scrambling existing associations. Delta-rule models subtract the current read before writing a new value, and Kimi Delta Attention (KDA) sharpens forgetting with channel-wise decay. But the active edit still uses a single scalar gate to control two different things: how much old content to erase on the key side and how much new content to commit on the value side. We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and KDA by inheriting adaptive forgetting and channel-wise decay while addressing their shared limitation, the scalar tie between erasing and writing. Gated Delta Rule-2 separates these roles with a channel-wise erase gate b_t and a channel-wise write gate w_t, reducing to KDA when both gates collapse to the same scalar and to Gated DeltaNet when the decay also collapses. We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training. At 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval. Its advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, where it improves the evaluated multi-key retrieval setting and remains strong in both recurrent and hybrid settings. Code is available at https://github.com/NVlabs/GatedDeltaNet-2.
Linear attention reformulates sequence modeling as recurrent state evolution, enabling efficient linear-time inference. Under the key-value associative paradigm, existing approaches restrict the role of the query to the readout operation, decoupling it from state evolution. We show that query-conditioned state readout induces a structured value prediction over accumulated memory that complements key-based retrieval. Based on this insight, we propose Q-Delta, a query-aware delta rule that integrates mixed key-query prediction errors into state evolution, enabling jointly corrective dynamics while preserving delta-rule efficiency. We establish stability guarantees for the resulting dynamics and derive a hardware-efficient chunkwise-parallel formulation with a custom Triton implementation. Empirical results demonstrate stable optimization, competitive throughput, and consistent improvements over strong baselines on language modeling and long-context retrieval tasks.
Sumin Park, Seojin Kim, Noseong Park
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.
Linear attention enables efficient long-context inference by compressing token history into a fixed-size recurrent memory. This compression makes each update a trade-off between incorporating new information and preserving useful associations. Models such as DeltaNet, Gated DeltaNet, and KDA predict write strength from the current token representation, without explicitly tracking uncertainty in the stored memory. Yet this uncertainty matters: a new observation should have greater influence when the existing association is uncertain and less when it is already well supported. We introduce Kalman Delta Networks (KDNs), a family of linear-attention models that explicitly track memory uncertainty to guide each update. By formulating associative memory as a linear-Gaussian state-space model, KDNs propagate both the memory estimate and its uncertainty, using the Kalman gain to balance accumulated evidence against the reliability of new observations. This formulation also recovers standard delta-rule updates by replacing tracked covariance with a token-predicted isotropic surrogate. To support hardware-efficient training and inference, we derive Diagonal KDN and Isotropic KDN, which retain one uncertainty value per key channel and per head, respectively. Their uncertainty updates admit associative scans with logarithmic parallel depth, requiring only O(dk) and O(1) auxiliary state per head. Across controlled pretraining at 750M and 1.3B parameters, both variants consistently improve perplexity and mean downstream accuracy over the evaluated state-of-the-art linear-attention baselines.