Dual DeltaNet: Value-Side Memory Reorganization for Linear Attention

Yixiao Qian; Yaxin Zhang; Song Chen; Jiaxu Liu; Shengze Cai; Chao Xu

cs.LGOct 28, 2026

Dual DeltaNet: Value-Side Memory Reorganization for Linear Attention

Authors: Yixiao Qian, Yaxin Zhang, Song Chen, Jiaxu Liu, Shengze Cai, Chao Xu

Organizations: College of Control Science and Engineering, Zhejiang University · Polytechnic Institute, Zhejiang University · Department of Mathematics, National University of Singapore · School of Science, Huzhou Normal University

Abstract

Linear attention and state space models (SSMs) offer efficient alternatives to softmax attention by maintaining a fixed-size recurrent state, but this design requires careful management to preserve useful historical associations and limit interference. The delta rule helps reduce interference through correction of the value associated with each key, but it does not explicitly account for how related value features are distributed across the state. We observe that a state $\boldsymbol{S}\in\mathbb{R}^{d_k\times d_v}$ can be viewed as a bank of $d_k$ addresses storing $d_v$ -dimensional values. While a key-side read $\boldsymbol{S}^\top\boldsymbol{k}$ retrieves the value associated with $\boldsymbol{k}$ , a value-side read $\boldsymbol{S}\boldsymbol{e}$ aggregates addresses whose stored value features match the selector $\boldsymbol{e}$ . Based on this observation, we propose Dual DeltaNet (DDN), which extends Kimi Delta Attention (KDA) with a two-sided delta rule. The key-side delta rule reads with the current key to correct its associated value, while the complementary value-side delta rule reads with selected features of the current value to reorganize the addresses it occupies. We derive a chunkwise parallel training algorithm and a rank-2 recurrence that increases arithmetic intensity during inference. Compared with KDA, GDN-2, and Mamba-3, DDN achieves the strongest overall performance across language modeling, zero-shot downstream tasks, and retrieval, while incurring modest training and inference overhead relative to KDA. Code: https://github.com/ReichtumQian/DualDeltaNet

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of the value-side read and delta rule. (a) The selector $\boldsymbol{e}_{t}$ extracts the address response $\boldsymbol{S}_{t}\boldsymbol{e}_{t}$ of selected value features, which occupy addresses 1 and 3 in this example. (b) Comparing this response with the target $(\boldsymbol{v}_{t}^{\top}\boldsymbol{e}_{t})\boldsymbol{k}_{t}$ produces a negative correction at addresses 1 and 3 and a positive correction at address 5, reorganizing the selected features toward the target address.

	$\displaystyle\tilde{\boldsymbol{v}}_{t}$	$\displaystyle\leftarrow\boldsymbol{v}_{t}-\boldsymbol{\ell}_{t},$
	$\displaystyle\tilde{\boldsymbol{k}}_{t}$	$\displaystyle\leftarrow(\tau_{t}-\beta_{t}\tilde{\boldsymbol{v}}_{t}^{\top}\boldsymbol{e}_{t})\boldsymbol{k}_{t}-\boldsymbol{r}_{t}.$

Algorithm 1 Rank-2 factorized form

Input	Output	FLOPs	Arithmetic intensity
$\begin{gathered}\boldsymbol{S}_{t-1}:(d_{k},d_{v})\\ \boldsymbol{\alpha}_{t},\boldsymbol{k}_{t},\boldsymbol{q}_{t}:(d_{k})\\ \boldsymbol{v}_{t}:(d_{v})\\ \beta_{t}:(1)\end{gathered}$	$\begin{gathered}\boldsymbol{S}_{t}:(d_{k},d_{v})\\ \boldsymbol{o}_{t}:(d_{v})\end{gathered}$	$7d_{k}d_{v}+\mathcal{O}(d_{k}+d_{v})$	$\displaystyle\frac{7d_{k}d_{v}+\mathcal{O}(d_{k}+d_{v})}{2(2d_{k}d_{v}+3d_{k}+2d_{v}+1)}\approx 1.75$

Table 1: Arithmetic intensity for (a) KDA and (b) DDN, measured in FLOPs/byte for 2-byte inputs and outputs. The batch and head dimensions cancel out. DDN’s higher arithmetic intensity can improve hardware utilization during memory-bound phases such as autoregressive decoding.

\begin{gathered}\boldsymbol{F}_{R,[n]}\leftarrow\boldsymbol{M}_{R,[n]}\boldsymbol{\Gamma}_{[n]}\boldsymbol{E}_{[n]},\\ \boldsymbol{Q}_{\mathrm{eff},[n]}\leftarrow\boldsymbol{Q}_{\pi,[n]}-\boldsymbol{G}_{L,[n]}\bar{\boldsymbol{U}}_{[n]},\\ \boldsymbol{\Phi}_{[n]}\leftarrow\boldsymbol{G}_{L,[n]}\boldsymbol{N}_{[n]}\boldsymbol{M}_{R,[n]}^{\top}.\end{gathered}

Algorithm 2 DDN chunkwise computation

Model	FW-Edu	WT2	LAMB.	LAMB.	PIQA	HellaS.	WinoGr.	ARC-E	ARC-C	OBQA	Avg.
	train ppl $\downarrow$	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc n $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$
130M models; 5B tokens
Mamba-3 SISO	15.10	44.58	68.20	26.37	63.11	31.97	50.91	50.46	20.39	18.80	37.43
Mamba-3 MIMO	14.93	43.89	73.03	25.97	62.08	32.22	52.64	51.01	20.73	19.60	37.75
GDN-2	15.22	42.79	67.88	26.41	62.51	32.28	51.70	50.93	20.14	19.20	37.60
KDA	14.79	42.53	69.91	25.69	62.95	32.14	52.09	49.66	20.14	19.00	37.38

Table 2: Results on language modeling and zero-shot downstream tasks at the 130M, 340M, and 1B scales after approximately 5B, 15B, and 50B FineWeb-Edu training tokens, respectively. All accuracy values are percentages. Bold and underlined values denote the best and second-best distinct scores within each model-size group in each column, respectively.

130M models; 5B tokens
Model	SWDE	SQuAD	FDA	TQA	NQ	DROP	Avg.
Mamba-3 SISO	5.85	18.97	0.82	35.13	4.47	11.60	12.81
Mamba-3 MIMO	6.57	16.89	1.36	34.83	4.72	13.51	12.98
GDN-2	10.35	21.35	2.09	37.03	5.61	16.77	15.53
KDA	8.82	20.07	1.91	37.92	5.70	16.68	15.18
DDN (ours)	10.62	21.95	2.63	39.40	5.96	16.53	16.18

Table 4: Real-world retrieval at the 130M, 340M, and 1B scales after approximately 5B, 15B, and 50B training tokens. Bold and underlined values denote the best and second-best distinct scores within each model-size group in each column.

Figure 3: Training throughput and peak allocated memory of 1B models on an NVIDIA H200. The horizontal axis denotes sequence length

\times

batch size. All baselines retain their default recomputation settings.

Prefill
Model	1024	2048	4096	8192
Mamba-3 SISO	0.34	0.68	1.35	2.69
Mamba-3 MIMO	0.40	0.81	1.64	3.35
GDN-2	0.40	0.80	1.58	3.16
KDA	0.35	0.70	1.39	2.81
DDN (ours)	0.39	0.79	1.57	3.15

Table 5: Prefill and prefill + decode latency in seconds for 1B models on an NVIDIA H200 with batch size 64. Generation length equals input length. Bold and underlined values denote the lowest and second-lowest distinct latencies in each column within each group, respectively.

Model	Downstream	Retrieval	NIAH
130M models; 5B tokens
KDA	37.38	15.18	38.10
DDN ( $\boldsymbol{e}_{t}=\hat{\boldsymbol{v}}_{t}$ )	37.41	15.87	39.45
DDN	38.03	16.18	40.57
340M models; 15B tokens
KDA	42.40	21.30	51.20

Table 6: Ablation results for 130M and 340M models trained on approximately 5B and 15B FineWeb-Edu tokens, respectively. Downstream, Retrieval, and NIAH report average accuracy in percent. Bold and underlined values denote the best and second-best scores at each scale.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

130M scale
Model	Depth	Heads	$d_{k}$	$d_{v}$	$d_{\mathrm{model}}$	$d_{\mathrm{MLP}}$	State size
Mamba-3 SISO	12	24	64	64	768	2,208	98,304
Mamba-3 MIMO	12	24	64	64	768	2,064	98,304
GDN-2	12	6	128	128	768	2,112	98,304
KDA	12	6	128	128	816	2,336	98,304
DDN (ours)	12	6	128	128	800	2,320	98,304

Appendix

Table 7: Architecture configurations. For the linear-attention models,

d_{k}

and

d_{v}

are the key and value dimensions. For Mamba-3, they denote the state and head dimensions, respectively.

Scale	130M	340M	1B
Training steps	9,500	30,000	95,000
Peak learning rate	$6\times 10^{-4}$	$4\times 10^{-4}$	$4\times 10^{-4}$
Warmup steps	512	1,024	1,024

Appendix

Table 8: Training steps, peak learning rates, and warmup steps of pretraining.

Linear Attention

Linear attention reduces sequence-length costs by replacing pairwise token interactions with kernel features, recurrent states, or compact memory.

See all 131 papers

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzLong-Context Language Modeling Recurrent Transformers
NVIDIA
Q-Delta: Beyond Key-Value Associative State Evolution
Jun 7, 2026Sumin Park, Seojin Kim, Noseong ParkLinear Attention Delta-Rule Linear Attention
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.
Kalman Delta Networks: Uncertainty-aware Associative Memory
Sep 7, 2026Ngoc Bui, Tinglin Huang, Rex YingMemory-Augmented Neural Networks Linear Attention
Department of Computer Science, Yale University

May 21, 2026J/K move · Enter open · S save

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Ali Hatamizadeh, Yejin Choi, Jan Kautz

NVIDIA

Long-Context Language Modeling Recurrent Transformers Linear Attention Delta-Rule Linear Attention Gated Delta Networks

Open paper AskPDF arXiv

Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory. The hard part is not just what to forget, but how to edit this compressed memory without scrambling existing associations. Delta-rule models subtract the current read before writing a new value, and Kimi Delta Attention (KDA) sharpens forgetting with channel-wise decay. But the active edit still uses a single scalar gate to control two different things: how much old content to erase on the key side and how much new content to commit on the value side. We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and KDA by inheriting adaptive forgetting and channel-wise decay while addressing their shared limitation, the scalar tie between erasing and writing. Gated Delta Rule-2 separates these roles with a channel-wise erase gate b_t and a channel-wise write gate w_t, reducing to KDA when both gates collapse to the same scalar and to Gated DeltaNet when the decay also collapses. We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training. At 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval. Its advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, where it improves the evaluated multi-key retrieval setting and remains strong in both recurrent and hybrid settings. Code is available at https://github.com/NVlabs/GatedDeltaNet-2.

Abstract

Linear Attention

Explore similar work

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Q-Delta: Beyond Key-Value Associative State Evolution

Kalman Delta Networks: Uncertainty-aware Associative Memory