DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding

Junqing Lin; Jingwei Sun; Guangzhong Sun

cs.DCAug 16, 2026

DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding

Authors: Junqing Lin, Jingwei Sun, Guangzhong Sun

Organizations: University of Science and Technology of China

Abstract

Linear attention models eliminate the quadratic prefix computation and context-growing KV cache of softmax attention by replacing pairwise token interactions with recurrent state updates. However, existing decoding implementations often materialize and write back the full recurrent state after every generated token, making state maintenance a major source of memory traffic, especially for models with large states and many heads. This paper presents DeltaLog, a recurrent-state decoding scheme that reduces this overhead without changing the model semantics. Specifically, DeltaLog represents the recurrent state as a dense base state together with a bounded log of recent compact updates. Most decode steps append only compact update factors to this log, while periodic merge steps fold the accumulated updates back into the dense base state. Thus, the model observes the same dense state as in eager decoding, but most full-state write-backs are replaced by lightweight append operations. We implement DeltaLog for GDN, KDA, and RWKV6 and integrate it into a prototype serving stack. Across these models, DeltaLog accelerates the recurrent-state update kernel by up to $1.86\times$ , reduces profiled recurrent-state write traffic by up to $7.83\times$ , and achieves $1.05$ -- $1.20\times$ end-to-end serving speedups over dense recurrent baselines.

Figures & tables

Panel of Figure 1 — Figure 1. The state-update tax. A token contributes compact factors with $O(BH(d_{k}+d_{v}))$ values, but eager materialization writes back the full $[B,H,d_{k},d_{v}]$ recurrent state, touching $O(BHd_{k}d_{v})$ elements per token.

Model	State Transform	Update Pair $(a_{t},b_{t})$	Decay Form
Linear Attention	$S_{t-1}$	$(\phi(k_{t}),v_{t})$	none
RetNet	$\gamma_{t}S_{t-1}$	$(k_{t},v_{t})$	scalar
GLA	$D(\lambda_{t})S_{t-1}$	$(k_{t},v_{t})$	diagonal
RWKV6	$D(\lambda_{t})S_{t-1}$	$(k_{t},v_{t})$	diagonal
GDN	$D(\lambda_{t})S_{t-1}$	$(k_{t},\beta_{t}(v_{t}-k_{t}^{\top}S_{t-1}))$	scalar
KDA	$D(\lambda_{t})S_{t-1}$	$(k_{t},\beta_{t}(v_{t}-k_{t}^{\top}S_{t-1}))$	diagonal

Table 1. Representative recurrent formulations written in the components used by our systems analysis. The columns list the model-specific ingredients that instantiate Eq. 3 .

Panel of Figure 2 — Figure 2. Recurrent-state materialization occupies a growing share of full-model decode latency.

Figure 3. Roofline view of eager dense recurrent-state updates. Their arithmetic intensity is far below machine balance, making them strongly memory-bound.

Figure 4. DeltaLog overview. Eager recurrent decoding materializes the state at every token. DeltaLog instead stores a base plus a bounded compact update log, appends while

L<C

, and materializes the logical state into the dense base when

L=C

Panel of Figure 5 — Figure 5. The append–merge kernel ownership path comprises three stages: shared replay preparation, parallel append decode across value tiles, and single-writer metadata commit. The merge path uses the same structure but writes back dense-base tiles.

Figure 6. Single-token decode performance on H200. Bars show TPOT, and lines show DeltaLog speedup over FLA.

Figure 7. Single-token decode performance on RTX 4090. Bars show TPOT, and lines show DeltaLog speedup over FLA.

GPU	Deployment	Memory	Peak Memory	Peak Tensor
GPU	Class	Memory	Bandwidth	Throughput
RTX 4090	commodity	24 GB	1008 GB/s	165.2 TFLOPS
H200 NVL	data-center	141 GB	4.8 TB/s	1671 TFLOPS

Table 2. Hardware configuration.

Operator	Shape	State	Log $(K/U)$	$M$
GDN	32/128/128	FP32	BF16/FP32	8
KDA	32/128/128	FP32	BF16/FP32	4
RWKV6	32/128/128	FP32	BF16/BF16	4

Table 3. Recurrent-kernel configurations. Shape reports

H/d_{k}/d_{v}

; log-decay metadata is FP32 for all operators.

Figure 8. Latency and profiled DRAM traffic for GDN at

B=128

and

M=8

. (a) reports dense decoding, append latency by log size, merge latency, and the amortized result. (b) reports the corresponding DRAM-traffic breakdown.

Figure 9. Split-replay ablation on H200 with CUDA Graph replay. Latencies are amortized over one merge cycle; GDN, KDA, and RWKV6 are shown at

M=4

and

M=8

Figure 10. Interval sweep for DeltaLog at

B=128

. (a) Amortized per-token latency. (b) Speedup over FLA.

Figure 11. Scaling with total recurrent-state size on H200.

Figure 12. Kernel correctness. (a) Output relative error. (b) Materialized recurrent-state relative error.

Model (Operator)	Method	Batch Size
Model (Operator)	Method	64	128	256
Qwen-35B (GDN)	Baseline	8.36	10.30	16.07
	DeltaLog	7.75	9.01	13.40
	Speedup	1.08 $\times$	1.14 $\times$	1.20 $\times$
Kimi-48B (KDA)	Baseline	7.66	8.92	13.98
	DeltaLog	7.33	8.13	12.54

Table 4. End-to-end decode TPOT (ms/token) on H200 NVL.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

Q-Delta: Beyond Key-Value Associative State Evolution
Jun 7, 2026Sumin Park, Seojin Kim, Noseong ParkLinear Attention Delta-Rule Linear Attention
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Sep 29, 2026Bingchen Yao, Haobo Xu, Haokun Lin +6LLM Quantization LLM Inference Acceleration
Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS +3
Kaczmarz Linear Attention
May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuLong-Context Language Modeling Recurrent Transformers
School of Mathematics and Statistics Xi’an Jiaotong University Xi’an, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing, China

Jun 7, 2026J/K move · Enter open · S save

Q-Delta: Beyond Key-Value Associative State Evolution

Sumin Park, Seojin Kim, Noseong Park

Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.

Linear Attention Delta-Rule Linear Attention Efficient Language Model Inference

Open paper AskPDF arXiv

Linear attention reformulates sequence modeling as recurrent state evolution, enabling efficient linear-time inference. Under the key-value associative paradigm, existing approaches restrict the role of the query to the readout operation, decoupling it from state evolution. We show that query-conditioned state readout induces a structured value prediction over accumulated memory that complements key-based retrieval. Based on this insight, we propose Q-Delta, a query-aware delta rule that integrates mixed key-query prediction errors into state evolution, enabling jointly corrective dynamics while preserving delta-rule efficiency. We establish stability guarantees for the resulting dynamics and derive a hardware-efficient chunkwise-parallel formulation with a custom Triton implementation. Empirical results demonstrate stable optimization, competitive throughput, and consistent improvements over strong baselines on language modeling and long-context retrieval tasks.

Abstract

LLM Inference Acceleration

Explore similar work

Q-Delta: Beyond Key-Value Associative State Evolution

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Kaczmarz Linear Attention