A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGA

Neelesh Gupta; Peter Wang; Rajgopal Kannan; Viktor K. Prasanna

cs.ARMar 6, 2026

A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGA

Authors: Neelesh Gupta, Peter Wang, Rajgopal Kannan, Viktor K. Prasanna

Organizations: University of Southern California, USA · DEVCOM Army Research Office, USA

Abstract

Gated DeltaNet (GDN) is a linear attention mechanism that replaces the growing KV cache with a fixed-size recurrent state. Hybrid LLMs like Qwen3-Next use 75% GDN layers and achieve competitive accuracy to attention-only models. However, at batch-1, GDN decode is memory-bound on GPUs since the full recurrent state must be round-tripped through HBM every token. We show that this bottleneck is architectural, not algorithmic, as all subquadratic sequence models exhibit arithmetic intensities below 1 FLOP/B at decode time, making them more memory-bound than standard Transformers. We present an FPGA accelerator that eliminates this bottleneck by holding the full 2 MB recurrent state persistently in on-chip BRAM, converting the workload from memory-bound to compute-bound. Our design fuses the GDN recurrence into a five-phase pipelined datapath that performs only one read and one write pass over each state matrix per token, exploits Grouped Value Attention for paired-head parallelism, and overlaps preparation, computation, and output storage via dataflow pipelining. We explore four design points on an AMD Alveo U55C using Vitis HLS, varying head-level parallelism from 2 to 16 value-heads per iteration. Our fastest configuration achieves 63 $μ$ s per token, 4.5 $\times$ faster than the GPU reference on NVIDIA H100 PCIe. Post-implementation power analysis reports 9.96 W on-chip, yielding up to 60 $\times$ greater energy efficiency per token decoded.

Figures & tables

Fig. 1: Batch-1 decode arithmetic intensity on the H100 FP32 roofline. All architectures fall deep in the memory-bound regime. MHSA Transformers [ 1 ] achieve an arithmetic intensity near 1 FLOP/B whereas subquadratic models (GDN [ 2 ] , DeltaNet [ 3 ] , Mamba [ 4 ] , Mamba-2 [ 5 ] ) fall well below 1 FLOP/B.

Fig. 2: Qwen3-Next [ 6 ] hybrid architecture. GDN layers (3:1 ratio) replace softmax attention with fixed-size recurrent state, making GDN decode the dominant per-token primitive.

	NVIDIA H100 PCIe	AMD Alveo U55C
Technology	TSMC 4N	TSMC 16 nm
FP32 peak	51 TFLOP/s	$\sim$ 2.7 TFLOP/s a
Memory BW	2.0 TB/s (HBM3)	460 GB/s (HBM2e)
On-chip memory	50 MB (L2 cache)	17.6 MB (BRAM)
BRAM blocks	—	4,032 $\times$ BRAM_18K
DSP slices	—	9,024 $\times$ DSP48E2

TABLE I: Platform Specifications

Quantity	GPU	FPGA (ours)
Compute (FLOPs)	$\sim$ 4.2 M
State I/O (bytes)	4,194,304	0
Token I/O (bytes)	49,664	49,664
Total off-chip I/O	$\sim$ 4.24 MB	$\sim$ 48.5 KB
Op. intensity (FLOP/B)	$\sim$ 1.0	$\sim$ 88

TABLE II: Per-Token Computational Profile (

h_{v}{=}32

d{=}128

, FP32)

Fig. 3: System architecture for $H_{\mathrm{iter}}{=}8$ . Per-token inputs arrive via three AXI ports (left) and are buffered in on-chip BRAM. The prepare stage computes gates $g_{t}$ , $\beta_{t}$ . Four GVA pairs each share a q/k datapath across two PEs; each PE executes the fused five-phase pipeline with $P_{K}{=}16$ column parallelism (MAC $\times$ 16, II $=$ 1). The 128 dual-port BRAM arrays (bottom) hold the persistent 2 MB state, partitioned by head and column bank, with all four iteration slices sharing the same physical banks at different addresses.

a Assumes constant $T_{\mathrm{iter}}\approx 2{,}106$ cycles.
$H_{\mathrm{iter}}$ (v-heads/iter)	2	4	8	16
$N_{\mathrm{iter}}$ (iterations)	16	8	4	2
GVA pairs/iter	1	2	4	8
Predicted $L$ @300 MHz ( $\mu$ s) a	124.2	67.7	39.4	25.3

TABLE III: Design-Space Configurations

Fig. 4: Per-token latency comparison for batch-1 GDN decode. All FPGA configurations outperform the GPU baseline.

H_{\mathrm{iter}}{=}8

is optimal;

H_{\mathrm{iter}}{=}16

regresses due to pipeline interval inflation.

Platform	Cycles	@300 MHz	@250 MHz	Speedup
H100 (GPU)	—	285		1.0 $\times$
$H_{\mathrm{iter}}{=}2$	42,538	141.7	170.2	2.0 $\times$
$H_{\mathrm{iter}}{=}4$	26,252	87.4	105.0	3.3 $\times$
$H_{\mathrm{iter}}{=}8$	18,978	63.2	75.9	4.5 $\times$
$H_{\mathrm{iter}}{=}16$	23,206	77.4	92.8	3.7 $\times$

TABLE IV: Per-Token Decode Latency (

\mu

s) and Speedup vs. GPU Baseline

a Board-level TDP [ 16 , 17 ] (conservative upper bound).
Platform	Power	Latency	Energy/token
H100 PCIe (GPU)	350 W a	285 $\mu$ s	99.8 mJ
$H_{\mathrm{iter}}{=}2$ (impl.)	9.96 W b	161.7 $\mu$ s	1.61 mJ
$H_{\mathrm{iter}}{=}4$	$\leq$ 150 W a	87.4 $\mu$ s	$\leq$ 13.1 mJ
$H_{\mathrm{iter}}{=}8$	$\leq$ 150 W a	63.2 $\mu$ s	$\leq$ 9.5 mJ
$H_{\mathrm{iter}}{=}16$	$\leq$ 150 W a	77.4 $\mu$ s	$\leq$ 11.6 mJ

TABLE V: Per-Token Energy Comparison

Fig. 5: FPGA resource utilization and latency across configurations. BRAM plateaus at 25%; DSP, FF, and LUT scale with

H_{\mathrm{iter}}

. Latency (right axis) reaches a minimum at

H_{\mathrm{iter}}{=}8

;

H_{\mathrm{iter}}{=}16

increases latency due to pipeline interval inflation despite fewer iterations.

	BRAM a		DSP		FF	LUT
	Used	%	Used	%	%	%
$H_{\mathrm{iter}}{=}2$	523	12	570	6	7	7
$H_{\mathrm{iter}}{=}4$	1,035	25	1,090	12	12	13
$H_{\mathrm{iter}}{=}8$	1,035	25	2,130	23	24	25
$H_{\mathrm{iter}}{=}16$	1,035	25	4,162	46	54	52
a BRAM_18K count; 4,032 available.

TABLE VI: FPGA Resource Utilization (Alveo U55C)

Panel of Figure 12 — Fig. 6: Post-implementation placement on Alveo U55C. Left: $H_{\mathrm{iter}}{=}2$ fits comfortably within SLR0 (12% BRAM, 6% DSP; 9.96 W on-chip power, 263 MHz). Right: $H_{\mathrm{iter}}{=}4$ saturates SLR0 and spills into SLR1 (77% SLR-level BRAM, 39% LUT), causing routing failure (88,725 unroutable signals) despite meeting post-synthesis timing at 2.370 ns.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding
Aug 16, 2026Junqing Lin, Jingwei Sun, Guangzhong SunLLM Inference Acceleration Linear Attention
University of Science and Technology of China
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10LLM Quantization LLM Inference Acceleration
UC Berkeley · University of Washington · MIT +2
Kaczmarz Linear Attention
May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuLong-Context Language Modeling Recurrent Transformers
School of Mathematics and Statistics Xi’an Jiaotong University Xi’an, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing, China

Aug 16, 2026J/K move · Enter open · S save

DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding

Junqing Lin, Jingwei Sun, Guangzhong Sun

University of Science and Technology of China

LLM Inference Acceleration Linear Attention

Open paper AskPDF arXiv

Linear attention models eliminate the quadratic prefix computation and context-growing KV cache of softmax attention by replacing pairwise token interactions with recurrent state updates. However, existing decoding implementations often materialize and write back the full recurrent state after every generated token, making state maintenance a major source of memory traffic, especially for models with large states and many heads. This paper presents DeltaLog, a recurrent-state decoding scheme that reduces this overhead without changing the model semantics. Specifically, DeltaLog represents the recurrent state as a dense base state together with a bounded log of recent compact updates. Most decode steps append only compact update factors to this log, while periodic merge steps fold the accumulated updates back into the dense base state. Thus, the model observes the same dense state as in eager decoding, but most full-state write-backs are replaced by lightweight append operations. We implement DeltaLog for GDN, KDA, and RWKV6 and integrate it into a prototype serving stack. Across these models, DeltaLog accelerates the recurrent-state update kernel by up to

1.86\times

, reduces profiled recurrent-state write traffic by up to

7.83\times

, and achieves

1.05

1.20\times

end-to-end serving speedups over dense recurrent baselines.

Abstract

LLM Inference Acceleration

Explore similar work

DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

Kaczmarz Linear Attention