Linear-attention language models replace a growing key-value cache with a recurrent state of fixed size. On an edge FPGA, this state can exceed the on-chip memory and must be paged to external memory. This paper asks whether such paging dominates the cost of decoding (generating tokens). We built a hardware tile that computes one complete layer of a custom 369-million-parameter hybrid Gated DeltaNet model on an Agilex 5 FPGA and pages its recurrent state. The model state needs 6.2 MiB, whereas the device has 0.87 MiB of block memory. Paging adds 26.0 MB of memory traffic per decoded token, at most 11.2 percent of the calculated total. On the board at 75 MHz, with synthetic weights, one token takes 11.15 ms in this layer. The matrix-vector products with the weights take about 62 percent of this time and the state phase, including the paging, 13 percent. The tile is one node of a multicast mesh of tri-mode processing elements. We measure the error of the fixed-point arithmetic against floating point and check that the hardware agrees bit for bit with a fixed-point reference. In simulation, the state logic agrees with the reference on all 18 recurrent layers. On the board, six builds pass all 30 runs of 64 tokens and two runs of 1,024 tokens. For this model and memory interface, the delivery of the weights limits the decoding performance, and the recurrent state does not. State paging is therefore practical for such a model on an edge FPGA.
Figures & tables
Figure 1: The application and the platform. Top: the multi-rate architecture. The acoustic front-end detects a hidden vehicle, the cache answers queries about an unchanged scene with the hit rate η , and a cache miss invokes the language model, of which the decode tile implements one recurrent layer. (a) The DK-A5E013BM16AEA development kit, which we used for all measurements. (b) Where each part runs. The host reaches the programmable logic only through JTAG, and the tile reaches its weights and its state pages only through the LPDDR4 memory.
Work
What is implemented
Recurrent state
Device and evidence
Exactness
Gupta et al. [ 5 ]
Gated DeltaNet decode kernel, five fused phases
2 MB, resident in block memory
Alveo U55C (datacenter). Cycle counts from high-level synthesis and power after implementation. No board run reported
—
DUET [ 31 ]
Mamba–Transformer hybrid, prefill and decode packages
kept in package-local buffers, residency not stated
28 nm ASIC synthesis and a cycle-accurate simulator. No prototype
FPGA (LightMamba, SpecMamba, LowRank-SSM, eMamba) or ASIC synthesis (MARCA, Mamba-X)
—
TRINE [ 26 ]
shared processing array, switched among systolic, SIMD and adder-tree dataflows
—
Alveo U50 and ZCU104 named. Switch cost reported as negligible in wall time, not in cycles
—
BitLogic [ 30 ]
logic-gate networks
—
FPGA, post-route figures. Verilator
bit-exact to its Python reference, in simulation
This work
complete Gated DeltaNet layer of a 369M hybrid model
paged, 26.0 MB per decoded token
Agilex 5 board. Six final builds pass 30 runs of 64 layer-tokens
state engine exact in 72 simulation runs. Kernel and complete layer checked on the board
Table 1: Comparison with the closest prior work on the four criteria of this paper. “Resident” means that the recurrent state stays on chip. “Paged” means that the state is read from and written to external memory in pages. Cells report what the cited works state. A dash means that the work does not address that criterion.
Figure 2: The mode controller of the processing element (Section 4 ). A mode switch costs D+3 cycles, where D is the number of cycles that the current engine needs to become idle (drain). Arrow labels read “condition / action”.
Figure 3: The implemented system. Left: the mesh, in which five nodes are tri-mode processing elements and node 5 is the decode tile. Right: the decode tile. The block names are those of the RTL: sys_csr is the command register file, pe_port with mm_pe is a processing element, ctrl_unit is the control datapath, matvec is the matrix-vector engine, and state_pager with gdn_head_core is the state pager with the state engine. The mesh carries commands and results only, and the weights and the state pages use separate 512-bit ports to the LPDDR4 memory. The peer port can forward flits to a second system.
Vocabulary projection in
Component
bf16
INT4
INT4 weights of the layers
148.45
148.45
Scales of the weights
9.28
9.28
Vocabulary projection
143.36
38.08
KV-cache reads of the attention layers
9.14
9.14
State read and write, implemented
25.98
25.98
Table 2: Calculated memory traffic per decoded token for the whole model, in MB, at a context of 744 tokens. The two columns differ in the number format of the vocabulary projection. The last row gives the state traffic without headers and padding, for comparison.
Shift limit ( SH_MAX )
Page rounding
Max Eo
Max ES
31 (reference)
no
4.69×10−4
4.40×10−4
3 (original RTL)
no
5.15×10−2
1.91×10−1
31 (reference)
yes
6.43×10−3
1.10×10−3
3 (original RTL)
yes
5.15×10−2
1.91×10−1
Table 3: Relative errors of the fixed-point state engine against fp32, computed with ( 4 ). Every row covers 18 recurrent layers, two prompts and 64 tokens. Page rounding, where used, is applied at initialization and after each token. These are software results.
Figure 4: Cache hit rate η of each of the 83 clips for three versions of the cache policy. Boxes span the quartiles, and whiskers span the full range. Diamonds mark the means, and points mark the individual clips. The dashed line is the feasibility threshold η≈0.80 (Section 3 ). Only the full policy places every clip above the threshold, and its lowest clip is at η=0.852 .
NL=64 , W=16
NL=128 , W=8 (chosen)
Cycles per layer-token
446,100
393,935
Time at 150 MHz
2.97 ms
2.63 ms
ALM
38,888 (83 %)
27,105 (58 %)
M20K
234 (65 %)
250 (70 %)
DSP
88 (23 %)
112 (30 %)
Fmax
155.6 MHz
158.8 MHz
Table 4: The two tile configurations that fit the device, from the design study. NL and W are the numbers of lanes of the matrix-vector engine and of the state engine. Cycle counts come from RTL simulation with the memory model, and times are calculated at 150 MHz. ALM, M20K and DSP are the logic, memory and multiplier blocks of the device, and Fmax is the maximum clock frequency. A third configuration, with NL=128 and W=16 , does not fit the device.
Config.
ALMs
M20Ks
DSPs
Fmax (MHz)
P
29,958–29,979
298
108
102.64–110.66
M
40,025–40,077
305
128
98.56–108.77
Table 5: The six final board builds. P is the tile alone, and M is the complete system (Fig. 3 ). All builds run at 75 MHz. The ranges span the three placement seeds. Fmax is the reported value for the clock of the tile.
Test
P passes
M passes
Layer-tokens
Runs after programming
15/15
15/15
1,920
Runs of 1,024 layer-tokens
1/1
1/1
2,048
Initial run and ten soft resets
11/11
11/11
352
Table 6: Results of the three board tests of the complete layer, for the configurations P and M.
Figure 5: Roofline plot of the memory link for the weights, from the experiment with the state on chip. It shows the rate of MACs as a function of the rate at which weight bytes arrive. The layer performs 12,736,512 MACs over a weight image of 6,796,032 bytes, which gives 1.874 MACs per byte (sloped line). The 128 lanes limit the rate to 128 MACs per cycle (horizontal line), and the two lines meet at 68.3 bytes per cycle. Filled markers are measurements on the device with bursts of 8 and 128 beats, and the open marker is the memory model. Dashed lines show the peak rate of the LPDDR4 memory, 8.53 GB/s, at 75 and 150 MHz.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Gated DeltaNet (GDN) is a linear attention mechanism that replaces the growing KV cache with a fixed-size recurrent state. Hybrid LLMs like Qwen3-Next use 75% GDN layers and achieve competitive accuracy to attention-only models. However, at batch-1, GDN decode is memory-bound on GPUs since the full recurrent state must be round-tripped through HBM every token. We show that this bottleneck is architectural, not algorithmic, as all subquadratic sequence models exhibit arithmetic intensities below 1 FLOP/B at decode time, making them more memory-bound than standard Transformers. We present an FPGA accelerator that eliminates this bottleneck by holding the full 2 MB recurrent state persistently in on-chip BRAM, converting the workload from memory-bound to compute-bound. Our design fuses the GDN recurrence into a five-phase pipelined datapath that performs only one read and one write pass over each state matrix per token, exploits Grouped Value Attention for paired-head parallelism, and overlaps preparation, computation, and output storage via dataflow pipelining. We explore four design points on an AMD Alveo U55C using Vitis HLS, varying head-level parallelism from 2 to 16 value-heads per iteration. Our fastest configuration achieves 63 μs per token, 4.5× faster than the GPU reference on NVIDIA H100 PCIe. Post-implementation power analysis reports 9.96 W on-chip, yielding up to 60× greater energy efficiency per token decoded.
Neelesh Gupta, Peter Wang, Rajgopal Kannan +1
University of Southern California, USA · DEVCOM Army Research Office, USA
Linear-attention language models keep their context in a recurrent state of fixed size, where an inference system stores one copy of this state for every sequence it decodes at the same time. Storing the state at lower precision therefore lets more sequences share an accelerator. Existing state quantizers round the state after every generated token and report that four bits lose accuracy unless the bit width is raised or the model is fine-tuned.\par We introduce PACE, a post-hoc four-bit state quantizer. PACE rounds the state only once every few generated tokens, as chunked kernels already do once per chunk during prefill, and keeps the tokens generated in between in a small buffer. Rounding less often reduces the error that accumulates in the state. Additionally, we find that the large entries of the gated-delta state gather in columns, because some channels of the update written at each token remain large across many tokens. PACE therefore shares each quantization scale within one column. It also keeps the heads that retain rounding error longest at eight bits. On four hybrid models from 4B to 35B parameters, PACE with mixed four- and eight-bit heads is competitive with eight-bit DAMP on the three larger models at about two thirds of its memory, PACE with four-bit heads throughout scores higher than Q-Mamba at equal memory, and rounding every few tokens preserves the decoding speedup of the buffer.
Linear attention models eliminate the quadratic prefix computation and context-growing KV cache of softmax attention by replacing pairwise token interactions with recurrent state updates. However, existing decoding implementations often materialize and write back the full recurrent state after every generated token, making state maintenance a major source of memory traffic, especially for models with large states and many heads. This paper presents DeltaLog, a recurrent-state decoding scheme that reduces this overhead without changing the model semantics. Specifically, DeltaLog represents the recurrent state as a dense base state together with a bounded log of recent compact updates. Most decode steps append only compact update factors to this log, while periodic merge steps fold the accumulated updates back into the dense base state. Thus, the model observes the same dense state as in eager decoding, but most full-state write-backs are replaced by lightweight append operations. We implement DeltaLog for GDN, KDA, and RWKV6 and integrate it into a prototype serving stack. Across these models, DeltaLog accelerates the recurrent-state update kernel by up to 1.86×, reduces profiled recurrent-state write traffic by up to 7.83×, and achieves 1.05--1.20× end-to-end serving speedups over dense recurrent baselines.