Linear-attention language models replace a growing key-value cache with a recurrent state of fixed size. On an edge FPGA, this state can exceed the on-chip memory and must be paged to external memory. This paper asks whether such paging dominates the cost of decoding (generating tokens). We built a hardware tile that computes one complete layer of a custom 369-million-parameter hybrid Gated DeltaNet model on an Agilex 5 FPGA and pages its recurrent state. The model state needs 6.2 MiB, whereas the device has 0.87 MiB of block memory. Paging adds 26.0 MB of memory traffic per decoded token, at most 11.2 percent of the calculated total. On the board at 75 MHz, with synthetic weights, one token takes 11.15 ms in this layer. The matrix-vector products with the weights take about 62 percent of this time and the state phase, including the paging, 13 percent. The tile is one node of a multicast mesh of tri-mode processing elements. We measure the error of the fixed-point arithmetic against floating point and check that the hardware agrees bit for bit with a fixed-point reference. In simulation, the state logic agrees with the reference on all 18 recurrent layers. On the board, six builds pass all 30 runs of 64 tokens and two runs of 1,024 tokens. For this model and memory interface, the delivery of the weights limits the decoding performance, and the recurrent state does not. State paging is therefore practical for such a model on an edge FPGA.
Figures & tables
Figure 1: The application and the platform. Top: the multi-rate architecture. The acoustic front-end detects a hidden vehicle, the cache answers queries about an unchanged scene with the hit rate η , and a cache miss invokes the language model, of which the decode tile implements one recurrent layer. (a) The DK-A5E013BM16AEA development kit, which we used for all measurements. (b) Where each part runs. The host reaches the programmable logic only through JTAG, and the tile reaches its weights and its state pages only through the LPDDR4 memory.
Work
What is implemented
Recurrent state
Device and evidence
Exactness
Gupta et al. [ 5 ]
Gated DeltaNet decode kernel, five fused phases
2 MB, resident in block memory
Alveo U55C (datacenter). Cycle counts from high-level synthesis and power after implementation. No board run reported
—
DUET [ 31 ]
Mamba–Transformer hybrid, prefill and decode packages
kept in package-local buffers, residency not stated
28 nm ASIC synthesis and a cycle-accurate simulator. No prototype
FPGA (LightMamba, SpecMamba, LowRank-SSM, eMamba) or ASIC synthesis (MARCA, Mamba-X)
—
TRINE [ 26 ]
shared processing array, switched among systolic, SIMD and adder-tree dataflows
—
Alveo U50 and ZCU104 named. Switch cost reported as negligible in wall time, not in cycles
—
BitLogic [ 30 ]
logic-gate networks
—
FPGA, post-route figures. Verilator
bit-exact to its Python reference, in simulation
This work
complete Gated DeltaNet layer of a 369M hybrid model
paged, 26.0 MB per decoded token
Agilex 5 board. Six final builds pass 30 runs of 64 layer-tokens
state engine exact in 72 simulation runs. Kernel and complete layer checked on the board
Table 1: Comparison with the closest prior work on the four criteria of this paper. “Resident” means that the recurrent state stays on chip. “Paged” means that the state is read from and written to external memory in pages. Cells report what the cited works state. A dash means that the work does not address that criterion.
Figure 2: The mode controller of the processing element (Section 4 ). A mode switch costs D+3 cycles, where D is the number of cycles that the current engine needs to become idle (drain). Arrow labels read “condition / action”.
Figure 3: The implemented system. Left: the mesh, in which five nodes are tri-mode processing elements and node 5 is the decode tile. Right: the decode tile. The block names are those of the RTL: sys_csr is the command register file, pe_port with mm_pe is a processing element, ctrl_unit is the control datapath, matvec is the matrix-vector engine, and state_pager with gdn_head_core is the state pager with the state engine. The mesh carries commands and results only, and the weights and the state pages use separate 512-bit ports to the LPDDR4 memory. The peer port can forward flits to a second system.
Vocabulary projection in
Component
bf16
INT4
INT4 weights of the layers
148.45
148.45
Scales of the weights
9.28
9.28
Vocabulary projection
143.36
38.08
KV-cache reads of the attention layers
9.14
9.14
State read and write, implemented
25.98
25.98
Table 2: Calculated memory traffic per decoded token for the whole model, in MB, at a context of 744 tokens. The two columns differ in the number format of the vocabulary projection. The last row gives the state traffic without headers and padding, for comparison.
Shift limit ( SH_MAX )
Page rounding
Max Eo
Max ES
31 (reference)
no
4.69×10−4
4.40×10−4
3 (original RTL)
no
5.15×10−2
1.91×10−1
31 (reference)
yes
6.43×10−3
1.10×10−3
3 (original RTL)
yes
5.15×10−2
1.91×10−1
Table 3: Relative errors of the fixed-point state engine against fp32, computed with ( 4 ). Every row covers 18 recurrent layers, two prompts and 64 tokens. Page rounding, where used, is applied at initialization and after each token. These are software results.
Figure 4: Cache hit rate η of each of the 83 clips for three versions of the cache policy. Boxes span the quartiles, and whiskers span the full range. Diamonds mark the means, and points mark the individual clips. The dashed line is the feasibility threshold η≈0.80 (Section 3 ). Only the full policy places every clip above the threshold, and its lowest clip is at η=0.852 .
NL=64 , W=16
NL=128 , W=8 (chosen)
Cycles per layer-token
446,100
393,935
Time at 150 MHz
2.97 ms
2.63 ms
ALM
38,888 (83 %)
27,105 (58 %)
M20K
234 (65 %)
250 (70 %)
DSP
88 (23 %)
112 (30 %)
Fmax
155.6 MHz
158.8 MHz
Table 4: The two tile configurations that fit the device, from the design study. NL and W are the numbers of lanes of the matrix-vector engine and of the state engine. Cycle counts come from RTL simulation with the memory model, and times are calculated at 150 MHz. ALM, M20K and DSP are the logic, memory and multiplier blocks of the device, and Fmax is the maximum clock frequency. A third configuration, with NL=128 and W=16 , does not fit the device.
Config.
ALMs
M20Ks
DSPs
Fmax (MHz)
P
29,958–29,979
298
108
102.64–110.66
M
40,025–40,077
305
128
98.56–108.77
Table 5: The six final board builds. P is the tile alone, and M is the complete system (Fig. 3 ). All builds run at 75 MHz. The ranges span the three placement seeds. Fmax is the reported value for the clock of the tile.
Test
P passes
M passes
Layer-tokens
Runs after programming
15/15
15/15
1,920
Runs of 1,024 layer-tokens
1/1
1/1
2,048
Initial run and ten soft resets
11/11
11/11
352
Table 6: Results of the three board tests of the complete layer, for the configurations P and M.
Figure 5: Roofline plot of the memory link for the weights, from the experiment with the state on chip. It shows the rate of MACs as a function of the rate at which weight bytes arrive. The layer performs 12,736,512 MACs over a weight image of 6,796,032 bytes, which gives 1.874 MACs per byte (sloped line). The 128 lanes limit the rate to 128 MACs per cycle (horizontal line), and the two lines meet at 68.3 bytes per cycle. Filled markers are measurements on the device with bursts of 8 and 128 beats, and the open marker is the memory model. Dashed lines show the peak rate of the LPDDR4 memory, 8.53 GB/s, at 75 and 150 MHz.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.