Lookup-table (LUT)-based neural networks (NNs) offer a compelling path to ultra-low-latency inference on FPGAs. However, existing LUT-based neural networks have largely been restricted to MLP-style topologies, leaving structured vision and audio tasks underexplored. We present ConvLUT, a family of fully unrolled LUT-mappable convolutional architectures targeting the MLPerf Tiny benchmarks. It is comprised of a standard convolutional model built around a hierarchical tree of LUTs and a depthwise separable model following the MobileNet design philosophy. These architectures are enabled by four key contributions: a unified LUT-based convolutional framework supporting standard, depthwise, and pointwise convolutions under explicit fan-in constraints; a LUT-tree that addresses the k2c receptive field dimensionality bottleneck; a channel-partitioned LUT pooling layer that replaces conventional spatial pooling; and a hardware-aware iterative pruning strategy for stable convergence at scale. Evaluated on CIFAR-10, Visual Wake Words, and Keyword Spotting, ConvLUT achieves competitive accuracy with fully unrolled inference latencies below ∼50 ns. On CIFAR-10 it reaches at least a 8844× latency reduction over non-logic-based prior implementations, and outperforms existing MLP-style LUT-based models on the same task by at least 20 pp in test accuracy.
Figures & tables
Figure 1. Traditional and proposed convolutional kernels. Top row: a standard dot-product kernel connects every input within the k2c receptive field. Bottom row: the proposed L-LUT kernel replaces the dot product with a NeuraLUT surrogate ( Andronic and Constantinides, 2024 ) . Each output channel learns its own sparsity pattern, which is shared across all spatial positions during the sliding operation. Fan-in is kept at F inputs per L-LUT.
Symbol
Description
c
Number of input channels
N
Number of output channels
k
Spatial kernel size ( k×k )
h,w
Input feature map height and width
F
L-LUT input fan-in
B
Activation bitwidth
Table 1. Summary of convolutional notation.
Figure 2. A sparse L-LUT convolutional layer with 3 input channels and N output channels. Each channel maintains its own learned sparsity pattern over the k2c receptive field. The resulting output volume has N channels, each encoding a distinct sparse projection of the local neighbourhood.
Figure 3. Convolution types supported by ConvLUT. In standard convolution ( 3(a) ), each L-LUT neuron selects a sparse subset of the full k2c receptive field. In depthwise convolution ( 3(b) ), neurons are restricted to a single input channel, with sparsity over the k2 spatial locations. In pointwise convolution ( 3(c) ), neurons operate across all channels at a single spatial location, with sparsity over the channel dimension. In all cases, each output channel maintains its own learned sparsity pattern, which is shared across spatial positions during the sliding operation.
Figure 4. LUT Tree Convolution: A standard convolutional layer produces N output channels, each learning a distinct sparse pattern over the shared k2c -dimensional receptive field (colored dots indicate active connections per channel). A subsequent pointwise layer with fixed structured sparsity aggregates groups of G=F channels into single L-LUT neurons (each channel occupies one fan-in input, so the group size is bounded by the fan-in budget F ). The effective receptive field (bottom bar) covers the union of all those channel patterns and some locations remain uncovered due to the finite fan-in budget F .
Figure 5. Channel-partitioned LUT pooling layer: The flattened convolutional output is partitioned by channel, with each L-LUT neuron constrained to receive inputs exclusively from a single channel’s spatial feature map. Output neurons are assigned to channels in an interleaved fashion, so that multiple neurons attend to each channel, compensating for information lost through pruning. Solid arrows indicate active connections after pruning to fan-in F ; dashed arrows indicate pruned connections. Each neuron implements a learned sparse spatial aggregation via the NeuraLUT surrogate ( Andronic and Constantinides, 2024 ) , replacing fixed max or average pooling.
Figure 6. Architecture overview of the two model families. Left: the CIFAR-10 CNN model, in which each block comprises one or two pointwise layers acting as LUT tree aggregators, as seen in Section 3.2 , followed by a strided standard convolution. The faded second pointwise layer is present only in the depth-3 variant. Right: the DS-CNN model used for VWW and KWS, following the MobileNet ( Howard et al., 2017 ) structure, in which each block pairs a pointwise layer for cross-channel mixing with a strided depthwise convolution for spatial downsampling. The LUT tree construction does not apply here. Both architectures share the same classifier head: a channel-partitioned LUT pooling layer (as described in Section 3.3 ), a fully connected L-LUT layer, and a group sum producing the final class scores.
Model
Bitwidth
Fan-in
Conv layers: unrolling factor × (channels per layer)
Table 2. Architecture configurations. The unrolling factor denotes the number of spatial windows per resolution stage, i.e. the number of parallel L-LUT instances implementing the same sparse kernel. The total number of L-LUTs equals the number of channels (or neurons), as each is mapped to a single L-LUT. Theoretically, each L-LUT occupies 3Y(2X−4−(−1)X) 6-LUTs as per ( Umuroglu et al., 2020 ) . For the input convolution: Y=Bh,X=BinFin ; for hidden convolutions: Y=Bh,X=BhFconv ; for pooling: Y=Bh,X=BhFfc ; and for the final FC layer: Y=Bout,X=BhFfc .
Figure 7. Implementation results for the CNN-S model. By designing the network architecture around the native LUT primitive, the resulting circuit maps compactly and efficiently onto the FPGA fabric, demonstrating that co-designing neural network architectures with the target hardware abstraction is key to achieving high resource efficiency and ultra-low latency.
Work
Acc. (%)
LUT (% util)
FF
BRAM
DSP
Latency (ns)
Throughput (FPS)
Fmax (MHz)
Area × Delay
FPGA
LL-ViT ( Nag et al., 2025 )
95.5
589,000
229,000
1,425
0
5,330,000,000
1,083
200
3.14×1015
VU9P
H-WNN L ∗ ( Jadhao et al., 2025 )
88.23
20,720
N/R
235
N/R
6,840,000,000
N/R
100
1.42×1013
Pynq-Z1
H-WNN S ∗ ( Jadhao et al., 2025 )
84.62
22,870
N/R
110
N/R
1,600,000,000
N/R
100
3.66×1012
Pynq-Z1
hls4ml TinyML ( Borras et al., 2022 )
83.5
28,544
49,215
42
4
27,300,000
N/R
N/R
8.82×1011
Pynq-Z2
FINN TinyML ( Borras et al., 2022 )
84.5
24,502
34,354
100
0
1,500,000
N/R
N/R
3.99×1010
Pynq-Z2
fpgaconvnet ( fpgaConvNet, 2024 )
86.0
47,000
N/R
93
211
410,000
N/R
143
1.93×1010
Z-7020
Table 3. Comparison of FPGA implementations on CIFAR-10. N/R = not reported. § No pipeline registers reported; single-cycle inference assumed. † Theoretical gate count without the group sum layer; FPGA utilization not reported. ∗ Results taken from out-of-context. The area delay column represents the product of LUTs and latency.
Work
Acc. (%)
LUT (% util)
FF
BRAM
DSP
Latency (ns)
Throughput (FPS)
Fmax (MHz)
Area × Delay
FPGA
H-WNN L ∗ ( Jadhao et al., 2025 )
81.52
22,870
N/R
110
N/R
1,600,000,000
N/R
100
3.66×1013
Pynq-Z1
H-WNN S ∗ ( Jadhao et al., 2025 )
78.22
17,740
N/R
42
N/R
640,000,000
N/R
100
1.14×1013
Pynq-Z1
fpgaconvnet ( fpgaConvNet, 2024 )
91.1
37,000
N/R
97
188
320,000
N/R
143
1.18×1010
Z-7020
FINN TinyML ( Borras et al., 2022 )
82.5
33,732
34,405
37
0
17,000
N/R
N/R
5.91×108
Pynq-Z2
DWN ∗ ( n=6 ) ( Bacellar et al., 2024 )
71.52
4,800
N/R
N/R
N/R
N/R
250M
250
N/R
Z-7045
ConvLUT DS-CNN-A ∗ (2 cycles)
88.99
144,711 (12%)
9,615
0
0
9
208M
208
1.30×106
VU9P
Table 4. Comparison of FPGA implementations on keyword spotting (KWS) using the Speech Commands dataset. N/R = not reported. ∗ Results taken from out-of-context.
Work
Acc. (%)
LUT (% util)
FF
BRAM
DSP
Latency (ns)
Throughput (FPS)
Fmax (MHz)
Area × Delay
FPGA
fpgaconvnet ( fpgaConvNet, 2024 )
84.8
34,000
123
N/R
189
9,490,000
N/R
111
3.23×1011
Z-7020
ConvLUT DS-CNN-C ∗ (2 cycles)
76.70
216,093 (18%)
35,059
0
0
7
302M
302
1.43×106
VU9P
ConvLUT DS-CNN-C ∗ (12 cycles)
76.70
388,163 (33%)
275,134
0
0
27
450M
450
1.05×107
VU9P
ConvLUT DS-CNN-D ∗ (3 cycles)
80.01
927,118 (37%)
37,236
0
0
23
129M
129
2.13×107
VU440
ConvLUT DS-CNN-D ∗ (16 cycles)
80.01
1,107,470 (44%)
333,359
0
0
63
256M
256
6.98×107
VU440
Table 5. Comparison of FPGA-based inference implementations on visual wakeword (VWW). N/R = not reported.
Figure 8. Effect of the per-stage retraining budget Et on CIFAR-10 validation accuracy during iterative pruning. Each sawtooth corresponds to one pruning stage: accuracy drops sharply as the fan-in is tightened from Ft to Ft+1 , then recovers during the Et -epoch retraining window. The dashed lines indicate the projected final accuracy for each schedule.
Pooling
Metric
CIFAR-10
KWS
VWW
Max
Val Acc (%)
55.78
77.77
74.50
Size
232×56
220×256
272×128
Avg
Val Acc (%)
60.29
82.21
74.91
Size
232×56
220×256
272×128
LUT
Val Acc (%)
70.91
83.14
77.47
(Ours)
Size
28×2240
28×1536
26×3840
Table 6. Comparison of pooling strategies for each dataset using toy model architectures. The L-LUT size is calculated based on the fan-in and bitwidth of each neuron. The number of L-LUTs is equal to the number of neurons in the pooling layer. Size is expressed as 2FB× number of L-LUTs.