Lookup-table (LUT)-based neural networks (NNs) offer a compelling path to ultra-low-latency inference on FPGAs. However, existing LUT-based neural networks have largely been restricted to MLP-style topologies, leaving structured vision and audio tasks underexplored. We present ConvLUT, a family of fully unrolled LUT-mappable convolutional architectures targeting the MLPerf Tiny benchmarks. It is comprised of a standard convolutional model built around a hierarchical tree of LUTs and a depthwise separable model following the MobileNet design philosophy. These architectures are enabled by four key contributions: a unified LUT-based convolutional framework supporting standard, depthwise, and pointwise convolutions under explicit fan-in constraints; a LUT-tree that addresses the k2c receptive field dimensionality bottleneck; a channel-partitioned LUT pooling layer that replaces conventional spatial pooling; and a hardware-aware iterative pruning strategy for stable convergence at scale. Evaluated on CIFAR-10, Visual Wake Words, and Keyword Spotting, ConvLUT achieves competitive accuracy with fully unrolled inference latencies below ∼50 ns. On CIFAR-10 it reaches at least a 8844× latency reduction over non-logic-based prior implementations, and outperforms existing MLP-style LUT-based models on the same task by at least 20 pp in test accuracy.
Figures & tables
Figure 1. Traditional and proposed convolutional kernels. Top row: a standard dot-product kernel connects every input within the k2c receptive field. Bottom row: the proposed L-LUT kernel replaces the dot product with a NeuraLUT surrogate ( Andronic and Constantinides, 2024 ) . Each output channel learns its own sparsity pattern, which is shared across all spatial positions during the sliding operation. Fan-in is kept at F inputs per L-LUT.
Symbol
Description
c
Number of input channels
N
Number of output channels
k
Spatial kernel size ( k×k )
h,w
Input feature map height and width
F
L-LUT input fan-in
B
Activation bitwidth
Table 1. Summary of convolutional notation.
Figure 2. A sparse L-LUT convolutional layer with 3 input channels and N output channels. Each channel maintains its own learned sparsity pattern over the k2c receptive field. The resulting output volume has N channels, each encoding a distinct sparse projection of the local neighbourhood.
Figure 3. Convolution types supported by ConvLUT. In standard convolution ( 3(a) ), each L-LUT neuron selects a sparse subset of the full k2c receptive field. In depthwise convolution ( 3(b) ), neurons are restricted to a single input channel, with sparsity over the k2 spatial locations. In pointwise convolution ( 3(c) ), neurons operate across all channels at a single spatial location, with sparsity over the channel dimension. In all cases, each output channel maintains its own learned sparsity pattern, which is shared across spatial positions during the sliding operation.
Figure 4. LUT Tree Convolution: A standard convolutional layer produces N output channels, each learning a distinct sparse pattern over the shared k2c -dimensional receptive field (colored dots indicate active connections per channel). A subsequent pointwise layer with fixed structured sparsity aggregates groups of G=F channels into single L-LUT neurons (each channel occupies one fan-in input, so the group size is bounded by the fan-in budget F ). The effective receptive field (bottom bar) covers the union of all those channel patterns and some locations remain uncovered due to the finite fan-in budget F .
Figure 5. Channel-partitioned LUT pooling layer: The flattened convolutional output is partitioned by channel, with each L-LUT neuron constrained to receive inputs exclusively from a single channel’s spatial feature map. Output neurons are assigned to channels in an interleaved fashion, so that multiple neurons attend to each channel, compensating for information lost through pruning. Solid arrows indicate active connections after pruning to fan-in F ; dashed arrows indicate pruned connections. Each neuron implements a learned sparse spatial aggregation via the NeuraLUT surrogate ( Andronic and Constantinides, 2024 ) , replacing fixed max or average pooling.
Figure 6. Architecture overview of the two model families. Left: the CIFAR-10 CNN model, in which each block comprises one or two pointwise layers acting as LUT tree aggregators, as seen in Section 3.2 , followed by a strided standard convolution. The faded second pointwise layer is present only in the depth-3 variant. Right: the DS-CNN model used for VWW and KWS, following the MobileNet ( Howard et al., 2017 ) structure, in which each block pairs a pointwise layer for cross-channel mixing with a strided depthwise convolution for spatial downsampling. The LUT tree construction does not apply here. Both architectures share the same classifier head: a channel-partitioned LUT pooling layer (as described in Section 3.3 ), a fully connected L-LUT layer, and a group sum producing the final class scores.
Model
Bitwidth
Fan-in
Conv layers: unrolling factor × (channels per layer)
Table 2. Architecture configurations. The unrolling factor denotes the number of spatial windows per resolution stage, i.e. the number of parallel L-LUT instances implementing the same sparse kernel. The total number of L-LUTs equals the number of channels (or neurons), as each is mapped to a single L-LUT. Theoretically, each L-LUT occupies 3Y(2X−4−(−1)X) 6-LUTs as per ( Umuroglu et al., 2020 ) . For the input convolution: Y=Bh,X=BinFin ; for hidden convolutions: Y=Bh,X=BhFconv ; for pooling: Y=Bh,X=BhFfc ; and for the final FC layer: Y=Bout,X=BhFfc .
Figure 7. Implementation results for the CNN-S model. By designing the network architecture around the native LUT primitive, the resulting circuit maps compactly and efficiently onto the FPGA fabric, demonstrating that co-designing neural network architectures with the target hardware abstraction is key to achieving high resource efficiency and ultra-low latency.
Work
Acc. (%)
LUT (% util)
FF
BRAM
DSP
Latency (ns)
Throughput (FPS)
Fmax (MHz)
Area × Delay
FPGA
LL-ViT ( Nag et al., 2025 )
95.5
589,000
229,000
1,425
0
5,330,000,000
1,083
200
3.14×1015
VU9P
H-WNN L ∗ ( Jadhao et al., 2025 )
88.23
20,720
N/R
235
N/R
6,840,000,000
N/R
100
1.42×1013
Pynq-Z1
H-WNN S ∗ ( Jadhao et al., 2025 )
84.62
22,870
N/R
110
N/R
1,600,000,000
N/R
100
3.66×1012
Pynq-Z1
hls4ml TinyML ( Borras et al., 2022 )
83.5
28,544
49,215
42
4
27,300,000
N/R
N/R
8.82×1011
Pynq-Z2
FINN TinyML ( Borras et al., 2022 )
84.5
24,502
34,354
100
0
1,500,000
N/R
N/R
3.99×1010
Pynq-Z2
fpgaconvnet ( fpgaConvNet, 2024 )
86.0
47,000
N/R
93
211
410,000
N/R
143
1.93×1010
Z-7020
Table 3. Comparison of FPGA implementations on CIFAR-10. N/R = not reported. § No pipeline registers reported; single-cycle inference assumed. † Theoretical gate count without the group sum layer; FPGA utilization not reported. ∗ Results taken from out-of-context. The area delay column represents the product of LUTs and latency.
Work
Acc. (%)
LUT (% util)
FF
BRAM
DSP
Latency (ns)
Throughput (FPS)
Fmax (MHz)
Area × Delay
FPGA
H-WNN L ∗ ( Jadhao et al., 2025 )
81.52
22,870
N/R
110
N/R
1,600,000,000
N/R
100
3.66×1013
Pynq-Z1
H-WNN S ∗ ( Jadhao et al., 2025 )
78.22
17,740
N/R
42
N/R
640,000,000
N/R
100
1.14×1013
Pynq-Z1
fpgaconvnet ( fpgaConvNet, 2024 )
91.1
37,000
N/R
97
188
320,000
N/R
143
1.18×1010
Z-7020
FINN TinyML ( Borras et al., 2022 )
82.5
33,732
34,405
37
0
17,000
N/R
N/R
5.91×108
Pynq-Z2
DWN ∗ ( n=6 ) ( Bacellar et al., 2024 )
71.52
4,800
N/R
N/R
N/R
N/R
250M
250
N/R
Z-7045
ConvLUT DS-CNN-A ∗ (2 cycles)
88.99
144,711 (12%)
9,615
0
0
9
208M
208
1.30×106
VU9P
Table 4. Comparison of FPGA implementations on keyword spotting (KWS) using the Speech Commands dataset. N/R = not reported. ∗ Results taken from out-of-context.
Work
Acc. (%)
LUT (% util)
FF
BRAM
DSP
Latency (ns)
Throughput (FPS)
Fmax (MHz)
Area × Delay
FPGA
fpgaconvnet ( fpgaConvNet, 2024 )
84.8
34,000
123
N/R
189
9,490,000
N/R
111
3.23×1011
Z-7020
ConvLUT DS-CNN-C ∗ (2 cycles)
76.70
216,093 (18%)
35,059
0
0
7
302M
302
1.43×106
VU9P
ConvLUT DS-CNN-C ∗ (12 cycles)
76.70
388,163 (33%)
275,134
0
0
27
450M
450
1.05×107
VU9P
ConvLUT DS-CNN-D ∗ (3 cycles)
80.01
927,118 (37%)
37,236
0
0
23
129M
129
2.13×107
VU440
ConvLUT DS-CNN-D ∗ (16 cycles)
80.01
1,107,470 (44%)
333,359
0
0
63
256M
256
6.98×107
VU440
Table 5. Comparison of FPGA-based inference implementations on visual wakeword (VWW). N/R = not reported.
Figure 8. Effect of the per-stage retraining budget Et on CIFAR-10 validation accuracy during iterative pruning. Each sawtooth corresponds to one pruning stage: accuracy drops sharply as the fan-in is tightened from Ft to Ft+1 , then recovers during the Et -epoch retraining window. The dashed lines indicate the projected final accuracy for each schedule.
Pooling
Metric
CIFAR-10
KWS
VWW
Max
Val Acc (%)
55.78
77.77
74.50
Size
232×56
220×256
272×128
Avg
Val Acc (%)
60.29
82.21
74.91
Size
232×56
220×256
272×128
LUT
Val Acc (%)
70.91
83.14
77.47
(Ours)
Size
28×2240
28×1536
26×3840
Table 6. Comparison of pooling strategies for each dataset using toy model architectures. The L-LUT size is calculated based on the fan-in and bitwidth of each neuron. The number of L-LUTs is equal to the number of neurons in the pooling layer. Size is expressed as 2FB× number of L-LUTs.
Field-programmable gate arrays (FPGAs) enable efficient neural-network inference, but most deployment flows either accelerate multiply-accumulate operations or convert pretrained quantized models into lookup tables (LUTs). We present DiffLUT-Net, an FPGA-native network connected by six-input LUTs that are trained from scratch. We jointly learn the 64 truth-table entries of a LUT and the source to each of its six input ports using a differentiable LUT function relaxation and hardware source selection. After training, the truth tables and connections are discretized, unused logic can be pruned, and the network is exported directly as synthesizable Verilog. Across five benchmarks, DiffLUT-Net achieves favorable accuracy-resource trade-offs. These results demonstrate the effectiveness of jointly learning LUT functions and sparse connectivity for compact FPGA-native inference. The code is available at https://github.com/TUDa-HWAI/DiffLUT-Network.
Achieving nanosecond-scale inference latency for deep neural networks (DNNs) has become a primary architectural concern for latency-critical applications. While Field-Programmable Gate Arrays (FPGAs) offer a promising substrate for low-latency inference, conventional FPGA accelerators remain arithmetic-centric, using LUTs primarily as building blocks for numerical operators and peripheral logic. In contrast, recent LUT-native neural networks treat LUTs as learnable neurons, revealing promising theoretical potential to exploit their intrinsic logic expressivity. However, existing methods are largely confined to algorithmic optimizations, failing to translate this theoretical potential into high-performance FPGA accelerators. Specifically, their differentiable formulations do not faithfully match FPGA LUT primitives, their physically-unaware topologies compromise routability and timing closure, and their lack of automated optimization flow hinders systematic design space exploration (DSE) and efficient hardware implementation. In this paper, we propose FPGN, an end-to-end physically-aware framework that closes the gap between LUT-native learning and latency-optimized FPGA implementation. FPGN addresses these challenges through (i) a hardware-aligned differentiable formulation for training FPGA-native LUT neurons, (ii) a structured LUT-native topology with a streaming hardware architecture to improve routing locality and timing closure, and (iii) a latency-driven compiler that leverages high-fidelity analytical Quality of Results models to automate DSE and hardware generation. Experiments show that FPGN achieves up to 205× latency reduction compared to representative FPGA-based BNN accelerators and up to 30× higher LUT efficiency than prior differentiable LUT-native networks, while maintaining competitive inference accuracy.
Lookup-table (LUT) based neural networks can deliver ultra-low latency and excellent hardware efficiency on FPGAs by mapping arithmetic operations directly onto the logic primitives. However, state-of-the-art LUT-aware training (LAT) approaches remain difficult to use in practice: they are often orders of magnitude slower to train than conventional networks, require non-trivial manual tuning for hardware efficiency, and lack an end-to-end workflow. This work presents HGQ-LUT, integrated in https://github.com/calad0i/HGQ2, a new LAT approach that achieves state-of-the-art hardware efficiency while accelerating training by over 100 times on modern GPUs. HGQ-LUT introduces LUT-Dense and LUT-Conv layers that are implemented with regular, accelerator-efficient tensor operations during training, which are then compiled into logic LUTs for hardware. By combining these layers with fine-grained, element-wise heterogeneous quantization (including zero-bit pruning) and a LUT-aware resource surrogate, HGQ-LUT enables the automatic exploration of accuracy-resource trade-offs without manual bit-width tuning. We further integrate HGQ-LUT into open-source toolchains, enabling unified design, compilation, and bit-exact verification of hybrid architectures that mix LUT-based with conventional arithmetic blocks. These features make LAT-based DNNs practical for real-world deployment, such as at the CERN Large Hadron Collider's experiments.
Chang Sun, Zhiqiang Que, Bakhtiar Zadeh +4
California Institute of Technology, Pasadena, CA, USA · University of Bristol, UK · Imperial College London, UK +1