Quantization Effects of Artificial Neural Networks for Embedded Edge-Computing Applications
Authors: Alperen Aksoy, Ilja Bekman, Vesselin Dimitrov, Qader Dorosti, Chimezie Eguzo, Sarah Fleitmann, Fabian Hader, Andre Zambanini, Stefan van Waasen
Organizations: Peter Grünberg Institute (PGI), Integrated Computing Architectures (ICA | PGI-4), Forschungszentrum Jülich GmbH, Germany · Center for Particle Physics Siegen, Department für Physik, Universität Siegen, Germany · Faculty of Engineering, JARA-FIT Institute for Quantum Information, Forschungszentrum Jülich GmbH and RWTH Aachen University, Germany · University Duisburg-Essen, Germany
This paper examines the use of Quantized Neural Networks (QNNs) for two resource-constrained scientific applications: automated calibration of semiconductor quantum bits (qubits) and scientific particle detectors. We evaluate the trade-offs between Post-Training Quantization (PTQ), Quantization-Aware Training (QAT), and ultra-low-bit Binary Neural Networks (BNNs) with respect to latency and resource usage. Our results demonstrate that PTQ and QAT are easily implementable solutions achieving a four-fold reduction in memory usage for Ushaped CNN (U-Net) architectures, whereas BNNs are better suitable for use cases with challenging latency requirements. For the training of non-differentiable custom BNNs, we propose a novel, hardware-constrained learning approach using Genetic Algorithms (GAs). We showcase a Look-Up Table (LUT)-based BNN architecture suitable for direct conversion to Very High-Speed Integrated Circuit Hardware Description Language (VHDL) via the HCL4BNN framework. This method achieves nanosecond-scale inference latencies at 10 ns to 15 ns without requiring specialized Digital Signal Processor (DSP) or Block RAM (BRAM) resources.
Figures & tables
Figure 1: Conventional workflow to deploy a QNN on an FPGA based on gradient training, quantization, and HLS conversion.
Figure 2: Proposed hardware-constrained learning approach with direct HDL generation.
Figure 3: Left: isolated simulated SiPM pulses (”Good” class). Right: two overlapping or distorted simulated SiPM pulses (”Ugly” class).
Figure 4: Left: Best training accuracy progression across generations for several GA runs, see Table 3 for validation results. Right: Progression of best accuracy and network sparsity (measured by the number of zero-weights) across generations for a longer GA run.
Figure 5: Simulated charge stability diagram with and without compensation of the sensor response to the voltage changes for gates P1 and P2 [ 8 ] . Further data examples can be found in [ 37 ] .
Model
Unquantized
PTQ
QAT
in %
in %
in %
UNet-447
85±28
83±29
85±28
UNet-38k
96±9
96±9
96±9
Table 1: Mean dice similarity on the test set with standard deviation on the test set.
# Parameters
9,112
4,268
3,147
2,062
1,013
Accuracy in %
88.7±3.7
89.1±1.2
87.5±1.7
85.3±1
78.3±1.2
Non-convergent
18 %
29 %
20 %
15 %
18 %
Table 2: Results of compressed CNN on MNIST dataset with small amount of parameters, binary weights and activations. Accuracy mean with standard deviation over normal fraction of runs is given in percent for converged runs. A fraction of training runs do not reach accuracy of 70 % after 10 epochs and are defined as stuck/non-convergent.
Figure 6: Illustration of the accuracy distributions of 5 compressed CNN (next to each other per accuracy bin) for training reruns on MNIST dataset after 10 epochs. Largest CNN (orange color, circle) distribution has the mean at 89% , Smallest CNN (purple color, down-triangle) peaks at 78% . Means and standard deviation points are calculated over the normal part of the distributions.
Model
Accuracy
Latency
LUT
FF
DSP
BRAM
Training Time
in %
in ns
×103
×103
18 k
min × cores
FINN
74±4
24850
30
20
106
5
∼2×16
hls4ml
93±2
3050
186
112
556
120
∼2×16
BNN a
63±3
15
58
1.5
0
0
∼300×90
BNN b
74±5
10
23
1.5
0
0
∼105×90
BNN b (static)
72±3
Table 3: Results and comparison of FINN 2DCNN ( (128-4-6-8-2) , Kernel [5,1], padding 2.0, int8) implementation, hls4ml CNN, and BNN a (128-64-128-2) , BNN b (128-32-32-2) , BNN c (128-16-64-2) with INT7 input quantization. BNN b is also run with a static training dataset.
Predicted
Good
Ugly
True
Good
947
53
Ugly
410
590
Table 4: Examples of confusion matrices for FINN (left) and hls4ml (right) for small validation sample. For the accuracies averaged over 30 runs see 3 .
Predicted
Good
Ugly
Either
True
Good
3052
1948
0
Ugly
1595
3400
5
Noise
0
0
5000
Table 5: Example confusion matrices for one BNN a 128-64-128-2 (left) and one BNN b 128-32-32-2 (right); Either (or undecided) encodes (1,1) or (0,0) tuples for prediction and random input value test for truth.
Figure 7: Illustration of the difference for static (left) and dynamic (right) SiPM training datasets of large (21k 4bit parameters) compressed CNN . The training dataset here is 32+32 waveforms and 640+640 for validation. Static training data leads to exaggerated over-fitting in this illustration.
Parameter
Value
Description
pop_size
300
Number of individuals per generation
ngen
50
Number of generations the GA is evolved for = stopping criterion
nmutbit
50
Expected number of mutated bits per genome (sets per-bit mutation probability, see Sec. 3.2.2 )
tourn_size
5
Tournament size used for parent selection
cxpb
0.5
Crossover probability per individual
cxpb_bit
1
Crossover probability per bit, given that crossover occurs
Table 6: Genetic algorithm hyperparameters and dataset generation settings used for BNN training.
Deep learning applications have been widely adopted on edge devices, to mitigate the privacy and latency issues of accessing cloud servers. Deciding the number of neurons during the design of a deep neural network to maximize performance is not intuitive. Particularly, many application scenarios are real-time and have a strict latency constraint, while conventional neural network optimization methods do not directly change the temporal cost of model inference for latency-critical edge systems. In this work, we propose a latency-oriented neural network learning method to optimize models for high accuracy while fulfilling the latency constraint. For efficiency, we also introduce a universal hardware-customized latency predictor to optimize this procedure to learn a model that satisfies the latency constraint by only a one-shot training process. The experiment results reveal that, compared to state-of-the-art methods, our approach can well-fit the 'hard' latency constraint and achieve high accuracy. Under the same training settings as the original model and satisfying a 34 ms latency constraint on the ImageNet-100 dataset, we reduce GoogLeNet's latency from 40.32 ms to 34 ms with a 0.14% accuracy reduction on the NVIDIA Jetson Nano. When coupled with quantization, our method can be further improved to only 0.04% drop for GoogLeNet. On the NVIDIA Jetson TX2, we compress VGG-19 from 119.98 ms to 34 ms and even improve its accuracy by 0.5%, and we scale GoogLeNet up from 20.27 ms to 34 ms and achieve higher accuracy by 0.78%. We also open source this framework at https://github.com/ntuliuteam/ZeroBN
Shuo Huai, Di Liu, Hao Kong +4
aSchool of Computer Science and Engineering, Nanyang Technological University, 639798, Singapore · cDepartment of Computer Science, Norwegian University of Science and Technology, 7491, Trondheim, Norway · bHP-NTU Digital Manufacturing Corporate Lab, Nanyang Technological University, 639798, Singapore +1
Lookup-table (LUT) based neural networks can deliver ultra-low latency and excellent hardware efficiency on FPGAs by mapping arithmetic operations directly onto the logic primitives. However, state-of-the-art LUT-aware training (LAT) approaches remain difficult to use in practice: they are often orders of magnitude slower to train than conventional networks, require non-trivial manual tuning for hardware efficiency, and lack an end-to-end workflow. This work presents HGQ-LUT, integrated in https://github.com/calad0i/HGQ2, a new LAT approach that achieves state-of-the-art hardware efficiency while accelerating training by over 100 times on modern GPUs. HGQ-LUT introduces LUT-Dense and LUT-Conv layers that are implemented with regular, accelerator-efficient tensor operations during training, which are then compiled into logic LUTs for hardware. By combining these layers with fine-grained, element-wise heterogeneous quantization (including zero-bit pruning) and a LUT-aware resource surrogate, HGQ-LUT enables the automatic exploration of accuracy-resource trade-offs without manual bit-width tuning. We further integrate HGQ-LUT into open-source toolchains, enabling unified design, compilation, and bit-exact verification of hybrid architectures that mix LUT-based with conventional arithmetic blocks. These features make LAT-based DNNs practical for real-world deployment, such as at the CERN Large Hadron Collider's experiments.
Chang Sun, Zhiqiang Que, Bakhtiar Zadeh +4
California Institute of Technology, Pasadena, CA, USA · University of Bristol, UK · Imperial College London, UK +1
Lightweight neural network accelerators are essential for edge devices with limited resources and power constraints. While quantization and binarization can efficiently reduce hardware cost, they still rely on the conventional Artificial Neural Network (ANN) computation pattern. The recently proposed Kolmogorov-Arnold Network (KAN) presents a novel network paradigm built on learnable nonlinear functions. However, it is computationally expensive for hardware deployment. Inspired by KAN, we propose BiKA, a multiply-free architecture that replaces nonlinear functions with binary, learnable thresholds, introducing an extremely lightweight computational pattern that requires only comparators and accumulators. Our FPGA prototype on Ultra96-V2 shows that BiKA reduces hardware resource usage by 27.73% and 51.54% compared with binarized and quantized neural network systolic array accelerators, while maintaining competitive accuracy. BiKA provides a promising direction for hardware-friendly neural network design on edge devices.
Yuhao Liu, Salim Ullah, Akash Kumar
Ruhr University Bochum, Germany · Dresden University of Technology, Germany · Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI Dresden/Leipzig), Germany