To deploy deep neural networks on edge hardware, highly efficient inference schemes are necessary that retain high accuracy. This work presents W16A16, a high precision (16-bit), fast speed, low energy quantization method. On a widely applied microcontroller architecture Armv7E-M, our proposed approach achieves faster speed and lower energy consumption on layer- and model-level compared to alternative quantization schemes. We analyze the architecture of Armv7E-M, explain the underlying principles behind the performance advantages of 16-bit approaches, and evaluate the empiric quantization errors for regression and classification tasks, as well as empiric time- and energy consumption in MCU deployment. We observe ca.\ 10 times lower quantization errors compared to 8-bit quantization schemes while achieving similar or better inference times and energy consumption.
Figures & tables
Figure 1: The asymmetric quantization of activation values.
Figure 2: The symmetric quantization of weight values.
Figure 3: The structure of a TCN block.
Figure 4: Instructions needed to perform MAC operations for four inputs and four weights in W8A8 quantization
Figure 5: Instructions needed to perform MAC operations for four inputs and four weights in W8A16 quantization
Figure 6: Instructions needed to perform MAC operations for four inputs and four weights in W16A16 quantization
Figure 7: The memory layout
Figure 8: A comparison of discontinuous array access in the original memory layout (top) versus the sequential array access after Seq2col (bottom).
Figure 9: An example application of Seq2Col, including causal left padding and dilation.
Figure 10: An overview of the requantization process, starting from input activations of a CNN layer, the MAC operations corresponding to the convolution, multiplication with the scaling factor of the layer, bit-shifting, and two-stage saturation to obtain a 16-bit activation for the next layer.
Figure 11: The multiplication of a 64-bit intermediate MAC result (accumulator) with a 32-bit multiplier for requantization using one intermediate UMULL for the lower bits and and one SMLAL operation for the higher bits, resulting in 94 bits spread across three registers, R3 for the low bits, R4 for the intermediate bits, and R5 for the high bits.
Table 1: Layer-wise, block-wise and End-to-End Quantization Error against FP32 Baseline on the C-MAPSS Jet Engine Dataset.
W16A16
W8A16
W8A8
RSME
0.0015
0.0161
0.1970
Table 2: Quantization Error of the three quantization approaches against the float32 reference for one layer, one block, and the full TCN on the NinaPro DB2 force regression data set.
FP32
W16A16
W8A16
W8A8
Accuracy
53.31%
53.29%
53.29%
41.63%
Table 3: Classification accuracy on NinaproDB2 gestures based on sEMG signals.
Configuration
W8A8
W8A16
W16A16
14×32×5,D=1
99867
123257
99436
32×32×5,D=1
144273
194531
164665
14×32×1,D=1
45861
53789
61374
32×1×1,D=1
5774
6570
6206
32×32×5,D=2
121790
164095
139373
32×32×5,D=4
77684
104667
88741
Table 4: Layer-level benchmarking cycles comparison across different quantization precisions STM32H7 ( L=30 ).
Configuration
W8A8
W8A16
W16A16
14×32×5,D=1
172195
222678
194077
32×32×5,D=1
304884
412798
349351
14×32×1,D=1
82826
103830
109196
32×1×1,D=1
10363
11816
11121
32×32×5,D=2
272183
349393
295622
32×32×5,D=4
173375
222449
188165
Table 5: Layer-level benchmarking cycles comparison across different quantization precisions on GD32F4 ( L=30 ).
Figure 13: model level benchmarking
Metric
Sleep
Cb.32
Cb.8
W8A8
W8A16
W16A16
Avg. I (mA)
53.33
189.30
178.34
197.14
208.15
212.06
Inference Time (ms)
–
15.11
13.68
2.77
3.42
2.82
Total Energy (mJ)
–
9.44
8.05
1.80
2.35
1.95
Table 6: Inference time and Energy Characterization on STM32H7.
Metric
Sleep
W8A8
W8A16
W16A16
Avg. I (mA)
85.96
122.43
124.36
120.74
Inference Time (ms)
–
14.46
20.59
15.90
Total Energy (mJ)
–
5.84
8.45
6.34
Table 7: Inference time and Energy Characterization on GD32F4.