The growing demand for machine learning applications in the context of the Internet of Things calls for new approaches to optimize the use of limited compute and memory resources. Despite significant progress that has been made w.r.t. reducing model sizes and improving efficiency, many applications still require remote servers to provide the required resources. However, such approaches rely on transmitting data from edge devices to remote servers, which may not always be feasible due to bandwidth, latency, or energy constraints. We propose a task-specific, trainable feature quantization layer that compresses the input features of a neural network. This can significantly reduce the amount of data that needs to be transferred from the device to a remote server. In particular, the layer allows each input feature to be quantized to a user-defined number of bits, enabling a simple on-device compression at the time of data collection. The layer is designed to approximate step functions with sigmoids, enabling trainable quantization thresholds. By concatenating outputs from multiple sigmoids, introduced as bitwise soft quantization, it achieves trainable quantized values when integrated with a neural network. We compare our method to full-precision inference as well as to several quantization baselines. Experiments show that our approach outperforms standard quantization methods, while maintaining accuracy levels close to those of full-precision models. In particular, depending on the dataset, compression factors of 5× to 16× can be achieved compared to 32-bit input without significant performance loss.
Figures & tables
Figure 1: The trainable feature quantization layer is integrated into a neural network to learn task-specific compressions for each input feature: (a) During training, the quantization layer (green rectangle) and the neural network are trained jointly on a remote server. The layer gives rise to an encoder-decoder composition \quanti=\deci∘\enci for the i -th input feature. (b) During inference, the encoder \enci is used to encode the i -th feature on the resource-constrained device using lightweight coding logic. The compressed features are then sent to the server, where the decoder \deci is used to decode the i -th feature. All decoded features are then used as input for the remaining neural network, which is executed on the remote server.
Figure 2: Schematic overview: Soft Quantization ( \quant\softmath ) and Bitwise Soft Quantization ( \quant\bwmath,\softmath ) are formed by summing or concatenating multiple soft step functions. Both are differentiable with respect to thresholds, enabling optimization during training. During inference, they are converted into Hard Quantization ( \quant ) and Bitwise Quantization ( \quant\bwmath ) via rounding.
Figure 3: Example of a quantization with bit width n=2 (i.e., M=3 thresholds) using minmax and quantile quantization on artificial skewed data. Left: Histogram of input data together with thresholds for minmax and quantile quantization. Center: Minmax and quantile quantization functions. Right: Histogram of quantized values for minmax and quantile quantization.
Name
Abbreviation
Source
Full Precision
FP
Baseline
Pre Minmax Quantization
Pr-MQ
Baseline
Pre Quantile Quantization
Pr-QQ
Baseline
Learnable Step Size Quantization
LSQ
[ 33 ]
Learnable Lookup Table with granule 4
LLT4
[ 34 ]
Learnable Lookup Table with granule 9
LLT9
[ 34 ]
Table 1: Models
Figure 4: MSE values per dataset per quantization level of the best-performing hyperparameter setting averaged over 10 data splits for all methods. The red line is the average over the full precision measurements.
Dataset
california
fried
superconduct
cpu_act
sulfur
wine
Mean
Bw-SQ
0.195
0.061
0.086
0.020
0.172
0.543
0.180
SQ
+12.15%
+3.74%
+5.86%
+28.34%
+13.36%
+7.36%
+11.80%
Bw-MQ
+57.43%
+8.03%
+16.42%
+5.74%
+21.21%
+4.86%
+18.95%
Bw-QQ
+9.91%
+6.64%
−0.75%
+112.85%
+33.49%
−1.36%
+26.80%
Table 2: Average MSE of Bitwise Soft Quantization per dataset together with average MSE ratio of ablation methods against Bitwise Soft Quantization per dataset. Values below 0 indicate better performance than Bitwise Soft Quantization.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Bit Width
Number of
Encoding Latency
Energy Consumption
Energy Consumption
Features
( μs )
( μAh )
( μJ )
2
6
10
0.00016
2.11
2
81
50
0.00073
9.79
3
6
10
0.00016
2.11
3
81
70
0.00102
13.54
4
6
10
0.00016
2.14
Appendix
Table 3: Latency and energy requirements for encoding of input features to bit widths 2, 3, and 4 on a micrcontroller with ESP32-S3 CPU.
Byte size
Latency ( ms )
Energy Consumption ( μAh )
Energy Consumption ( μJ )
8 bytes
21
0.6733
8968.34
32 bytes
140
4.0779
54318.13
128 byte
250
8.4043
111945.38
Appendix
Table 4: Energy and latency requirements for sending different amounts of data via CoAP protocol from a microcontroller equipped with ESP32-S3 CPU.
Dataset
Short Name
Source
N
K
Cˉ
OpenML ID
California Housing
california
[ 38 ]
20640
8
8880
43979
Fried
fried
[ 39 , 40 ]
40768
10
1001
564
Superconduct
superconduct
[ 42 ]
21263
81
7449
43174
CPU Activity Database
cpu_act
8192
21
2192
197
Sulfur
sulfur
[ 41 ]
10081
6
8636
23515
Wine Quality
wine
[ 43 ]
6497
11
241
287
Appendix
Table 5: Datasets used where N denotes the size of the dataset, K the number of features and Cˉ the average number of characteristics (i.e. distinct values) per feature.
Figure 5: Visualization of different temperature schedules. All schedules start with a temperature value τinit=1 and end with a temperature value τend after nepochs epochs.
Schedule
τend
0.0001
0.001
0.01
Linear
0.6315
0.6515
0.6421
Warmup
0.6425
0.6456
0.6604
Cosine
0.6401
0.6412
0.6320
Cyclical
0.6526
0.6473
0.6503
Exponential
0.6300
0.6201
0.6412
Appendix
Table 6: Average validation MSE of Bitwise Soft-Quantization on wine dataset for 2 bit quantization for different temperature schedules.
Schedule
τend
0.0001
0.001
0.01
Linear
0.1615
0.1790
0.1672
Warmup
0.1848
0.1741
0.1737
Cosine
0.1335
0.1336
0.1449
Cyclical
0.1595
0.1636
0.1896
Exponential
0.1197
0.1112
0.1230
Appendix
Table 7: Average validation MSE of Bitwise Soft-Quantization on superconduct dataset for 7 bit quantization for different temperature schedules.
Name
Values
Description
dropout rate
[0.0, 0.2, 0.4, 0.5]
dropout rate for neurons during training
learning rate
[0.001, 0.0001]
learning rate for optimizer
hidden layers
[5, 6, 8, 10]
number of hidden layers
hidden neurons
[32, 64, 128, 256, 512,
number of neurons per hidden layer
1024, 2048, 4096, 8192]
num epochs
[30, 50, 70]
number of epochs to train
Appendix
Table 8: Search Space for Hyperparameter Tuning
Figure 6: Distribution of the k-fold values for the best average hyperparameter setup per quantization method for a 2-bit quantization level.
Figure 7: Distribution of the k-fold values for the best average hyperparameter setup per quantization method for a 3-bit quantization level.
Figure 8: Distribution of the k-fold values for the best average hyperparameter setup per quantization method for a 4-bit quantization level.
Figure 9: Distribution of the k-fold values for the best average hyperparameter setup per quantization method for a 5-bit quantization level.
Figure 10: Distribution of the k-fold values for the best average hyperparameter setup per quantization method for a 6-bit quantization level.
Figure 11: Distribution of the k-fold values for the best average hyperparameter setup per quantization method for a 7-bit quantization level.
Figure 12: Distribution of the k-fold values for the best average hyperparameter setup per quantization method for a 8-bit quantization level.
dataset
bits
FP
Comparison models
Ours
Pr-MQ
Pr-QQ
LLT4
LLT9
LSQ
Bw-SQ
california
2
0.186
0.593
0.338
0.424
0.396
0.356
0.290
3
0.186
0.376
0.279
0.278
0.273
0.259
0.210
4
0.186
0.326
0.213
0.219
0.222
0.216
0.185
5
0.186
0.279
0.199
0.204
0.208
0.207
0.171
6
0.186
0.254
0.189
0.197
0.197
0.195
0.173
Appendix
Table 9: Average MSE for selected hyperparameter setting per dataset and bit width. Minimal values of baseline and our models per row are bold, values within a range of 2.5% from the minimal value are underlined.
dataset
bits
FP
Comparison models
Ours
Pr-MQ
Pr-QQ
LLT4
LLT9
LSQ
Bw-SQ
california
2
[0.163, 0.210]
[0.565, 0.621]
[0.302, 0.374]
[0.387, 0.462]
[0.357, 0.435]
[0.322, 0.389]
[0.257, 0.323]
3
[0.163, 0.210]
[0.349, 0.402]
[0.248, 0.310]
[0.241, 0.315]
[0.243, 0.303]
[0.226, 0.291]
[0.178, 0.241]
4
[0.163, 0.210]
[0.298, 0.354]
[0.188, 0.238]
[0.196, 0.241]
[0.190, 0.254]
[0.189, 0.243]
[0.154, 0.216]
5
[0.163, 0.210]
[0.245, 0.313]
[0.171, 0.226]
[0.178, 0.230]
[0.178, 0.239]
[0.177, 0.236]
[0.147, 0.194]
6
[0.163, 0.210]
[0.219, 0.289]
[0.164, 0.213]
[0.163, 0.232]
[0.172, 0.221]
[0.163, 0.226]
[0.142, 0.204]
Appendix
Table 10: 95%-confidence intervals of 10 -fold results for selected hyperparameter settings per dataset and bit width.
dataset
bits
Comparison models
Ours
SQ
Bw-MQ
Bw-QQ
Bw-SQ
california
2
0.288
0.595
0.329
0.290
3
0.217
0.377
0.265
0.210
4
0.214
0.312
0.207
0.185
5
0.208
0.262
0.182
0.171
6
0.200
0.230
0.177
0.173
Appendix
Table 11: Ablation Study: Average MSE for selected hyperparameter setting per dataset and bit width. Minimal values are bold, values within a range of 2.5% from the minimal value are underlined.
Deep neural networks have witnessed remarkable advancements in recent years and have become integral to various applications. However, alongside these developments, training and deployment of neural network models on embedding and edge devices face significant challenges due to limited memory and computational resources. These problems can be addressed with deep neural network compression, which involves a trade-off between model size and performance. In this paper, we propose a novel method for model compression through two phases. First, we utilize model compression techniques, such as pruning and quantization, to significantly reduce the model size. Then, we use Mixture of Experts to route the previously compressed models to enhance performance while maintaining a balance in inference efficiency. MoEs consist of multiple expert models (i.e., compressed models) that are moderately sized and deliver stable performance. Experimental results on several benchmark datasets show that our method successfully compresses CNN models which achieves substantial reductions in FLOPs and parameters with a negligible accuracy drop.
PQuantML is a new open-source, hardware-aware neural network model compression library tailored to end-to-end workflows. Motivated by the need to deploy performant models to environments with strict latency constraints, PQuantML simplifies training of compressed models by providing a unified interface to apply pruning and quantization, either jointly or individually. The library implements multiple pruning methods with different granularities, as well as fixed-point quantization with support for High-Granularity Quantization. We evaluate PQuantML on representative tasks such as the jet substructure classification, so-called jet tagging, an on-edge problem related to real-time LHC data processing. Using various pruning methods with fixed-point quantization, PQuantML achieves substantial parameter and bit-width reductions while maintaining accuracy. The resulting compression is further compared against existing tools, such as QKeras and HGQ.
Roope Niemi, Anastasiia Petrovych, Arghya Ranjan Das +9
European Center for Nuclear Research (CERN), CH-1211 Geneva, Switzerland · Purdue University, West Lafayette, IN 47907, USA · California Institute of Technology, Pasadena, CA 91125, United States +4
Quantization plays a crucial role in efficiently deploying deep learning models on resource-constrained devices. Post-training quantization does not require access to the original dataset or retraining the full model. Current methods that achieve high performance close to baseline results typically require INT8 fixed-point integers. However, achieving higher model compression with lower bit-widths often leads to significant performance degradation. In this paper, we propose VLCQ, a post-training quantization method that relaxes the fixed-point encoding constraint, enabling more flexible exploration of quantization techniques through variable-length encoding. This approach achieves competitive or improved performance while reducing bit-widths, without requiring access to training data or model fine-tuning. Extensive experiments on image classification, segmentation, and object detection demonstrate that VLCQ improves model compression and accuracy compared with state-of-the-art post-training quantization approaches. For per-channel quantization, VLCQ surpasses FP32 accuracy and Piece-Wise Linear Quantization (PWLQ) on most evaluated models, achieving up to 6x compression compared with FP32 and up to 1.7x compared with PWLQ. When prioritizing compression with limited accuracy loss, VLCQ achieves up to 12.25x compression compared with FP32 with less than 4% performance degradation. For per-tensor quantization, VLCQ is competitive with Data-Free Quantization (DFQ), while providing greater flexibility to achieve lower bit rates across different tasks and models.
Reem Abdel-Salam, Ahmed H. Abdel-Gawad, Amr G. Wassal
Computer Engineering Department, Faculty of Engineering, Cairo University, Giza, Egypt