Quantization plays a crucial role in efficiently deploying deep learning models on resource-constrained devices. Post-training quantization does not require access to the original dataset or retraining the full model. Current methods that achieve high performance close to baseline results typically require INT8 fixed-point integers. However, achieving higher model compression with lower bit-widths often leads to significant performance degradation. In this paper, we propose VLCQ, a post-training quantization method that relaxes the fixed-point encoding constraint, enabling more flexible exploration of quantization techniques through variable-length encoding. This approach achieves competitive or improved performance while reducing bit-widths, without requiring access to training data or model fine-tuning. Extensive experiments on image classification, segmentation, and object detection demonstrate that VLCQ improves model compression and accuracy compared with state-of-the-art post-training quantization approaches. For per-channel quantization, VLCQ surpasses FP32 accuracy and Piece-Wise Linear Quantization (PWLQ) on most evaluated models, achieving up to 6x compression compared with FP32 and up to 1.7x compared with PWLQ. When prioritizing compression with limited accuracy loss, VLCQ achieves up to 12.25x compression compared with FP32 with less than 4% performance degradation. For per-tensor quantization, VLCQ is competitive with Data-Free Quantization (DFQ), while providing greater flexibility to achieve lower bit rates across different tasks and models.
Figures & tables
Methods
Limitations
PWLQ Fang et al. (2020)
Divides distribution into multiple regions, thus having higher bit-rate b+1 , when having 2b levels (if b=8, the dense region will have 256 levels, while each of the sparse regions will have 127 levels which result in a total of 512 levels) Uses a small dataset for activation calibration
DFQ Nagel et al. (2019)
Performance degrades beyond 6 bits Achieves good results for per-tensor quantization only Requires access to a small amount of data for high-bias absorption and bias correction steps
ACIQ Banner et al. (2019)
Reports result for VGG16 and Resnet18 for Image classification task only. Experiments were conducted for per-channel settings only
MultiPoint Liu et al. (2021)
Approximate error by quantizing it and Adding it to original tensor
AdaQuant Hubara et al. (2020) , AdaRound Nagel et al. (2020) , and BRECQ Li et al. (2021b) : Need access to calibration dataset for fine-tuning
Table 1: Drawbacks of the current state-of-the-art methods.
Figure 1: The weights distribution of selected channels in the first convolution layer in the 10th Inverted Residual
Figure 2: VLCQ motivation compared to uniform quantization.
Figure 3: The weights distribution in a different layer in MobileNetV2 model. The mean of the distribution is almost centered around zero while the minimum and maximum of the distribution range for different convolution layers.
Figure 4: Non-uniform quantization through breakpoints and variable step size.
Rounding Technique
MSE of Weights
MSE of Activation
Accuracy
Nearest
0.161
0.007
71.752 %
Floor
0.115
0.006
71.814 %
Ceil
0.239
0.009
71.752 %
Table 2: Comparison of ImageNet validation accuracy, MSE of weights and activation among different rounding schemes, for INT8 uniform quantization using MobileNetV2 model
Figure 5: Golomb Code decoder design.
Figure 6: Accelerator design to incorporate variable-length encoding.
Figure 7: Memory Layout to speed up the decoding process.
Figure 8: Relationship between bit-rate and number of bins vs top-1 accuracy for per-channel quantization. The Red Line refers to the relation between Accuracy and the number of bins and the blue line refers to the relationship between bit-rate and the number of bins.
MobileNetV2
ResNet18
ResNet50
Method
Number of bins
Bit-rate (W)
Accuracy
Bit-rate (W)
Accuracy
Bit-rate (W)
Accuracy
FP32
71.880%
FP32
69.758%
FP32
76.150%
PWLQ
512
9
71.730%
9
69.622%
9
76.100%
128
7
71.580%
7
69.602%
7
76.080%
32
5
69.220%
5
69.200%
5
75.620%
16
4
55.170%
4
67.076%
4
73.840%
Table 3: Top-1 accuracy (%) for various methods. Comparison of accuracy on weights quantized post-training models, per-channel methods . Activations in this case are unquantized and stored with full precision.
MobileNetV2
ResNet18
ResNet50
Method
Number of bins
Bit-rate (W)
Accuracy
Bit-rate (W)
Accuracy
Bit-rate (W)
Accuracy
FP32
71.880%
FP32
69.760%
FP32
76.130%
PWLQ
512
9
70.218%
9
69.622%
9
76.048%
256
8
70.628%
8
69.568%
8
75.932%
128
7
65.700%
7
69.602%
7
75.680%
32
5
1.000%
5
69.200%
5
69.000%
Table 4: Top-1 accuracy (%) for various methods. Comparison of accuracy on per-tensor quantized weights, post-training. Activations in this case are unquantized and stored with full precision.
MobileNetV2
ResNet18
ResNet50
Method
Number of bins
Target bit-rate
Bit-rate (W)
Accuracy FP32: 71.880%
Bit-rate (W)
Accuracy FP32: 69.760%
Bit-rate (W)
Accuracy FP32: 76.130%
VLCQ MP
256
6
5.854
71.674%
4.533
69.632%
4.651
76.008%
256
5
4.468
70.616%
3.670
69.076%
3.860
75.610%
256
4
3.554
63.278%
2.716
66.206%
2.898
74.168%
VLCQ MP SS
256
6
5.885
71.784%
5.325
69.618%
5.423
76.020%
256
5
5.193
71.348%
4.615
69.378%
4.704
76.010%
Table 5: Top-1 accuracy (%) for MobileNetV2, ResNet18, and ResNet50 when using per-region bit-allocation mixed precision strategy, per-region per-layer bit-allocation mixed precision strategy, and per-region bit-allocation mixed precision strategy with scale selection. Model weights are quantized, but activations here are unquantized and kept in full precision.
MobileNetV2
ResNet18
ResNet50
Method
Number of bins
Bit-rate (W)
Accuracy
Bit-rate (W)
Accuracy
Bit-rate (W)
Accuracy
AdaRound
FP32
71.720%
69.680%
76.070%
16
4
69.780%
4
69.710%
4
75.230%
AdaQuant
FP32
72.490%
71.080%
77.000%
16
4
44.780%
4
68.820%
4
75.220%
VLCQ
FP32
71.880%
69.760%
76.130%
Table 6: Top-1 accuracy (%) for VLCQ against various PTQ methods with calibration data. Comparison of accuracy on per-channel quantized weights, post-training. Activations in this case are unquantized and stored with full precision.
Figure 9: Top-1 accuracy (%) and bit-widths for the indicated model. Comparison of accuracy on weights quantized post-training models, per-channel methods. Activations in this case are unquantized and stored with full precision.
Technique
Bit-rate
Number of Bins
Model Size (13.5MB)
FLOPs
PWLQ
8
256
3.5726
26,932,832
7
128
3.136
6
64
2.7
Uniform Quant.
8
256
3.51755
13,574,608
7
128
3.0815
6
64
2.645
Table 7: The memory size of the model and FLOPs for MobileNetV2 for per-channel quantization.
Technique
Number of bins
Bit-rate
PWLQ
512
8.464
64
6.103
32
5.125
DFQ
256
8.527
BRECQ
16
4.010
Table 8: Bit-rate estimation using variable length encoding in different quantization techniques for MobileNetV2 .
Method
Accuracy
Bit-width Weights
Bit-width Activation
BRECQ
71.420%
4
8
55.580%
2
BRECQ SS Min-Max
72.004%
4
56.19%
2
BRECQ 2-Scales
72.306%
4
65.804%
2
Table 9: Top-1 accuracy (%) for MobileNetV2 (FP32: 72.490%) for BRECQ original model and when integrated to the proposed scale selection, and 2 scales method.
MobileNetV2
Method
Per channel
Number of bins
Bit rate (W)
mIOU
FP32
70.75%
PWLQ
✓
512
9
70.76%
✓
128
7
70.08%
✓
32
5
56.49%
VLCQ
✓
256
6.458
70.16%
Table 10: Comparison of mIOU on Pascal VOC semantic segmentation dataset.
MobileNetV2
Method
Per channel
Number of bins
Bit rate (W)
mAP
FP32
68.6%
PWLQ
✓
512
9
68.51%
✓
128
7
68.42%
✓
32
5
68.32%
VLCQ
✓
256
6.368
68.71%
Table 11: Comparison of mAP on Pascal VOC object detection dataset.
YOLOv11n
YOLOv8n
YOLOv5s
YOLOv3
Method
Number of bins
Bit-rate (W)
mAP
Bit-rate (W)
mAP
Bit-rate (W)
mAP
Bit-rate (W)
mAP
FP32
0.548
FP32
0.521
FP32
0.594
FP32
0.684
VLCQ
256
5.96
0.547
5.89
0.52
5.91
0.593
5.59
0.684
128
5.16
0.542
5.09
0.518
5.11
0.592
4.82
0.684
64
4.35
0.536
4.28
0.509
4.3
0.588
4
0.683
32
3.52
0.516
3.4
0.486
3.4
0.577
3.016
0.679
Table 12: Comparison of mAP on Pascal VOC object detection dataset for Yolo models.
Beijing Institute of Computer Technology and Application, Beijing, China · School of Elite Engineering, Northwestern Polytechnical University, Shaanxi, China