Deploying vision-language models (VLMs) on mobile devices is challenging due to their significant memory and compute requirements. We present a framework for quantizing VLMs for efficient inference on resource-constrained hardware. Our approach combines a quantization pipeline that uses the model itself to generate training data and does not require access to the training setup, with a novel 2.7-bit-per-parameter format supporting efficient execution on Arm CPUs. We validate our approach by compressing the Llama 3.2 11B Vision Instruct model to 3.7 GB with 8-bit activations, preserving strong performance on a set of standard visual question answering tasks.
Figures & tables
Figure 1: Trade-off between overall task performance and model compression under direct casting, GPTQ, and QAT. Under our QAT scheme, S3D8 is approximately 22% smaller than block-scaled INT at matched task performance in our evaluation. Directly casting model weights to fewer than 3.5 bits per parameter causes significant model degradation. While both GPTQ and QAT can improve performance, QAT is superior at fewer bits per parameter. The dashed line shows the original bfloat16 ( 21.3 GB) model performance.
Figure 2: Left: The S3D8 encoding flow. Starting from bfloat16 data, a scale is chosen based on the absolute maximum of each output channel. Rounding and quantizing the scaled elements produces the channel- INT8 compute format. This is further compressed to S3D8 by extracting the sign and storing the index of the closest centroid. Note that this logical view does not portray the physical layout of the data and lookup tables (see ). Right: S3D8 centroids trained on unit Normal weights. The 32 centroids in blue are trained on the absolute values of the rescaled weights. The 3 sign indicator bits provide the same centroids reflected into each of the 8 orthants ( 4 shown).
Format
Size [MB]
bpp
VQAv2
ChartQA
DocVQA
AI2D
Avg.
bfloat16
21,340
16.00
0.754
0.747
0.844
0.631
0.744
S3D8
3,569
2.68
0.702
0.648
0.740
0.554
0.661
INT
3,620
2.71
0.579
0.303
0.258
0.249
0.347
student-t
3,620
2.71
0.647
0.598
0.670
0.344
0.565
lloyd-max
3,459
2.59
0.638
0.510
0.444
0.151
0.436
Table 1: Comparison of QAT formats at similar model sizes. S3D8 achieves favorable downstream performance against similarly-sized INT , student-t , and lloyd-max formats. VQAv2, ChartQA, and AI2D report accuracy, while DocVQA reports ANLS. Note that the model sizes here do not include vocabulary and metadata. Further results can be found in .
Figure 3: Quantization-aware training results.
Shape
GMAC/s
(m,k,n)
bfloat16
INT8
S3D8
Pixel 8a 5 -core
(1,4096,14336)
13.6
26.5
33.8
(128,4096,14336)
76.3
201.1
197.7
(1601,1280,5120)
116.5
206.1
211.8
Graviton4
Table 2: Compute rate of selected matrix multiply shapes across different weight formats. S3D8 uses a fused kernel for m=1 and a cast followed by INT8 multiply for m>1 . See , and for further results.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Format
Element quantizer
Block shape
g
INT
Uniform signed integer grid
(1,g)
{32,64,128}
student-t
Scalar per-tensor LUT from a Student- t cube-root-density quantizer
(1,g)
{32,64,128}
lloyd-max
Scalar per-tensor LUT fit by Lloyd-Max/k-means
(1,∗)
–
Appendix
Table 3: Scalar quantization baselines used in the format sweep. The block shape (1,g) applies one scale to groups along the input dimension, while (1,∗) denotes per-output-channel scaling.
Figure 4: GPTQ scaling ablation. The absmax and affine parameterizations are evaluated using K∈{4,6,8,12,16} and group sizes g∈{32,64,128} . The dashed line shows the original bfloat16 model performance.
VCIP, College of Computer Science, Nankai University · School of Computer Science and Engineering, Tianjin University of Technology · OPPO Research Institute +3