ProQuant: Progressive Quantization-aware Training for Edge MLLMs

Yufei Xue; Yushi Huang; Jiawei Shao; Pingcheng Dong; Yonghao Tan; Shiyao Li; Kwang-Ting Cheng; Jun Zhang

cs.CVNov 5, 2026

ProQuant: Progressive Quantization-aware Training for Edge MLLMs

Authors: Yufei Xue, Yushi Huang, Jiawei Shao, Pingcheng Dong, Yonghao Tan, Shiyao Li, Kwang-Ting Cheng, Jun Zhang

Organizations: The Hong Kong University of Science and Technology · Northwestern Polytechnical University · Institute of Artificial Intelligence (TeleAI), China Telecom · Tsinghua University

Abstract

Multi-modal large language models (MLLMs) with impressive perception and understanding capabilities will enable a range of downstream applications. However, their substantial computational and memory requirements hinder the real-world implementation, particularly for resource-constrained edge devices. Quantization has emerged as an effective solution for deploying large models. While post-training quantization (PTQ) successfully retains the performance at moderate bit-width (e.g., W8A8), it suffers from severe performance degradation under low-bit quantization (e.g., W4A8), especially for small-scale models that are more suitable for edge deployment. In this paper, we propose ProQuant, a novel weight-activation quantization-aware training (QAT) framework tailored for low-bit MLLMs. First, to align the limited finetuning dataset distribution with the pretrained dataset distribution, we propose a self-relabeling (SRL) scheme to resample the responses of the open-source multimodal dataset. Second, we are the first to take a decoupled view of multi-modal (MM) understanding and language modeling abilities. We propose a progressive block activation (PBA) mechanism to prioritize the MM understanding recovery. We also introduce a vision-sensitive factor $\vecτ$ that effectively identifies the MM-critical layers. Extensive experiments across various benchmarks, covering models with 2B-8B parameters, show that ProQuant significantly outperforms existing methods. For example, our W4A8 Qwen3-VL-2B-Instruct improves average accuracy by impressively 6.6% and achieves performance comparable to full precision counterparts with only approximately 1% performance drop. Code is available at https://github.com/yufeixue-ai/ProQuant/.

Figures & tables

Figure 2: Overview of the proposed ProQuant . Left: examples of responses from GPT-4 [ Bubeck et al., 2023 ] and re-sampled via the SRL scheme (Section 3.1 ) using Qwen3-VL-2B-Instruct (see more details in Appendix B ). Right: QAT with the PBA mechanism, where quantization is first activated for decoding layers with the larger vision-sensitivity factor $\tau$ . As training proceeds, quantization is gradually enabled for the remaining layers in descending order of $\tau$ (Section 3.2 ).

Panel of Figure 3 — Figure 3: Accuracy of Qwen3-VL-2B-Instruct W4A8 checkpoints on HellaSwag [ Zellers et al., 2019 ] (left) and AI2D [ Kembhavi et al., 2016 ] (right) benchmarks over training iterations.

Figure 4: Loss trajectories of TSB/VSB-prioritized QAT over the first 300 steps.

Figure 5: Normalized vision sensitivity $\vec{\tau}$ and weight gradient norm $\lVert\mathbf{G}_{w}\rVert_{\ell_{1}}$ across decoding layers.

Figure 6: Training loss comparison of Qwen3-VL-2B-Instruct between naive QAT and ProQuant under W4A8 and W4A4. PBA exhibits faster warm-up convergence toward the pre-trained level, rapid recovery during PBA, and better final convergence.

Arch	#BW	Method	AI2D	ChQ	DocV	M3U	SciQ	Seed	TxtV	Avg. ( $\uparrow$ )	MME
Qwen2-VL-2B	BF16	-	70.04	72.16	89.40	39.89	74.23	69.98	79.38	70.73	1870.19
	W4A8	SmoothQ	62.66	53.68	76.59	34.00	66.47	63.20	69.17	60.82	1736.61
		MBQ	65.12	62.08	79.76	35.22	68.99	64.64	71.91	63.96	1810.68
		MQuant	67.36	70.44	87.39	38.33	–	–	77.72	–	–
		LLM-QAT	66.94	68.20	87.14	35.56	69.37	68.27	77.94	67.63	1931.70
		ProQuant	69.30	70.32	88.10	39.00	72.79	69.19	78.60	69.61	1830.28

Table 1: Weight-activation quantization results for Qwen VL series. The headers represent model architectures, bitwidth, quantization methods, AI2D, ChartQA, DocVQA, MMMU, ScienceQA, SeedBench, TextVQA, average accuracy, and MME scores, respectively. The superscript

{\dagger}

indicates down_proj activations are quantized to 8-bit under W4A4 settings.

Method	AI2D	ChQ	DocV	M3U	SciQ	Seed	TxtV	Avg. ( $\uparrow$ )	MME
Qwen2-VL-2B-Instruct								70.73	1870.19
MBQ	69.88	72.28	86.89	39.33	74.09	68.82	77.13	69.77	1884.59
ProQuant	69.66	72.08	88.91	38.89	73.21	69.88	79.34	70.28	1864.33
Qwen2.5-VL-3B-Instruct								76.33	2149.00
MBQ	78.27	83.00	92.69	46.78	82.55	71.12	78.60	76.14	2156.88
ProQuant	78.30	83.52	92.25	48.11	82.34	71.08	78.42	76.29	2102.85

Table 2: W8A8 quantization results on three model backbones. Values in red denote the corresponding full-precision (BF16) performance.

#BW	Method	AI2D	ChQ	DocV	M3U	SciQ	Seed	TxtV	Avg. ( $\uparrow$ )	MME
W4g128	MBQ	64.35	74.88	91.23	36.67	68.57	70.22	76.94	68.98	1955.42
	ProQuant	71.76	76.44	91.26	38.67	77.08	72.11	77.97	72.18	1958.53
W3g128	MBQ	32.25	61.00	76.21	28.00	46.59	62.91	65.48	53.21	1350.49
	ProQuant	65.90	71.92	88.29	35.67	68.45	70.23	75.60	68.01	1922.82

Table 3: Weight-only quantization results of MBQ and ProQuant under W4g128 and W3g128 settings.

#BW	Method	AI2D	ChQ	DocV	M3U	SciQ	Seed	TxtV	Avg. ( $\uparrow$ )	MME
FP	-	83.71	84.80	95.53	50.22	92.88	75.31	82.48	80.70	2381.20
W4A8	MBQ	77.62	75.40	92.71	45.78	87.31	72.55	78.00	75.62	2162.47
	ProQuant	82.19	84.12	95.20	49.33	91.44	75.16	81.18	79.80	2325.49
W4A4	MBQ	2.20	0.00	0.03	26.22	3.56	8.30	0.00	5.76	0.00
	ProQuant	62.44	57.36	88.22	40.89	76.77	67.53	75.32	66.93	1877.45

Table 4: W/A quantization results on Qwen3-VL- 8B -Instruct .

#BW	Qwen2-VL-2B		Qwen2.5-VL-3B		Qwen3-VL-2B
	Avg.	$\mathbf{\Delta}$	Avg.	$\mathbf{\Delta}$	Avg.	$\mathbf{\Delta}$
W4A4 ${}^{\phantom{{\dagger}}}$	54.45	–	55.08	–	59.83	–
W4A4 †	55.89	+1.44	64.47	+9.39	63.04	+3.21

Table 5: Mixed-precision quantization results on three backbones. The superscript

{\dagger}

denotes the W4A4 † setting, where down_proj activations are kept at 8-bit.

Panel of Figure 7 — Figure 7: The step-by-step ablation studies on SRL and PBA.

Metric	Prior	Avg. ( $\uparrow$ )
Random	-	71.88
$\lVert\mathbf{G}_{w}\rVert_{\ell_{1}}$	Lower	72.10
$\lVert\mathbf{G}_{w}\rVert_{\ell_{1}}$	Higher	72.19
$\vec{\tau}$	Lower	72.03
$\vec{\tau}$ (Ours)	Higher	72.54

Table 6: Ablation studies on block activation metric

Table 7: Ablation studies on SRL labeler, PBA scheduler, and block granularity.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Extended example of resampled datasets in Section 3 .

#BW	SQ Init	Avg. ( $\uparrow$ )	MME
W4A8	✓	71.41	1956.58
	✗	72.54	1962.04
W4A4	✗	50.87	1459.14
	✓	59.83	1652.63

Appendix

Table A1 : ProQuant w/ or w/o SmoothQuant.

Supervision	Formulation	Avg. ( $\uparrow$ )
KL Div.	$\mathcal{L}_{\text{KL}}=\mathcal{D}(P_{t}\|\|P_{s})$	71.76
Reversed KL	$\mathcal{L}_{\text{ReKL}}=\mathcal{D}(P_{s}\|\|P_{t})$	71.41
CAKLD Du et al. [2024]	$\mathcal{L}_{\text{CAKLD}}=\alpha\mathcal{L}_{\text{ReKL}}+(1-\alpha)\mathcal{L}_{\text{KL}}$	71.94
Multi-obj. Guo et al. [2025]	$\mathcal{L}_{\text{QAT}}=\mathcal{L}_{\text{CAKLD}}+\beta\mathcal{L}_{\text{CE}}$	72.54

Appendix

Table A2 : QAT performance under different supervisions.

Figure A2 : System-level architecture of our benchmark edge MLLM accelerator.

Figure A3 : Test environment. A Linux-based ZU5EV FPGA board supplies data and instructions to the test-chip board and receives results. The PC supports debugging and control, the DC supply powers the system board, and the oscilloscope observes state signals. The voltage tuning circuit supports 0.6–1.0 V; this range is not a statement of the operating voltage used in every experiment.

Stage	#V Tok.	#T Tok.	BF16		W4A8				W4A4
			Lat.	Eng.	Lat.	Spd.	Eng.	Sav.	Lat.	Spd.	Eng.	Sav.
Prefill	256	256	877.02	2400.87	530.80	1.65 $\times$	1020.90	2.35 $\times$	350.51	2.50 $\times$	779.96	3.08 $\times$
		1024	1977.79	4817.34	1144.89	1.73 $\times$	1888.29	2.55 $\times$	632.23	3.13 $\times$	1164.54	4.14 $\times$
	1024	256	2559.89	6045.12	1726.99	1.48 $\times$	3116.07	1.94 $\times$	1214.34	2.11 $\times$	2392.32	2.53 $\times$
		1024	3766.04	8654.54	2441.82	1.54 $\times$	4069.73	2.13 $\times$	1522.48	2.47 $\times$	2831.59	3.06 $\times$
Decode	256	256	124.55	493.04	31.79	3.92 $\times$	124.33	3.97 $\times$	31.40	3.97 $\times$	123.95	3.98 $\times$

Appendix

Table A7 : Performance comparison of BF16, W4A8, and W4A4 for Qwen2-VL-2B-Instruct in both prefill and decode stages. “#V Tok.” and “#T Tok.” denote the numbers of vision tokens and text tokens, respectively; “Lat.” and “Eng.” denote latency (ms) and energy (mJ), respectively; “Spd.” and “Sav.” denote the speedup and energy saving relative to the BF16 baseline, respectively.

Stage	#V Tok.	#T Tok.	BF16		W4A8				W4A4
			Lat.	Eng.	Lat.	Spd.	Eng.	Sav.	Lat.	Spd.	Eng.	Sav.
Prefill	256	256	1663.57	4802.11	882.53	1.89 $\times$	1580.31	3.04 $\times$	507.40	3.28 $\times$	1030.43	4.66 $\times$
		1024	3959.49	10486.84	2132.48	1.86 $\times$	3347.68	3.13 $\times$	1088.50	3.64 $\times$	1859.23	5.64 $\times$
	1024	256	4439.11	11419.90	2612.11	1.70 $\times$	4280.74	2.67 $\times$	1568.12	2.83 $\times$	2792.29	4.09 $\times$
		1024	6972.05	17529.33	4034.06	1.73 $\times$	6226.26	2.82 $\times$	2193.81	3.18 $\times$	3719.84	4.71 $\times$
Decode	256	256	248.69	984.06	63.25	3.93 $\times$	247.63	3.97 $\times$	62.45	3.98 $\times$	246.85	3.99 $\times$

Appendix

Table A8 : Performance comparison of BF16, W4A8, and W4A4 for Qwen2.5-VL-3B-Instruct in both prefill and decode stages.

Stage	#V Tok.	#T Tok.	BF16		W4A8				W4A4
			Lat.	Eng.	Lat.	Spd.	Eng.	Sav.	Lat.	Spd.	Eng.	Sav.
Prefill	256	256	869.62	2227.72	498.60	1.74 $\times$	860.56	2.59 $\times$	300.42	2.89 $\times$	569.67	3.91 $\times$
		1024	2090.36	4832.36	1190.86	1.76 $\times$	1791.66	2.70 $\times$	612.83	3.41 $\times$	1038.96	4.65 $\times$
	1024	256	2444.41	5570.82	1544.91	1.58 $\times$	2530.13	2.20 $\times$	966.88	2.53 $\times$	1777.43	3.13 $\times$
		1024	3798.92	8333.40	2370.94	1.60 $\times$	3559.28	2.34 $\times$	1313.97	2.89 $\times$	2231.03	3.74 $\times$
Decode	256	256	140.46	556.20	35.89	3.91 $\times$	141.56	3.93 $\times$	35.89	3.91 $\times$	141.56	3.93 $\times$

Appendix

Table A9 : Performance comparison of BF16, W4A8, and W4A4 for Qwen3-VL-2B-Instruct in both prefill and decode stages.

LLM Quantization

Large language model quantization reduces numerical precision to lower memory use and accelerate training or inference.

See all 413 papers

LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
Sep 28, 2025Shubhang Bhatnagar, Andy Xu, Kar-Han Tan +1Mixed-Precision Quantization LLM Quantization
University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles +1
EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation
Apr 10, 2026Shu-Hao Zhang, Le-Tong Huang, Xiang-Sheng Deng +5Quantization-Aware Distillation LLM Quantization
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210063, China · School of Intelligent Science and Technology, Nanjing University, Suzhou 215163, China · Microsoft AI, Beijing 100080, China +1
Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs
Aug 27, 2026Tanzila Rahman, Mehran Taghian Jazi, Yunke Peng +10Activation Quantization Low-Bit Quantization
Huawei

Sep 28, 2025J/K move · Enter open · S save

LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models

Shubhang Bhatnagar, Andy Xu, Kar-Han Tan, Narendra Ahuja

University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles+1

Mixed-Precision Quantization LLM Quantization VLM Quantization Multimodal Large Language Models Multimodal Foundation Models Post-Training Quantization+1

Open paper AskPDF arXiv

Large Language Models (LLMs) with multimodal capabilities have revolutionized vision-language tasks, but their deployment often requires huge memory and computational resources. Post-training quantization (PTQ) has successfully compressed language models to as low as 1-bit precision, its effectiveness for multimodal LLMs (MLLMs) remains unexplored. In this paper, we present the first method for ultra-low-bit (<4-bit) quantization of MLLMs. Our analysis reveals that multimodal tokens and intermediate layer activations produced by them exhibit significantly higher entropy compared to text tokens, indicating greater functional complexity that makes MLLMs less tolerant to ultra-low bit quantization. However, this entropy varies significantly across layers, with some layers producing lower-entropy activation distributions that we empirically show can better tolerate ultra-low bit quantization. Existing PTQ methods optimize weight quantization within each layer but apply the same target precision uniformly, ignoring this variation in complexity across layers. Building on this insight, we propose LUQ: Layerwise Ultra-Low Bit Quantization, which characterizes each transformer layer's functional complexity via its output activation entropy and selectively applies ultra-low bit quantization to layers encoding simpler, more compressible functions. We also show that multimodal calibration (image and text tokens) boosts VQA performance in the ultra-low bit regime. Evaluated on LLaVA-1.5 and Qwen-2.5-VL across 9 VQA benchmarks, LUQ models use 40% and 31% less memory than their 4-bit counterparts while exhibiting less than 10% degradation on MME.

Abstract

LLM Quantization

Explore similar work

LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models

EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation

Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs