Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization

Chenwei Jia; Baoting Li; Xuchong Zhang; Mingzhuo Wei; Bochen Lin; Hongbin Sun

cs.CVFeb 27, 2026

Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization

Authors: Chenwei Jia, Baoting Li, Xuchong Zhang, Mingzhuo Wei, Bochen Lin, Hongbin Sun

Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence Institute of Artificial Intelligence and Robotics Xi’an Jiaotong University

Abstract

Post-Training Quantization (PTQ) has emerged as an effective technique for alleviating the substantial computational and memory overheads of Vision-Language Models (VLMs) by compressing both weights and activations without retraining the full model. Existing PTQ methods primarily rely on static identification and global compensation of sensitive or outlier channels, yet they often overlook the distributional differences of these important channels across inputs, leading to unsatisfactory quantization. In this work, we observe that the distributions and occurrence frequencies of important channels vary significantly both across modalities and among tokens, even within the same modality. Accordingly, we propose Quant Experts (QE), a token-aware adaptive error compensation with mixture-of-experts for VLMs quantization. QE divides the important channels into token-independent and token-dependent groups. For the former, a shared expert is designed for most tokens to compensate for global quantization error using a low-rank adapter. For the latter, routed experts including multiple routed low-rank adapters are elaborated to compensate for local quantization error related to specific tokens. Extensive experiments demonstrate that QE consistently enhances task accuracy across various quantization settings and model scales, ranging from 2B to 70B parameters, while maintaining performance comparable to full-precision models.

Figures & tables

Figure 1: Illustration of Different Quantization Granularities. $\mathcal{D}$ is the calibration data, and $\mathbf{W}_{q}$ is the quantized weight matrix. SmoothQuant [ 44 ] employs channel-wise with fixed scaling coefficients to achieve global quantization. MBQ [ 22 ] presents a modality-aware strategy that focuses on sensitivities in input modalities. In contrast, we propose a token-aware adaptive quantization that considers both global and local error reconstruction.

Figure 2: Visualization of a Transformer block in Qwen2VL-2B, illustrating token value distributions (top) and the positions of important channels (bottom). In the top panel, brightness reflects token magnitude, while in the bottom panel, highlighted positions denote important channels.

Figure 3: Visualization of a Transformer block in Qwen2VL-2B, illustrating important channel behavior. The top panel shows activation frequencies, while the bottom depicts average activation values. Red crosses mark important channel positions identified by the static global method on the calibration dataset.

Figure 4: The framework of Quant Experts (QE). The token-independent channels are model by a shared expert, while token-dependent channels are captured by multiple routed experts. A lightweight low-rank adapter is implemented for each expert.

Figure 5: Illustration of the Inference Computation Process of QE.

Method	#W	#A	MMMU	OCRBench	ScienceQA	TextVQA	VizWiz	AI2D	ChartQA	DocVQA	InfoVQA	MMStar	MuriBench	Avg. ( $\uparrow$ )
Qwen2VL-2B	16	16	39.89	74.90	76.96	77.72	65.73	70.01	72.04	87.28	58.49	43.46	26.19	62.97
RTN	4	6	34.00	59.80	64.70	67.58	55.62	59.26	56.08	75.78	45.12	40.67	31.19	53.62
SQ (ICML’23)	4	6	30.44	59.60	65.25	65.88	53.90	59.16	40.44	70.73	39.36	38.20	30.00	50.27
LQER (ICML’24)	4	6	33.00	65.80	68.32	69.37	55.91	62.56	62.68	81.02	48.92	37.82	29.77	55.92
MBQ (CVPR’25)	4	6	34.44	61.10	67.08	69.45	57.19	60.91	60.08	76.24	43.13	42.61	29.77	54.73
QE	4	6	33.78	68.20	71.84	73.18	59.62	65.45	64.60	82.75	51.84	42.04	32.88	58.74

Table 1: Main results on the model of Qwen2VL-2B.

Method	#W	#A	MMMU	OCRBench	ScienceQA	TextVQA	VizWiz	AI2D	ChartQA	DocVQA	InfoVQA	MMStar	MuriBench	Avg. ( $\uparrow$ )
InternVL2-8B	16	16	48.00	76.90	97.12	76.91	60.61	82.09	82.60	89.97	66.92	59.36	36.12	70.60
RTN	4	6	37.00	69.20	93.16	69.96	55.34	73.77	74.16	83.81	55.82	50.03	30.12	62.94
SQ (ICML’23)	4	6	40.44	69.50	94.84	69.83	50.91	75.13	74.92	84.28	56.77	49.59	32.00	63.47
LQER (ICML’24)	4	6	40.22	72.20	94.74	71.83	58.03	76.49	77.88	85.80	60.47	49.69	30.85	65.29
MBQ (CVPR’25)	4	6	43.67	71.00	95.49	70.26	52.90	77.59	75.64	84.21	58.09	53.89	32.27	65.00
QE	4	6	44.89	74.30	96.23	74.77	59.00	79.40	80.48	87.77	63.12	55.27	34.15	68.13

Table 2: Main results on the model of InternVL2-8B.

Setting	Method	MMMU	OCRBench	ScienceQA	TextVQA	VizWiz
FP16	-	61.44	78.70	91.22	82.26	76.27
W4A6	RTN	51.33	59.10	85.52	72.12	64.29
	SQ	53.33	61.20	86.96	74.42	63.45
	LQER	52.33	59.60	86.71	73.84	66.64
	MBQ	52.67	69.70	86.32	76.08	67.99
	QE	58.11	76.60	90.33	79.27	73.91

Table 3: Main results of Qwen2VL-72B model (higher is better).

Figure 6: Illustration of the Co-Occurrence-Based Clustering in a Transformer Block of Qwen2VL-2B. (a) Similarity matrix $\mathbf{S}^{l}$ showing mutual co-occurrence among token-dependent channels, with brightness indicating similarity. (b) Channels with strong co-occurrence are grouped into the same cluster. (c) t-SNE [ 30 ] projection demonstrates that the clustering effectively captures their co-occurrence relations.

Setting	Component	MMMU ( $\uparrow$ )	ScienceQA ( $\uparrow$ )
FP16	-	39.89	76.95
W4A6	routed experts (REs)	34.56	68.72
	shared expert (SE)	35.22	69.61
	SE + random routing	35.89	70.00
	SE + random clustering	35.33	69.71
	QE (SE+REs)	36.89	70.85

Table 4: Ablation study results on Qwen2VL-2B model.

Model	Ref.	MMMU	OCRBench	ScienceQA	TextVQA	VizWiz
2B	✗	33.78	68.20	71.84	73.18	59.62
2B	✓	36.89	69.60	70.85	73.30	60.58
7B	✗	45.44	73.00	79.87	71.63	64.18
7B	✓	44.00	74.60	80.61	77.58	65.11

Table 5: Ablation study of Refinement on Routed Experts. “✓” indicates applied refinement; “✗” indicates none.

$N_{r}$	OCRBench	TextVQA	VizWiz	Avg. ( $\uparrow$ )
2	68.40	73.14	59.70	67.08
4	68.50	73.13	60.41	67.35
8	69.60	73.30	60.58	67.83
16	69.90	73.52	60.75	68.06

Table 6: Impact of the number of routed experts on the performance of Qwen2VL-2B under the W4A6 quantization setting.

Complexity	Origin	QE
Computation	$sd^{2}$	$sd^{2}+sd(2r+N_{r})$
Memory	$d^{2}$	$d^{2}+rd(1+N_{r})$

Table 7: Complexity analysis of the linear layer in QE method.

Shape (IC $\times$ OC)		W4A6	W4A8	W3A16
3584 $\times$	3584	$3.56\times$	$3.50\times$	$4.10\times$
3584 $\times$	18944	$3.60\times$	$3.59\times$	$4.50\times$
18944 $\times$	3584	$3.84\times$	$3.77\times$	$4.50\times$

Table 8: NPU speedup ratios of QE for Qwen2VL-7B linear layers compared with the fp16 model, measured during the prefill stage with a sequence length of

s{=}128

. “IC” and “OC” denote the input and output channel dimensions, respectively.

Method	#W	#A	MMMU	OCRBench	ScienceQA	TextVQA	VizWiz	AI2D	ChartQA	DocVQA	InfoVQA	MMStar	MuriBench	Avg. ( $\uparrow$ )
Qwen2VL-7B	16	16	50.78	79.50	84.83	81.48	68.56	80.60	81.68	91.68	69.77	57.74	42.92	71.78
RTN	4	6	39.00	59.50	75.41	66.93	56.64	69.62	72.00	78.73	52.91	49.97	38.27	59.91
SQ (ICML’23)	4	6	41.00	63.90	77.09	67.94	57.48	70.92	68.84	78.78	52.90	50.48	35.65	60.45
LQER (ICML’24)	4	6	42.56	65.60	77.24	71.87	64.44	71.44	74.04	81.57	56.86	49.75	41.85	63.38
MBQ (CVPR’25)	4	6	40.56	62.70	79.67	70.91	51.48	71.31	72.20	81.99	54.71	47.37	34.54	60.68
QE	4	6	45.44	73.00	79.87	71.63	64.18	75.58	74.16	83.60	60.45	52.63	42.19	65.70

Table 9: Main results on the model of Qwen2VL-7B.

Method	#W	#A	MMMU	OCRBench	ScienceQA	TextVQA	VizWiz	AI2D	ChartQA	DocVQA	InfoVQA	MMStar	MuriBench	Avg. ( $\uparrow$ )
InternVL2-2B	16	16	34.33	75.30	94.30	72.58	45.94	72.83	74.84	84.84	53.23	48.20	28.46	62.26
RTN	4	6	30.44	67.10	86.47	66.17	41.66	63.41	66.48	77.41	43.91	40.11	26.85	55.46
SQ (ICML’23)	4	6	31.89	69.30	88.25	67.24	40.17	64.41	65.48	79.94	46.94	41.62	25.69	56.45
LQER (ICML’24)	4	6	30.78	70.90	88.35	67.81	39.33	65.64	68.92	79.78	45.73	41.75	28.27	57.02
MBQ (CVPR’25)	4	6	31.33	70.90	90.53	68.54	41.39	67.52	70.20	80.99	47.95	45.26	25.46	58.19
QE	4	6	32.11	72.80	92.12	70.41	43.81	68.69	70.52	82.00	48.69	45.18	28.69	59.55

Table 10: Main results on the model of InternVL2-2B.

Model	Setting	Method	MMLU ( $\uparrow$ )
Qwen2VL-2B	FP16	-	52.79
	W4A6	LQER	44.37
	W4A6	QE	47.21
	W4A8	LQER	46.60
	W4A8	QE	50.35
Qwen2VL-7B	FP16	-	67.88

Table 11: The results of quantized Qwen2VL on the MMLU benchmark.

L	V	M	OCRBench	ScienceQA	TextVQA	VizWiz	Avg. ( $\uparrow$ )
-	-	-	74.90	76.95	77.72	65.73	73.83
✓	-	-	68.20	71.84	73.18	59.62	68.21
✓	✓	-	65.90	70.75	72.09	59.83	67.14
✓	✓	✓	66.40	70.10	71.77	59.28	66.89

Table 12: Quantization results of different modules in Qwen2VL-2B. The symbol “-‘’ indicates full precision (FP16), while ✓ denotes W4A6 quantization. L, V, and M correspond to the VLM, visual encoder, and merger module, respectively.

$k$	MMMU	OCRBench	VizWiz	Avg. ( $\uparrow$ )
4	34.89	67.30	58.93	53.71
8	35.33	68.90	58.95	54.39
16	35.11	68.50	59.92	54.51
32	36.11	69.30	60.24	55.22
64	34.44	70.10	59.57	54.70

Table 13: Impact of the number of important channels on the performance of Qwen2VL-2B under the W4A6 quantization setting.

VLM Quantization

Vision-language model quantization reduces model precision to lower memory and computation while preserving multimodal capabilities.

See all 52 papers

SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models
Aug 5, 2025Yufei Xue, Yushi Huang, Lunjie Zhu +2Vision-Language Models VLM Quantization
Institute of Artificial Intelligence (TeleAI), China Telecom · The Hong Kong University of Science and Technology
Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients
Mar 18, 2026Ziwei Xiang, Fanhu Zeng, Hongjian Fang +6VLM Quantization Post-Training Quantization
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, UCAS · Beijing National Research Center for Information Science and Technology +3
Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
May 19, 2026Yi Zhong, Haotong Qin, Xindong Zhang +2Vision-Language Models VLM Quantization
VCIP, College of Computer Science, Nankai University · D-ITET, ETH Zürich · OPPO Research Institute +1

Aug 5, 2025J/K move · Enter open · S save

SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models

Yufei Xue, Yushi Huang, Lunjie Zhu, Jiawei Shao+1

Institute of Artificial Intelligence (TeleAI), China Telecom · The Hong Kong University of Science and Technology

Vision-Language Models VLM Quantization Efficient VLM Inference Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated remarkable capabilities across diverse language tasks, motivating their extension to vision-language models (VLMs) for multimodal understanding. However, billion-parameter VLMs incur substantial memory and computational costs that hinder deployment in resource-constrained settings. Post-training quantization (PTQ) compresses models and accelerates inference without retraining, yet remains underexplored for VLMs. We identify two intrinsic VLM activation properties in PTQ: (1) visual over-representation, where vision tokens are excessive and often redundant, and (2) the modality gap separating text and vision tokens in the latent feature space. Prior methods largely overlook these properties, leading to quantization performance degradation. To address this mismatch, we propose SalQ-VLM, an importance-aware PTQ framework that prioritizes salient tokens and suppresses redundant vision tokens during calibration. We derive a gradient-driven importance factor that captures token-level importance variance and is theoretically grounded in the relationship among loss perturbation, activation errors, and output gradients. SalQ-VLM obtains this factor through a single lightweight block-wise gradient-caching pass and incorporates it into the layer-wise reconstruction objective. Because SalQ-VLM modifies only calibration, it adds no inference-time operations and remains compatible with existing high-performance kernels. Extensive evaluations across benchmarks and backbones show that SalQ-VLM consistently outperforms strong PTQ baselines, especially under ultra-low-bit quantization. Notably, it improves MME-RealWorld accuracy by 16.45% under INT2g128 quantization.

Abstract

VLM Quantization

Explore similar work

SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models

Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients

Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models