Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients

Ziwei Xiang; Fanhu Zeng; Hongjian Fang; Rui-Qi Wang; Renxing Chen; Yanan Zhu; Yi Chen; Peipei Yang

cs.CVMar 18, 2026

Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients

Authors: Ziwei Xiang, Fanhu Zeng, Hongjian Fang, Rui-Qi Wang, Renxing Chen, Yanan Zhu, Yi Chen, Peipei Yang,

Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, UCAS · Beijing National Research Center for Information Science and Technology · Institute of Artificial Intelligence, USTB ·

Abstract

Large Vision Language Models (LVLMs) have achieved remarkable success in a range of downstream tasks that require multimodal interaction, but their capabilities come with substantial computational and memory overhead, which hinders practical deployment. Among numerous acceleration techniques, post-training quantization is a popular and effective strategy for reducing memory cost and accelerating inference. However, existing LVLM quantization methods typically measure token sensitivity at the modality level, which fails to capture the complex cross-token interactions and falls short in quantitatively measuring the quantization error at the token level. As tokens interact within the model, the distinction between modalities gradually diminishes, suggesting the need for fine-grained calibration. Inspired by axiomatic attribution in mechanistic interpretability, we introduce a fine-grained quantization strategy on Quantization-aware Integrated Gradients (QIG), which leverages integrated gradients to quantitatively evaluate token sensitivity and push the granularity from modality level to token level, reflecting both inter-modality and intra-modality dynamics. Extensive experiments on multiple LVLMs under both W4A8 and W3A16 settings show that our method improves accuracy across models and benchmarks with negligible latency overhead. For example, under 3-bit weight-only quantization, our method improves the average accuracy of LLaVA-onevision-7B by 1.60%, reducing the gap to its full-precision counterpart to only 1.33%. The code is available at https://github.com/ucas-xiang/QIG.

Figures & tables

Figure 1: Token-level quantization sensitivity across layers in the form of heatmap and curves. At layers 1 and 16, we show both the token-level sensitivity heatmap and its channel-averaged line curve for special , vision , and text tokens, measured using our Quantization-aware Integrated Gradients (QIG).

Figure 2: Visualization of activation distributions in InternVL2-8B during calibration. We visualize two representative layers and four linear sub-layers. In each panel, the horizontal axis denotes token positions in the multimodal sequence and the vertical axis indexes hidden channels; color encodes the average activation magnitude per token–channel pair over the calibration set. The plots reveal four recurring phenomena: massive activations, layer heterogeneity, sub-layer divergence, and token variability. These patterns indicate that coarse modality-level sensitivity modeling is insufficient, motivating our token-level sensitivity weighting.

Sensitivity Type	Granularity	Accuracy (%)
Gradient-based	Modality-level	57.36
	Token-level	55.78
	Token-level (+ special)	55.65
Attention-based	Modality-level	56.43
	Token-level	57.12
	Token-level (+ special)	57.52

Table 1: Comparison of modality-level and token-level sensitivity estimation strategies on VizWiz (W4A8, InternVL2-8B).

Figure 3: Comparison between modality-balanced quantization and our fine-grained quantization. Different colors indicate token types. Unlike MBQ, which assigns modality-level sensitivity, our method computes token-level sensitivity via Quantization-aware Integrated Gradients (QIG) during calibration, enabling more effective quantization.

Model	Bitwidth	Method	VizWiz	MMMU	ChartQA	AI2D	ScienceQA	Avg.
LLaVA-onevision-7B	FP16	-	60.41	49.22	80.04	81.31	95.88	73.37
	W3A16	RTN	59.12	43.67	68.88	78.92	94.55	69.03
		GPTQ	54.87	42.33	73.72	76.81	92.12	67.97
		AWQ	58.65	42.89	74.08	77.92	82.20	67.15
		MBQ	57.99	44.00	76.84	78.47	94.89	70.44
		QIG (Ours)	62.82	45.78	77.20	79.11	95.29	72.04

Table 2: Overall comparison of full-precision and post-training quantization methods on three representative LVLMs under W3A16 and W4A8. RTN and SQ are naive PTQ baselines, MBQ is the modality-balanced baseline, and QIG is the proposed fine-grained quantization method. Bold numbers indicate the best performance, and underlined numbers indicate the second best in each column.

Model	Bitwidth	Method	ChartQA	MMMU	VizWiz
	FP16	-	86.44	52.78	65.65
	W4A8	MBQ	84.44	49.78	63.51
InternVL2-26B	W4A8	Ours	85.24	50.22	63.91
	W3A16	MBQ	84.48	51.67	63.33
	W3A16	Ours	85.12	50.89	64.14

Table 3: Quantization on InternVL2-26B: MBQ vs. Ours under W3A16/W4A8.

baseline	Attribution objective	ChartQA	VizWiz
$0$	$f(x)$	73.87	61.73
$0$	$f(x)-f(0)$	74.30	62.31
$x^{q}$	$f(x)$	74.12	61.52
$x^{q}$	$f(x)-f(x^{q})$	74.52	62.82

Table 4: Ablation of the integrated-gradients configuration for token-wise sensitivities, varying the reference baseline

x^{\prime}

(0 vs.

x^{q}

) and attribution objective (task output

f(x)

vs. quantization-error outputs

f(x)-f(0)

f(x)-f(x^{q})

). Results are on LLaVA-onevision-7B with W4A8; the last row is our QIG formulation and performs best (higher is better).

Model	Bitwidth	Method	ChartQA	AI2D	VizWiz
[3pt] LLaVA -onevision -7B	FP16	-	80.04	81.31	60.41
	W3A16	GPTQ	73.72	76.81	54.87
	W3A16	+ Ours	74.12	76.65	56.95
[6pt] InternVL2 -8B	FP16	-	82.64	82.42	60.86
	W3A16	GPTQ	76.40	76.65	59.79
	W3A16	+ Ours	78.12	78.47	60.57

Table 5: Results of combining our fine-grained quantization with GPTQ under the W3A16.

Model Size	GPU Hours
Model Size	MBQ	Leave One Out	Ours
InternVL2-8B	0.55	2.07 (+91 min)	0.58 (+2.0 min)
InternVL2-26B	0.95	4.20 (+ 195 min)	0.99 (+2.5 min)

Table 6: Quantization time (in GPU hours) of differnet models using a single A800 80GB GPU. Fine-Grained Quantization incurs negligible overhead compared to baseline methods.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	VizWiz	MMMU	ScienceQA
No Clipping	54.32	41.37	93.28
Top5 zero	57.20	43.56	94.10
Top5 average	57.25	44.78	94.18
IQR Clipping (Ours)	59.10	45.00	94.25

Appendix

Table A1: Ablation on sensitivity stabilization strategies. Our IQR Clipping achieves the best overall performance on LLaVA-OneVision-7B under W4A8 quantization.

	PPL $\downarrow$	PIQA $\uparrow$	ARC-e $\uparrow$	ARC-c $\uparrow$	MMLU $\uparrow$
GPTQ	6.24	75.46	67.00	40.10	30.05
+ Ours	6.19	75.95	67.17	39.85	32.01

Appendix

Table A2 : Comparison of GPTQ and Our Fine-Grained Quantization on LLaMA-2-7B (3bit).

Bitwidth	Calib. Size	Method	DocVQA	ChartQA	OCRBench	Avg.
W4A8	128	MBQ	84.48	77.28	70.60	77.45
	128	Ours	88.60	77.52	76.80	80.97
	256	MBQ	84.87	76.68	71.50	77.68
	256	Ours	89.13	77.04	77.00	81.06

Appendix

Table A3: Results on Qwen2-VL-7B using OCR-specific calibration data. Our method shows significant robustness improvements over MBQ in text-rich scenarios.

Figure A1 : The baseline fails to identify the film and produces an incomplete answer, whereas our fine-grained quantization successfully preserves the correct semantic prediction and matches the full-precision model.

Figure A2 : The baseline fails to answer the question and provides no reasoning, whereas our fine-grained quantization preserves both correctness and detailed visual justification, closely matching the full-precision model.

Figure A3 : The baseline provides only a minimal and overly generic description, missing most visual details, whereas our fine-grained quantization preserves rich scene understanding and produces a comprehensive description close to the full-precision model.

VLM Quantization

Vision-language model quantization reduces model precision to lower memory and computation while preserving multimodal capabilities.

See all 52 papers

SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models
Aug 5, 2025Yufei Xue, Yushi Huang, Lunjie Zhu +2Vision-Language Models VLM Quantization
Institute of Artificial Intelligence (TeleAI), China Telecom · The Hong Kong University of Science and Technology
Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization
Feb 27, 2026Chenwei Jia, Baoting Li, Xuchong Zhang +3VLM Quantization Neural Network Quantization
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence Institute of Artificial Intelligence and Robotics Xi’an Jiaotong University
QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Models
Apr 3, 2026Xinhao Wang, Zhonyu Xia, Zhiwei Lin +2Visual Token Pruning LVLM Pruning
Wangxuan Institute of Computer Technology, Peking University

Aug 5, 2025J/K move · Enter open · S save

SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models

Yufei Xue, Yushi Huang, Lunjie Zhu, Jiawei Shao+1

Institute of Artificial Intelligence (TeleAI), China Telecom · The Hong Kong University of Science and Technology

Vision-Language Models VLM Quantization Efficient VLM Inference Post-Training Quantization Neural Network Quantization

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated remarkable capabilities across diverse language tasks, motivating their extension to vision-language models (VLMs) for multimodal understanding. However, billion-parameter VLMs incur substantial memory and computational costs that hinder deployment in resource-constrained settings. Post-training quantization (PTQ) compresses models and accelerates inference without retraining, yet remains underexplored for VLMs. We identify two intrinsic VLM activation properties in PTQ: (1) visual over-representation, where vision tokens are excessive and often redundant, and (2) the modality gap separating text and vision tokens in the latent feature space. Prior methods largely overlook these properties, leading to quantization performance degradation. To address this mismatch, we propose SalQ-VLM, an importance-aware PTQ framework that prioritizes salient tokens and suppresses redundant vision tokens during calibration. We derive a gradient-driven importance factor that captures token-level importance variance and is theoretically grounded in the relationship among loss perturbation, activation errors, and output gradients. SalQ-VLM obtains this factor through a single lightweight block-wise gradient-caching pass and incorporates it into the layer-wise reconstruction objective. Because SalQ-VLM modifies only calibration, it adds no inference-time operations and remains compatible with existing high-performance kernels. Extensive evaluations across benchmarks and backbones show that SalQ-VLM consistently outperforms strong PTQ baselines, especially under ultra-low-bit quantization. Notably, it improves MME-RealWorld accuracy by 16.45% under INT2g128 quantization.

Abstract

VLM Quantization

Explore similar work

SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models

Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization

QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Models