QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Models

Xinhao Wang; Zhonyu Xia; Zhiwei Lin; Zhe Li; Yongtao Wang

cs.CVApr 3, 2026

QAPruner: Quantization-Aware Vision Token Pruning for Multimodal Large Language Models

Authors: Xinhao Wang, Zhonyu Xia, Zhiwei Lin, Zhe Li, Yongtao Wang

Organizations: Wangxuan Institute of Computer Technology, Peking University

Abstract

Multimodal Large Language Models (MLLMs) have shown strong reasoning ability, but their high computational and memory costs hinder deployment in resource-constrained settings. While Post-Training Quantization (PTQ) and vision token pruning are standard compression techniques, they are usually treated as independent optimizations. In this paper, we show that these two techniques are strongly coupled: naively applying semantic-based token pruning to PTQ-optimized MLLMs can discard activation outliers that are important for numerical stability and thus worsen quantization errors in low-bit regimes (e.g., W4A4). To address this issue, we propose a quantization-aware vision token pruning framework. Our method introduces a lightweight hybrid sensitivity metric that combines simulated group-wise quantization error with outlier intensity. By combining this metric with standard semantic relevance scores, the method retains tokens that are both semantically informative and robust to quantization. Experiments on standard LLaVA architectures show that our method consistently outperforms naive integration baselines. At an aggressive pruning ratio that retains only 12.5% of visual tokens, our framework improves accuracy by 2.24% over the baseline and even surpasses dense quantization without pruning. To the best of our knowledge, this is the first method that explicitly co-optimizes vision token pruning and PTQ for accurate low-bit MLLM inference.

Figures & tables

Figure 1: Motivation teaser of quantization-aware vision token pruning on a real ScienceQA sample. Panel (a) shows the input image. Panel (b) shows the token-level outlier scores, with darker red cells indicating greater quantization sensitivity. Panel (c) shows the tokens kept by semantic-only pruning, which misses the highest-scoring outlier token and leads the quantized model to predict Rhode Island . Panel (d) shows our selection, which preserves that token and recovers the correct answer South Carolina .

Panel of Figure 2 — Figure 2: Overview of the proposed quantization-aware vision token pruning framework. Given input visual tokens and the query text, the model computes three complementary signals: group-wise quantization error, global outlier intensity, and semantic relevance. The first two signals are combined into a quantization sensitivity score $\mathbf{S}^{Q}$ , which is further fused with the semantic pruning score $\mathbf{S}^{P}$ to produce the final score for selecting the top- $K$ visual tokens.

	Bits	Method	Tokens	Subject			Context Modality			Average
	Bits	Method		NAT	SOC	LAN	TXT	IMG	NO	Average
LLaVA-7B	FP	-	256	89.43	95.61	85.09	88.61	87.41	88.15	89.60
	W4A4	QVLM	256	77.62	89.99	78.18	75.95	77.74	80.77	80.36
		With Prune	128	77.62	87.63	77.73	75.90	76.45	80.77	79.75
		Ours	128	78.69	90.55	77.82	77.22	78.98	80.77	80.95 (+1.20)
		With Prune	32	77.58	82.34	77.73	75.76	74.07	80.77	78.61

Table 1: Comparisons with quantization methods and vision tokens pruning methods for LLaVA-v1.3 and LLaVA-v1.5 7B models on the ScienceQA dataset. Question classes: NAT = natural science, SOC = social science, LAN = language science, TXT = text context, IMG = image context, NO = no context.

Panel of Figure 3 — Figure 3: Normalized accuracy retention versus retained visual-token ratio for LLaVA-7B and LLaVA-13B under W4A4 PTQ. Each curve is normalized by the dense W4A4 baseline of its model. As the token budget shrinks, semantic-only pruning degrades more noticeably, while our method remains consistently closer to, and often above, the dense PTQ baseline.

Methods	G1 Accuracy $\uparrow$	G7 Accuracy $\uparrow$	Average. $\uparrow$
-	80.73	74.82	78.61
$L_{\infty}$ Norm	82.42	75.81	80.05
Token-wise AbsMax	82.42	76.07	80.15
Variance	82.67	75.68	80.17
$L_{1}$ Norm	82.67	76.27	80.38
$L_{2}$ Norm	82.93	76.20	80.52

Table 2: Ablation study about methods to calculate quantization sensitivity

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 265 papers

Towards Joint Quantization and Token Pruning of Vision-Language Models
Apr 19, 2026Xinqing Li, Xin He, Xindong Zhang +3Vision-Language Models VLM Quantization
VCIP, College of Computer Science, Nankai University · School of Computer Science and Engineering, Tianjin University of Technology · OPPO Research Institute +3
SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models
Jul 28, 2026Yuchen Wang, Qihui Zhu, Yang Liu +2Efficient Multimodal Inference Multimodal Large Language Models
MoE Key Lab of BIPC, NEL-BITA, University of Science and Technology of China, Hefei, China · ChangXin Memory Technologies, Hefei, China · APKL of BIIP, IAI, Hefei Comprehensive National Science Center, Hefei, China
P4Q: Co-designing Token Pruning and Quantization for Vision-Language Model Acceleration
Sep 28, 2026Haizhao Jing, Zhenhao Shang, Haokui Zhang +2VLM Quantization Efficient VLM Inference
Northwest Polytechnical University · Intellifusion

Apr 19, 2026J/K move · Enter open · S save

Towards Joint Quantization and Token Pruning of Vision-Language Models

Xinqing Li, Xin He, Xindong Zhang, Ming-Ming Cheng+2

VCIP, College of Computer Science, Nankai University · School of Computer Science and Engineering, Tianjin University of Technology · OPPO Research Institute+3

Vision-Language Models VLM Quantization Efficient VLM Inference Low-Bit Quantization Visual Token Pruning Neural Network Compression+1

Open paper AskPDF arXiv

Deploying Vision-Language Models (VLMs) under aggressive low-bit inference remains challenging because inference cost is dominated by the long visual-token prefix during prefill and the growing KV cache during autoregressive decoding. Token pruning and low-bit quantization are complementary for reducing these costs, yet naive stage-wise combinations are often brittle due to a mismatch between quantization calibration and pruning execution. We present a collaborative quantization-and-pruning framework that unifies low-bit inference and deterministic visual-token pruning in a single deployable pipeline. The framework introduces the Quantization Unified Offline Token Allocator (QUOTA), which converts low-bit calibration signals into a layer-wise token allocation schedule and materializes it as a pruning recipe. Token importance is evaluated under deployed W4A4 operators with a quantized KV cache by combining activation magnitude, attention cues, and an explicit low-bit risk signal, enabling consistent budgeted top-

k

selection. Experiments on standard VLM benchmarks show improved robustness over stage-wise baselines under the same low-bit regime, achieving 95.65% average retention while retaining only 30% of visual tokens, compared with about 94.3% retention for representative stage-wise combinations. The code will be released.

Abstract

Visual Token Pruning

Explore similar work

Towards Joint Quantization and Token Pruning of Vision-Language Models

SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models

P4Q: Co-designing Token Pruning and Quantization for Vision-Language Model Acceleration