Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decoding

Jiwan Kim; Kibum Kim; Wonjoong Kim; Byung-Kwan Lee; Chanyoung Park

cs.CVApr 14, 2026

Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decoding

Authors: Jiwan Kim, Kibum Kim, Wonjoong Kim, Byung-Kwan Lee, Chanyoung Park

Organizations: Korea Advanced Institute of Science and Technology (KAIST) · NVIDIA

Abstract

Recently, visual token pruning has been studied to handle the vast number of visual tokens in Multimodal Large Language Models. However, we observe that while existing pruning methods perform reliably on simple visual understanding, they struggle to effectively generalize to complex visual reasoning tasks, a critical gap underexplored in previous studies. Through a systematic analysis, we identify Relevant Visual Information Shift (RVIS) during decoding as the primary failure driver. To address this, we propose Decoding-stage Shift-aware Token Pruning (DSTP), a training-free add-on framework that enables existing pruning methods to align visual tokens with shifting reasoning requirements during the decoding stage. Extensive experiments demonstrate that DSTP significantly mitigates performance degradation of pruning methods in complex reasoning tasks, while consistently yielding performance gains even across visual understanding benchmarks. Furthermore, DSTP demonstrates effectiveness across diverse state-of-the-art architectures, highlighting its generalizability and efficiency with minimal computational overhead.

Figures & tables

Figure 1: (a) Performance retention rates of various token pruning methods on Qwen3-VL [ 4 ] and InternVL3.5 [ 42 ] across VQA and VMR. (b)–(f) Attention heatmaps for a MathVerse [ 52 ] sample on Qwen3-VL relative to the text token whose attention shifts drastically, reflecting the reasoning context. 3 3 3 Regarding how the specific text token for the attention heatmap is selected, please refer to Sec. 3 . Full sentences are provided to offer complete context for each reasoning step.

Figure 2: (a) Cosine similarity of visual attention distributions between the prefill stage ( $l=0$ ) and each decoding step. (b) Proportion of samples maintaining attention similarity above thresholds throughout the entire decoding process.

Figure 3: Average number of RVIS occurrences across various answer lengths.

Figure 4: Distribution of RVIS occurrences for VQA and VMR. $N$ indicates the sample count for each bin.

Figure 5: Success rate of FastV [ 9 ] across different RVIS frequencies.

Figure 6: Overall framework of DSTP . (a) Prefill-stage protocol of base pruning methods. (b) RISD and CPTS modules at Decoding-stage. RISD monitors attention similarity at each decoding step, invoking CPTS for context-preserving token swapping when RVIS is detected. (c) Overall flow of DSTP throughout the entire decoding process.

Qwen3-VL-4B [ 4 ]
Method	MathVerse	WeMath	DynaMath	LogicVista	MMMU-Pro	Acc. (%)
Vanilla (Full Tokens)	61.29	48.29	66.48	49.22	37.63	100%
Retain 33.3% Tokens ( $\downarrow$ 66.7%)
FastV (ECCV’24)	32.23 (52.6%)	25.90 (53.6%)	38.76 (58.3%)	30.64 (62.3%)	19.41 (51.6%)	55.68%
w/ DSTP	52.54 (85.7%)	42.19 (87.4%)	51.00 (76.7%)	41.16 (83.6%)	30.52 (81.1%)	82.90%
DivPrune (CVPR’25)	33.90 (55.3%)	29.32 (60.7%)	41.94 (63.1%)	30.64 (62.3%)	13.08 (34.8%)	55.24%

Table 1: Visual Reasoning Performance of Qwen3-VL-4B and InternVL3.5-8B under different token retention ratios with various pruning methods and DSTP . Values in parentheses are the proportion relative to the upper bound.

Method	SQA	$\text{VQA}^{\text{T}}$	GQA	Acc.
Vanilla	93.42	81.57	61.82	100%
FastV	87.98	74.82	60.04	94.3%
w/ DSTP	91.84	77.95	60.93	97.5%
DivPrune	86.12	71.36	58.07	91.2%
w/ DSTP	91.36	74.11	60.42	95.5%
VisionZip	90.41	77.10	60.68	96.5%

Table 2: Visual Understanding Performance of Qwen3-VL-4B with 33.3% retention ratio.

Row	Detect	Swap		Qwen3 VL		InternVL3.5
Row	Detect	Strategy	Ratio $\downarrow$	MV	MP	MV	MP
(a)	Full visual tokens			61.29	37.63	42.00	39.42
(b)	FastV (33.3%)			32.23	19.41	21.57	18.96
(c)	Random	CPTS	38.1%	37.69	22.36	31.09	22.18
(d)	Avg	CPTS	38.1%	51.51	27.68	39.08	32.77
(e)	RISD	Full	100%	53.04	29.55	40.96	34.16

Table 3: Ablation for RISD and CPTS.

Figure 7: Success rate of FastV and DSTP across different RVIS frequencies.

Figure 8: Performance comparison of FastV versus DSTP across various computational cost on MathVerse [ 52 ]

Figure 9: Visualization of visual token selection. (a) and (b) show FastV results at 33.3% and 66.6% token retention ratios. (c)-(e) illustrate DSTP results at a 33.3% token retention ratio. White and black tokens represent retained and pruned tokens respectively. Red tokens denote those retained by DSTP but pruned even in the 66.6% vanilla FastV.

Figure 10: Hyper-parameter experiments on Generation Length $L$ and threshold $\tau$ .

Method	Total Latency $\downarrow$	GPU Mem. $\downarrow$	Accuracy $\uparrow$	Lat./Ex. $\downarrow$	TPS $\uparrow$	Avg Tokens $\downarrow$
Qwen3-VL-4B
Vanilla	27:51:28	18.6G	61.29	127.27s	19.30	2419.4
FastV	26:07:27	14.6G	32.23	119.35s	24.99	2931.7
FastV + DSTP	22:26:49	15.1G	52.54	102.55s	24.11	2412.7
InternVL-3.5-8B
Vanilla	2:34:53	28.9G	42.00	11.79s	31.55	377.3

Table 4: Efficiency analysis of DSTP on FastV across Qwen3-VL and InternVL3.5.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure I : Attention heatmaps for ScienceQA (VQA) samples. Decoding steps progress from left to right.

Figure II : Attention heatmaps for MathVerse (VMR) samples. Decoding steps progress from left to right.

Figure III : Temporal occurrence distribution of RVIS.

Figure IV : Temporal occurrence distribution of RVIS.

Method	Qwen3-VL
Method	MathVerse	MMMU-Pro
Vanilla	61.29	37.63
FastV (33.3%)	32.23	19.41
DSTP w/o Reversion	44.04	22.03
DSTP	52.54	30.52

Appendix

Table I : Ablation for Reversion Strategy on FastV.

Thinking MLLMs			Large-scale MLLMs
Method	MathVerse	MMMU-Pro	Method	MathVerse	MMMU-Pro
Qwen3-VL-4B-Thinking			Qwen3-VL-8B
Vanilla (Full Tokens)	73.01	40.51	Vanilla (Full Tokens)	62.81	42.31
FastV (33.3%)	40.28	29.77	FastV (33.3%)	37.64	18.64
w/ DSTP	58.49	36.52	w/ DSTP	51.16	33.31
InternVL3.5-8B-Thinking			InternVL3.5-14B

Appendix

Table II : Generalization performance of DSTP on Thinking-based MLLMs and Large-scale MLLMs using MathVerse and MMMU-Pro benchmarks. All experiments are conducted with a token retention ratio of 33.3%.

Method	Qwen3-VL-4B		InternVL3.5-8B
Method	G1	G2	G1	G2
FastV	60.13	42.94	83.61	52.87
DivPrune	81.26	41.14	54.80	35.53
VisionZip	52.68	26.04	60.16	41.73
Total Avg.	64.69	36.71	66.19	43.38

Appendix

Table III : Average Performance Improvement (%) of Proposed Method over Baselines (Retain 33.3% Tokens).

Upper Bound, Full Visual Tokens (100%)
Model	Qwen3-VL-4B [ 4 ]				InternVL3.5-8B [ 42 ]
Method	ScienceQA	TextVQA	GQA	Acc. (%)	ScienceQA	TextVQA	GQA	Acc. (%)
Vanilla	93.42	81.57	61.82	100%	96.70	77.59	61.24	100%
Retain 33.3% Tokens ( $\downarrow$ 66.7%)
FastV (ECCV’24)	87.98 (94.2%)	74.82 (91.7%)	60.04 (97.1%)	94.3%	81.60 (84.4%)	64.92 (83.7%)	57.08 (93.2%)	87.1%
w/ DSTP	91.84 (98.3%)	77.95 (95.6%)	60.93 (98.6%)	97.5%	87.65 (90.6%)	67.74 (87.3%)	58.71 (95.9%)	91.3%

Appendix

Table IV : Visual Understanding Performance of Qwen3-VL-4B and InternVL3.5-8B under different token retention ratios with various VTP methods and our method. Values in parentheses are the proportion relative to the upper bound.

Ratio	Method	Prefill	Decoding	Total
Ratio	Method	(TFLOPs)	(TFLOPs)	(TFLOPs)
100%	Vanilla	1.637	2.158	3.795
33.3%	FastV	0.546	2.130	2.676
33.3%	w/ DSTP	0.546	2.137	2.683
22.2%	FastV	0.363	2.126	2.489
22.2%	w/ DSTP	0.363	2.133	2.496

Appendix

Table V : TFLOPs comparison on Qwen3-VL-4B.

Figure V : Performance of DSTP on MathVerse across various decoding token budgets, with FastV as the pruning baseline.

Figure VI : Example 1: Robustness in numeric information extraction. Comparison highlighting DSTP ’s ability to maintain high fidelity in reading numerical figures where baseline models like FastV exhibit semantic drift.

Figure VII : Example 2: Structural parsing of geometric primitives. DSTP demonstrates a deeper understanding of spatial layouts, correctly identifying structure, unlike FastV which misinterprets the geometric arrangement.

Figure VIII : Example 3: Discerning selection states. Comparison of model performance on multiple-choice tasks. DSTP correctly recognizes the intended choice, while FastV fails to correlate the visual marker with the corresponding text.

Figure IX : Example 4: Another discerning selection states.

Figure X : Example 5: Visual content recognition under token pruning. DSTP recognizes the painting itself, while FastV misunderstands the painting as related tokens are discarded.

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 276 papers

When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Model
Sep 28, 2026Minchan Kang, Kyeonghye Park, Seoyoung Cho +2Efficient VLM Inference Visual Token Pruning
Korea Advanced Institute of Science and Technology (KAIST) · Hanbat National University
TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference
Jun 25, 2026Tinghao Wang, Yichen Guo, Rui Huang +9Efficient VLM Inference Multimodal Large Language Models
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · University of Electronic Science and Technology of China · Nanyang Technological University +1
Decoupled Similarity for Task-Aware Token Pruning in Large Vision-Language Models
Apr 13, 2026Kexin Ma, Jing Xiao, Chaofeng Chen +4Visual Token Pruning Efficient VLM Inference
School of Artificial Intelligence, Wuhan University, Wuhan, China · Department of Computer Science, University of Exeter, Exeter, United Kingdom · Chinese Academy of Sciences, Beijing, China; Wuhan AI Research, Wuhan, China +1

Sep 28, 2026J/K move · Enter open · S save

When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Model

Minchan Kang, Kyeonghye Park, Seoyoung Cho, Daeshik Kim+1

Korea Advanced Institute of Science and Technology (KAIST) · Hanbat National University

Efficient VLM Inference Visual Token Pruning Visual Token Compression

Open paper AskPDF arXiv

Visual token pruning has been widely studied as a practical approach to reducing the computational cost of large vision-language models. However, it struggles to preserve essential visual information, which can lead to substantial performance degradation. In particular, image-based token selection can overlook task-relevant details, while text-guided token selection may fail to capture the text--visual relationships needed for complex reasoning. We find that applying textual guidance too early can limit its ability to identify answer-relevant visual regions, whereas text-to-visual attention becomes more informative at intermediate decoder depths. This finding motivates our training-free method, which separates early vision-guided pruning from deferred text-guided reselection. We first prune visual tokens using vision-encoder attention, retain additional candidates until the decoder midpoint, and then use text-to-visual attention to determine the final visual-token set. Across eight benchmarks and three models, our method outperforms the best-performing baselines by an average of 11.10 and 16.84 percentage points in performance recovery at 80% and 90% pruning, respectively, with comparable or lower LLM-prefill latency than most baselines. The source code is publicly available at https://github.com/kmc3661/DeFT

Abstract

Visual Token Pruning

Explore similar work

When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Model

TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference

Decoupled Similarity for Task-Aware Token Pruning in Large Vision-Language Models