Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanyang Technological University, Singapore · University of Electronic Science and Technology of China · De Artificial Intelligence Lab · University of Michigan, Ann Arbor
Large vision-language models (LVLMs) achieve strong multimodal understanding, but the hundreds to thousands of visual tokens they process impose substantial computational overhead, motivating training-free visual token pruning. In this work, we conduct two complementary analyses of visual token pruning. First, we measure the feature-space coverage of tokens retained before cross-modal fusion and find that aggressive pruning discards substantial visual information. Second, we track text-to-visual attention across decoder layers and find that the visual tokens considered important change substantially with depth, making one-shot pruning decisions unreliable. Together, these findings show that effective pruning should preserve broad visual coverage before fusion and progressively refine the retained tokens as cross-modal evidence evolves during fusion. We therefore propose STAR-Pro (STage-Wise Adaptive Token Reduction with Progressive Refinement), a training-free two-stage framework. Its Adaptive Stage applies pivoted QR to construct an over-budget feature-coverage candidate pool, while its Progressive Stage uses evolving text-to-visual attention at selected decoder layers to prune a nested survivor set under a target layer-average token budget. Extensive experiments across seven LVLMs spanning multiple architectures and 18 image and video benchmarks demonstrate the effectiveness of STAR-Pro under aggressive pruning. On LLaVA-Video-7B, STAR-Pro reduces visual tokens by 90.5%, retains 92.7% of baseline performance, and achieves a 2.24× measured inference speedup. Code is available at https://github.com/EasonAI-5589/starpro.
Figures & tables
Figure 1: STAR-Pro at ∼90% visual token pruning. Each radar axis is normalised to the strongest plotted method (100); below each panel: STAR-Pro vs. the strongest baseline in the corresponding main table, both as percentages of baseline performance.
Figure 2: Empirical Study. (a) Empirical Study on Adaptive Stage. We measure the visual feature retention ratio η=∥PSX∥F2/∥X∥F2 of the K retained tokens. Across all budgets, STAR-Pro preserves more visual information than CDPruner and DivPrune, with a larger advantage under more aggressive pruning. The PCA insets visualize the first six selections of each method for one image: STAR-Pro spans the largest region of the feature space and reaches the most distinct feature clusters. (b) Empirical Study on Progressive Stage. At each decoder layer ℓ , we measure the IoU continuity of the top- K attended set Sℓ with layers a distance d away, defined as 21[IoU(Sℓ,Sℓ−d)+IoU(Sℓ,Sℓ+d)] . Across gaps d=1 – 5 , continuity stays well below 1 , confirming visual token importance changes with depth (LLaVA-1.5-7B, K=64 , mean over 500 MME images).
Figure 3: Overview of STAR-Pro. The Adaptive Stage (§ 3.2 ) uses pivoted QR to build a feature-coverage candidate pool. At scheduled pruning layers, the Progressive Stage (§ 3.3 ) uses evolving text-to-visual attention to progressively refine the pool.
Method
LLaVA-1.5-7B
LLaVA-1.5-13B
LLaVA-NeXT-7B
LLaVA-NeXT-13B
576 tokens
576 tokens
2880 tokens
2880 tokens
Compression
77.8%
88.9%
94.4%
77.8%
88.9%
94.4%
77.8%
88.9%
94.4%
77.8%
88.9%
94.4%
FastV (ECCV24)
92.5
75.6
–
94.1
85.0
–
95.4
80.4
–
96.4
86.9
–
PDrop (CVPR25)
94.8
75.2
–
97.8
88.7
–
97.0
83.1
–
98.5
91.5
–
SparseVLM (ICML25)
96.4
87.8
–
98.0
93.1
–
98.9
94.0
–
99.9
96.8
–
VisionZip (CVPR25)
96.7
93.1
87.9
96.3
93.1
87.4
99.4
95.8
90.5
94.8
96.4
92.6
Table 1: Evaluation on LLaVA Series. All numbers report Rel. (%), performance relative to the corresponding baseline. Ties at the displayed one-decimal precision are marked jointly best. “–”: not applicable.
Method
MLVU
LVB
VMME
Acc.
Rel.
Retain 64×32 Tokens ( ↓ 81.1%)
FastV (ECCV24)
58.5
52.4
56.0
55.6
87.7
SparseVLM (ICML25)
60.7
53.7
59.0
57.8
91.1
CDPruner (NeurIPS25)
63.0
56.5
60.5
60.0
94.6
DivPrune (CVPR25)
61.5
56.4
59.3
59.1
93.1
VScan (TMLR26)
62.4
57.8
60.9
60.4
95.2
Table 2: Evaluation on Video. MLVU/LVB/VMME report m-avg/val/w/o-subtitle. Acc. is the equal average of the three reported benchmark scores; Rel. (%) normalizes Acc. by the unpruned baseline’s three-benchmark mean (63.4).
Method
AI2D
POPE
HallBench
MME
MMB EN
MMB CN
MMStar
SQA
Acc.
Rel.
Qwen3-VL-8B-Instruct (Total: 1,296 Tokens)
Baseline
84.1
89.7
56.2
2406
86.3
86.1
67.1
94.6
85.6
100
Layer-Average Budget T=256 ( ↓ 80.2%)
FastV (ECCV24)
69.3
81.7
43.0
1913
77.3
76.7
51.1
82.0
72.1
84.3
CDPruner (NeurIPS25)
75.1
87.6
41.9
2034
79.7
78.7
50.9
83.7
74.9
87.6
DivPrune (CVPR25)
80.0
89.4
46.9
2236
83.6
82.3
58.3
88.0
80.0
93.6
Table 3: Evaluation on Advanced Architectures. Budgets are layer-average visual-token counts. Acc. denotes the average across eight benchmarks (MME divided by 20); Rel. (%) is performance relative to the corresponding baseline.
Timed component
Latency (ms)
Δ
Full
STAR-Pro
ms
%
Vision encoder + projector
662.8
659.1
3.7
0.37
Adaptive Stage: pivoted QR
0.0
40.4
−40.4
−3.99
Auxiliary selection ops.
0.0
0.2
−0.2
−0.02
Decoder prefill
968.8
95.9
872.9
86.15
Residual runtime
188.9
11.9
177.0
17.47
Table 4: Adaptive-Stage overhead on LLaVA-Video-7B at T=16 . End-to-end speedup: 2.24 × .
Figure 8
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Decoder layers
Nominal budget T
Adaptive pool 2T
Progressive checkpoints
LLaVA-1.5-7B
32
128 / 64 / 32
256 / 128 / 64
ℓ12→74/37/17,ℓ20→36/18/10
LLaVA-1.5-13B
40
128 / 64 / 32
256 / 128 / 64
ℓ15→64/32/16,ℓ30→32/16/8
LLaVA-NeXT-7B
32
640 / 320 / 160
1,280 / 640 / 320
ℓ12→367/182/91,ℓ20→182/92/46
LLaVA-NeXT-13B
40
640 / 320 / 160
1,280 / 640 / 320
ℓ15→320/160/80,ℓ30→160/80/40
LLaVA-Video-7B
28
32 / 16
64 / 32
ℓ12→11/5,ℓ21→4/3
Qwen3-VL-8B-Instruct
36
256 / 128
512 / 256
ℓ14→128/64,ℓ24→64/32
Appendix
Table 5: Architecture-specific pruning checkpoints used by STAR-Pro. Decoder layers are named one-based: an entry ℓ→m prunes after layer ℓ , so the following layer receives m visual tokens. In the zero-based implementation this is the boundary before decoder index ℓ . Each schedule starts from the 2T candidate pool produced by the Adaptive Stage. Slash-separated survivor counts correspond in order to the slash-separated budgets in the same row. For LLaVA-Video, token counts are reported per frame; integer allocation gives observed layer averages of 31.96 and 16.07 for the nominal 32- and 16-token budgets.
Method
SEED
GQA
SQA IMG
VQA Text
POPE
MME
MMB EN
MMB CN
AI2D
MMMU
Acc.
Rel.
Upper Bound: All 576 tokens (100%)
Baseline
66
61.9
69.5
58.2
85.9
1508.3
64.7
58.1
55.5
35
63.02
100.0
Retain 128 Tokens ( ↓ 77.8%)
FastV (ECCV24)
59.66
54
69.2
56.4
68.2
1376.5
63.0
55.9
53.6
34.3
58.31
92.5
PDrop (CVPR25)
59.74
57.1
70.1
56.7
77.5
1434.1
62.3
55.3
52.7
34.1
59.72
94.8
SparseVLM (ICML25)
61.72
57.3
69.0
56.3
83.1
1401.3
62.6
56.9
54.7
35.9
60.76
96.4
Appendix
Table 6: Performance comparison on LLaVA-1.5-7B. Acc. denotes the average across 10 benchmarks, with MME divided by 20. Rel. (%) is performance relative to the corresponding baseline.
Method
SEED
GQA
SQA IMG
VQA Text
POPE
MME
MMB EN
MMB CN
AI2D
MMMU
Acc.
Rel.
Upper Bound: All 576 tokens (100%)
Baseline
68.22
63.3
72.8
61.2
86
1531.2
68.5
63.5
60.8
36.4
65.73
100.0
Retain 128 Tokens ( ↓ 77.8%)
FastV (ECCV24)
55.5
58.3
74.2
58.6
75.5
1460.6
66.1
62.3
58.1
36.9
61.85
94.1
PDrop (CVPR25)
65.6
61
73.3
60.2
83.4
1494.3
66.7
62.7
59.1
36.1
64.28
97.8
SparseVLM (ICML25)
65.3
59.6
74.3
59.3
85
1487.9
68.4
62.6
58.1
37.4
64.44
98.0
Appendix
Table 7: Performance comparison on LLaVA-1.5-13B. Acc. denotes the average across 10 benchmarks, with MME divided by 20. Rel. (%) is performance relative to the corresponding baseline.
Method
SEED
GQA
SQA IMG
VQA Text
POPE
MME
MMB EN
MMB CN
AI2D
MMMU
Acc.
Rel.
Upper Bound: All 2880 tokens (100%)
Baseline
69.66
62.5
67.5
60.3
86.8
1511.8
65.8
57.3
64.7
35.2
64.54
100.0
Retain 640 Tokens ( ↓ 77.8%)
FastV (ECCV24)
64.25
58.9
67.4
58.1
79.5
1412.6
63.1
53.5
65.1
35
61.55
95.4
PDrop (CVPR25)
64.68
60
66.7
57.8
83.8
1475.9
64.1
55.2
64.6
35.1
62.58
97.0
SparseVLM (ICML25)
67.17
61.2
67.6
59.7
85.3
1456.8
65.9
58.6
64.7
35.1
63.81
98.9
Appendix
Table 8: Performance comparison on LLaVA-NeXT-7B. Acc. denotes the average across 10 benchmarks, with MME divided by 20. Rel. (%) is performance relative to the corresponding baseline.
Method
SEED
GQA
SQA IMG
VQA Text
POPE
MME
MMB EN
MMB CN
AI2D
MMMU
Acc.
Rel.
Upper Bound: All 2880 tokens (100%)
Baseline
71.58
64.4
73.1
63.2
85.3
1539.5
68.5
61.2
70.1
35.7
67.0
100.0
Retain 640 Tokens ( ↓ 77.8%)
FastV (ECCV24)
67.4
60.9
71.7
60.7
80.2
1516.7
65.5
59.9
67.7
35.8
64.6
96.4
PDrop (CVPR25)
68.8
62.8
71.7
62.1
84.4
1559.1
66.6
60.8
69
35.9
66.0
98.5
SparseVLM (ICML25)
69.7
62.7
72.5
62.8
85.6
1562.7
68.5
64.0
68.1
37.6
67.0
99.9
Appendix
Table 9: Performance comparison on LLaVA-NeXT-13B. Acc. denotes the average across 10 benchmarks, with MME divided by 20. Rel. (%) is performance relative to the corresponding baseline.
Method
MLVU
LongVideoBench
Video-MME
Acc.
Rel.
Metric
m-avg
val
perception
relation
w/o sub
short
medium
long
Baseline ( 64×169 tokens, 100%)
Baseline
67.7
59.0
65.0
53.8
63.6
76.6
61.2
53.1
63.4
100.0
Retain 64×32 Tokens ( ↓ 81.1%)
FastV (ECCV24)
58.5
52.4
57.0
48.5
56.0
63.8
55.9
48.4
55.6
87.7
SparseVLM (ICML25)
60.7
53.7
58.1
49.9
59.0
69.8
56.9
50.3
57.8
91.1
Appendix
Table 10: Full LLaVA-Video-7B results with 64 frames per video. Acc. uses the same three benchmark-level scores as the main paper (MLVU m-avg, LongVideoBench val, and Video-MME w/o subtitle); Rel. (%) normalizes Acc. by the corresponding unpruned mean (63.4). The remaining columns provide the associated sub-metrics.
Figure 6: Pruning-schedule robustness on LLaVA-1.5-7B at T=64 . (a) Mean-4 over the 100 measured pairs, shown directly (left) and with the CDPruner Mean-4 reference plane at 68.1 (right); the star marks the deployed (12,20) schedule. (b) Cross-sections of the same surface, with regions at or above the reference shown in green and regions below it shown in red.
Figure 7: Robustness and stability to the candidate-pool multiplier α . Each panel plots STAR-Pro’s aggregate Rel. (%) as α sweeps {1,1.5,2,2.5,3} , for one model (rows) at one layer-averaged token budget (columns). The star marks α=2 , the deployed configuration used throughout the paper. All plotted points—including the α=2 star—are the per-model candidate-multiplier sweep values reported in Tables 13 , 14 , 15 and 16 , where the α=2 row is highlighted in each. The dashed line is the best competing method at the same budget in the main comparison tables. Rel. is the 10-benchmark Acc. (MME/20) normalized by each model’s unpruned baseline; exact values are listed in those tables.
(b) Number of Progressive Stage pruning layers P ( α=2 )
P=0
Adaptive only
87.21
59.68
56.79
1422.5
86.12
58.13
56.03
1384.7
83.94
55.72
54.28
1353.1
Appendix
Table 11: Matched-compute controls on LLaVA-1.5-7B. Every arm has the nominal layer-average visual-token budget T . Block (a) reports Random and Stride candidate pools under a fixed diagnostic Progressive schedule. Block (b) varies the number of Progressive-Stage pruning layers; P=0 denotes Adaptive-only selection, and the listed P∈{3,4} placements are diagnostic equal-depth schedules. The Ours row uses query-agnostic pivoted-QR coverage with the deployed P=2 , ℓ=(12,20) schedule and reports the corresponding main-paper scores. MME is unnormalised and should be read alongside POPE, GQA, and VQA Text .
Table 12: Robustness to the per-layer token allocation. For each model the pruning layers and α=2 are fixed at their deployed values; only the per-layer retained counts change. Depl. is the deployed schedule and Token-var. a moderately more front-heavy re-allocation ( K1:K2≈2.5:1 ) with the same layers and the same exact layer-average budget. The Config column gives 2T→K1→K2 . Acc. is the average over the 10 benchmarks with MME divided by 20; Rel. (%) is Acc. relative to the corresponding unpruned baseline.
Config.
Tavg
SEED
GQA
SQA IMG
VQA Text
POPE
MME
MMB EN
MMB CN
AI2D
MMMU
Acc.
Rel.
Layer-average budget T=128 ( ↓ 77.8%)
α=1
128
62.99
59.68
68.67
56.79
87.21
1388.6
61.31
56.19
54.05
35.44
61.18
97.1
α=1.5
128
63.87
60.10
68.62
57.57
87.47
1421.7
62.16
57.39
54.66
36.00
61.89
98.2
STAR-Pro ( α=2 )
128
64.4
60.8
68.9
57.2
87.3
1456.3
62.4
57.7
55.0
36.7
62.32
98.9
α=2.5
128
65.07
58.31
68.47
54.95
87.06
1480.1
63.52
58.33
54.79
36.33
62.08
98.5
α=3
144
64.10
50.99
68.52
50.61
80.00
1469.4
62.84
57.13
55.21
36.00
59.89
95.0
Appendix
Table 13: Candidate-multiplier sweep on LLaVA-1.5-7B with the two Progressive-Stage checkpoints fixed at {12,20} . The highlighted STAR-Pro ( α=2 ) row is the deployed configuration used throughout the paper; its scores are taken from the main comparison table ( Table 6 ). Tavg is the realized layer-average visual-token count (it exceeds T only where the strict average constraint is infeasible, at α=3 ). Acc. is the 10-benchmark average with MME/20, and Rel. (%) normalizes Acc. by the full-token baseline.
Config.
Tavg
SEED
GQA
SQA IMG
VQA Text
POPE
MME
MMB EN
MMB CN
AI2D
MMMU
Acc.
Rel.
Layer-average budget T=128 ( ↓ 77.8%)
α=1
128
64.34
59.21
73.08
58.65
86.83
1761.8
66.33
62.03
58.00
36.44
65.30
99.3
α=1.5
128
65.50
59.93
73.08
59.53
87.01
1493.7
66.24
62.11
57.71
36.11
64.19
97.7
STAR-Pro ( α=2 )
128
66.1
60.2
72.7
59.5
87.2
1492.7
67.0
63.1
57.9
35.4
64.37
97.9
α=2.5
128
66.97
60.88
73.23
59.87
86.94
1493.4
67.35
62.29
59.49
36.33
64.80
98.6
α=3
128
67.22
58.73
73.33
57.48
86.30
1488.1
67.09
62.29
59.29
36.33
64.25
97.7
Appendix
Table 14: Candidate-multiplier sweep on LLaVA-1.5-13B with the two Progressive-Stage checkpoints fixed at {15,30} . The highlighted STAR-Pro ( α=2 ) row is the deployed configuration; its scores are taken from the main comparison table ( Table 7 ). Tavg is the realized layer-average visual-token count. Acc. is the 10-benchmark average with MME/20, and Rel. (%) normalizes Acc. by the full-token baseline.
Config.
Tavg
SEED
GQA
SQA IMG
VQA Text
POPE
MME
MMB EN
MMB CN
AI2D
MMMU
Acc.
Rel.
Layer-average budget T=640 ( ↓ 77.8%)
α=1
640
67.66
61.97
67.97
58.19
87.31
1472.0
65.48
57.90
65.22
35.33
64.06
99.3
α=1.5
640
68.53
62.62
68.37
59.51
87.56
1491.5
65.31
57.99
65.09
34.89
64.44
99.9
STAR-Pro ( α=2 )
640
68.95
62.4
68.8
59.1
87.6
1479.1
66.0
58.0
65.6
32.2
64.26
99.6
α=2.5
640
69.06
59.30
69.01
53.40
87.95
1488.2
65.65
58.08
64.93
34.33
63.61
98.6
α=3
720
67.27
49.05
68.72
43.81
81.91
1471.8
63.69
56.01
64.35
34.33
60.27
93.4
Appendix
Table 15: Candidate-multiplier sweep on LLaVA-NeXT-7B with the two Progressive-Stage checkpoints fixed at {12,20} . The highlighted STAR-Pro ( α=2 ) row is the deployed configuration; its scores are taken from the main comparison table ( Table 8 ). Tavg is the realized layer-average visual-token count (it exceeds T only where the strict average constraint is infeasible, at α=3 ). Acc. is the 10-benchmark average with MME/20, and Rel. (%) normalizes Acc. by the full-token baseline.
Config.
Tavg
SEED
GQA
SQA IMG
VQA Text
POPE
MME
MMB EN
MMB CN
AI2D
MMMU
Acc.
Rel.
Layer-average budget T=640 ( ↓ 77.8%)
α=1
640
69.22
63.50
72.19
60.79
86.61
1529.1
67.26
63.75
68.17
38.00
66.59
99.4
α=1.5
640
70.36
64.30
72.88
61.75
86.78
1528.3
68.11
63.14
68.91
38.00
67.06
100.1
STAR-Pro ( α=2 )
640
71.33
64.2
72.8
62.2
87.4
1545.2
68.5
63.1
69.8
35.8
67.2
100.3
α=2.5
640
71.38
63.72
73.03
62.30
87.00
1548.0
67.60
62.63
69.46
36.89
67.14
100.2
α=3
640
71.30
60.51
72.88
55.15
86.42
1554.3
67.77
62.46
69.62
35.44
65.93
98.4
Appendix
Table 16: Candidate-multiplier sweep on LLaVA-NeXT-13B with the two Progressive-Stage checkpoints fixed at {15,30} . The highlighted STAR-Pro ( α=2 ) row is the deployed configuration; its scores are taken from the main comparison table ( Table 9 ). Tavg is the realized layer-average visual-token count. Acc. is the 10-benchmark average with MME/20, and Rel. (%) normalizes Acc. by the full-token baseline.
Large Vision-Language Models (LVLMs) incur high computational costs from redundant visual tokens. Although training-free attention-based multi-layer pruning in the vision encoder stage has been explored as an effective strategy, we find that pruning in shallow layers consistently degrades performance. In this paper, we aim to understand this problem and seek a solution. By analyzing attention patterns across network depth, we find that shallow layers primarily function as edge detectors with chaotic attention maps, while deeper layers transition through local subject recognition and unstable semantic aggregation. To address the misalignment between pruning strategies and network stages, we propose STD, a hierarchical token pruning framework that adapts token selection mechanisms to the functional role of each network stage. STD employs High-Frequency Spectral Analysis in shallow layers to deterministically preserve structural edges, uses Gaussian-Smoothed Attention in intermediate layers to maintain spatial coherence, and introduces a Stability-Adaptive Trigger in deep layers to execute pruning only during semantically stable phases. Extensive experiments show that STD outperforms state-of-the-art pruning methods by 1.1% on LLaVA-1.5-7B with 88.9% token reduction, while also being plug-and-play and highly effective when combined with other methods, and by 2.1% on LLaVA-NeXT-7B with 94.4% reduction, delivering a 3.9x speed-up in the prefilling stage. Our code will be released at https://github.com/Twilight03/STD.
Shuo Zhang, Jintao Tong, Yixiong Zou +2
School of Computer Science and Technology, Huazhong University of Science and Technology
Large Vision-Language Models (LVLMs) face significant computational inefficiencies caused by the large number of visual tokens. Existing visual token pruning methods mainly focus on either retaining individually important tokens or selecting mutually diverse ones. In this work, we revisit visual token pruning from a coverage perspective and formulate it as a biased attention coverage maximization problem. The key idea is to select a compact token subset whose encoder-side outgoing attention can jointly cover the image while assigning higher coverage priority to more informative regions. From this perspective, we propose ACPruner, a training-free visual token pruning framework for efficient LVLM inference. ACPruner first estimates token importance by combining intra-modal saliency and inter-modal relevance, then derives token-wise coverage from attention patterns within the vision encoder, and finally performs greedy selection to maximize the proposed coverage objective. Extensive experiments across multiple LVLM backbones, including LLaVA-1.5-7B/13B, LLaVA-NeXT-7B/13B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B, show that ACPruner consistently achieves strong performance retention while delivering substantial end-to-end inference speedups.
Xu Li, Yuxuan Liang, Yi Zheng +6
College of Computer Science and Artificial Intelligence Fudan University, Shanghai, China
Visual token pruning is a promising approach to reducing the inference cost of large vision-language models (LVLMs), yet aggressive token reduction often causes substantial performance degradation. We identify three key factors behind this degradation: text-guided selection bias, information loss from discarded tokens, and positional distortion caused by sequence compaction. Based on these observations, we propose VPRune, a training-free pre-LLM pruning framework consisting of visual-only diversity selection, similarity-guided token recycling, and position-preserving restoration. Experiments on FastVLM-1.5B across multiple vision-language benchmarks demonstrate that VPRune achieves a favorable accuracy--compression trade-off, with particularly pronounced advantages under aggressive compression. Furthermore, evaluations on edge-device show that VPRune effectively reduces end-to-end inference latency while maintaining superior task performance, demonstrating its practicality for resource-constrained LVLM deployment.
Guangchuan Lv, Dianxing Shi, Dingjie Fu
Northeastern University · Beihang University · Huazhong University of Science and Technology