HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit

Hao Wu; Yingqi Fan; Jinyang Dai; Junlong Tong; Yunpu Ma; Xiaoyu Shen

cs.CVFeb 27, 2026

HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit

Authors: Hao Wu, Yingqi Fan, Jinyang Dai, Junlong Tong, Yunpu Ma, Xiaoyu Shen

Organizations: Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · University of Science and Technology of China · Shanghai Jiao Tong University · Munich Center for Machine Learning, LMU Munich

Abstract

The quadratic computational cost of processing vision tokens in Multimodal Large Language Models (MLLMs) hinders their widespread adoption. While progressive vision token pruning offers a promising solution, current methods misinterpret shallow layer functions and use rigid schedules, which fail to unlock the full efficiency potential. To address these issues, we propose HiDrop, a framework that aligns token pruning with the true hierarchical function of MLLM layers. HiDrop features two key innovations: (1) Late Injection, which bypasses passive shallow layers to introduce visual tokens exactly where active fusion begins; and (2) Concave Pyramid Pruning with an Early Exit mechanism to dynamically adjust pruning rates across middle and deep layers. This process is optimized via an inter-layer similarity measure and a differentiable top-k operator. To ensure practical efficiency, HiDrop further incorporates persistent positional encoding, FlashAttention-compatible token selection, and parallel decoupling of vision computation to eliminate hidden overhead associated with dynamic token reduction. Extensive experiments show that HiDrop compresses about 90% visual tokens while matching the original performance and accelerating training by 1.72 times. Our work not only sets a new state-of-the-art for efficient MLLM training and inference but also provides valuable insights into the hierarchical nature of multimodal fusion. The code is released at https://github.com/EIT-NLP/HiDrop.

Figures & tables

Figure 1: Comparison of progressive vision token pruning methods. (a) FastV conducts single-stage pruning at an early layer. (b) TwigVLM performs early pruning and removes remaining vision tokens at deeper layers. (c) PDrop applies progressive pruning with uniform ratios and intervals. (d) HiDrop introduces vision tokens only at the end of shallow layers, prunes them in a non-uniform progressive manner in middle layers, and removes remaining vision tokens before deep layers. (e) HiDrop prunes vision tokens by about $4.8\times$ more aggressively than state-of-the-art progressive pruning method with negligible performance drop.

Figure 2: Layer-wise representational dynamics , with the left panel showing intra-modal refinement, and the right panel highlighting cross-modal interaction intensity.

Figure 3: Left: Vision token reduction curves under different $\bm{p}$ values, where lower $\bm{p}$ enforces stronger pruning. Right: Model performance remains stable even under high compression rates, demonstrating robustness of our pruning strategy.

Figure 4: Early vision exit analysis under different masking ratios.

Figure 5: Overview of HiDrop . (a) Framework illustration, shallow layers focus on vision-independent reasoning, middle layers progressively prune redundant tokens through differentiable top- $k$ selection, and deep layers enable early vision exit. (b) Comparison between hard top- $k$ and differentiable top- $k$ , which achieves adaptive selection and better information preservation.

Figure 6: ILVAS curves for different window sizes, extended results in Appendix. G.5 .

Method	MME P	MMB	MMB CN	GQA	VQA v2	SQA I	VizWiz	TextVQA	POPE	SEED I	MMStar	Avg(%)
Upper Bound, 576 Tokens (100%)
LLaVA-1.5-7B	1510.7	64.3	58.3	62.0	78.5	66.8	50.0	58.2	85.9	66.1	-	-
LLaVA-1.5-7B ∗	1506.5	64.7	58.1	61.9	78.5	69.5	50.1	58.2	86.8	66.2	33.7	100.0
Retain 80 Tokens in Average ( $\downarrow$ 86.1%)
FastV	1214.4	57.3	47.8	51.3	66.6	68.8	51.3	52.1	73.7	52.9	31.1	87.9
PDrop †	1133.1	53.6	41.1	50.8	67.2	69.1	46.9	51.5	72.0	50.8	30.8	84.5

Table 1: Performance comparisons with three pruning ratios on 11 benchmarks. All methods are applied on the same base model LLaVA-1.5-7B . The best result for each benchmark and pruning ratio is bolded . Dashed lines separate training-free (above) and training-based (below) methods within each block. The

*

denotes results reproduced using the official checkpoints;

\dagger

denotes training-based methods evaluated under training-free settings;

\ddagger

denotes training-free methods evaluated under training-based settings.

Model	Method	Avg. Vis. Tokens	Train hours	Infer TFlops	Prefill Latency (ms)	MME P	MMB	GQA	VizWiz	VQA T	Avg(%)
LLaVA-1.5- MobileLLaMA-2.7B	Vanilla	576	108.4	1.52	35.3	1258.2	57.0	59.4	32.6	48.6	100.0
	PDrop	270	50.3	0.70	28.7	1231.1	54.3	57.0	30.9	47.5	96.3
	ours	64 $\downarrow$ 206	45.6	0.17	25.4/25.1/22.0	1206.6	53.1	56.1	30.4	47.2	94.8 $\downarrow$ 1.5
LLaVA-1.5-7B	Vanilla	576	159.3	3.82	63.6	1506.5	64.7	61.9	50.1	58.2	100.0
	PDrop	270	107.3	1.78	43.7	1490.1	63.9	61.7	52.4	57.7	100.2
	ours	64 $\downarrow$ 206	94.4	0.42	32.6/31.8/28.8	1474.3	63.2	60.5	52.6	55.2	98.6 ${\downarrow 1.6}$

Table 2: Efficiency comparison across three LLM backbones within the LLaVA-1.5 framework. Prefill latency (ms) is reported as actual / decoupled visual-KV / fewer dropping stages.

Figure 7: Ablation across visual perception layers comparing Late Injection , Fixed-Entry Span , and Equal-Depth Window , confirming that our setting is the most efficient. The full per-benchmark results are provided in Appendix G.1 , Table 8 .

Model	Train	Top-K	MME P	MMB	GQA	VQA v2	VizWiz	VQA T	MMStar	Avg(%)
LLaVA-1.5-7B	-	-	1506.5	64.7	61.9	78.5	50.1	58.2	33.7	100.0
LLaVA-1.5-7B + Top- $K$	PT+FT	Hard	1436.9	64.2	59.7	76.4	50.1	55.7	33.9	97.7
	PT+FT	Diff.	1484.7	65.5	60.2	76.3	52.7	56.2	34.3	99.7
	FT	Hard	1482.7	65.0	60.3	76.5	46.8	55.9	33.4	97.5
	FT	Diff.	1471.7	65.2	59.9	76.5	47.1	56.2	34.8	98.1

Table 3: Performance comparison of LLaVA variants with Hard vs. Differentiable Top-

K

Operators. PT and FT denote pretrain and finetune, respectively.

Model	MME P	MMB	GQA	VQA v2	VizWiz	VQA T	MMStar	Avg(%)
LLaVA-1.5-7B	1506.5	64.7	61.9	78.5	50.1	58.2	33.7	100.0
Last token (1-rounds)	1424.7	65.3	59.6	75.6	49.0	55.5	33.2	97.1
Last token (n-rounds)	1484.7	65.5	60.2	76.3	52.7	56.2	34.3	99.7
Last token (n-rounds, L2 norm)	1447.0	65.2	59.7	76.3	48.8	55.9	34.0	97.9
All token	1414.8	65.0	59.0	74.8	51.4	56.6	34.3	98.1
All token (L2 norm)	1424.0	65.5	59.9	75.2	53.2	56.6	35.5	99.6

Table 4: Effect of different strategies for estimating vision token saliency.

Model	MME P	MMB	GQA	VQA v2	VizWiz	VQA T	MMStar	Avg(%)
LLaVA-1.5-7B	1506.5	64.7	61.9	78.5	50.1	58.2	33.7	100.0
Persistent PE	1414.4	63.7	61.3	76.6	52.1	55.6	32.0	97.6
Compacted PE	1452.3	64.6	61.1	76.8	48.9	55.1	30.3	96.4
Group PE	1442.2	63.9	60.4	76.2	51.2	55.5	31.1	97.0

Table 5: Effect of position encoding (PE) schemes under shallow–middle–deep compression.

Figure 8: Ablation across filter layers, confirming that our setting is the most efficient. The full per-benchmark results are provided in Appendix G.5 , Table 12 .

Model	Data Scale	MME P	MMB	GQA	VQA v2	VizWiz val	VQA T	MMStar	Avg(%)
LLaVA-1.5-7B	665k	1506.5	64.7	61.9	78.5	54.4	58.2	33.7	100.0
HiDrop	1M	1446.4	63.7	59.8	75.6	56.3	54.4	32.7	97.0
LLaVA-1.5-7B	665k	1526.1	68.7	62.7	79.2	61.2	58.8	38.2	100.0
HiDrop	1M	1453.9	66.2	59.5	76.1	60.7	55.4	36.9	96.3

Table 6: Effect of instruction fine-tuning data scale (HiDrop retains 48 visual tokens in average).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Blocks	Heads	Hidden Dim	FFN Dim
MobileLLaMA 2.7B	32	32	2560	6912
Vicuna-7B-v1.5	32	32	4096	11008
Vicuna-13B-v1.5	40	40	5120	13824

Appendix

Table 7: Detailed settings of LLM backbones.

Panel of Figure 9 — Figure 9: Layer-wise representational dynamics, where each subfigure consists of a left panel showing intra-modal refinement and a right panel highlighting cross-modal interaction intensity

#	Model	MME P	MMB	MMB CN	GQA	VQA v2	SQA I	VizWiz	TextVQA	POPE	SEED I	MMStar	Avg(%)
1	Baseline	1506.5	64.7	58.1	61.9	78.5	69.5	50.1	58.2	86.8	66.2	33.7	100.0
Late Injection
2	I 5	1441.9	66.0	59.9	62.4	78.5	69.5	51.9	55.9	86.5	65.6	34.0	100.1
3	I 6	1442.5	65.8	58.3	62.4	78.5	69.3	51.5	56.5	86.8	66.1	33.3	99.7
4	I 7	1413.8	66.2	58.8	62.2	78.3	69.8	48.6	56.6	86.4	65.3	33.1	99.0
5	I 8	1424.1	65.1	58.2	62.7	78.3	69.1	50.7	57.3	87.1	65.9	31.9	99.1

Appendix

Table 8: Complete per-benchmark results corresponding to Fig. 7 . I i denotes visual injection at layer

i

, E i denotes early visual exit at layer

i

Model	Train	Topk	MME P	MMB	MMB CN	GQA	VQA v2	SQA I	VizWiz	TextVQA	POPE	SEED I	MMStar	Avg(%)
LLaVA-1.5-7B	-	-	1506.5	64.7	58.1	61.9	78.5	69.5	50.1	58.2	86.8	66.2	33.7	100.0
576 $\to$ 64 $\to$ 8 $\to$ 1
LLaVA-1.5-7B + TopK	PT+FT	Hard	1436.9	64.2	57.0	59.7	76.4	70.4	50.1	55.7	86.5	63.1	33.6	98.0
	PT+FT	Diff.	1484.7	65.5	56.3	60.2	76.3	71.5	52.7	56.2	86.2	63.3	34.3	99.3
	FT	Hard	1482.7	65.0	54.9	60.3	76.5	69.9	46.8	55.9	86.0	63.5	33.4	97.5
	FT	Diff.	1471.7	65.2	56.6	59.9	76.5	70.7	47.1	56.2	85.9	63.2	34.8	98.2

Appendix

Table 9: Performance comparison of LLaVA variants with Hard vs. Differentiable top-

k

Operators. PT and FT denote pretrain and finetune, respectively.

Model	MME P	MMB	MMB CN	GQA	VQA v2	SQA I	VizWiz	TextVQA	POPE	SEED I	MMStar	Avg(%)
LLaVA-1.5-7B	1506.5	64.7	58.1	61.9	78.5	69.5	50.1	58.2	86.8	66.2	33.7	100.0
Last token (1-R)	1424.7	65.3	56.9	59.6	75.6	71.0	49.0	55.5	86.2	63.0	33.2	97.7
Last token (n-R)	1484.7	65.5	56.3	60.2	76.3	71.5	52.7	56.2	86.2	63.3	34.3	99.3
Last token (n-R, L2)	1447.0	65.2	56.8	59.7	76.3	70.6	48.8	55.9	86.5	63.5	34.0	98.2
All token	1414.8	65.0	59.2	59.0	74.8	70.3	51.4	56.6	86.4	63.4	34.3	98.6
All token (L2)	1424.0	65.5	58.7	59.9	75.2	68.9	53.2	56.6	87.0	64.7	35.5	99.6

Appendix

Table 10: Different strategies for scoring visual tokens. Last-token variants are computed using repeated attention from the last text token, while all-token variants aggregate attention from all text tokens, with or without L2-norm weighting.

Model	MME P	MMB	MMB CN	GQA	VQA v2	SQA I	VizWiz	TextVQA	POPE	SEED I	MMStar	Avg(%)
LLaVA-1.5-7B	1506.5	64.7	58.1	61.9	78.5	69.5	50.1	58.2	86.8	66.2	33.7	100.0
Persistent PE	1414.4	63.7	56.7	61.3	76.6	67.0	52.1	55.6	86.9	65.2	32.0	97.8
Compacted PE	1452.3	64.6	56.1	61.1	76.8	67.9	48.9	55.1	86.5	64.6	30.3	96.9
Group PE	1442.2	63.9	55.4	60.4	76.2	67.6	51.2	55.5	86.9	63.6	31.1	97.1

Appendix

Table 11: Effect of position encoding (PE) schemes under shallow–middle–deep compression. Persistent PE with fixed RoPE indices performs best overall, while resetting indices (Compacted PE) leads to accuracy degradation.

#	Model	MME P	MMB	MMB CN	GQA	VQA v2	SQA I	VizWiz	TextVQA	POPE	SEED I	MMStar	Avg(%)
1	Baseline	1506.5	64.7	58.1	61.9	78.5	69.5	50.1	58.2	86.8	66.2	33.7	100.0
2	{12,15,18,21}	1431.0	64.6	59.5	61.4	77.4	67.4	46.5	56.2	86.7	65.3	32.0	97.7
3	{10, 12 ,16,18}	1452.9	61.3	55.2	60.7	76.8	67.6	49.3	54.1	86.4	64.5	31.7	96.5
4	{10, 13 ,16,18}	1459.3	64.8	56.8	60.0	76.3	68.1	49.3	55.3	86.6	64.5	29.9	96.9
5	{10, 14 ,16,18}	1469.5	65.0	56.2	60.9	76.7	69.0	50.8	55.1	86.1	64.7	33.1	98.3
6	{10, 15 ,16,18}	1468.9	64.9	57.0	61.6	77.2	68.6	50.0	56.2	86.8	64.5	30.6	97.9

Appendix

Table 12: Per-benchmark results for different filtering layer configurations under the concave pyramid dropping policy. The ILVAS-based set

\{10,14,16,18\}

achieves the best trade-off.

Panel of Figure 10 — Figure 10: ILVAS curve over the middle layers on a model configured with the late injection and early exit. (a)–(f) sweep top- $k\!\in\!\{5,10,20,50,100,200\}$ , and each curve compares observation windows $n=4$ and $n=8$ . Consistent valleys across $K$ indicate layers with strong filtering ability, i.e., candidates for the pruning set $\mathcal{F}$ .

Explore similar work

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs
Jun 30, 2026Yuhao Wang, Mu Qiao, Haiwen Diao +5Efficient VLM Inference Multimodal Large Language Models
School of Future Technology, Dalian University of Technology, Dalian 116024, China · S-Lab, Nanyang Technological University, Singapore 639798 · OPPO Research Institute, Shenzhen 518000, China +1
HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models
Apr 9, 2026Qihui Zhu, Tao Zhang, Yuchen Wang +9Visual Token Pruning Efficient VLM Inference
University of Science and Technology of China · ChangXin Memory Technologies, Inc · Huawei Noah’s Ark Lab
CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models
Apr 14, 2026Yunkai Dang, Yizhu Jiang, Yifan Jiang +4Visual Token Pruning Multimodal Large Language Models
School of Artificial Intelligence Science and Technology, Nanjing University.

Jun 30, 2026J/K move · Enter open · S save

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

Yuhao Wang, Mu Qiao, Haiwen Diao, Yunzhi Zhuge+4

School of Future Technology, Dalian University of Technology, Dalian 116024, China · S-Lab, Nanyang Technological University, Singapore 639798 · OPPO Research Institute, Shenzhen 518000, China+1

Efficient VLM Inference Multimodal Large Language Models Multimodal Token Compression Visual Token Pruning

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attention distributions, giving rise to a phenomenon we term Attention Logit Collapse. To address this issue, we propose ERA, an Entropy-guided visual token pruning framework with Rectified Attention for efficient MLLMs. Specifically, ERA comprises three crucial components: Dual-view Entropy Pruning (DEP), Bias-aware Token Recycling (BTR), and Logit-preserving Attention Rectification (LAR). First, DEP identifies representative anchor tokens by jointly modeling visual diversity and head-wise saliency. BTR then recycles pruned tokens into their corresponding anchors while estimating a cluster-level logit bias. Building upon this, LAR injects the estimated bias into attention logits, effectively rectifying the collapse induced by token reduction. Together, these components preserve visual evidence even under aggressive compression, enabling robust performance across single-image, multi-image, and video settings on a wide range of MLLMs. Beyond delivering practical acceleration, ERA establishes logit-preserving visual token pruning as a principled framework for efficient MLLMs, unifying theoretical foundation, algorithmic design, and practical deployment. The code is at https://github.com/924973292/ERA.

Abstract

Explore similar work

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models