EntropyPrune: Matrix Entropy Guided Visual Token Pruning for Multimodal Large Language Models

Yahong Wang; Juncheng Wu; Zhangkai Ni; Chengmei Yang; Yihang Liu; Longzhen Yang; Yuyin Zhou; Ying Wen

cs.CVFeb 19, 2026

EntropyPrune: Matrix Entropy Guided Visual Token Pruning for Multimodal Large Language Models

Authors: Yahong Wang, Juncheng Wu, Zhangkai Ni, Chengmei Yang, Yihang Liu, Longzhen Yang, Yuyin Zhou, Ying Wen,

Organizations: Tongji University · University of California, Santa Cruz · East China Normal University · Shanghai Eye Disease Prevention and Treatment Center

Abstract

Multimodal large language models (MLLMs) incur substantial inference cost due to the processing of hundreds of visual tokens per image. Although token pruning has proven effective for accelerating inference, determining when and where to prune remains largely heuristic. Existing approaches typically rely on static, empirically selected layers, which limit interpretability and transferability across models. In this work, we introduce a matrix-entropy perspective and identify an "Entropy Collapse Layer" (ECL), where the information content of visual representations exhibits a sharp and consistent drop, which provides a principled criterion for selecting the pruning stage. Building on this observation, we propose EntropyPrune, a novel matrix-entropy-guided token pruning framework that quantifies the information value of individual visual tokens and prunes redundant ones without relying on attention maps. Moreover, to enable efficient computation, we exploit the spectral equivalence of dual Gram matrices, reducing the complexity of entropy computation and yielding up to a 64x theoretical speedup. Extensive experiments on diverse multimodal benchmarks demonstrate that EntropyPrune consistently outperforms state-of-the-art pruning methods in both accuracy and efficiency. On LLaVA-1.5-7B, our method achieves a 68.2% reduction in FLOPs while preserving 96.0% of the original performance. Furthermore, EntropyPrune generalizes effectively to high-resolution and video-based models, highlighting the strong robustness and scalability in practical MLLM acceleration. The code will be publicly available at https://github.com/YahongWang1/EntropyPrune.

Figures & tables

Figure 1: (a) Comparison between vanilla LLaVA-1.5-7B and EntropyPrune. Correct answers are highlighted in green , while hallucinations are marked in red . By removing low-information tokens, EntropyPrune encourages the model to concentrate on more critical details ( e.g. , the person’s state and the car’s color). (b) Performance comparison. The radial-axis visualization of min-max normalized scores shows that EntropyPrune consistently outperforms competitive methods, including FastV, DART, DivPrune, and CDPruner.

Figure 2: Layer-wise matrix entropy of visual tokens (query and key states) in LLaVA-1.5-7B and LLaVA-Next-7B across eight datasets. A consistent layer-wise trend is observed across different datasets, with a precipitous entropy drop after the second layer.

Figure 3: Eigenvalue distribution of the query and key states covariance matrices. L

i

represents the

i-th

layer. We visualize the magnitude of eigenvalues across different layers of LLaVA-1.5-7B. The rapid decay observed in the eigenvalues distribution indicates a low-rank structure within the matrices.

Figure 4: Overview of the proposed EntropyPrune. (a) When to Prune identifies the “Entropy Collapse Layer” by detecting a sharp drop in layer-wise matrix entropy. (b) What to Prune details the calculation of matrix entropy based on the token’s feature. (c) EntropyPrune Pipeline illustrates the overall workflow: after reaching “Entropy Collapse Layer”, the matrix entropy of each visual token is calculated to guide the pruning of low-entropy ones.

Upper Bound, 576 Tokens (100%)
Method	MMB	MME	VQA ${}^{\text{T}}$	MMB ${}^{\text{C}}$	MMstar	Acc.(%)	Rel.(%)	FLOPs
LLaVA-1.5-7B	64.6	1862	58.3	58.1	33.1	56.1	100.0	8.80T
Retain 192 Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 66.7\%)}}$
FastV (ECCV24)	63.2	1796	56.4	57.4	31.1	54.4	97.0	3.99T
PDrop (CVPR25)	63.3	1765	56.2	56.5	30.7	53.9	96.1	3.79T
DART (EMNLP25)	63.7	1833	57.0	57.1	31.4	54.9	97.9	3.69T

Table 1: Performance of LLaVA-1.5-7B with EntropyPrune. The vanilla number of vision tokens is 576. Acc. represents the average accuracy across all benchmarks. Rel. denotes the relative performance retained compared to the original model. The best performance is highlighted in bold .

Upper Bound, 2880 Tokens (100%)
Method	VQA ${}^{\text{T}}$	MMVet	OCR ${}^{\text{B}}$	MMMU	AI2D	Acc.
LLaVA-Next	61.3	44.0	43.7	28.0	65.6	48.5
Retain 320 Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 88.9\%)}}$
FastV	49.0	31.1	21.8	29.3	60.2	38.3
PDrop	53.2	29.4	27.1	27.3	61.5	39.7
DART	58.0	39.3	30.5	26.7	62.2	43.3

Table 2: Performance of LLaVA-Next-7B with EntropyPrune. The best performance is highlighted in bold , while the second best is underlined .

Upper Bound (100%)
Method	MMB	MMB ${}^{\text{C}}$	MMstar	MMMU	AI2D	Acc.
Qwen2.5-VL	79.8	81.0	56.3	58.0	80.8	71.2
Retain 25% Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 75\%)}}$
FastV	77.7	77.2	46.7	42.7	74.5	63.8
CDPruner	74.3	77.7	47.8	42.9	71.6	62.9
EntropyPrune	77.9	79.6	50.6	52.1	74.1	66.9

Table 3: Performance of EntropyPrune on Qwen-2.5-VL-7B.

Upper Bound (100%)
Method	MSVD		MSRVTT		Avg.
	Acc.	Score	Acc.	Score	Acc.	Score
Video-LLaVA	53.1	3.2	35.6	2.5	44.4	2.9
Retain 50% Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 50\%)}}$
CDPruner	41.9	2.6	30.1	2.2	36.0	2.4
EntropyPrune	52.8	3.2	36.0	2.5	44.4	2.9

Table 4: Performance of EntropyPrune on Video-LLaVA-7B.

Method	Prefill (s)	Lat. (s)	KV (MB)	Mem. (GB)	FLOPs (%)	Score
Upper Bound, 576 Tokens (100%)
LLaVA-1.5-7B	381.8	459.6	288.0	16.2	100	1862
Retain 192 Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 66.7\%)}}$
FastV	297.5	370.3	96.3	16.0	45.3	1796
CDPruner	360.4	415.7	96.0	19.8	42.7	1786
EntropyPrune	273.3	358.2	95.8	15.8	42.3	1844

Table 5: Efficiency analysis of different pruning methods on LLaVA-1.5-7B.

Figure 5: Ablation study on pruning layer selection. Experiments are conducted on TextVQA and MMB when retaining 192 tokens. Applying pruning at Entropy Collapse Layer (Layer 2) consistently yields the best performance across all methods.

Upper Bound, 576 Tokens (100%)
Method	MMB	MME	SQA	VQA ${}^{\text{T}}$	Rel.(%)
LLaVA-1.5-7B	64.6	1862	69.5	58.3	100.0
Retain 192 Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 66.7\%)}}$
FastV	64.3	1821	68.9	57.0	98.6
DART	64.4	1833	69.0	57.0	98.8
DivPrune	63.2	1795	68.5	57.5	97.9

Table 6: Ablation study on token selection strategies.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Upper Bound, 576 Tokens (100%)
Method	MMB	MME	VQA ${}^{\text{T}}$	MMB ${}^{\text{C}}$	MMstar	Acc.(%)	Rel.(%)	FLOPs
LLaVA-1.5-7B	64.6	1862	58.3	58.1	33.1	56.1	100.0	8.80T
Retain 192 Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 66.7\%)}}$
FastV (ECCV24)	63.2	1796	56.4	57.4	31.1	54.4	97.0	3.99T
PDrop (CVPR25)	63.3	1765	56.2	56.5	30.7	53.9	96.1	3.79T
DART (EMNLP25)	63.7	1833	57.0	57.1	31.4	54.9	97.9	3.69T

Appendix

Table 7: Performance of LLaVA-1.5-7B with EntropyPrune-q and EntropyPrune-k. The vanilla number of vision tokens is 576. Acc. represents the average accuracy across all benchmarks. Rel. denotes the relative performance retained compared to the original model. The best performance is highlighted in bold , while the second best is underlined .

Retain 20% Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 80.0\%)}}$
Method	MME	VQA ${}^{\text{T}}$	MMStar
FitPrune	1775	55.7	30.1
EntropyPrune	1782	56.5	31.8
Method	MME	MMB	MMStar
Retain 128 Tokens $\mathbf{{\color[rgb]{0.1328,0.5469,0.1328}(\downarrow 77.8\%)}}$
SparseVLM	1703	60.5	30.7

Appendix

Table 8: Comparisons with recent visual token pruning methods on LLaVA-1.5-7B.

Figure 6: Layer-wise matrix entropy of visual tokens (query and key states) in Qwen-2.5-VL-7B across eight datasets. A consistent entropy collapse is observed after the first layer across different datasets and feature states, demonstrating that the entropy collapse phenomenon also exists in non-LLaVA architectures.

Token Pruning

Token pruning removes redundant tokens from transformer inputs or intermediate representations to reduce computation, memory use, and inference time.

See all 55 papers

Jun 30, 2026

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attention distributions, giving rise to a phenomenon we term Attention Logit Collapse. To address this issue, we propose ERA, an Entropy-guided visual token pruning framework with Rectified Attention for efficient MLLMs. Specifically, ERA comprises three crucial components: Dual-view Entropy Pruning (DEP), Bias-aware Token Recycling (BTR), and Logit-preserving Attention Rectification (LAR). First, DEP identifies representative anchor tokens by jointly modeling visual diversity and head-wise saliency. BTR then recycles pruned tokens into their corresponding anchors while estimating a cluster-level logit bias. Building upon this, LAR injects the estimated bias into attention logits, effectively rectifying the collapse induced by token reduction. Together, these components preserve visual evidence even under aggressive compression, enabling robust performance across single-image, multi-image, and video settings on a wide range of MLLMs. Beyond delivering practical acceleration, ERA establishes logit-preserving visual token pruning as a principled framework for efficient MLLMs, unifying theoretical foundation, algorithmic design, and practical deployment. The code is at https://github.com/924973292/ERA.

Yuhao Wang, Mu Qiao, Haiwen Diao +5

School of Future Technology, Dalian University of Technology, Dalian 116024, China · S-Lab, Nanyang Technological University, Singapore 639798 · OPPO Research Institute, Shenzhen 518000, China +1

Efficient VLM Inference Multimodal Large Language Models Multimodal Token Compression Visual Token Pruning

Ask PDF

Sep 1, 2026

SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models

Despite their strong multimodal understanding ability, multimodal large language models (MLLMs) incur substantial computational overhead when processing long visual token sequences. To reduce inference costs, recent studies have explored visual token pruning through vision-centric or text-guided strategies. However, these methods often overlook high-norm outlier tokens, i.e., tokens with abnormally large feature norms, leading to suboptimal pruning decisions. In this work, we show that such high-norm outlier tokens are highly redundant in both feature and spatial dimensions, yet are often mistakenly preserved as informative cues by existing methods. Motivated by this observation, we propose SinkPruner, a training-free visual token pruning framework for efficient MLLM inference. SinkPruner follows a coarse-to-fine design with two key modules: a visual sanitizer that filters high-norm redundancies and alleviates attention sink and attention dispersion, and a text-guided pruner that further retains tokens semantically aligned with the text query. Extensive experiments on twelve image-language and four video-language benchmarks demonstrate the effectiveness, efficiency, and generalizability of our framework. Notably, SinkPruner preserves 96.5% (91.8%) of the original performance of LLaVA-1.5 (Qwen2.5-VL) under an 89% token reduction. Experiments further indicate that our visual sanitizer exhibits promising transferability in enhancing the performance of existing pruning methods. Our code is available at https://github.com/LaVi-Lab/SinkPruner.

Shiyu Li, Zi-Yuan Hu, Shijia Huang +3

1The Chinese University of Hong Kong · 2Weitu AI · 3Peking University

Efficient VLM Inference Multimodal Large Language Models Attention Sinks Multimodal Token Compression Visual Token Pruning

Ask PDF

Sep 1, 2026

S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

Visual token pruning reduces the inference overhead of multimodal large language models (MLLMs) by retaining only a subset of visual tokens. Existing methods usually select tokens based on importance or redundancy. However, we observe that these criteria produce stable spatial biases across inputs and do not always outperform simple Uniform Grid sampling, highlighting the value of broad spatial coverage. Motivated by this, we propose S

^2

Prune, a training-free pruning method that preserves spatial coverage while adapting token density to local image structure. We first divide the image into regions and assign at least one token to each region to preserve coverage. The remaining token budget is then distributed according to Laplacian variation, giving more tokens to regions with richer structure. We then use Early Representation Change (ERC), computed from the first decoder block, to select representative tokens within each region. We evaluate S

^2

Prune across diverse settings and two MLLM architectures. On Qwen2.5-VL-7B-Instruct, it achieves the highest average accuracy among the evaluated training-free pruning methods. With only 32 of the original 576 visual tokens, it still retains 79.3% of the full-model performance. Code is available at https://github.com/yuanyuanjia71-spec/S2Prune.

Yuanyuan Jia, Shunpu Tang, Qianqian Yang

College of Information Science and Electronic Engineering, Zhejiang University Hangzhou, China

Efficient VLM Inference Multimodal Large Language Models Visual Token Pruning LVLM Pruning

Ask PDF

Abstract

Token Pruning

Explore similar work

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models

S2^22Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

S $^2$ Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models