Transformers are strong baselines in both vision and language because self-attention captures long-range dependencies across tokens. However, the cost of self-attention grows quadratically with the number of tokens. Patch pruning mitigates this cost by estimating per-patch importance and removing redundant patches. To identify informative patches for pruning, we introduce a criterion based on the Shannon entropy of the attention distribution. Low-entropy patches, which receive selective and concentrated attention, are kept as important, while high-entropy patches with attention spread across many locations are treated as redundant. We also extend the criterion from Shannon to Rényi entropy, which emphasizes sharp attention peaks and supports pruning strategies that adapt to task needs and computational limits. In experiments on fine-grained image recognition, where patch selection is critical, our method reduced computation while preserving accuracy. Moreover, adjusting the pruning policy through the Rényi entropy measure yields further gains and improves the trade-off between accuracy and computation.
Figures & tables
Figure 1: Overview of our key idea. The left figure shows an attention entropy map, where red indicates higher entropy and blue indicates lower entropy. We observe that low attention entropy corresponds to foreground regions and high attention entropy to background, which helps identify informative patches. Based on this, we use attention entropy as the pruning criterion, as illustrated on the right.
Figure 2: Overall pipeline of the Rényi attention entropy pruning. The pruning procedure is described in Section 3.3 .
Figure 3: Visualization of Rényi attention entropy ( α=2.0 ) for each Transformer block in DeiT-S. This visualization shows that attention entropy depends on Transformer layer depth, and lower entropy corresponds to foreground regions.
Configurations and baseline
Keep rate r
Dataset
Method
DeiT-S
0.9
0.8
0.7
0.6
0.5
0.4
0.3
IN-100
EViT
86.74
86.06
86.04
86.46
85.68
85.32
84.50
83.92
Ours (Shannon)
86.74
86.40
85.74
86.14
85.66
85.54
84.70
84.16
Ours (Rényi, α=2.0 )
86.74
86.34
86.46
85.60
85.50
85.36
84.78
84.30
Ours (Rényi, α=5.0 )
86.74
85.88
86.20
86.30
85.78
85.44
85.18
84.00
Ours (Rényi, α=10.0 )
86.74
86.26
86.02
86.72
85.72
85.68
84.82
84.20
Table 1: Top-1 accuracy(%) with DeiT-S on ImageNet-100 ( IN-100 ), FGVC Aircraft ( Aircraft ), and Oxford Flowers102 ( Flowers ) under different keep rates r . The DeiT-S column reports scores without patch pruning. The bold scores indicate the best performance at each keep rate.
Configurations and baseline
Keep rate r
Dataset
Method
DeiT-B
0.9
0.8
0.7
0.6
0.5
0.4
0.3
IN-100
EViT
86.82
87.04
86.64
86.36
86.20
85.02
84.32
83.82
Ours (Shannon)
86.82
86.48
86.58
86.18
85.38
84.80
84.12
83.02
Ours (Rényi, α=2.0 )
86.82
86.62
86.40
86.44
85.72
85.20
84.40
83.14
Ours (Rényi, α=5.0 )
86.82
86.44
86.54
86.14
85.48
84.88
84.34
83.02
Ours (Rényi, α=10.0 )
86.82
86.68
86.42
86.16
85.96
84.70
84.50
83.02
Table 2: Top-1 accuracy(%) with DeiT-B on ImageNet-100 ( IN-100 ), FGVC Aircraft ( Aircraft ), and Oxford Flowers102 ( Flowers ) under different keep rates r . The DeiT-B column reports scores without patch pruning. The bold scores indicate the best performance at each keep rate.
Figure 4: Visualizations of patch pruning results for EViT and Rényi attention entropy-based approach on ImageNet-100 , FGVC Aircraft , and Oxford Flowers102 . From left to right: input image, pruning results at Blocks 4, 7, and 10. The keep rate is r=0.7 , and for our method we show the results with the tuned α .
Figure 5: Visualization of Shannon and Rényi attention entropies. For each DeiT-S block, the figure shows histograms of Shannon attention entropy ( α=1.0 ) and Rényi attention entropy at different α orders. Blue indicates informative patches that are kept, and red indicates redundant patches that are pruned. The results show that the Rényi order controls peak emphasis and allows the characterization of the attention distribution to be adapted to the task.
r
acc(%)
GFLOPs
images/s
DeiT-S
1.0
86.74
4.61
1111.4
EViT
0.7
86.46 -0.28
3.04 -34.1%
1947.1 +75.2%
0.5
85.32 -1.42
2.31 -49.8%
2535.8 +128.2%
0.3
83.92 -2.82
1.82 -60.5%
3150.1 +183.4%
Ours
0.7
86.72 -0.02
3.00 -34.8%
1730.3 +55.7%
0.5
86.68 -0.06
2.29 -50.4%
2257.5 +103.1%
Table 3: Comparison of computational cost with existing methods in terms of the best score of validation top-1 accuracy (acc), FLOPs, and throughput (images/s), by varying the keep rate r .
Figure 6: Results for attention entropy (top) and attention distance [ 19 ] (bottom). All values represent averages over 500 samples.
Vision Transformers (ViTs) achieve state-of-the-art performance but suffer from the O(N2) complexity of self-attention, making inference costly for high-resolution inputs. To address this bottleneck, token pruning has emerged as a critical technique to accelerate inference. Most existing methods rely on the [CLS] token to estimate patch importance. However, we argue that the [CLS] token can be unreliable in early layers where semantic representations are still immature. As a result, pruning in the early layer often leads to inaccurate importance estimation and unnecessary information loss. In this work, we propose a training-free token importance metric, namely Col-Ln, which is derived from Rényi entropy that enables the identification of informative tokens from the first layer of the network, thereby enabling more reliable pruning in token reduction. Extensive experiments on ViTs and Large Vision-Language Models (LVLMs) demonstrate that our approach consistently outperforms state-of-the-art pruning methods across diverse benchmarks.
Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning, but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). In this paper, we analyze the effectiveness of attention mechanisms in both vision encoders and LLMs. We find that vision encoders suffer from attention sink, leading to poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to-vision attention demonstrates resistance to this bias and enables effective pruning guidance in middle layers. Based on these observations, we propose LearnPruner, a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. Experimental results show that our LearnPruner can preserve approximately 95% of the original performance while using only 5.5% of vision tokens, and achieve 3.2× inference acceleration, demonstrating a superior accuracy-efficiency trade-off.
Transformer architectures have achieved strong results in natural language processing and computer vision, but their deployment is limited by high computational, memory and storage costs, motivating research into model compression techniques, such as pruning, while preserving performance. Prior work has shown substantial redundancy in large language models (LLMs), and redundancy in vision transformers has also been investigated, but it remains less systematically analyzed than in LLMs. In this work, we extend similarity-based redundancy analysis to vision models by measuring cosine similarity between layer inputs and outputs to identify transformations with minimal impact. We study redundancy patterns across several transformer architectures and find that deep vision transformers such as DINOv2 and SwinV2, like language models, exhibit significant redundancy. We then evaluate how this redundancy can be exploited to improve efficiency, analyzing the trade-off between inference speed and performance degradation under different pruning strategies. Our results show that moderate pruning can yield substantial acceleration with limited impact on accuracy, and that 20% of the DINOv2 blocks can be removed without any retraining, for a 1.24x throughput gain and a 20% lower energy per image at a relative accuracy loss below 1% on ImageNet-1K. We further propose a model selection pipeline that automatically identifies the best pruned configuration for a given dataset, reducing FLOPs by 13% to 27% on average across architectures with an average accuracy change between -0.3 and +1.0 points.