Vision Transformers (ViTs) achieve state-of-the-art performance but suffer from the O(N2) complexity of self-attention, making inference costly for high-resolution inputs. To address this bottleneck, token pruning has emerged as a critical technique to accelerate inference. Most existing methods rely on the [CLS] token to estimate patch importance. However, we argue that the [CLS] token can be unreliable in early layers where semantic representations are still immature. As a result, pruning in the early layer often leads to inaccurate importance estimation and unnecessary information loss. In this work, we propose a training-free token importance metric, namely Col-Ln, which is derived from Rényi entropy that enables the identification of informative tokens from the first layer of the network, thereby enabling more reliable pruning in token reduction. Extensive experiments on ViTs and Large Vision-Language Models (LVLMs) demonstrate that our approach consistently outperforms state-of-the-art pruning methods across diverse benchmarks.
Figures & tables
Figure 1: Visual Comparison of Token Pruning. We compare standard [CLS] attention and our proposed metric in the pruning process across initial layers ( L0 – L5 ). While [CLS] suffers from semantic immaturity and erroneously prunes critical foreground patches, our method preserves significantly more of the main subject while accurately eliminating redundant background tokens.
Keep rate ( r )
[CLS]
Random
0.7
66.1
73.2
0.8
74.0
77.6
0.9
79.0
80.0
Table 1: EViT on ViT-Small Accuracy (%). Pruning is applied at layers (0, 3, 6).
Figure 2: (Left) Conventional methods rely on the [CLS] token as a global proxy, utilizing its attention weights (the top row of the matrix) to estimate patch importance. (Right) Our Col-Ln metric focuses on collective consensus, calculating the column-wise ℓn -norm of an attention matrix.
Table 4Table 5
Keep Rate ( r )
Pruning Layers
[CLS]
Ours
Random
GFLOPs
ViT-Small
–
81.4
81.4
81.4
4.6
0.7
3, 6, 9
78.9
79.0
77.3
3.0
0, 3, 6
66.1
75.0
73.2
2.3
0.8
3, 6, 9
80.5
80.5
79.5
3.5
0, 3, 6
74.0
78.6
77.6
2.9
0.9
3, 6, 9
81.3
81.3
80.6
4.0
Table 9: Comparison of different metrics integrated into EViT on ViT-Small .
Keep Rate ( r )
Pruning Layers
[CLS]
Ours
Random
GFLOPs
ViT-Base
–
84.5
84.5
84.5
17.6
0.7
3, 6, 9
83.0
83.0
81.0
11.6
0, 3, 6
69.7
79.9
77.8
8.8
0.8
3, 6, 9
83.9
84.1
82.7
13.3
0, 3, 6
76.7
82.1
81.4
11.3
0.9
3, 6, 9
84.5
84.5
83.9
15.4
Table 10: Comparison of different metrics integrated into EViT on ViT-Base .
Keep Rate ( r )
Pruning Layers
[CLS]
Ours
Random
GFLOPs
ViT-Small
–
81.4
81.4
81.4
4.6
0.7
0, 3, 6
66.1
75.0
73.2
2.3
1, 4, 7
76.3
77.0
74.8
2.5
2, 5, 8
77.9
78.3
75.9
2.8
3, 6, 9
78.9
79.0
77.3
3.0
4, 7, 10
79.6
79.8
78.3
3.3
Table 11: The Trend on ViT-Small .
Figure 3: Training Loss Comparison on the EViT Framework.
Model
Pruning Layers
[CLS]
Ours
Random
GFLOPs
ViT-S
0, 3, 6
77.2
79.3
77.3
2.3
3, 6, 9
81.0
81.1
79.4
3.0
Table 13: EViT fine-tuning result with a 0.7 keep rate. Our method significantly outperforms the [CLS] metric in early-layer pruning (0, 3, 6).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Heatmap-based Attention Visualization. This figure provides the attention heatmaps for the samples shown in Fig. 1 of the main paper. We compare the standard [CLS] attention (top rows) with our proposed Col-Ln metric (bottom rows) across the initial six layers ( L0 – L5 ) of a ViT-Base model. Red colors indicate higher importance scores.
Figure 5: Layer-wise attention visualization on ViT-Base with different norm The top row displays traditional [CLS] attention, which is messy and noisy in early layers (L0–L3). In contrast, the bottom rows show our column-wise ℓn -norms. As the norm order n increases (from ℓ1 to ℓ4 ), the heatmaps become progressively sharper and more concentrated on the salient object.
Keep Rate ( r )
Pruning Layers
[CLS]
Ours
Random
GFLOPs
DeiT-Small
–
79.8
79.8
79.8
4.6
0.7
3, 6, 9
78.5
78.5
77.5
3.0
0, 3, 6
64.7
75.1
74.5
2.3
0.8
3, 6, 9
79.3
79.3
78.5
3.5
0, 3, 6
72.1
77.7
77.3
2.9
0.9
3, 6, 9
79.7
79.7
79.3
4.0
Appendix
Table 14: EViT inference on DeiT-Small .
Keep Rate ( r )
Pruning Layers
[CLS]
Ours
Random
GFLOPs
DeiT-Base
–
82.0
82.0
82.0
4.6
0.7
3, 6, 9
80.6
80.6
78.8
3.0
0, 3, 6
71.3
76.6
75.7
2.3
0.8
3, 6, 9
81.3
81.3
80.3
3.5
0, 3, 6
77.0
79.5
79.0
2.9
0.9
3, 6, 9
81.7
81.8
81.2
4.0
Appendix
Table 15: EViT inference on DeiT-Base .
Hyperparameter
Value
Backbone Model
vit_small_patch16_augreg
Input Resolution
224×224
Total Batch Size
2048
Optimizer
AdamW
Learning Rate (Base)
2×10−5
Min Learning Rate
2×10−6
Appendix
Table 16: Hyperparameters for EViT Fine-tuning. These settings were applied to the ViT-Small (augreg) backbone used in our comparative analysis.