Random Wins All: Rethinking Grouping Strategies for Vision Tokens
Authors: Qihang Fan, Yuang Ai, Huaibo Huang, Ran He
Organizations: MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Since Transformers are introduced into vision architectures, their quadratic complexity has always been a significant issue that many research efforts aim to address. A representative approach involves grouping tokens, performing self-attention calculations within each group, or pooling the tokens within each group into a single token. To this end, various carefully designed grouping strategies have been proposed to enhance the performance of Vision Transformers. Here, we pose the following questions: Are these carefully designed grouping methods truly necessary? Is there a simpler and more unified token grouping method that can replace these diverse methods? Therefore, we propose the random grouping strategy, which involves a simple and fast random grouping strategy for vision tokens. We validate this approach on multiple baselines, and experiments show that random grouping almost outperforms all other grouping methods. When transferred to downstream tasks, such as object detection, random grouping demonstrates even more pronounced advantages. In response to this phenomenon, we conduct a detailed analysis of the advantages of random grouping from multiple perspectives and identify several crucial elements for the design of grouping strategies: positional information, head feature diversity, global receptive field, and fixed grouping pattern. We demonstrate that as long as these four conditions are met, vision tokens require only an extremely simple grouping strategy to efficiently and effectively handle various visual tasks. We also validate the effectiveness of our proposed random method across multiple modalities, including visual tasks, point cloud processing, and vision-language models. Code will be available at https://github.com/qhfan/random.
Figures & tables
Figure 1: Comparison among baselines and our methods. Random Grouping strategy outperforms all baselines.
Figure 2: Comparison among different token grouping strategies.
Figure 3: Illustration of the random grouping strategy.
Model
Params (M)
FLOPs (G)
Throughput ↑ (imgs/s)
Acc (%)
Plain Backbone
DeiT-T [ 32 ]
6
1.3
6433
72.2
Random-DeiT-T
6
1.1
6682
73.1( +0.9 )
DeiT-S [ 32 ]
22
4.6
3122
79.8
Random-DeiT-S
22
4.3
3313
80.9( +1.1 )
DeiT-B [ 32 ]
87
17.6
1226
81.8
Table 1: Comparison of random grouping strategy with various baselines. The random grouping strategy not only delivers strong performance but also offers faster inference speed compared to other baselines.
Backbone
Params (M)
FLOPs (G)
Mask R-CNN 1×
Params (M)
FLOPs (G)
RetinaNet 1×
APb
AP50b
AP75b
APm
AP50m
AP75m
APb
AP50b
AP75b
APSb
APMb
APLb
Partition-Based Backbone
Swin-T [ 23 ]
48
267
43.7
66.6
47.7
39.8
63.3
42.7
38
248
41.7
63.1
44.3
27.0
45.3
54.7
Random-Swin-T
48
267
46.0
68.1
50.5
41.9
65.3
45.4
38
248
44.3
65.8
46.9
29.7
47.9
57.8
Swin-S [ 23 ]
69
359
45.7
67.9
50.4
41.1
64.9
44.2
60
339
44.5
66.1
47.4
29.8
48.5
59.1
Random-Swin-S
69
359
48.0
69.5
51.9
43.2
67.1
47.0
60
339
46.6
67.8
48.9
30.5
49.3
62.3
Table 2: Comparison of random grouping strategy with various baselines using RetinaNet and Mask R-CNN on COCO val2017 object detection and instance segmentation.
Model
Semantic FPN 80K
Upernet 160K
Params (M)
FLOPs (G)
mIoU (%)
Params (M)
FLOPs (G)
mIoU ss (%)
Plain Backbone
DeiT-S
–
–
–
52
1099
43.0
Random-DeiT-S
–
–
–
52
1099
44.5
Partition-Based Backbone
Swin-T [ 23 ]
–
–
–
60
945
44.5
Table 3: Comparison of random grouping strategy with various baselines using Semantic FPN and UperNet.
Model
Params (M)
Latency ↓ (ms)
mIoU (%)
PTv3 [ 40 ]
46
88
77.6
Random-PTv3 [ 40 ]
46
68
77.8
Table 4: Comparison of random grouping strategy and baseline on point cloud segmentation.
Model
VQA T
MME P
GQA
POPE
VQAv2
SEED I
LLaVA-1.5
58.2
1511
62.0
86.1
78.5
66.1
+Random Pattern
58.6
1578
62.3
87.4
78.5
66.9
LLaVA-1.6
64.9
1519
64.2
86.5
81.8
70.2
+Random Pattern
65.2
1593
64.5
87.6
82.1
70.6
Table 5: Appliying random pattern to LLaVA-1.5/1.6.
Model
Params(M)
FLOPs(G)
Acc(%)
Swin-T [ 23 ]
28
4.5
81.3
RPE CPE
28
4.5
81.6
CPE APE
28
4.5
80.5
CPE LePE
28
4.5
81.7
w/o PE
28
4.5
80.1( -1.6 )
Random-Swin-T
28
4.5
82.7
Table 6: Effect of position information.
Figure 4: Comparison of attention maps with and without positional information. Positional information is particularly crucial for the random grouping method, which lacks local biases.
Figure 5: Similarity curves between adjacent heads in different Models
Figure 6: Comparison of each head with a unique random tensor and all heads sharing a single random tensor. Sharing a single random tensor ( P ) significantly reduces the diversity between different heads.
Model
Params(M)
FLOPs(G)
Acc(%)
DeiT-S [ 32 ]
22
4.6
79.8
Random-DeiT-S
22
4.3
80.9
multi P single P
22
4.3
79.2( -1.7 )
Swin-T [ 23 ]
28
4.5
81.3
Random-Swin-T
28
4.5
82.7
multi P single P
28
4.5
80.5( -2.2 )
Table 7: Effect of head feature diversity.
Model
Params(M)
FLOPs(G)
Acc(%)
DeiT-S [ 32 ]
22
4.6
79.8
Random-DeiT-S
22
4.3
80.9
global region
22
4.3
79.3( -1.6 )
Swin-T [ 23 ]
28
4.5
81.3
Random-Swin-T
28
4.5
82.7
global region
28
4.5
81.5( -1.2 )
Table 8: Effect of global receptive field.
Model
Params(M)
FLOPs(G)
Acc(%)
DeiT-S [ 32 ]
22
4.6
79.8
Random-DeiT-S
22
4.3
80.9
Fully Random
22
4.3
74.3( -6.6 )
Swin-T [ 23 ]
28
4.5
81.3
Random-Swin-T [ 23 ]
28
4.5
82.7
Fully Random
28
4.5
76.4( -6.3 )
Table 9: Effect of the fixed grouping pattern.
Model
Params (M)
FLOPs (G)
Throughput (imgs/s)
Acc (%)
Swin-T
28
4.5
1738
81.3
RPE CPE
28
4.5
1698
81.6
Fully Random(single P )
28
4.5
1922
71.2
+fixed pattern
28
4.5
1922
77.6
+multi P
28
4.5
1917
80.1
+CPE
28
4.5
1866
82.7
Table 10: Roadmap from fully random grouping backbone to Random-Swin. Inference speed are measured on A100 GPU.
Vision Transformers achieve strong performance but incur high computational costs from quadratic token interactions. Training-free token reduction methods improve efficiency, but their robustness remains underexplored. We propose Prune-and Merge ViT, a training-free framework for off-the-shelf models. It combines class attention-based pruning with similarity-based merging to reduce redundancy while preserving semantic information. To evaluate robustness we introduce two metrics. Relative Accuracy Drop (RAD) measures relative accuracy degradation under transformations, and Prediction Flip Rate (PFR) quantifies prediction consistency. On pretrained ViT and DeiT, Prune-and-Merge reduces computation competitively while maintaining robustness and better accuracy than existing token reduction methods. Our analysis suggests that hybrid token reduction does not introduce hallucination-like behavior and preserves prediction stability for common image transformations.
Vatsal, Soumil Malhotra, Priyanshu Chahal +1
Department of Mathematics, Indian Institute of Technology Roorkee
Vision Transformers incur high computational costs when processing long visual token sequences. Existing reduction methods typically rank tokens independently and discard those deemed unimportant or redundant. We propose a reconstruction-guided framework for visual token condensation (ReConV), which summarizes the complete visual sequence with a compact Summary Bank (SMB) and retains tokens whose information is not adequately captured by the SMB, as indicated by their reconstruction residuals. At each compression layer, a lightweight position-conditioned decoder reconstructs the current tokens, leaving shared information compressed in the SMB while explicitly preserving poorly reconstructed details. Experiments on image classification and multimodal understanding demonstrate favorable performance-efficiency trade-offs: ReConV surpasses the uncompressed baseline with 48% fewer GMACs on ImageNet and enables reusable image-level representations for efficient multi-turn inference. Code is available at TokenDrop.
Xiaoya Tang, Shilong Liu, Daniel Smolyak +3
Scientific Computing and Imaging Institute University of Utah · Princeton AI Lab Princeton University · National Institutes of Health
Leading flexible vision tokenizers achieve SOTA quality at an extreme cost, relying on parameter-heavy backbones and slow, multi-step generative decoders. We depart from this complex, spatial-token paradigm and introduce a simple, lightweight, and fast channel-wise flexible-length tokenizer. Our method treats each latent channel as a visual token, enabling a parameter-efficient CNN-Transformer hybrid backbone. Furthermore, employing a stochastic tail-dropping paradigm during training naturally forces channels to organize by semantic importance. This allows for flexible compression at inference by simply retaining the first k channels, and naturally enables variable-length autoregressive image generation. We validate our approach through extensive experiments on ImageNet, demonstrating consistent quality across diverse token budgets. The results establish a new quality-efficiency frontier: our model achieves state-of-the-art perceptual quality (rFID 2.92) while being 8.6× faster in decoding and 2.1× smaller (159M params) than the next-best alternative. Our work establishes channel-wise tokenization as a powerful and practical paradigm for efficient visual representation. Project page: https://channeltok.github.io