Random Wins All: Rethinking Grouping Strategies for Vision Tokens

Qihang Fan; Yuang Ai; Huaibo Huang; Ran He

cs.CVFeb 28, 2026

Random Wins All: Rethinking Grouping Strategies for Vision Tokens

Authors: Qihang Fan, Yuang Ai, Huaibo Huang, Ran He

Organizations: MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

Abstract

Since Transformers are introduced into vision architectures, their quadratic complexity has always been a significant issue that many research efforts aim to address. A representative approach involves grouping tokens, performing self-attention calculations within each group, or pooling the tokens within each group into a single token. To this end, various carefully designed grouping strategies have been proposed to enhance the performance of Vision Transformers. Here, we pose the following questions: Are these carefully designed grouping methods truly necessary? Is there a simpler and more unified token grouping method that can replace these diverse methods? Therefore, we propose the random grouping strategy, which involves a simple and fast random grouping strategy for vision tokens. We validate this approach on multiple baselines, and experiments show that random grouping almost outperforms all other grouping methods. When transferred to downstream tasks, such as object detection, random grouping demonstrates even more pronounced advantages. In response to this phenomenon, we conduct a detailed analysis of the advantages of random grouping from multiple perspectives and identify several crucial elements for the design of grouping strategies: positional information, head feature diversity, global receptive field, and fixed grouping pattern. We demonstrate that as long as these four conditions are met, vision tokens require only an extremely simple grouping strategy to efficiently and effectively handle various visual tasks. We also validate the effectiveness of our proposed random method across multiple modalities, including visual tasks, point cloud processing, and vision-language models. Code will be available at https://github.com/qhfan/random.

Figures & tables

Figure 1: Comparison among baselines and our methods. Random Grouping strategy outperforms all baselines.

Figure 2: Comparison among different token grouping strategies.

Figure 3: Illustration of the random grouping strategy.

Model	Params (M)	FLOPs (G)	Throughput $\uparrow$ (imgs/s)	Acc (%)
Plain Backbone
DeiT-T [ 32 ]	6	1.3	6433	72.2
Random-DeiT-T	6	1.1	6682	73.1( +0.9 )
DeiT-S [ 32 ]	22	4.6	3122	79.8
Random-DeiT-S	22	4.3	3313	80.9( +1.1 )
DeiT-B [ 32 ]	87	17.6	1226	81.8

Table 1: Comparison of random grouping strategy with various baselines. The random grouping strategy not only delivers strong performance but also offers faster inference speed compared to other baselines.

Backbone	Params (M)	FLOPs (G)	Mask R-CNN $1\times$						Params (M)	FLOPs (G)	RetinaNet $1\times$
Backbone	Params (M)	FLOPs (G)	$AP^{b}$	$AP^{b}_{50}$	$AP^{b}_{75}$	$AP^{m}$	$AP^{m}_{50}$	$AP^{m}_{75}$	Params (M)	FLOPs (G)	$AP^{b}$	$AP^{b}_{50}$	$AP^{b}_{75}$	$AP^{b}_{S}$	$AP^{b}_{M}$	$AP^{b}_{L}$
Partition-Based Backbone
Swin-T [ 23 ]	48	267	43.7	66.6	47.7	39.8	63.3	42.7	38	248	41.7	63.1	44.3	27.0	45.3	54.7
Random-Swin-T	48	267	46.0	68.1	50.5	41.9	65.3	45.4	38	248	44.3	65.8	46.9	29.7	47.9	57.8
Swin-S [ 23 ]	69	359	45.7	67.9	50.4	41.1	64.9	44.2	60	339	44.5	66.1	47.4	29.8	48.5	59.1
Random-Swin-S	69	359	48.0	69.5	51.9	43.2	67.1	47.0	60	339	46.6	67.8	48.9	30.5	49.3	62.3

Table 2: Comparison of random grouping strategy with various baselines using RetinaNet and Mask R-CNN on COCO val2017 object detection and instance segmentation.

Model	Semantic FPN 80K			Upernet 160K
Model	Params (M)	FLOPs (G)	mIoU (%)	Params (M)	FLOPs (G)	mIoU ss (%)
Plain Backbone
DeiT-S	–	–	–	52	1099	43.0
Random-DeiT-S	–	–	–	52	1099	44.5
Partition-Based Backbone
Swin-T [ 23 ]	–	–	–	60	945	44.5

Table 3: Comparison of random grouping strategy with various baselines using Semantic FPN and UperNet.

Model	Params (M)	Latency $\downarrow$ (ms)	mIoU (%)
PTv3 [ 40 ]	46	88	77.6
Random-PTv3 [ 40 ]	46	68	77.8

Table 4: Comparison of random grouping strategy and baseline on point cloud segmentation.

Model	VQA T	MME P	GQA	POPE	VQAv2	SEED I
LLaVA-1.5	58.2	1511	62.0	86.1	78.5	66.1
+Random Pattern	58.6	1578	62.3	87.4	78.5	66.9
LLaVA-1.6	64.9	1519	64.2	86.5	81.8	70.2
+Random Pattern	65.2	1593	64.5	87.6	82.1	70.6

Table 5: Appliying random pattern to LLaVA-1.5/1.6.

Model	Params(M)	FLOPs(G)	Acc(%)
Swin-T [ 23 ]	28	4.5	81.3
RPE $\xrightarrow{}$ CPE	28	4.5	81.6
CPE $\xrightarrow{}$ APE	28	4.5	80.5
CPE $\xrightarrow{}$ LePE	28	4.5	81.7
w/o PE	28	4.5	80.1( -1.6 )
Random-Swin-T	28	4.5	82.7

Table 6: Effect of position information.

Figure 4: Comparison of attention maps with and without positional information. Positional information is particularly crucial for the random grouping method, which lacks local biases.

Figure 5: Similarity curves between adjacent heads in different Models

Figure 6: Comparison of each head with a unique random tensor and all heads sharing a single random tensor. Sharing a single random tensor (

P

) significantly reduces the diversity between different heads.

Model	Params(M)	FLOPs(G)	Acc(%)
DeiT-S [ 32 ]	22	4.6	79.8
Random-DeiT-S	22	4.3	80.9
multi $P$ $\xrightarrow{}$ single $P$	22	4.3	79.2( -1.7 )
Swin-T [ 23 ]	28	4.5	81.3
Random-Swin-T	28	4.5	82.7
multi $P$ $\xrightarrow{}$ single $P$	28	4.5	80.5( -2.2 )

Table 7: Effect of head feature diversity.

Model	Params(M)	FLOPs(G)	Acc(%)
DeiT-S [ 32 ]	22	4.6	79.8
Random-DeiT-S	22	4.3	80.9
global $\xrightarrow{}$ region	22	4.3	79.3( -1.6 )
Swin-T [ 23 ]	28	4.5	81.3
Random-Swin-T	28	4.5	82.7
global $\xrightarrow{}$ region	28	4.5	81.5( -1.2 )

Table 8: Effect of global receptive field.

Model	Params(M)	FLOPs(G)	Acc(%)
DeiT-S [ 32 ]	22	4.6	79.8
Random-DeiT-S	22	4.3	80.9
Fully Random	22	4.3	74.3( -6.6 )
Swin-T [ 23 ]	28	4.5	81.3
Random-Swin-T [ 23 ]	28	4.5	82.7
Fully Random	28	4.5	76.4( -6.3 )

Table 9: Effect of the fixed grouping pattern.

Model	Params (M)	FLOPs (G)	Throughput (imgs/s)	Acc (%)
Swin-T	28	4.5	1738	81.3
RPE $\xrightarrow{}$ CPE	28	4.5	1698	81.6
Fully Random(single $P$ )	28	4.5	1922	71.2
+fixed pattern	28	4.5	1922	77.6
+multi $P$	28	4.5	1917	80.1
+CPE	28	4.5	1866	82.7

Table 10: Roadmap from fully random grouping backbone to Random-Swin. Inference speed are measured on A100 GPU.

Explore similar work

Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression
Oct 13, 2026Vatsal, Soumil Malhotra, Priyanshu Chahal +1Vision Transformer Efficient ViTs
Department of Mathematics, Indian Institute of Technology Roorkee
Bottlenecked semantic compression by representation reconstruction
Oct 12, 2026Xiaoya Tang, Shilong Liu, Daniel Smolyak +3Visual Token Compression Efficient ViTs
Scientific Computing and Imaging Institute University of Utah · Princeton AI Lab Princeton University · National Institutes of Health
ChannelTok: Efficient Flexible-Length Vision Tokenization
Jun 3, 2026Sukriti Paul, Arpit Bansal, Tom GoldsteinImage Tokenization Visual Tokenization
University of Maryland, College Park

Oct 13, 2026J/K move · Enter open · S save

Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression

Vatsal, Soumil Malhotra, Priyanshu Chahal, Sanjeev Kumar

Department of Mathematics, Indian Institute of Technology Roorkee

Vision Transformer Efficient ViTs Visual Token Compression Visual Token Pruning Visual Token Merging

Open paper AskPDF arXiv

Vision Transformers achieve strong performance but incur high computational costs from quadratic token interactions. Training-free token reduction methods improve efficiency, but their robustness remains underexplored. We propose Prune-and Merge ViT, a training-free framework for off-the-shelf models. It combines class attention-based pruning with similarity-based merging to reduce redundancy while preserving semantic information. To evaluate robustness we introduce two metrics. Relative Accuracy Drop (RAD) measures relative accuracy degradation under transformations, and Prediction Flip Rate (PFR) quantifies prediction consistency. On pretrained ViT and DeiT, Prune-and-Merge reduces computation competitively while maintaining robustness and better accuracy than existing token reduction methods. Our analysis suggests that hybrid token reduction does not introduce hallucination-like behavior and preserves prediction stability for common image transformations.

Abstract

Explore similar work

Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression

Bottlenecked semantic compression by representation reconstruction

ChannelTok: Efficient Flexible-Length Vision Tokenization