ConvRoute-ViT: Convolutional Token Routing for Efficient Vision Transformers

Manish Das; Shiv Ram Dubey; Satish Kumar Singh

cs.CVOct 28, 2026

ConvRoute-ViT: Convolutional Token Routing for Efficient Vision Transformers

Authors: Manish Das, Shiv Ram Dubey, Satish Kumar Singh

Organizations: IIIT Allahabad, Prayagraj, India

Abstract

Vision Transformers (ViTs) achieve strong image recognition performance, but their self-attention increases quadratically with the number of input tokens and makes them computationally expensive. The standard implementation of ViT models treats all patch tokens with equal importance across every layer of the model. This uniform treatment introduces redundant computation and may reduce the model's focus on the most discriminative object regions. In this paper, we propose ConvRoute-ViT, a convolutional token routing mechanism for efficient vision transformers that performs token reduction through convolutional routing. Our model preserves the most relevant patch tokens according to class-token attention at selected layers, while the remaining tokens are spatially reorganized and compressed into a compact set of convolutional summary tokens. This design reduces the sequence length without completely removing the contextual information of the less informative tokens. By applying the routing process hierarchically, the proposed framework constructs compact and discriminative representations for deeper layers. Experiments on CIFAR-100 and TinyImageNet show that ConvRoute-ViT improves computational efficiency while maintaining classification accuracy compared with the base ViT.

Figures & tables

Fig. 1: Overview of the proposed ConvRoute-ViT architecture. The input image is converted into patch tokens with positional encoding and a CLS token, creating an input size of $B\times N\times D$ . After initial transformer processing up to $k^{th}$ transformer layers, the ConvRoute Token Routing Module computes the token importance score and retains the top-R important tokens. The remaining tokens are passed through the ConvRoute Compressor where they are spatially reshaped to generate compact ConvSummary tokens, resulting in $B\times M\times D$ and are processed by transformer layers from $k\!+\!1\!\rightarrow\!l$ . A second ConvRoute Token Routing Module is applied at layer $l$ to further reduce the sequence and is then passed through the subsequent transformer blocks for final classification.

$(k,l)$	$(\rho_{k},\rho_{l})=(0.6,0.7)$		$(\rho_{k},\rho_{l})=(0.7,0.8)$		$(\rho_{k},\rho_{l})=(0.8,0.9)$		$(\rho_{k},\rho_{l})=(0.9,0.5)$		$(\rho_{k},\rho_{l})=(0.9,0.7)$
(2,4)	73.70 ( -0.34 )	55.56 ( -1.24 )	74.06 ( +0.02 )	55.70 ( -1.10 )	74.70 ( +0.66 )	56.62 ( -0.18 )	74.35 ( +0.31 )	56.20 ( -0.60 )	74.07 ( +0.03 )	56.06 ( -0.74 )
(2,5)	73.76 ( -0.28 )	55.74 ( -1.06 )	74.18 ( +0.14 )	56.58 ( -0.22 )	74.70 ( +0.66 )	58.04 ( +1.24 )	74.97 ( +0.93 )	56.90 ( +0.10 )	75.27 ( +1.23 )	57.18 ( +0.38 )
(2,6)	73.86 ( -0.18 )	56.08 ( -0.72 )	75.54 ( +1.50 )	56.74 ( -0.06 )	75.71 ( +1.67 )	56.74 ( -0.06 )	74.84 ( +0.80 )	57.22 ( +0.42 )	75.03 ( +0.99 )	57.70 ( +0.90 )
(2,7)	74.19 ( +0.15 )	55.98 ( -0.82 )	74.21 ( +0.17 )	56.26 ( -0.54 )	75.05 ( +1.01 )	56.96 ( +0.16 )	75.15 ( +1.11 )	56.20 ( -0.60 )	74.93 ( +0.89 )	57.26 ( +0.46 )
(3,5)	74.33 ( +0.29 )	55.78 ( -1.02 )	74.54 ( +0.50 )	56.66 ( -0.14 )	74.88 ( +0.84 )	56.58 ( -0.22 )	74.74 ( +0.70 )	57.10 ( +0.30 )	75.42 ( +1.38 )	57.28 ( +0.48 )
(3,6)	74.50 ( +0.46 )	56.34 ( -0.46 )	74.90 ( +0.86 )	56.70 ( -0.10 )	74.90 ( +0.86 )	57.28 ( +0.48 )	75.25 ( +1.21 )	57.86 ( +1.06 )	75.15 ( +1.11 )	57.76 ( +0.96 )

TABLE I: Comparison of ConvRoute-ViT under different routing-layer positions

(k,l)

and keep-ratio settings

(\rho_{k},\rho_{l})

. Rows denote routing-layer pairs, and each column reports CIFAR-100 and TinyImageNet accuracies for each keep-ratio pair. The base ViT achieves 74.04% on CIFAR-100 and 56.80% on TinyImageNet. Blue : CIFAR-100. Green : TinyImageNet.

Fig. 2: Attention heatmaps for representative CIFAR-100 samples [ 24 ] . Each example consists of three panels arranged from left to right: the input image, the base ViT attention map, and the ConvRoute-ViT attention map.

Fig. 3: Comparison of prediction confidence between the proposed model and the base ViT on CIFAR-100. Input images are shown on the left, and the top-3 predictions on the right ( blue : proposed; green : base ViT).

$(\rho_{k},\rho_{l})$	(0.4,0.5)	(0.5,0.4)	(0.5,0.5)	(0.5,0.6)	(0.6,0.5)	(0.6,0.6)	(0.6,0.7)	(0.7,0.6)	(0.7,0.7)	(0.7,0.8)	(0.8,0.7)	(0.8,0.8)	(0.8,0.9)	(0.9,0.5)	(0.9,0.6)	(0.9,0.7)	(0.9,0.8)	(0.9,0.9)
CIFAR-100	74.28	74.34	74.55	74.25	74.70	74.55	74.86	74.75	74.87	74.65	75.05	74.40	75.39	75.00	75.01	75.07	74.01	75.06
TinyImageNet	55.48	56.36	55.34	56.56	56.82	56.96	56.16	56.90	57.12	56.78	56.32	57.92	57.18	56.88	57.20	57.70	57.22	57.26
GFLOPs	0.1239	0.1287	0.1325	0.1353	0.1395	0.1433	0.1472	0.1504	0.1553	0.1602	0.1653	0.1712	0.1763	0.1797	0.1697	0.1618	0.1736	0.1858

TABLE II: Effect of keep-ratio pair settings

(\rho_{k},\rho_{l})

at the 4th and 6th routing layers. Columns shows different pairs, and rows report CIFAR-100, TinyImageNet, and GFLOPs. The baseline achieves 74.04% on CIFAR-100, 56.80% on TinyImageNet, and 0.1890 GFLOPs.

Model	GFLOPs	CIFAR-100	TinyImageNet
ResNet-50 [ 27 ]	0.7	71.03	56.02
EfficientNet-B3 [ 28 ]	0.2	68.77	48.85
MobileNetV2-1.4 [ 29 ]	0.1	69.92	54.12
CvT-21 [ 30 ]	1.7	63.32	49.97
HSViT-C3A4 [ 26 ]	1.3	72.46	56.73
CaiT † [ 31 ]	0.6	72.44	57.59

TABLE III: Comparison with representative CNN and Transformer baselines on CIFAR-100 and TinyImageNet. Results are taken from [ 26 ] , and models marked with † are implemented from scratch.

Fig. 4: Four representative CIFAR-100 examples illustrating the proposed ConvRoute mechanism. From left to right: input image, tokens removed at layer $k$ , additional tokens removed at layer $l$ , and the final compact token representation, which is further processed and passed for final classification.

Vision Transformer

Vision Transformers (ViTs) use self-attention over image patches to learn visual representations for recognition, segmentation, and other tasks.

See all 370 papers

RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT
Jun 6, 2026Kyumin Choi, Ikbeom JangVision Transformer Efficient ViTs
Hankuk University of Foreign Studies Seoul, South Korea
Bottlenecked semantic compression by representation reconstruction
Oct 12, 2026Xiaoya Tang, Shilong Liu, Daniel Smolyak +3Visual Token Compression Efficient ViTs
Scientific Computing and Imaging Institute University of Utah · Princeton AI Lab Princeton University · National Institutes of Health
Representative Attention For Vision Transformers
May 14, 2026Yuntong Li, Hainuo Wang, Hengxing Liu +2Vision Transformer Efficient ViTs
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China

Jun 6, 2026J/K move · Enter open · S save

RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT

Kyumin Choi, Ikbeom Jang

Hankuk University of Foreign Studies Seoul, South Korea

Vision Transformer Efficient ViTs Visual Token Merging Visual Token Pruning Image Classification Visual Token Compression

Open paper AskPDF arXiv

Vision Transformers (ViTs) achieve strong performance but suffer from high computational costs due to quadratic self-attention complexity. Although token reduction techniques such as pruning and merging mitigate this, they typically overlook how representations evolve across network depth. We propose RAPID, a depth-aware token reduction framework that adapts reduction strategies to the layer-wise characteristics of token representations. The primary methodological contribution is a bifurcated strategy: in shallow-to-middle layers, RAPID employs a redundancy-similarity aware pruning metric to eliminate over-represented local patterns. As features transition to global semantic concepts in deeper layers, the framework shifts to an importance-similarity aware merging mechanism. This stage leverages classification (CLS) token attention weights to protect semantically critical tokens while fusing less important but similar neighbors. Empirical validation on ImageNet-1K using ViT and DeiT architectures demonstrates that RAPID establishes a superior accuracy-compression Pareto frontier compared to plug-and-play baselines such as ToMe and ToFu. RAPID is particularly robust in aggressive compression regimes, achieving up to 4.29% higher accuracy than ToMe at extreme reduction rates. Our framework provides a training-free template for optimizing vision models by aligning reduction strategies with hierarchical feature evolution.

Abstract

Vision Transformer

Explore similar work

RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT

Bottlenecked semantic compression by representation reconstruction

Representative Attention For Vision Transformers