Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression

Vatsal; Soumil Malhotra; Priyanshu Chahal; Sanjeev Kumar

cs.CVOct 13, 2026

Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression

Authors: Vatsal, Soumil Malhotra, Priyanshu Chahal, Sanjeev Kumar

Organizations: Department of Mathematics, Indian Institute of Technology Roorkee

Abstract

Vision Transformers achieve strong performance but incur high computational costs from quadratic token interactions. Training-free token reduction methods improve efficiency, but their robustness remains underexplored. We propose Prune-and Merge ViT, a training-free framework for off-the-shelf models. It combines class attention-based pruning with similarity-based merging to reduce redundancy while preserving semantic information. To evaluate robustness we introduce two metrics. Relative Accuracy Drop (RAD) measures relative accuracy degradation under transformations, and Prediction Flip Rate (PFR) quantifies prediction consistency. On pretrained ViT and DeiT, Prune-and-Merge reduces computation competitively while maintaining robustness and better accuracy than existing token reduction methods. Our analysis suggests that hybrid token reduction does not introduce hallucination-like behavior and preserves prediction stability for common image transformations.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the Prune-and-Merge ViT architecture. The framework processes tokens through a dual-stage reduction at layers $l\in\{3,6,9\}$ . Stage 1 (pruning) filters out background tokens based on class-attention scores $k_{p}$ , while Stage 2 (merging) aggregates redundant salient tokens using bipartite matching $k_{m}$ .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	DeiT-Tiny				DeiT-Small				DeiT-Base
	Baseline: 88.42				Baseline: 92.62				Baseline: 93.64
Keep Rates	0.9	0.7	0.5	0.25	0.9	0.7	0.5	0.25	0.9	0.7	0.5	0.25
Random Drop	87.94	84.02	75.22	33.18	92.06	90.52	85.00	50.34	92.68	90.62	82.14	25.86
EViT	88.44	86.42	80.34	52.54	92.46	91.80	89.02	70.16	93.76	93.06	89.00	51.32
ToMe	88.44	86.70	82.92	–	92.42	91.54	89.96	–	93.26	92.00	85.30	–
Top-K	88.48	86.28	80.14	52.02	92.46	91.60	89.16	68.78	93.74	93.18	88.86	53.04

Appendix

Table 1: Top-1 Accuracy comparison for DeiT and ViT models under different keep rates.

	DeiT-Tiny				DeiT-Small				DeiT-Base
Keep Rates	0.9	0.7	0.5	0.25	0.9	0.7	0.5	0.25	0.9	0.7	0.5	0.25
Balanced	88.06	86.96	84.00	67.62	92.30	91.86	90.60	79.58	93.56	92.58	89.72	47.88
Pruning-heavy	88.28	86.60	82.66	68.00	92.26	91.96	90.40	79.76	93.44	92.96	90.24	52.44
Merging-heavy	88.00	86.88	84.38	74.98	92.28	92.06	90.26	84.84	93.46	92.70	88.70	68.58

Appendix

Table 2: Balanced vs pruning-heavy vs merging-heavy hybrid strategies (Accuracy %)

	DeiT-Tiny				DeiT-Small				DeiT-Base
	Baseline: 1.079				Baseline: 4.250				Baseline: 16.867
Keep Rates	0.9	0.7	0.5	0.25	0.9	0.7	0.5	0.25	0.9	0.7	0.5	0.25
EViT	0.989	0.752	0.576	0.432	3.817	2.892	2.203	1.634	14.985	11.336	8.608	6.349
ToMe	0.983	0.746	0.571	–	3.787	2.863	2.178	–	14.865	11.216	8.507	–
Top-K	0.981	0.744	0.570	0.428	3.786	2.862	2.177	1.618	14.863	11.215	8.506	6.287
Hybrid (Ours)	0.992	0.753	0.577	0.429	3.823	2.893	2.204	1.624	15.005	11.337	8.608	6.309

Appendix

Table 3: GFLOPs comparison for DeiT and ViT models under different keep rates.

Model	0.25	0.50	0.70	0.90
ViT-Base	0.3611	0.0422	0.0099	0.0022
ViT-Tiny	0.4105	0.0943	0.0272	0.0059

Appendix

Table 4: Average

\Delta

PFR across augmentations.

Model	Comparison	0.25	0.50	0.70	0.90
Tiny	Raw	0.073	0.023	0.003	$-$ 0.004
	ToMe	–	$-$ 0.003	$-$ 0.003	$-$ 0.003
	EViT	$-$ 0.030	$-$ 0.002	0.000	$-$ 0.003
Base	Raw	0.045	0.009	0.001	$-$ 0.001
	ToMe	–	0.002	$-$ 0.001	$-$ 0.001
	EViT	$-$ 0.026	0.000	0.001	$-$ 0.002

Appendix

Table 5: Relative Accuracy Drop (RAD).

	DeiT-Tiny				DeiT-Small				DeiT-Base
	Baseline: 1.629				Baseline: 3.092				Baseline: 12.772
Keep Rates	0.9	0.7	0.5	0.25	0.9	0.7	0.5	0.25	0.9	0.7	0.5	0.25
EViT	1.241	0.984	0.751	0.747	3.256	2.390	1.763	1.253	11.559	8.771	6.728	5.058
ToMe	1.042	0.888	0.900	–	2.952	2.217	1.621	–	10.437	7.958	6.092	–
Top-K	1.156	0.903	0.729	0.633	2.892	2.138	1.569	1.311	10.107	7.660	5.845	4.421
Hybrid (Ours)	1.110	1.116	1.249	1.107	2.930	2.165	1.628	1.193	10.189	7.626	5.885	4.532

Appendix

Table 6: Latency comparison (ms/img)

	DeiT-Tiny				DeiT-Small				DeiT-Base
	Baseline: 613.92				Baseline: 323.46				Baseline: 78.30
Keep Rates	0.9	0.7	0.5	0.25	0.9	0.7	0.5	0.25	0.9	0.7	0.5	0.25
EViT	805.73	1016.41	1330.76	1338.06	307.11	418.34	567.28	797.83	86.51	114.02	148.62	197.69
ToMe	960.08	1126.47	1110.56	–	338.74	451.12	616.72	–	95.81	125.66	164.14	–
Top-K	864.74	1107.43	1371.64	1579.34	345.74	467.62	637.31	762.66	98.94	130.56	171.08	226.19
Hybrid (Ours)	917.30	895.66	800.48	903.08	341.34	461.98	614.11	838.28	98.14	131.13	169.92	220.63

Appendix

Table 7: Throughput comparison (imgs/sec)

Model	Augmentation	Raw	0.25	0.50	0.70	0.90
ViT-Base	Rotation	0.0472	0.4528	0.0938	0.0552	0.0458
	Color Jitter	0.0128	0.3552	0.0524	0.0250	0.0162
	Horizontal Flip	0.0112	0.3464	0.0516	0.0206	0.0130
ViT-Tiny	Rotation	0.1856	0.6086	0.2902	0.2162	0.1898
	Color Jitter	0.0416	0.4516	0.1394	0.0698	0.0498
	Horizontal Flip	0.0356	0.4342	0.1160	0.0584	0.0408

Appendix

Table 8: Prediction Flip Rate (PFR) for raw and hybrid models across augmentations and keep rates.

Model	Augmentation	0.25	0.50	0.70	0.90
ViT-Base	Rotation	0.4056	0.0466	0.0080	0.0014
	Color Jitter	0.3424	0.0396	0.0122	0.0034
	Horizontal Flip	0.3352	0.0404	0.0094	0.0018
ViT-Tiny	Rotation	0.4230	0.1046	0.0306	0.0042
	Color Jitter	0.4100	0.0978	0.0282	0.0082
	Horizontal Flip	0.3986	0.0804	0.0228	0.0052

Appendix

Table 9: Absolute change in Prediction Flip Rate (

\Delta

PFR) between raw and hybrid models.

Explore similar work

RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT
Jun 6, 2026Kyumin Choi, Ikbeom JangVision Transformer Efficient ViTs
Hankuk University of Foreign Studies Seoul, South Korea
When Token Compression Breaks: Structural Pruning vs. Token Reduction for Robust ViT Segmentation under High Compression
Jul 2, 2026Tien-Phat Nguyen, Ngai-Man CheungDistribution Shift Robustness Vision Transformer
Temasek Laboratories, Singapore University of Technology and Design, Singapore
AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers
May 26, 2026Semi Lee, Hyejin Go, Hyesong ChoiToken Merging Self-Attention
Electronic Engineering · Soongsil University · Seoul, South Korea

Jun 6, 2026J/K move · Enter open · S save

RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT

Kyumin Choi, Ikbeom Jang

Hankuk University of Foreign Studies Seoul, South Korea

Vision Transformer Efficient ViTs Visual Token Merging Visual Token Pruning Image Classification Visual Token Compression

Open paper AskPDF arXiv

Vision Transformers (ViTs) achieve strong performance but suffer from high computational costs due to quadratic self-attention complexity. Although token reduction techniques such as pruning and merging mitigate this, they typically overlook how representations evolve across network depth. We propose RAPID, a depth-aware token reduction framework that adapts reduction strategies to the layer-wise characteristics of token representations. The primary methodological contribution is a bifurcated strategy: in shallow-to-middle layers, RAPID employs a redundancy-similarity aware pruning metric to eliminate over-represented local patterns. As features transition to global semantic concepts in deeper layers, the framework shifts to an importance-similarity aware merging mechanism. This stage leverages classification (CLS) token attention weights to protect semantically critical tokens while fusing less important but similar neighbors. Empirical validation on ImageNet-1K using ViT and DeiT architectures demonstrates that RAPID establishes a superior accuracy-compression Pareto frontier compared to plug-and-play baselines such as ToMe and ToFu. RAPID is particularly robust in aggressive compression regimes, achieving up to 4.29% higher accuracy than ToMe at extreme reduction rates. Our framework provides a training-free template for optimizing vision models by aligning reduction strategies with hierarchical feature evolution.

Abstract

Explore similar work

RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT

When Token Compression Breaks: Structural Pruning vs. Token Reduction for Robust ViT Segmentation under High Compression

AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers