Retraining-Free Depth Pruning of Vision Transformers: A Cross Domain Study of Layer Redundancy

Alessandro Viespoli; Loris Nanni

cs.CVOct 10, 2026

Retraining-Free Depth Pruning of Vision Transformers: A Cross Domain Study of Layer Redundancy

Authors: Alessandro Viespoli, Loris Nanni

Organizations: University of Padua

Abstract

Transformer architectures have achieved strong results in natural language processing and computer vision, but their deployment is limited by high computational, memory and storage costs, motivating research into model compression techniques, such as pruning, while preserving performance. Prior work has shown substantial redundancy in large language models (LLMs), and redundancy in vision transformers has also been investigated, but it remains less systematically analyzed than in LLMs. In this work, we extend similarity-based redundancy analysis to vision models by measuring cosine similarity between layer inputs and outputs to identify transformations with minimal impact. We study redundancy patterns across several transformer architectures and find that deep vision transformers such as DINOv2 and SwinV2, like language models, exhibit significant redundancy. We then evaluate how this redundancy can be exploited to improve efficiency, analyzing the trade-off between inference speed and performance degradation under different pruning strategies. Our results show that moderate pruning can yield substantial acceleration with limited impact on accuracy, and that 20% of the DINOv2 blocks can be removed without any retraining, for a 1.24x throughput gain and a 20% lower energy per image at a relative accuracy loss below 1% on ImageNet-1K. We further propose a model selection pipeline that automatically identifies the best pruned configuration for a given dataset, reducing FLOPs by 13% to 27% on average across architectures with an average accuracy change between -0.3 and +1.0 points.

Figures & tables

Dataset	#Classes	#Samples	Format	Image Size	Protocol
ImageNet-1K	1000	50,000	RGB	variable	-
CIFAR-10	10	60,000	RGB	$32\times 32$	OG
LCZ42	17	400,673	RGB	$32\times 32$	OG
CrossD	44	94,116	RGB	$256\times 256$	OG
Zoolake	35	17,943	RGB	variable	70:15:15
Laryngeal	4	1,320	RGB	$100\times 100$	OG

Table 1: Summary of the datasets used in this study.

	Joint			Attn			MLP			Block
Architecture	Drop 4	Drop 8	Drop 12	Drop 4	Drop 8	Drop 12	Drop 4	Drop 8	Drop 12	Drop 4	Drop 8	Drop 12
DINOv2	0.569	0.151	0.151	0.603	0.192	0.603	0.003	0.003	0.003	1.000	1.000	1.000
DINOv3	0.015	0.024	0.024	0.001	0.001	0.001	0.005	0.002	0.001	0.001	0.001	0.001
ViT	0.259	0.424	0.192	0.328	0.010	0.001	0.259	0.424	0.015	0.791	0.054	0.001
SwinV2	0.950	0.950	0.547	1.000	1.000	1.000	0.010	0.004	0.004	0.038	0.128	0.622

Table 2: Holm-adjusted

p

-values of the Wilcoxon signed-rank test between pruning methods and the baseline across 4 architectures. At drop counts 4 and 8, Joint coincides with Attn on DINOv2 and SwinV2 and with MLP on ViT; their adjusted values differ only because the Holm correction also includes drop count 12.

Arch	Method	Drop	HL (pp)	90% CI	Outcome
DINOv2	Block	4	$-0.04$	$[-0.25,+0.17]$	equivalent
DINOv2	Block	8	$+0.01$	$[-0.94,+0.61]$	equivalent
DINOv2	Attn	4	$+0.05$	$[-0.16,+0.82]$	equivalent
DINOv2	Attn	8	$+0.58$	$[+0.06,+1.93]$	equivalent
DINOv2	Joint	4	$+0.05$	$[-0.16,+0.82]$	equivalent
DINOv2	Joint	8	$+0.58$	$[+0.06,+1.93]$	equivalent

Table 3: Conclusive outcomes of the equivalence analysis at drop counts 4, 8 and 12 over the 12 downstream datasets. HL is the Hodges-Lehmann estimate of the median paired accuracy difference (pruned

-

baseline, percentage points) with its 90% confidence interval. A configuration is equivalent when the two one-sided Wilcoxon tests against the pre-specified margin of

\pm 2

points both reject at

p<0.05

. Otherwise, it is superior when its interval lies entirely above zero and the Holm-adjusted two-sided test is significant, and inferior in the symmetric case below zero. Inconclusive configurations are reported in Appendix D .

		DINOv2					DINOv3					SwinV2					ViT
Method	Drop	Avg	Speedup ( $\uparrow$ )	$\gamma$ ( $\downarrow$ )	GFLOPs	Energy	Avg	Speedup ( $\uparrow$ )	$\gamma$ ( $\downarrow$ )	GFLOPs	Energy	Avg	Speedup ( $\uparrow$ )	$\gamma$ ( $\downarrow$ )	GFLOPs	Energy	Avg	Speedup ( $\uparrow$ )	$\gamma$ ( $\downarrow$ )	GFLOPs	Energy
Block	4	82.10	1.10 $\times$	$-0.013$	540	1.993	77.18	1.19 $\times$	0.473	103	0.552	81.41	1.14 $\times$	$-0.123$	35	0.610	77.29	1.43 $\times$	0.059	24	0.107
	8	81.98	1.24 $\times$	0.001	480	1.772	57.07	1.48 $\times$	0.679	82	0.443	82.13	1.33 $\times$	$-0.080$	28	0.523	65.58	2.62 $\times$	0.107	12	0.058
	12	81.20	1.41 $\times$	0.023	420	1.556	53.24	1.96 $\times$	0.389	62	0.335	80.67	1.50 $\times$	$-0.016$	22	0.463	14.29	14.71 $\times$	0.060	0.5	0.008
	16	81.01	1.64 $\times$	0.019	360	1.338	46.48	2.88 $\times$	0.241	41	0.227	71.52	1.85 $\times$	0.125	15	0.376	–	–	–	–	–
	28	51.94	3.22 $\times$	0.165	181	0.684	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–

Table 5: Efficiency of pruned models across the 4 architectures. Avg is the mean accuracy over the 12 downstream datasets, Speedup is the throughput relative to the unpruned baseline,

\gamma

is the SDR, and Energy is the energy per image in joules, computed as the average board power during the throughput runs divided by the throughput. The unpruned baselines reach an average accuracy of 81.99% for DINOv2, 84.92% for DINOv3, 80.02% for SwinV2 and 79.31% for ViT, with 599.57, 123.11, 41.66 and 35.43 GFLOPs and 2.201, 0.658, 0.693 and 0.155 J per image respectively. Configurations that coincide by construction (Joint with Attn or MLP at 4 and 8 removals) were benchmarked separately, and their speedup differences of about 1% reflect measurement noise. Dashes mark drop counts not available for a backbone.

Dataset	DINOv2					DINOv3					SwinV2					ViT
Dataset	Unpruned (%)	Selected (%)	Method	Drop	FLOPs $\downarrow$ (%)	Unpruned (%)	Selected (%)	Method	Drop	FLOPs $\downarrow$ (%)	Unpruned (%)	Selected (%)	Method	Drop	FLOPs $\downarrow$ (%)	Unpruned (%)	Selected (%)	Method	Drop	FLOPs $\downarrow$ (%)
CIFAR-10	99.32	99.30	Block †	4	9.98	99.14	99.14	baseline	0	0.00	97.38	97.38	baseline	0	0.00	93.75	93.75	baseline	0	0.00
Zoolake	95.77	96.06	MLP †	8	12.94	95.62	95.62	baseline	0	0.00	93.57	94.17	MLP	4	10.31	93.31	93.68	MLP	4	20.99
Laryngeal	98.48	99.24	Joint †	24	28.45	97.35	97.35	baseline	0	0.00	95.83	98.48	MLP	8	20.62	91.29	95.45	Block	4	32.83
Pest	98.90	97.24	Joint †	16	16.25	97.51	97.51	baseline	0	0.00	87.85	93.37	MLP	8	20.62	95.86	95.86	baseline	0	0.00
InfLarynge	91.67	96.25	Joint †	40	50.64	93.33	95.00	Joint	20	42.78	95.83	96.25	Joint	24	46.07	95.42	95.42	MLP	4	20.99

Table 6: Model selection results on the 12 downstream datasets. For each architecture, Unpruned is the test accuracy of the unpruned model and Selected the test accuracy of the configuration selected on the validation split, both trained with the third-phase protocol, followed by the method, drop count and FLOPs reduction of the selected configuration. Baseline marks datasets on which no pruned configuration exceeded the unpruned model on the validation split, and

\dagger

marks the model returned by the full pipeline.

Explore similar work

Jul 4, 2026

Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective

While prior studies have successfully compressed vision Transformers (ViTs) through various pruning techniques, most have concentrated on width pruning to achieve significant reductions in model size. Depth pruning, which removes entire layers from a ViT, is notoriously difficult for accuracy recovery despite its potential to deliver higher speedups, limiting the acceleration achieved by existing joint width-and-depth pruning methods. In this work, we reveal that the failure of existing depth pruning methods lies in their neglect of heterogeneity between different layers, and we introduce HetDPT, a heterogeneity-aware depth pruning method that avoids dimension mismatch. Comprehensive experiments on ImageNet-1K, CIFAR-100, COCO, and ADE20K validate our method: HetDPT achieves a 1.58

\times

speedup for DeiT-B while maintaining accuracy and a 1.39

\times

speedup for DeiT-S with nearly no accuracy degradation. Furthermore, when combined with width pruning, HetDPT+ sets a new state-of-the-art record in extreme ViT pruning, enhancing the acceleration ratio from 4.24

\times

to 5.19

\times

for the Isomorphic-Pruning-2.6G configuration while maintaining near-lossless accuracy; our code is available at https://github.com/Efficient-AI-for-All/HetDPT.

Zhenfeng Su, Kang Zhao, Han Bao +4

School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · Huawei Technologies Co., Ltd., Shenzhen, China · Tsinghua University, Beijing, China +1

Vision Transformer Efficient ViTs Neural Network Pruning

Ask PDF

Oct 13, 2026

Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression

Vision Transformers achieve strong performance but incur high computational costs from quadratic token interactions. Training-free token reduction methods improve efficiency, but their robustness remains underexplored. We propose Prune-and Merge ViT, a training-free framework for off-the-shelf models. It combines class attention-based pruning with similarity-based merging to reduce redundancy while preserving semantic information. To evaluate robustness we introduce two metrics. Relative Accuracy Drop (RAD) measures relative accuracy degradation under transformations, and Prediction Flip Rate (PFR) quantifies prediction consistency. On pretrained ViT and DeiT, Prune-and-Merge reduces computation competitively while maintaining robustness and better accuracy than existing token reduction methods. Our analysis suggests that hybrid token reduction does not introduce hallucination-like behavior and preserves prediction stability for common image transformations.

Vatsal, Soumil Malhotra, Priyanshu Chahal +1

Department of Mathematics, Indian Institute of Technology Roorkee

Vision Transformer Efficient ViTs Visual Token Compression Visual Token Pruning Visual Token Merging

Ask PDF

Jun 6, 2026

RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT

Vision Transformers (ViTs) achieve strong performance but suffer from high computational costs due to quadratic self-attention complexity. Although token reduction techniques such as pruning and merging mitigate this, they typically overlook how representations evolve across network depth. We propose RAPID, a depth-aware token reduction framework that adapts reduction strategies to the layer-wise characteristics of token representations. The primary methodological contribution is a bifurcated strategy: in shallow-to-middle layers, RAPID employs a redundancy-similarity aware pruning metric to eliminate over-represented local patterns. As features transition to global semantic concepts in deeper layers, the framework shifts to an importance-similarity aware merging mechanism. This stage leverages classification (CLS) token attention weights to protect semantically critical tokens while fusing less important but similar neighbors. Empirical validation on ImageNet-1K using ViT and DeiT architectures demonstrates that RAPID establishes a superior accuracy-compression Pareto frontier compared to plug-and-play baselines such as ToMe and ToFu. RAPID is particularly robust in aggressive compression regimes, achieving up to 4.29% higher accuracy than ToMe at extreme reduction rates. Our framework provides a training-free template for optimizing vision models by aligning reduction strategies with hierarchical feature evolution.

Kyumin Choi, Ikbeom Jang

Hankuk University of Foreign Studies Seoul, South Korea

Vision Transformer Efficient ViTs Visual Token Merging Visual Token Pruning Image Classification Visual Token Compression

Ask PDF