Adaptive MLP Pruning for Large Vision Transformers

Chengchao Shen

cs.CVMar 9, 2026

Adaptive MLP Pruning for Large Vision Transformers

Authors: Chengchao Shen

Organizations: Central South University

Abstract

Large vision transformers present impressive scalability, as their performance can be well improved with increased model capacity. Nevertheless, their cumbersome parameters results in exorbitant computational and memory demands. By analyzing prevalent transformer structures, we find that multilayer perceptron (MLP) modules constitute the largest share of the model's parameters. In this paper, we propose an Adaptive MLP Pruning (AMP) method to substantially reduce the parameters of large vision transformers without obvious performance degradation. First, we adopt Taylor based method to evaluate neuron importance of MLP. However, the importance computation using one-hot cross entropy loss ignores the potential predictions on other categories, thus degrading the quality of the evaluated importance scores. To address this issue, we introduce label-free information entropy criterion to fully model the predictions of the original model for more accurate importance evaluation. Second, we rank the hidden neurons of MLP by the above importance scores and apply binary search algorithm to adaptively prune the ranked neurons according to the redundancy of different MLP modules, thereby avoiding the predefined compression ratio. Experimental results on several state-of-the-art large vision transformers, including CLIP and DINOv2, demonstrate that our method achieves roughly 40% parameter and FLOPs reduction in a near lossless manner. Moreover, when the models are not finetuned after pruning, our method outperforms other pruning methods by significantly large margin. The source code and trained weights are available at https://github.com/visresearch/AMP.

Figures & tables

Figure 1: The overview of the proposed method. First, the importance scores of hidden neurons are evaluated by Taylor based method. Then, we rank the hidden neurons by the obtained importance scores. Afterwards, we conduct binary search to adaptively prune the hidden neurons for MLP modules in transformer. Finally, the pruned model is guided by the original model using knowledge distillation to recover performance.

Figure 2: One-hot cross entropy vs information entropy for neuron importance evaluation. Our proposed information entropy exploits all predictions of the model for more accurate importance evaluation.

Figure 3: Adaptive MLP Pruning. In each pruning step, we conduct binary search algorithm to adaptively reduce the search range of optimal hidden size into half according to information entropy $\mathcal{E}$ , until the maximum pruning step number reaches. If the increment of information entropy within the range of $\Delta\mathcal{E}$ , we further prune the hidden neurons of MLP. Otherwise, we reduce the number of pruned neurons in the previous step.

Method	#Params	FLOPs	Throughout	IN-1K	IN-Adv	IN-R	IN-V2	IN-Ske	ObjectNet	Avg. Acc.
OpenCLIP-g ( Cherti et al., 2023 )	1.01B	0.52T	150.8 imgs/s	78.5%	60.8%	90.2%	71.7%	67.5%	69.2%	73.0%
OpenCLIP-g (ours, prune)	0.62B	0.32T	222.3 imgs/s	65.0%	34.0%	74.4%	57.7%	43.9%	47.9%	53.8%
OpenCLIP-g (ours, distill)	0.62B	0.32T	222.3 imgs/s	78.4%	60.8%	90.5 %	71.5%	67.9%	69.4%	73.1%
OpenCLIP-G ( Wortsman, 2023 )	1.84B	0.95T	90.6 imgs/s	80.1%	69.3%	92.1%	73.6%	68.9%	73.0%	76.2%
OpenCLIP-G (ours, prune)	1.17B	0.60T	135.8 imgs/s	70.2%	37.1%	78.3%	62.7%	50.9%	51.1%	58.4%
OpenCLIP-G (ours, distill)	1.17B	0.60T	135.8 imgs/s	80.1%	68.4%	91.9%	73.4%	68.9%	72.8%	75.9%

Table 1: The performance comparison on zero-shot image classification tasks of various ImageNet variants and ObjectNet. “#Params” denotes the number of vision encoder’s parameters, excluding the parameters of text encoder. Throughout is averaged over 10 runs on single A6000 GPU with batch size of 1000. “prune” and “distill” indicate the pruned model without finetuning and the distilled one, respectively.

		Zero-Shot Text Retrieval (text $\rightarrow$ image)						Zero-Shot Image Retrieval (image $\rightarrow$ text)
Method	#Params	Flickr30K			COCO			Flickr30K			COCO			MR
Method	#Params	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	MR
OpenCLIP-g ( Cherti et al., 2023 )	1.01B	91.5	98.9	99.5	66.4	86.0	91.8	77.5	94.1	96.7	48.7	73.2	81.4	83.8
OpenCLIP-g (ours, prune)	0.62B	83.4	95.7	97.7	56.6	79.3	86.3	69.5	88.8	93.2	41.6	66.6	76.3	77.9
OpenCLIP-g (ours, distill)	0.62B	91.6	99.1	99.6	66.2	86.5	91.9	77.5	94.1	96.7	49.3	73.9	82.0	84.0
OpenCLIP-G ( Wortsman, 2023 )	1.84B	92.6	99.4	99.8	66.9	87.2	92.8	79.8	95.1	97.0	51.3	74.8	82.9	85.0

Table 2: The performance comparison on zero-shot retrieval task of Flickr30K and COCO datasets. “R@K” is short for Recall@K. “MR” is short for mean recall, which is average value of all recall metrics.

	OpenCLIP-g		EVA-CLIP-E
Method	prune	distill	prune	distill
original	-	73.0%	-	80.9%
random pruning	0.4%	67.4%	0.4%	75.8%
$\ell_{2}$ norm pruning	1.6%	69.3%	0.7%	78.3%
Taylor pruning ( Molchanov et al., 2017 )	9.6%	70.6%	1.5%	79.1%
SAViT ( Zheng et al., 2022 )	10.3%	70.9%	2.1%	79.3%

Table 3: Performance comparison of different pruning strategies on average zero-shot classification accuracy of ImageNet variants and ObjectNet. “original” denotes the original model without compression.

Method	#Params	FLOPs	kNN (%)
OpenCLIP-g ( Cherti et al., 2023 )	1.01B	0.52T	81.7
OpenCLIP-g (ours, prune)	0.62B	0.32T	72.8
OpenCLIP-g (ours, distill)	0.62B	0.32T	81.7
OpenCLIP-G ( Wortsman, 2023 )	1.84B	0.95T	82.9
OpenCLIP-G (ours, prune)	1.17B	0.60T	74.7
OpenCLIP-G (ours, distill)	1.17B	0.60T	82.8

Table 4: The performance comparison on ImageNet-1K using kNN evaluation protocol. The outputs of the last transformer block are used to evaluate the kNN performance.

Criterion	#Params	IN-1K	IN-Adv	IN-R	IN-V2	IN-Ske	ObjectNet	Avg. Acc.
Cross Entropy	0.65B	60.2%	30.8%	69.6%	53.9%	40.3%	44.5%	50.0%
Information Entropy (ours)	0.62B	65.0%	34.0%	74.4%	57.7%	43.9%	47.9%	53.8%

Table 5: The effect of different criteria on zero-shot classification tasks.

Criterion	IN-1K	IN-Adv	IN-R	IN-V2	IN-Ske	ObjectNet	Avg. Acc.
Taylor Pruning	8.2%	2.9%	14.5%	5.8%	2.7%	9.6%	7.3%
Binary Search (ours)	65.0%	34.0%	74.4%	57.7%	43.9%	47.9%	53.8%

Table 6: The effect of binary search on zero-shot classification tasks.

Threshold $\Delta\mathcal{E}$	$1\times 10^{-4}$	$1\times 10^{-3}$	$5\times 10^{-3}$	$1\times 10^{-2}$	$2\times 10^{-2}$	$4\times 10^{-2}$	$1\times 10^{-1}$
#Params	0.73B	0.72B	0.69B	0.66B	0.61B	0.54B	0.43B
Avg. Acc.	62.5%	61.5%	57.6%	53.8%	48.5%	41.7%	31.3%

Table 7: The effect of entropy threshold on zero-shot classification tasks.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model	dist. strategy	temperature $\tau$	$\Delta\mathcal{E}$	batch_size	base_lr	min_lr	beta1	beta2
OpenCLIP-g	DDP	$\frac{1}{15}$	0.01	512	$5\times 10^{-5}$	$1\times 10^{-7}$	0.90	0.95
OpenCLIP-G	DDP	$\frac{1}{15}$	$2\times 10^{-4}$	512	$2\times 10^{-5}$	$1\times 10^{-6}$	0.90	0.95
EVA-CLIP-E	FSDP	$\frac{1}{20}$	$2\times 10^{-3}$	512	$1\times 10^{-5}$	$1\times 10^{-6}$	0.90	0.95
EVA-CLIP-8B	FSDP	$\frac{1}{20}$	0.02	160	$1\times 10^{-5}$	$1\times 10^{-6}$	0.90	0.95
DINOv2-g	DDP	$\frac{1}{20}$	0.01	512	$7\times 10^{-6}$	$1\times 10^{-7}$	0.90	0.95

Appendix

Table 8: The hyperparameters of our method. “DDP” and “FSDP” denote distributed data parallel strategy and fully sharded data parallel strategy, respectively.

Figure 4: The relation between MLP hidden size and information entropy.

Explore similar work

Oct 10, 2026

Retraining-Free Depth Pruning of Vision Transformers: A Cross Domain Study of Layer Redundancy

Transformer architectures have achieved strong results in natural language processing and computer vision, but their deployment is limited by high computational, memory and storage costs, motivating research into model compression techniques, such as pruning, while preserving performance. Prior work has shown substantial redundancy in large language models (LLMs), and redundancy in vision transformers has also been investigated, but it remains less systematically analyzed than in LLMs. In this work, we extend similarity-based redundancy analysis to vision models by measuring cosine similarity between layer inputs and outputs to identify transformations with minimal impact. We study redundancy patterns across several transformer architectures and find that deep vision transformers such as DINOv2 and SwinV2, like language models, exhibit significant redundancy. We then evaluate how this redundancy can be exploited to improve efficiency, analyzing the trade-off between inference speed and performance degradation under different pruning strategies. Our results show that moderate pruning can yield substantial acceleration with limited impact on accuracy, and that 20% of the DINOv2 blocks can be removed without any retraining, for a 1.24x throughput gain and a 20% lower energy per image at a relative accuracy loss below 1% on ImageNet-1K. We further propose a model selection pipeline that automatically identifies the best pruned configuration for a given dataset, reducing FLOPs by 13% to 27% on average across architectures with an average accuracy change between -0.3 and +1.0 points.

Alessandro Viespoli, Loris Nanni

University of Padua

Efficient ViTs Neural Network Pruning

Ask PDF

Mar 11, 2026

Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Models

Large Vision-Language Models (LVLMs) incur substantial inference costs due to the processing of a vast number of visual tokens. Existing methods typically struggle to model progressive visual token reduction as a multi-step decision process with sequential dependencies and often rely on hand-engineered scoring rules that lack adaptive optimization for complex reasoning trajectories. To overcome these limitations, we propose TPRL, a reinforcement learning framework that learns adaptive pruning trajectories through language-guided sequential optimization tied directly to end-task performance. We formulate visual token pruning as a sequential decision process with explicit state transitions and employ a self-supervised autoencoder to compress visual tokens into a compact state representation for efficient policy learning. The pruning policy is initialized through learning from demonstrations and subsequently fine-tuned using Proximal Policy Optimization (PPO) to jointly optimize task accuracy and computational efficiency. Our experimental results demonstrate that TPRL removes up to 66.7% of visual tokens and achieves up to a 54.2% reduction in FLOPs during inference while maintaining a near-lossless average accuracy drop of only 0.7%. Code is released at mypink{https://github.com/MagicVicCoder/TPRL}.

Sihan Cao, Jianwei Zhang, Pengcheng Zheng +7

School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Robotics and Advanced Manufacture, Harbin Institute of Technology, Shenzhen, China · College of Information and Control Engineering, Xi’an University of Architecture and Technology, Xi’an, China

Efficient VLM Inference Visual Token Pruning Visual Token Compression RL for VLMs

Ask PDF

Apr 14, 2026

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) suffer from substantial computational overhead due to the high redundancy in visual token sequences. Existing approaches typically address this issue using single-layer Vision Transformer (ViT) features and static pruning strategies. However, such fixed configurations are often brittle under diverse instructions. To overcome these limitations, we propose CLASP, a plug-and-play token reduction framework based on class-adaptive layer fusion and dual-stage pruning. Specifically, CLASP first constructs category-specific visual representations through multi-layer vision feature fusion. It then performs dual-stage pruning, allocating the token budget between attention-salient pivot tokens for relevance and redundancy-aware completion tokens for coverage. Through class-adaptive pruning, CLASP enables prompt-conditioned feature fusion and budget allocation, allowing aggressive yet robust visual token reduction. Extensive experiments demonstrate that CLASP consistently outperforms existing methods across a wide range of benchmarks, pruning ratios, and MLLM architectures. Code will be available at https://github.com/Yunkaidang/CLASP.

Yunkai Dang, Yizhu Jiang, Yifan Jiang +4

School of Artificial Intelligence Science and Technology, Nanjing University.

Visual Token Pruning Multimodal Large Language Models Multimodal Token Compression

Ask PDF