Efficient ViTs

ViT: Vision Transformer

Efficient Vision Transformer research seeks to make visual models practical at high resolutions and on limited hardware without losing the accuracy needed for classification, detection, and segmentation. Common approaches replace or constrain full self-attention with linear, windowed, or hybrid attention, and reduce work through token pruning or merging, structured pruning, quantization, and adaptive early exits. Researchers evaluate these methods on benchmarks such as ImageNet, COCO, and ADE20K, as well as on edge devices and dense prediction tasks where fine details matter. A central tension is that fewer FLOPs or tokens do not always mean faster real-world inference, and aggressive compression can harm accuracy, robustness, or small-region detail.

Latest papers

243 papers29 papers in the last four weeks, up 32% on the four weeks before. 0.2% of all new papers.

Adaptive MLP Pruning for Large Vision Transformers
Mar 9, 2026Chengchao ShenNeural Network Compression Neural Network Pruning
Central South University
StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models
Mar 7, 2026Duy M. H. Nguyen, Tuan A. Tran, Duong Nguyen +17Token Merging Image Segmentation
University of Stuttgart · DFKI · Max Planck Research School +10
ButterflyViT: 354 $\times$ Expert Compression for Edge Vision Transformers
Mar 6, 2026Aryan KarmoreEfficient ViTs Vision Transformer
Indian Institute of Information Technology, Nagpur
Rank-Factorized Implicit Neural Bias: Scaling Super-Resolution Transformer with FlashAttention
Mar 6, 2026Dongheon Lee, Seokju Yun, Jaegyun Im +1Image Super-Resolution Attention Mechanisms
University of Seoul · KAIST AI
OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer
Mar 6, 2026Si-Yu Lu, Po-Ting Chen, Hui-Che Hsu +3Streaming 3D Reconstruction 3D ViTs
National Taiwan University · National Taiwan University of Science and Technology
DAGE: Dual-Stream Architecture for Efficient and Fine-Grained Geometry Estimation
Mar 4, 2026Tuan Duc Ngo, Jiahui Huang, Seoung Wug Oh +4Multi-View 3D Reconstruction Video Depth Estimation
UMass Amherst · Adobe Research · TU Crete
ATD: Improved Transformer with Adaptive Token Dictionary for Image Restoration
Mar 3, 2026Leheng Zhang, Wei Long, Yawei Li +3Image Restoration Efficient ViTs
School of Computer Science and Engineering (School of Cyber Security), University of Electronic Science and Technology of China (UESTC), China · Computer Vision Lab, ETH Zürich, Zürich, Switzerland and with the Integrated Systems Laboratory, ETH Zürich, Zürich, Switzerland
GRAD-Former: Gated Robust Attention-based Differential Transformer for Change Detection
Mar 1, 2026Durgesh Ameta, Ujjwal Mishra, Praful Hambarde +1Differential Attention Remote Sensing Change Detection
School of Computing, IIIT Una HP-177209, India · Center of Artificial Intelligence and Robotics (CAIR), IIT Mandi, Mandi, HP 175005, India
Neural Discrimination-Prompted Transformers for Efficient UHD Image Restoration and Enhancement
Mar 1, 2026Cong Wang, Jinshan Pan, Liyan Wang +2All-in-One Image Restoration Efficient ViTs
Shenzhen Campus of Sun Yat-Sen University, China · The Hong Kong Polytechnic University, Hong Kong · Nanjing University of Science and Technology, China +2
BornoViT: A Novel Efficient Vision Transformer for Bengali Handwritten Basic Characters Classification
Feb 28, 2026Rafi Hassan Chowdhury, Naimul Haque, Kaniz FatihaOptical Character Recognition Efficient ViTs
Department of Computer Science and Engineering, Islamic University of Technology, Gazipur 1704, Bangladesh
Vision-TTT: Efficient and Expressive Visual Representation Learning with Test-Time Training
Feb 28, 2026Quan Kong, Yanru Xiao, Yuhao Shen +1Visual Representation Learning Efficient ViTs
Zhejiang University, Hangzhou, China · Amazon Web Services, USA
Random Wins All: Rethinking Grouping Strategies for Vision Tokens
Feb 28, 2026Qihang Fan, Yuang Ai, Huaibo Huang +1Vision Transformer Efficient ViTs
MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
RAViT: Resolution-Adaptive Vision Transformer
Feb 27, 2026Martial Guidez, Stefan Duffner, Christophe GarciaEfficient ViTs Vision Transformer
INSA Lyon, CNRS, Université Claude Bernard Lyon 1, LIRIS, UMR5205, 69621 Villeurbanne, France
UTPTrack: Towards Simple and Unified Token Pruning for Visual Tracking
Feb 27, 2026Hao Wu, Xudong Wang, Jialiang Zhang +5Visual Object Tracking Visual Token Pruning
Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · Shanghai Jiao Tong University +2
Self-Attention And Beyond the Infinite: Towards Linear Transformers with Infinite Self-Attention
Feb 26, 2026Giorgio Roffo, Hazem Abdelkawy, Nilli Lavie +1Self-Attention Efficient ViTs
Equixly API Security, Italy · Toyota Motor Europe, Belgium · MindVisionLabs, UK +2
Scaling Vision Transformers: Evaluating DeepSpeed for Image-Centric Workloads
Feb 24, 2026Huy Trinh, Rebecca Ma, Zeqi Yu +1Vision Transformer Efficient ViTs
Electrical & Computer Eng University of Waterloo
Le-DETR: Revisiting Real-Time Detection Transformer with Efficient Encoder Design
Feb 24, 2026Jiannan Huang, Aditya Kane, Fengzhe Zhou +2Real-Time Object Detection Efficient Object Detection
Beijing Jiaotong University
Token-UNet: A New Case for Transformers Integration in Efficient and Interpretable 3D UNets for Brain Imaging Segmentation
Feb 23, 2026Louis Fabrice Tshimanga, Andrea Zanola, Federico Del Pup +13D Medical Image Segmentation Hybrid CNN-Transformer Architectures
A Computationally Efficient Multidimensional Vision Transformer
Feb 23, 2026Alaa El Ichi, Khalide JbilouVision Transformer Efficient ViTs
Universit´e du Littoral Cote d’Opale, LMPA, 50 rue F. Buisson, 62228 Calais-Cedex, France.
ZACH-ViT: Regime-Dependent Inductive Bias in Compact Vision Transformers for Medical Imaging
Feb 20, 2026Athanasios AngelakisVision Transformer Efficient ViTs
BioML Lab, RI CODE, UniBw, Munich, Germany · EDS, Amsterdam UMC, Amsterdam, Netherlands
RefineFormer3D: Efficient 3D Medical Image Segmentation via Adaptive Multi-Scale Transformer with Cross Attention Fusion
Feb 18, 2026Kavyansh Tyagi, Vishwas Rathi, Puneet Goyal3D Medical Image Segmentation 3D ViTs
National Institute of Technology Kurukshetra Department of Electronics and Communication Engineering · National Institute of Technology Kurukshetra Department of Computer Engineering · Indian Institute of Technology Ropar Department of Computer Science and Engineering
AFFMAE: Scalable Vision Pre-Training for High-Resolution Microscopy Segmentation on Desktop Hardware
Feb 18, 2026David Smerkous, Zian Wang, Behzad NajafianMasked Autoencoders Self-Supervised Pre-Training
Department of Laboratory Medicine & Pathology, University of Washington, Seattle WA, USA · Electrical Engineering & Computer Science, Oregon State University, Corvallis, OR, USA · Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle WA, USA
ToaSt: Token Channel Selection and Structured Pruning for Efficient ViT
Feb 17, 2026Hyunchan Moon, Cheonjun Park, Steven L. WaslanderChannel Pruning Efficient ViTs
LG Electronics, Seoul, Republic of Korea · Hankuk University of Foreign Studies, Yongin, Republic of Korea · University of Toronto, Toronto, Canada
VariViT: A Vision Transformer for Variable Image Sizes
Feb 16, 2026Aswathi Varma, Suprosanna Shit, Chinmay Prabhakar +5Vision Transformer Medical Image Representation Learning
Department of Neuroradiology, Technical University of Munich · Institute for Artificial Intelligence and Informatics in Medicine, Technical University of Munich · Department of Quantitative Biomedicine, University of Zurich +1
From Local Windows to Adaptive Candidates via Individualized Exploratory: Rethinking Attention for Image Super-Resolution
Jan 13, 2026Chunyu Meng, Wei Long, Shuhang GuEfficient ViTs Sparse Attention
University of Electronic Science and Technology of China
Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers
Jul 21, 2025Ian Chuang, Jinyu Zou, Andrew Lee +2Visuomotor Policy Learning Efficient ViTs
University of California, Berkeley · 2Tongji University · University of California, Davis
DETRPose: Real-Time End-to-End Multi-Person Pose Estimation via Modified Transformer Decoder and Novel Denoising Keypoints
Jun 16, 2025Sebastian Janampa, Marios PattichisDetection Transformer Efficient ViTs
The University of New Mexico
GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers
Jun 13, 2025Guang Liang, Xinyao Liu, Jianxin WuQuantization-Aware Training 4-Bit Quantization
State Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · University of Science and Technology of China, Hefei, China
DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Foundation Models
Jun 12, 2025Shicheng Yin, Kaixuan Yin, Yang Liu +2Vision Foundation Models Efficient ViTs
Sun Yat-sen University, China
Fast and Accurate Image Restoration and Generation with Rank Enhanced Linear Attention
May 22, 2025Yuang AiEfficient ViTs Linear Attention
University of Chinese Academy of Sciences, Beijing, China

Mar 9, 2026J/K move · Enter open · S save

Adaptive MLP Pruning for Large Vision Transformers

Chengchao Shen

Central South University

Neural Network Compression Neural Network Pruning Efficient ViTs Vision Transformer

Open paper AskPDF arXiv

Large vision transformers present impressive scalability, as their performance can be well improved with increased model capacity. Nevertheless, their cumbersome parameters results in exorbitant computational and memory demands. By analyzing prevalent transformer structures, we find that multilayer perceptron (MLP) modules constitute the largest share of the model's parameters. In this paper, we propose an Adaptive MLP Pruning (AMP) method to substantially reduce the parameters of large vision transformers without obvious performance degradation. First, we adopt Taylor based method to evaluate neuron importance of MLP. However, the importance computation using one-hot cross entropy loss ignores the potential predictions on other categories, thus degrading the quality of the evaluated importance scores. To address this issue, we introduce label-free information entropy criterion to fully model the predictions of the original model for more accurate importance evaluation. Second, we rank the hidden neurons of MLP by the above importance scores and apply binary search algorithm to adaptively prune the ranked neurons according to the redundancy of different MLP modules, thereby avoiding the predefined compression ratio. Experimental results on several state-of-the-art large vision transformers, including CLIP and DINOv2, demonstrate that our method achieves roughly 40% parameter and FLOPs reduction in a near lossless manner. Moreover, when the models are not finetuned after pruning, our method outperforms other pruning methods by significantly large margin. The source code and trained weights are available at https://github.com/visresearch/AMP.