cs.CVNov 13, 2026

SWAM: Saliency-Weighted Asymmetric Merging for Efficient Vision Transformers

Authors: Utkarsh Kumar Singh, Manish Das, Shiv Ram Dubey, Satish Kumar Singh

Organizations: Computer Vision and Biometrics Lab, Department of Information Technology, Indian Institute of Information Technology Allahabad, Prayagraj, Uttar Pradesh-211015, India

Abstract

Vision Transformers (ViTs) achieve strong performance on image recognition tasks, yet their quadratic self-attention complexity over input tokens creates prohibitive computational costs for practical deployment. Token reduction methods offer a practical solution by progressively reducing the token sequence during the forward pass. We propose Saliency-Weighted Asymmetric Merging (SWAM), a training-based token merging framework that extends the Bipartite Soft Matching (BSM) with two saliency-aware mechanisms. We introduce a two-step matching formulation in which cosine similarity decides candidate pairs, while a multiplicative maximum saliency penalty regulates their selection for merging. We then further use asymmetric exponential saliency-weighted aggregation, which allows more salient tokens to contribute more strongly to the merged representation. A lightweight MLP-based saliency prediction module (75k parameters per stage, 1% of backbone parameters) supervised via attention rollout is trained end-to-end jointly with a DeiT-Small backbone using a composite cross-entropy, knowledge-distillation, and saliency-alignment objective. On CIFAR-100, TinyImageNet, and ImageNet-100, under a conservative schedule that reduces tokens from 196 to 100 patch tokens, with the CLS token retained throughout, 27.5% FLOPs reduction and 1.35 times speedup, SWAM matches or exceeds baseline across the reported results. Under an aggressive schedule (196 to 70 tokens with the CLS token kept throughout, 33.8% FLOPs reduction, and 1.46 times speedup), accuracy gains are achieved on ImageNet-100 (+0.23%) with minimal losses on CIFAR-100 and TinyImageNet.

Figures & tables

Explore similar work

CardsList
  1. AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers

    May 26, 2026Semi Lee, Hyejin Go, Hyesong ChoiToken MergingSelf-Attention

  2. Prune-and-Merge: Training-Free Scaling of Vision Transformers

    Oct 13, 2026Vatsal, Soumil Malhotra, Priyanshu Chahal +1Vision TransformerEfficient ViTs

  3. ASAP: Attention Sink Anchored Pruning

    May 21, 2026Jaehyuk Lee, Hanyoung Kim, Yanggee Kim +1Self-AttentionEfficient ViTs