STRAP-ViT: Segregated Tokens with Randomized -- Transformations for Defense against Adversarial Patches in ViTs

Nandish Chattopadhyay; Anadi Goyal; Chandan Karfa; Anupam Chattopadhyay

cs.CVMar 13, 2026

STRAP-ViT: Segregated Tokens with Randomized -- Transformations for Defense against Adversarial Patches in ViTs

Authors: Nandish Chattopadhyay, Anadi Goyal, Chandan Karfa, Anupam Chattopadhyay

Organizations: Indian Institute of Technology, Guwahati, India · Nanyang Technological University, Singapore

Abstract

Adversarial patches are physically realizable localized noise, which are able to hijack Vision Transformers (ViT) self-attention, pulling focus toward a small, high-contrast region and corrupting the class token to force confident misclassifications. In this paper, we claim that the tokens which correspond to the areas of the image that contain the adversarial noise, have different statistical properties when compared to the tokens which do not overlap with the adversarial perturbations. We use this insight to propose a mechanism, called STRAP-ViT, which uses Jensen-Shannon Divergence as a metric for segregating tokens that behave as anomalies in the Detection Phase, and then apply randomized composite transformations on them during the Mitigation Phase to make the adversarial noise ineffective. The minimum number of tokens to transform is a hyper-parameter for the defense mechanism and is chosen such that at least 50% of the patch is covered by the transformed tokens. STRAP-ViT fits as a non-trainable plug-and-play block within the ViT architectures, for inference purposes only, with a minimal computational cost and does not require any additional training cost/effort. STRAP-ViT has been tested on multiple pre-trained vision transformer architectures (ViT-base-16 and DinoV2) and datasets (ImageNet and CalTech-101), across multiple adversarial attacks (Adversarial Patch, LAVAN, GDPA and RP2), and found to provide excellent robust accuracies lying within a 2-3% range of the clean baselines, and outperform the state-of-the-art.

Figures & tables

Fig. 1 : STRAP-ViT in action: Demonstrating the detection and transformation of tokens which overlap with the adversarial patch using the proposed STRAP-ViT defense technique. In this case, the minimum number of tokens to modify is two ( $K=2$ ), such that at least $50\%$ of the patch is covered, to render the patch ineffective.

Fig. 2 : Illustration of the statistical separability of the tokens which overlap with the adversarial patch and other tokens from the clean part of the image, using entropy as a metric.

Fig. 3 : STRAP-ViT pipeline: The two-stage adversarial defense framework comprising of the Detection using Token Segregation through Jensen-Shannon Divergence and Mitigation using Token Transformation through composite randomised transformations, preceded by ViT Pre-processing Tokenization and succeeded by Vit Forward Pass.

Attack	Patch Size	Setting	ImageNet Top-1	Top-5	Caltech Top-1	Top-5
Clean Baseline			80.5%	95.4%	90.4%	96.9%
STRAP-ViT on Clean			80.1% (-0.4% $\downarrow$ )	95.1% (-0.3% $\downarrow$ )	88.6% (-1.8% $\downarrow$ )	95.7% (-1.2% $\downarrow$ )
Adversarial Patch (GoogleAP) [ 5 ]
	40 $\times$ 40	No Defence	3.6%	16.4%	5.3%	14.9%
		STRAP-ViT	78.8% (+75.2% $\uparrow$ )	94.2% (+77.8% $\uparrow$ )	87.6% (+82.3% $\uparrow$ )	95.4% (+80.5% $\uparrow$ )
	45 $\times$ 45	No Defence	2.9%	13.6%	1.8%	17.9%

TABLE I : Performance of STRAP-ViT against adversarial patch attacks on ViT-B/16 across ImageNet and Caltech-101. Gains are shown as improvements over the no-defence accuracy.

Attack	Patch Size	Setting	ImageNet Top-1	Top-5	Caltech Top-1	Top-5
Clean Baseline			83.8%	99.3%	95.0%	99.8%
STRAP-ViT on Clean			86.0% (+2.2% $\uparrow$ )	97.3% (-2.0% $\downarrow$ )	93.5% (-1.5% $\downarrow$ )	96.9% (-2.9% $\downarrow$ )
Adversarial Patch (GoogleAP) [ 5 ]
	40 $\times$ 40	No Defence	7.2%	20.2%	8.1%	16.7%
		STRAP-ViT	83.9% (+76.7% $\uparrow$ )	96.7% (+76.5% $\uparrow$ )	91.3% (+83.2% $\uparrow$ )	98.0% (+81.3% $\uparrow$ )
	45 $\times$ 45	No Defence	7.5%	18.5%	5.6%	20.4%

TABLE II : Performance Analysis of STRAP-ViT on Image Classification Task with the DinoV2 backbone model across datasets (ImageNet and Caltech-101), for three adversarial attacks (Adversarial Patch, LAVAN and GDPA) with varying patch sizes.

Patches	Patch Size	Setting	ImageNet Top-1	Top-5	Caltech Top-1	Top-5
Clean Baseline			80.5%	95.4%	90.4%	96.9%
STRAP-ViT on Clean			80.1% (-0.4% $\downarrow$ )	95.1% (-0.3% $\downarrow$ )	88.6% (-1.8% $\downarrow$ )	95.7% (-1.2% $\downarrow$ )
RP2 – TWO Patches
	40 $\times$ 40	No Defence	31.9%	34.9%	28.1%	37.2%
		STRAP-ViT (K=4)	77.2% (+45.3% $\uparrow$ )	92.3% (+57.4% $\uparrow$ )	88.8% (+60.7% $\uparrow$ )	94.6% (+57.4% $\uparrow$ )
RP2 – THREE Patches

TABLE III : Performance Analysis of STRAP-ViT on Image Classification Task with the ViT-base-16 model across datasets (ImageNet and Caltech-101), for the RP2 [ 8 ] attack with multiple patches.

Defense	Patch-Size	Robust Acc	Defense	Patch-Size	Robust Acc
Smoothed ViT-B [ 23 ]	32x32	39	JEDI [ 31 ]	40x40	70.6
Certifiable Defense with ViT [ 21 ]	32x32	41.7	Dimension Reduction [ 33 ]	40x40	62.6
PatchCleanser [ 22 ]	32x32	62.1	ODDR [ 32 ]	40x40	74.3
LGS [ 19 ]	40x40	70.6	STRAP (ours)	40x40	78.8

TABLE IV : Performance Comparison of STRAP-ViT with the related works, tested on the ViT model, on the ImageNet dataset with the Adversarial Patch [ 5 ] .

Fig. 4 : A plot of the performance analysis of STRAP-ViT, with multiple adversarial patches ( Adversarial Patch, LAVAN, GDPA) and patch sizes, across two datasets (ImageNet and CalTech-101), also including the clean baseline and the impact of applying STRAP-ViT on clean samples, for reference and comparison.

Explore similar work

Adversarial Decoys: Misdirecting Attention-Based Defenses in ViT
Jul 8, 2026Giulia Marchiori Pietrosanti, Giulio Rossolini, Giorgio ButtazzoAdversarial Patch Attacks Adversarial Attacks
Department of Excellence in Robotics and AI, Sant’Anna School of Advanced Study, Italy
MOAT: Model-Agnostic Randomized Transformations for preventing Efficiency Degradation Attacks on ViTs
Aug 5, 2026Anadi Goyal, Nandish Chattopadhyay, Chandan Karfa +2Adversarial Attacks Vision Transformer
Indian Institute of Technology Guwahati, India · Nanyang Technological University, Singapore · Prince of Songkla University, Phuket Campus
A Mechanistic Analysis of Adversarial Fine-tuning of Vision Transformers
May 28, 2026Hannah Gao, Isha Agarwal, Dylan Hadfield-Menell +1Adversarial Training Fine-Tuning
Massachusetts Institute of Technology · MIT Cambridge, MA

Jul 8, 2026J/K move · Enter open · S save

Adversarial Decoys: Misdirecting Attention-Based Defenses in ViT

Giulia Marchiori Pietrosanti, Giulio Rossolini, Giorgio Buttazzo

Department of Excellence in Robotics and AI, Sant’Anna School of Advanced Study, Italy

Adversarial Patch Attacks Adversarial Attacks Vision Transformer Adversarial Robustness

Open paper AskPDF arXiv

Vision Transformers (ViTs) remain vulnerable to localized adversarial attacks, e.g., adversarial patches, while recent test-time defenses mitigate them by suppressing image tokens with abnormally high attention scores. These defenses exploit a strong coupling between attention and adversarial effectiveness: adversarial tokens often need to attract substantial attention to influence the prediction. We introduce adversarial decoys, independently optimized image patches that redirect the attention, and therefore related defenses, toward selected target tokens. Rather than jointly optimizing misclassifications and defense evasion, our approach decouples the two objectives: the original adversarial region induces the incorrect prediction, while a separate decoy manipulates the attention ranking used by the defense. A layer-wise objective increases target-token attention and promotes these tokens above competing non-target ones. Since the decoy is optimized independently of the underlying attack, the method is attack-agnostic and can be easily integrated with any existing adversarial patch attack. Experiments on ImageNet across multiple ViT architectures and attacks show that decoys can redirect high attention scores away from the true adversarial region while preserving much of the attack effectiveness. These results reveal a fundamental limitation of using attention magnitude as an indicator of adversarial relevance.

Abstract

Explore similar work

Adversarial Decoys: Misdirecting Attention-Based Defenses in ViT

MOAT: Model-Agnostic Randomized Transformations for preventing Efficiency Degradation Attacks on ViTs

A Mechanistic Analysis of Adversarial Fine-tuning of Vision Transformers