Backdoor Directions in Vision Transformers

Sengim Karayalcin; Marina Krcek; Pin-Yu Chen; Stjepan Picek

cs.CVMar 11, 2026

Backdoor Directions in Vision Transformers

Authors: Sengim Karayalcin, Marina Krcek, Pin-Yu Chen, Stjepan Picek

Organizations: Leiden University · Radboud University · IBM Research · University of Zagreb Faculty of Electrical Engineering and Computing

Abstract

This paper investigates how Backdoor Attacks are represented within Vision Transformers (ViTs). By assuming knowledge of the trigger, we identify a specific ``trigger direction'' in the model's activations that corresponds to the internal representation of the trigger. We confirm the causal role of this linear direction by showing that interventions in both activation and parameter space consistently modulate the model's backdoor behavior across multiple datasets and attack types. Using this direction as a diagnostic tool, we trace how backdoor features are processed across layers. Our analysis reveals distinct qualitative differences: static-patch triggers follow a different internal logic than stealthy, distributed triggers. We further examine the link between backdoors and adversarial attacks, specifically testing whether PGD-based perturbations (de-)activate the identified trigger mechanism. Finally, we propose a data-free, weight-based detection scheme for stealthy-trigger attacks. Our findings show that mechanistic interpretability offers a robust framework for diagnosing and addressing security vulnerabilities in computer vision.

Figures & tables

Figure 1: Derivation of the backdoor (BD) direction at layer $l$ (top). PCA projections of internal activations for clean and backdoored images at an intermediate layer alongside the resulting BD direction (bottom).

	$N$	Baseline			Steering CLS		Steering All		Orthog.
		ASR	RA	CA	$ASR_{+\hat{r}}$	$RA_{-\hat{r}}$	$ASR_{+\hat{r}}$	$RA_{-\hat{r}}$	ASR	RA	CA
CIFAR10	12	97.6	2.3	95.4	59.3	22.8	96.5	70.6	0.3	84.6	93.9
CIFAR100	13	96.9	2.6	78.6	62.1	19.7	87.5	39.5	15.9	51.4	78.0
TINY	8	99.0	0.8	70.7	89.0	21.6	99.5	51.9	1.3	56.4	70.4
Overall	33	97.7	2.1	82.8	67.6	21.3	93.7	53.8	6.7	64.7	82.0

Table 1: Average Steering and Orthogonalization for

\hat{r}

results by dataset (excluding models with baseline ASR

<0.9

Figure 2: Steering results on CIFAR100 with 0.1 poisoning rate.

Figure 3: Steering SSBA across datasets and poisoning rates (0.05 top, 0.1 bottom).

Figure 4: Backdoor steering for Tiny-ImageNet for DeiT-S/Swin-S

Attack	Clean			Backdoor
	0.01	0.05	0.1	0.01	0.05	0.1
BADNET	0.6%	5.6%	7.7%	33.2%	32.9%	33.0%
BLENDED	0.2%	0.2%	0.2%	25.0%	22.2%	22.3%
WANET	1.9%	1.9%	4.1%	28.9%	25.1%	26.0%
SSBA	0.2%	0.0%	0.1%	26.2%	25.1%	25.9%
BPP	29.2%	31.4%	41.5%	18.7%	21.7%	21.9%

Table 2: Percentage of AE Predictions for Target/Original Class - CIFAR100

Figure 5: Distribution of cosine similarities across layers for CIFAR100 with poisoning rate 0.05, starting from clean images

Figure 6: Distribution of cosine similarities across layers for CIFAR100 with poisoning rate 0.05, starting from backdoored images.

Figure 7: $Z$ -scores for CIFAR100 models at poisoning rate 0.05.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Steering results on CIFAR-100 over seven attacks and three poisoning rates.

Figure 9: Steering results on Tiny-ImageNet over three attacks and three poisoning rates.

Figure 10: Steering results on CIFAR-10 over six attacks and three poisoning rates.

Figure 11: Distributions of predicted classes for backdoored models on CIFAR-100. Starting from clean images (above) and poisoned images (below).

Figure 12: Distributions of predicted classes for backdoored models on Tiny-ImageNet. Starting from clean images (above) and poisoned images (below).

Figure 13: Distributions of predicted classes for backdoored models on CIFAR-10. Starting from clean images (above) and poisoned images (below).

Poisoning Rate	0.01	0.05	0.1
CIFAR10 Rate	12	11	10
CIFAR100 Rate	9	8	7
Tiny-ImageNet Rate	15	14	13

Appendix

Table 6: Tables for each setting, for AEs starting from clean images

Poisoning Rate	0.01	0.05	0.1
CIFAR10 Rate	20	21	22
CIFAR100 Rate	17	18	19
Tiny-ImageNet Rate	23	24	25

Appendix

Table 16: Tables for each setting, for AEs starting from backdoored images

Figure 14: $Z$ -scores for CIFAR-10 dataset across different attacks and poisoning rates.

Figure 15: $Z$ -scores for CIFAR-100 dataset across different attacks and poisoning rates.

Figure 16: $Z$ -scores for Tiny-ImageNet dataset across different attacks and poisoning rates.

Explore similar work

PASTA: A Patch-Agnostic Twofold-Stealthy Backdoor Attack on Vision Transformers
Apr 21, 2026Dazhuang Liu, Yanqi Qiao, Rui Wang +2Adversarial Attacks Vision Transformer
Delft University of Technology, Delft, the Netherlands. · University of Turku, Turku, Finland
ATAAT: Adaptive Threat-Aware Adversarial Tuning Framework against Backdoor Attacks on Vision-Language-Action Models
May 9, 2026Kewei Chen, Yayu Long, Shuai Li +1Adversarial Attacks Adversarial Attacks on VLA Models
Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences · Chongqing School, University of Chinese Academy of Sciences · Faculty of Information Technology and Electrical Engineering, University of Oulu, Finland
What Helps---and What Hurts: Bidirectional Explanations for Vision Transformers
Mar 2, 2026Qin Su, Tie LuoTransformer Interpretability Explainable Image Classification
Department of Electrical and Computer Engineering University of Kentucky, Lexington, KY, USA

Apr 21, 2026J/K move · Enter open · S save

PASTA: A Patch-Agnostic Twofold-Stealthy Backdoor Attack on Vision Transformers

Dazhuang Liu, Yanqi Qiao, Rui Wang, Kaitai Liang+1

Delft University of Technology, Delft, the Netherlands. · University of Turku, Turku, Finland

Adversarial Attacks Vision Transformer Backdoor Attacks

Open paper AskPDF arXiv

Vision Transformers (ViTs) have achieved remarkable success across vision tasks, yet recent studies show they remain vulnerable to backdoor attacks. Existing patch-wise attacks typically assume a single fixed trigger location during inference to maximize trigger attention. However, they overlook the self-attention mechanism in ViTs, which captures long-range dependencies across patches. In this work, we observe that a patch-wise trigger can achieve high attack effectiveness when activating backdoors across neighboring patches, a phenomenon we term the Trigger Radiating Effect (TRE). We further find that inter-patch trigger insertion during training can synergistically enhance TRE compared to single-patch insertion. Prior ViT-specific attacks that maximize trigger attention often sacrifice visual and attention stealthiness, making them detectable. Based on these insights, we propose PASTA, a twofold stealthy patch-wise backdoor attack in both pixel and attention domains. PASTA enables backdoor activation when the trigger is placed at arbitrary patches during inference. To achieve this, we introduce a multi-location trigger insertion strategy to enhance TRE. However, preserving stealthiness while maintaining strong TRE is challenging, as TRE is weakened under stealthy constraints. We therefore formulate a bi-level optimization problem and propose an adaptive backdoor learning framework, where the model and trigger iteratively adapt to each other to avoid local optima. Extensive experiments show that PASTA achieves 99.13% attack success rate across arbitrary patches on average, while significantly improving visual and attention stealthiness (144.43x and 18.68x) and robustness (2.79x) against state-of-the-art ViT defenses across four datasets, outperforming CNN- and ViT-based baselines.

Abstract

Explore similar work

PASTA: A Patch-Agnostic Twofold-Stealthy Backdoor Attack on Vision Transformers

ATAAT: Adaptive Threat-Aware Adversarial Tuning Framework against Backdoor Attacks on Vision-Language-Action Models

What Helps---and What Hurts: Bidirectional Explanations for Vision Transformers