VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer

Yanning Hou; Peiyuan Li; Zirui Liu; Yitong Wang; Yanran Ruan; Jianfeng Qiu; Ke Xu

cs.CVMar 9, 2026

VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer

Authors: Yanning Hou, Peiyuan Li, Zirui Liu, Yitong Wang, Yanran Ruan, Jianfeng Qiu, Ke Xu

Organizations: College of Intelligence Science and Technology, National University of Defense Technology, Changsha, China · School of Artificial Intelligence, Anhui University, Hefei, China · State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Anhui University, Hefei, China

Abstract

Zero-shot anomaly detection (ZSAD) requires detecting and localizing anomalies without access to target-class anomaly samples. Mainstream methods rely on vision-language models (VLMs) such as CLIP: they build hand-crafted or learned prompt sets for normal and abnormal semantics, then compute image-text similarities for open-set discrimination. While effective, this paradigm depends on a text encoder and cross-modal alignment, which can lead to training instability and parameter redundancy. This work revisits the necessity of the text branch in ZSAD and presents VisualAD, a purely visual framework built on Vision Transformers. We introduce two learnable tokens within a frozen backbone to directly encode normality and abnormality. Through multi-layer self-attention, these tokens interact with patch tokens, gradually acquiring high-level notions of normality and anomaly while guiding patches to highlight anomaly-related cues. Additionally, we incorporate a Spatial-Aware Cross-Attention (SCA) module and a lightweight Self-Alignment Function (SAF): SCA injects fine-grained spatial information into the tokens, and SAF recalibrates patch features before anomaly scoring. VisualAD achieves state-of-the-art performance on 13 zero-shot anomaly detection benchmarks spanning industrial and medical domains, and adapts seamlessly to pretrained vision backbones such as the CLIP image encoder and DINOv2. Code: https://github.com/7HHHHH/VisualAD

Figures & tables

Figure 1: Exploratory study motivating this work. (a) AnomalyCLIP derives normal/abnormal prototypes via trainable text prompts and a text encoder. (b) A purely visual variant removes the text branch and directly learns two visual prototypes, achieving comparable or slightly better results on VisA and MVTec with 99% fewer parameters. Radar and line plots show that our variant maintains similar accuracy but with much smoother evaluation curves, whereas AnomalyCLIP oscillates noticeably.

Figure 2: Overview of VisualAD. Two learnable global tokens (anomaly and normal) are inserted into a frozen ViT. For intermediate layers $\ell$ , SCA uses a few anchor queries with positional encoding and token-guided gating to aggregate localized spatial evidence, yielding enhanced tokens $\tilde{\mathbf{t}}_{a}^{(\ell)},\tilde{\mathbf{t}}_{n}^{(\ell)}$ . In parallel, each layer’s patch features are recalibrated by a SAF, denoted as $\mathcal{F}_{\ell}$ . The cosine-similarity difference between enhanced tokens and recalibrated patches forms layer-wise anomaly maps, which are upsampled and summed to obtain the final anomaly map; the image-level score is the mean of the top- $k$ responses.

Domain	Metric	Dataset	WinCLIP [ 23 ]	APRIL-GAN [ 10 ]	CLIP-AD [ 11 ]	AnomalyCLIP [ 58 ]	AdaCLIP [ 9 ]	VisualAD(CLIP [ 41 ] )	VisualAD(DINOv2 [ 37 ] )
Domain	Metric	Dataset	ViT-L/14@336px	ViT-L/14@336px	ViT-L/14@336px	ViT-L/14@336px	ViT-L/14@336px	ViT-L/14@336px	ViT-L/14
Industrial	Image-level (AUROC, $F_{1}$ -max, AP)	MVTec-AD [ 6 ]	(90.4, 92.7 , 95.6)	(86.1, 90.4, 93.6)	(74.1, 86.3, 88.1)	(91.6, 92.7 , 96.2)	( 92.0 , 92.7 , 96.4 )	( 92.2 , 93.2 , 96.7 )	(90.1, 92.4, 94.8)
		VisA [ 60 ]	(75.6, 78.2, 78.8)	(77.4, 78.6, 80.9)	(66.2, 74.3, 71.4)	(81.0, 80.3, 84.4)	(79.7, 79.63, 83.2)	( 84.7 , 82.5 , 87.6 )	( 83.1 , 81.4 , 86.8 )
		BTAD [ 35 ]	(68.2, 67.8, 70.9)	(73.7, 68.7, 69.9)	(66.7, 65.9, 67.3)	(88.7, 86.0, 90.6)	( 90.0 , 87.2 , 91.5 )	( 94.9 , 93.9 , 97.0 )	(88.2, 84.7, 89.7)
		KSDD2 [ 47 ]	(93.5, 86.4, 94.2)	(90.4, 82.9, 92.0)	(81.7, 75.0, 85.5)	(91.9, 84.5, 93.4)	(94.9, 90.3, 96.2)	( 98.0 , 93.9 , 98.3 )	( 97.7 , 93.1 , 98.1 )
		DAGM [ 49 ]	(91.8, 75.8, 79.5)	(94.4, 80.3, 83.9)	(62.1, 37.1, 32.3)	(98.0, 90.6, 92.4)	( 98.3 , 91.5 , 94.2 )	( 99.5 , 95.0 , 97.8 )	(93.2, 83.9, 86.1)

Table 1: Comparison of zero-shot anomaly detection methods on industrial and medical benchmarks. For each dataset, we report image-level metrics (AUROC,

F_{1}

-max, AP) and pixel-level metrics (AUROC,

F_{1}

-max, AP, PRO). VisualAD(CLIP) and VisualAD(DINOv2) denote our method instantiated with CLIP and DINOv2 backbones, respectively. The best and second-best results for each metric and dataset are highlighted in red and blue .

Figure 3: Qualitative comparison of anomaly segmentation results for different ZSAD methods. The first five columns show images from industrial datasets, while the last three columns correspond to medical datasets.

Ablation	SCA	SAF	Focal	Dice	Ctr	Image-level	Pixel-level
Module	✗	✓	✓	✓	✓	(82.3, 85.4)	(95.3, 27.4)
	✓	✗	✓	✓	✓	(50.5, 58.9)	(87.9, 3.5)
	✗	✗	✓	✓	✓	(48.0, 56.1)	(38.3, 0.8)
Loss	✓	✓	✗	✓	✓	(83.5, 86.8)	(95.3, 26.6)
	✓	✓	✓	✗	✓	(83.4, 86.5)	(95.3, 27.3)
	✓	✓	✓	✓	✗	(84.2, 87.0)	(95.4, 27.6)

Table 2: Ablation study on different modules and loss components using ViT-L/14@336px. Both Image-level and Pixel-level metrics are reported as (AUROC, AP).

Figure 4: Radar plot comparing different pretrained ViT backbones under the ZSAD setting. We evaluate CLIP and DINO-based variants from small to large model scales. Larger backbones and higher input resolutions generally lead to consistent improvements in both pixel-level and sample-level metrics, with CLIP’s ViT-L/14@336px giving the strongest sample-level accuracy and DINO’s ViT-g/14 delivering the highest pixel-level precision.

Table 7

Positional modeling	Pixel-level	Image-level
None (no PE)	(93.0, 29.0, 22.5)	(82.4, 78.1, 84.5)
Learnable 1D Absolute	( 95.8 , 34.6 , 28.4 )	( 84.7 , 82.5 , 87.6 )
Fixed 1D Sinusoidal	( 95.2 , 33.8 , 27.6 )	( 84.1 , 81.9 , 86.8 )
Learnable 2D Absolute	(95.0, 33.2, 27.0)	(83.8, 81.6, 86.0)
Fixed 2D Sinusoidal	(94.6, 32.4, 26.1)	(83.4, 80.8, 85.4)
Learnable 2D Relative Bias	(94.4, 31.9, 25.7)	(83.0, 80.2, 84.9)

Table 5: Effect of positional modeling in the SCA module on VisA. Metrics are reported as (AUROC,

F_{1}

-max, AP). Best and second-best results are highlighted in bold and underline .

Figure 5: PCA visualization of feature distributions under three configurations. From left to right: vanilla CLIP features, CLIP with normal/anomaly tokens, and CLIP with both tokens and the MLP-based transformation. The anomaly cluster (dark) becomes progressively more compact and moves farther from the normal cluster, while the variance concentrates along a single dominant axis, indicating increasingly discriminative representations for anomaly separation.

Figure 6: Layer-wise anchor attention maps on the carpet category. Anchors within the same layer attend to similar regions, whereas attention patterns differ across layers: mid-level layers focus on fine-grained defect boundaries, while deeper layers emphasize broader normal regions. This complementarity between layers provides both local anomaly cues and holistic normal context for accurate localization.

Explore similar work

Anomaly-Aware Vision-Language Adapters for Zero-Shot Anomaly Detection
May 12, 2026Muhammad Aqeel, Maham Nazir, Uzair Khan +2VLM Adaptation Medical Image Anomaly Detection
Dept. of Engineering for Innovation Medicine, University of Verona, Italy · School of Computer Science and Engineering, Beihang University, China · Dept. of Engineering for Innovation +1
VFAD: Variational Semantic Prompting Meets Frequency-Adaptive Representation Learning for Zero-Shot Anomaly Detection
Jul 31, 2026Peng Chen, Kaige Li, Wei Wang +5VLM Adaptation Frequency-Domain Feature Learning
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China
DuoAD: Leveraging [CLS] Dual Characteristics for Training-Free Few-Shot Anomaly Detection
Jul 27, 2026Jyun-Ze Tang, Po-Han Huang, Ming-Ching Chang +2Few-Shot Anomaly Detection Anomaly Localization
Inventec Corporation, Taipei, Taiwan · University at Albany, State University of New York, NY, USA

May 12, 2026J/K move · Enter open · S save

Anomaly-Aware Vision-Language Adapters for Zero-Shot Anomaly Detection

Muhammad Aqeel, Maham Nazir, Uzair Khan, Marco Cristani+1

Dept. of Engineering for Innovation Medicine, University of Verona, Italy · School of Computer Science and Engineering, Beihang University, China · Dept. of Engineering for Innovation+1

VLM Adaptation Medical Image Anomaly Detection Industrial Anomaly Detection Vision Foundation Model Adaptation Multimodal Anomaly Detection Zero-Shot Anomaly Detection

Open paper AskPDF arXiv

Zero-shot anomaly detection aims to identify defects in unseen categories without target-specific training. Existing methods usually apply the same feature transformation to all samples, treating normal and anomalous data uniformly despite their fundamentally asymmetric distributions, compact normals versus diverse anomalies. We instead exploit this natural asymmetry by proposing AVA-DINO, an anomaly-aware vision-language adaptation framework with dual specialized branches for normal and anomalous patterns that adapt frozen DINOv3 visual features. During training on auxiliary data, the two branches are learned jointly with a text-guided routing mechanism and explicit routing regularization that encourages branch specialization. At test time, only the input image and fixed, predefined language descriptions are used to dynamically combine the two branches, enabling an asymmetric activation. This design prevents degenerate uniform routing and allows context-specific feature transformations. Experiments across nine industrial and medical benchmarks demonstrate state-of-the-art performance, achieving 93.5% image-AUROC on MVTec-AD and strong cross-domain generalization to medical imaging without domain-specific fine-tuning. https://github.com/aqeeelmirza/AVA-DINO

Abstract

Explore similar work

Anomaly-Aware Vision-Language Adapters for Zero-Shot Anomaly Detection

VFAD: Variational Semantic Prompting Meets Frequency-Adaptive Representation Learning for Zero-Shot Anomaly Detection

DuoAD: Leveraging [CLS] Dual Characteristics for Training-Free Few-Shot Anomaly Detection