Latent Anomaly Knowledge Excavation: Unveiling Sparse Sensitive Neurons in Vision-Language Models

Shaotian Li; Shangze Li; Chuancheng Shi; Wenhua Wu; Yanqiu Wu; Xiaohan Yu; Fei Shen; Tat-Seng Chua

cs.CVApr 9, 2026

Latent Anomaly Knowledge Excavation: Unveiling Sparse Sensitive Neurons in Vision-Language Models

Authors: Shaotian Li, Shangze Li, Chuancheng Shi, Wenhua Wu, Yanqiu Wu, Xiaohan Yu, Fei Shen, Tat-Seng Chua

Organizations: Macquarie University · Nanjing University of Science and Technology · The University of Sydney · National University of Singapore

Abstract

Large-scale vision-language models (VLMs) exhibit remarkable zero-shot capabilities, yet the internal mechanisms driving their anomaly detection (AD) performance remain poorly understood. Current methods predominantly treat VLMs as black-box feature extractors, assuming that anomaly-specific knowledge must be acquired through external adapters or memory banks. In this paper, we challenge this assumption by arguing that anomaly knowledge is intrinsically embedded within pre-trained models but remains latent and under-activated. We hypothesize that this knowledge is concentrated within a sparse subset of anomaly-sensitive neurons. To validate this, we propose latent anomaly knowledge excavation (LAKE), a training-free framework that identifies and elicits these critical neuronal signals using only a minimal set of normal samples. By isolating these sensitive neurons, LAKE constructs a highly compact normality representation that integrates visual structural deviations with cross-modal semantic activations. Extensive experiments on industrial AD benchmarks demonstrate that LAKE achieves state-of-the-art performance while providing intrinsic, neuron-level interpretability. Ultimately, our work advocates for a paradigm shift: redefining anomaly detection as the targeted activation of latent pre-trained knowledge rather than the acquisition of a downstream task.

Figures & tables

Fig. 1: Comparison of anomaly detection paradigms. Unlike existing approaches that rely on external black-box modeling, our method intrinsically elicits anomaly awareness by isolating and activating sparse sensitive neurons within the frozen network.

Fig. 2: The LAKE framework. Features from layer $l$ are projected into a variance-filtered sensitive subspace ( $I_{sens}$ ) and compared against a normal gallery ( $\mathcal{G}$ ) to compute the visual score ( $S_{vis}$ ). Simultaneously, deeper features ( $P^{(l^{\prime})}$ ) are aligned with text embeddings ( $t_{norm},t_{anom}$ ) to extract the semantic score ( $S_{text}$ ). Both scores are fused via weight $\alpha$ to produce the unified anomaly score ( $S$ ).

Method	MVTec-AD [ 42 ]			VisA [ 43 ]			BTAD [ 44 ]
Method	AUROC	$F_{1}$ -max	AP	AUROC	$F_{1}$ -max	AP	AUROC	$F_{1}$ -max	AP
OpenCLIP [ 45 ]	74.1	88.5	89.1	60.2	73.0	66.2	25.7	66.0	49.8
WinCLIP [ 46 ]	90.4	92.7	95.6	75.6	78.2	78.8	68.2	67.8	70.9
CLIP-AD [ 47 ]	74.1	86.3	88.1	66.2	74.3	71.4	66.7	65.9	67.3
AnomalyCLIP [ 7 ]	91.6	92.7	96.2	81.0	80.3	84.4	88.7	86.0	90.6
AdaCLIP [ 8 ]	92.2	92.7	96.4	79.7	79.6	83.2	90.0	87.2	91.5

TABLE I: Image-level anomaly detection results on the MVTec-AD, VisA, and BTAD benchmarks. Evaluation metrics include AUROC,

F_{1}

-max, and AP. The highest scores are in bold, alongside relative performance gains over the OpenCLIP model.

Method	MVTec-AD [ 42 ]				VisA [ 43 ]				BTAD [ 44 ]
Method	AUROC	$F_{1}$ -max	AP	PRO	AUROC	$F_{1}$ -max	AP	PRO	AUROC	$F_{1}$ -max	AP	PRO
OpenCLIP [ 45 ]	35.6	2.8	–	6.9	43.6	1.5	–	14.0	41.2	6.8	–	10.8
WinCLIP [ 46 ]	82.3	24.8	18.2	62.0	73.2	9.0	5.4	51.1	72.7	18.5	12.9	27.3
CLIP-AD [ 47 ]	77.9	26.3	21.1	55.7	93.0	24.1	17.9	80.2	80.9	24.1	18.3	41.4
AnomalyCLIP [ 7 ]	91.0	38.9	34.4	81.7	95.4	27.6	20.7	86.4	93.0	47.1	41.5	71.0
AdaCLIP [ 8 ]	88.5	43.9	41.0	47.6	95.1	33.8	29.2	71.3	87.7	42.3	36.6	17.1

TABLE II: Pixel-level anomaly localization results on three datasets. Evaluated metrics are AUROC,

F_{1}

-max, AP, and PRO. The highest scores are shown in italics, alongside the relative improvements achieved over the OpenCLIP model.

Fig. 3: Qualitative comparison of anomaly localization with state-of-the-art (SOTA) methods.

Fig. 4: Ablation of support set size on image-level anomaly detection.

Fig. 5: Ablation of support set size on pixel-level anomaly localization.

Fig. 6: Ablation studies on key hyperparameters. Impact of the top- $k$ selection at the image level (Left). Effect of the top- $k$ selection at the pixel level (Middle). Performance variations across different values of the parameter $\alpha$ (Right).

Model	AUROC $\uparrow$	$F_{1}$ -max $\uparrow$	AP $\uparrow$	PRO $\uparrow$
Image-Level Performance
LAKE (Random 100)	81.6	83.2	83.5	-
LAKE (Top 100)	94.7	93.9	96.7	-
Pixel-Level Performance
LAKE (Random 100)	78.5	20.4	20.0	45.8
LAKE (Top 100)	93.7	50.8	45.5	88.9

TABLE III: Quantitative evaluation of neuron selection: Top-K vs. Random selection strategies on MVTec-AD under 64-shot.

Fig. 7: Qualitative evaluation of neuron specificity for anomaly excavation. Heatmaps generated by random neurons exhibit severe background noise, whereas our explicitly identified anomaly-sensitive neurons yield highly precise localizations that tightly align with the ground truth.

Fig. 8: Cross-category overlap analysis. The t-SNE demonstrates the significant spatial overlap of identified anomaly-sensitive neurons across five diverse MVTec-AD categories, highlighting their generalized activation patterns.

Image-Level Performance
Model	AUROC $\uparrow$	$F_{1}$ -max $\uparrow$	AP $\uparrow$	PRO $\uparrow$
WinCLIP [ 46 ]	90.4	92.7	95.6	-
+ Ours	92.8 (+2.4%)	93.4 (+0.7%)	95.8 (+0.2%)	-
ReMP-AD [ 51 ]	96.8	95.5	–	-
+ Ours	96.9 (+0.1%)	96.2 (+0.7%)	96.9	-
Pixel-Level Performance

TABLE IV: Performance Comparison at Image and Pixel Levels. The best results are highlighted in bold.

Method	AUROC $\uparrow$	$F_{1}$ -max $\uparrow$	AP $\uparrow$	PRO $\uparrow$
Image-Level Performance
WinCLIP [ 46 ]	72.7	90.7	91.6	-
CLIP-AD [ 47 ]	72.1	91.3	91.5	-
AnomalyCLIP [ 7 ]	69.0	90.6	90.1	-
AdaCLIP [ 8 ]	80.0	90.2	94.1	-
VisualAD (CLIP) [ 30 ]	80.8	91.8	94.7	-

TABLE V: Anomaly detection performance on Brain-AD. The best results are highlighted in bold.

Multimodal Anomaly Detection

Multimodal anomaly detection identifies unusual patterns by combining complementary signals such as images, depth, video, text, and sensor data.

See all 97 papers

VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer
Mar 9, 2026Yanning Hou, Peiyuan Li, Zirui Liu +4Zero-Shot Anomaly Detection Vision Foundation Model Adaptation
College of Intelligence Science and Technology, National University of Defense Technology, Changsha, China · School of Artificial Intelligence, Anhui University, Hefei, China · State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Anhui University, Hefei, China
AnomalyAgent: Training-Free Agentic Models for Zero-/Few-Shot Anomaly Detection
May 28, 2026Yi Zhang, Jiawen Zhu, Lele Fu +1Few-Shot Anomaly Detection Tool Use in VLMs
Singapore Management University, Singapore · Sun Yat-sen University, China
Reasoning-Driven Anomaly Detection and Localization with Image-Level Supervision
Mar 28, 2026Yizhou Jin, Yuezhu Feng, Jinjin Zhang +3Anomaly Localization Multimodal Anomaly Detection
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Hangzhou Innovation Institute, Beihang University, Hangzhou, China

Mar 9, 2026J/K move · Enter open · S save

VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer

Yanning Hou, Peiyuan Li, Zirui Liu, Yitong Wang+3

College of Intelligence Science and Technology, National University of Defense Technology, Changsha, China · School of Artificial Intelligence, Anhui University, Hefei, China · State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Anhui University, Hefei, China

Zero-Shot Anomaly Detection Vision Foundation Model Adaptation Medical Image Anomaly Detection

Open paper AskPDF arXiv

Zero-shot anomaly detection (ZSAD) requires detecting and localizing anomalies without access to target-class anomaly samples. Mainstream methods rely on vision-language models (VLMs) such as CLIP: they build hand-crafted or learned prompt sets for normal and abnormal semantics, then compute image-text similarities for open-set discrimination. While effective, this paradigm depends on a text encoder and cross-modal alignment, which can lead to training instability and parameter redundancy. This work revisits the necessity of the text branch in ZSAD and presents VisualAD, a purely visual framework built on Vision Transformers. We introduce two learnable tokens within a frozen backbone to directly encode normality and abnormality. Through multi-layer self-attention, these tokens interact with patch tokens, gradually acquiring high-level notions of normality and anomaly while guiding patches to highlight anomaly-related cues. Additionally, we incorporate a Spatial-Aware Cross-Attention (SCA) module and a lightweight Self-Alignment Function (SAF): SCA injects fine-grained spatial information into the tokens, and SAF recalibrates patch features before anomaly scoring. VisualAD achieves state-of-the-art performance on 13 zero-shot anomaly detection benchmarks spanning industrial and medical domains, and adapts seamlessly to pretrained vision backbones such as the CLIP image encoder and DINOv2. Code: https://github.com/7HHHHH/VisualAD

Abstract

Multimodal Anomaly Detection

Explore similar work

VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer

AnomalyAgent: Training-Free Agentic Models for Zero-/Few-Shot Anomaly Detection

Reasoning-Driven Anomaly Detection and Localization with Image-Level Supervision