Reasoning-Driven Anomaly Detection and Localization with Image-Level Supervision

Yizhou Jin; Yuezhu Feng; Jinjin Zhang; Peng Wang; Qingjie Liu; Yunhong Wang

cs.CVMar 28, 2026

Reasoning-Driven Anomaly Detection and Localization with Image-Level Supervision

Authors: Yizhou Jin, Yuezhu Feng, Jinjin Zhang, Peng Wang, Qingjie Liu, Yunhong Wang

Organizations: State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Hangzhou Innovation Institute, Beihang University, Hangzhou, China

Abstract

Multimodal large language models (MLLMs) have recently demonstrated remarkable reasoning and perceptual abilities for anomaly detection. However, most approaches remain confined to image-level anomaly detection and textual reasoning, while pixel-level localization still relies on external vision modules and dense annotations. In this work, we activate the intrinsic reasoning potential of MLLMs to perform anomaly detection, pixel-level localization, and interpretable reasoning solely from image-level supervision, without any auxiliary components or pixel-wise labels. Specifically, we propose Reasoning-Driven Anomaly Localization (ReAL), which extracts anomaly-related tokens from the autoregressive reasoning process and aggregates their attention responses to produce pixel-level anomaly maps. We further introduce a Consistency-Guided Reasoning Optimization (CGRO) module that leverages reinforcement learning to align reasoning tokens with visual attentions, resulting in more coherent reasoning and accurate anomaly localization. Extensive experiments on four public benchmarks demonstrate that our method significantly improves anomaly detection, localization, and interpretability. Remarkably, despite relying solely on image-level supervision, our approach achieves performance competitive with MLLM-based methods trained under dense pixel-level supervision. Code is available at https://github.com/YizhouJin313/ReADL.

Figures & tables

Figure 1: Visualization of token-to-patch attention maps for certain reasoning tokens. Only a small subset of tokens (highlighted in red) exhibits focused attention on the true anomaly regions (red boxes), and these tokens are also semantically related to anomaly concepts (e.g., “scratch”, “mark”).

Figure 2: Overview of the proposed reasoning-driven anomaly detection framework. Given an input image, the MLLM generates a reasoning process and a final anomaly answer. The ReAL module selects anomaly-related tokens based on semantic relevance $S_{\text{T}}$ and spatial entropy $S_{\text{I}}$ , then aggregates their visual attentions to obtain a pixel-level anomaly map. During training, these selected tokens are further used by the CGRO module to leverage reinforcement learning jointly driven by a reasoning–localization consistency reward and R1-based accuracy and format rewards, enabling the model to align reasoning tokens with their corresponding visual attentions while simultaneously improving reasoning correctness, structural quality, and anomaly localization accuracy.

Model	Param.	Anno.	Image-level					Pixel-level					Reasoning
Model	Param.	Anno.	AVG	SDD	DTD	WFDD	MVTec	AVG	SDD	DTD	WFDD	MVTec	MVTec-COT
GPT-4o-mini [ 15 ]	/	/	70.9, 71.4	66.2, 92.8	79.4, 70.3	66.4, 63.5	71.5, 59.0	N/A	N/A	N/A	N/A	N/A	19.0, 65.2
GPT-4o [ 15 ]	/	/	79.4, 85.3	77.5, 90.7	89.2, 90.2	78.2, 78.6	72.7, 81.7	N/A	N/A	N/A	N/A	N/A	19.9, 60.4
GPT-4.1	/	/	87.2 , 88.4	84.4, 92.1	94.3, 94.3	86.8, 86.0	83.3, 81.2	N/A	N/A	N/A	N/A	N/A	20.8, 69.9
GPT-5-nano	/	/	80.3, 81.6	82.6, 88.6	90.4, 87.2	80.2, 79.7	68.0, 70.7	N/A	N/A	N/A	N/A	N/A	14.0, 50.0
Gemini-2.5-Flash [ 9 ]	/	/	79.9, 76.3	71.8, 49.5	88.6, 91.3	82.6, 82.2	76.5, 82.3	N/A	N/A	N/A	N/A	N/A	17.6, 68.1

Table 1: Comparison of image-level and pixel-level detection performance and reasoning performance under zero-shot settings. Detection results are reported as (AUROC, ACC), and reasoning results as (ROUGE-L, SBERT). The column ‘Anno.’ denotes the supervision type used in training, where ‘I’ represents image-level supervision, ‘P’ denotes pixel-level annotations, and ‘T’ indicates task-specific instruction-tuning data designed for anomaly detection. * indicates anomaly maps are generated with ReAL.

Figure 3: Qualitative comparison of Qwen2.5-VL-7B , Qwen2.5-VL-7B+R1 , and Qwen2.5-VL-7B+CGRO .

Model	Image-level			Pixel-level
Model	AUROC	AUPR	ACC	AUROC	AUPR	ACC
Vanilla	63.4	59.4	63.3	64.7	2.6	73.0
Vanilla + ReAL	63.4	59.4	63.3	61.7	5.1	85.6
Vanilla + CGRO	83.9	76.7	86.9	72.7	5.9	92.6
Full	83.9	76.7	86.9	80.7	13.3	97.1

Table 2: Ablation on ReAL and CGRO. Vanilla denotes the original Qwen2.5-VL-7B . All metrics are macro-averaged across four AD datasets.

Spatial entropy $S_{\text{I}}$	Semantic relevant $S_{\text{T}}$	AUROC	AUPR	ACC
		72.7	5.9	92.6
✓		74.1	11.8	89.2
	✓	76.7	14.0	90.2
✓	✓	80.7	13.3	97.1

Table 3: Ablation on token selection in ReAL. Base model is Qwen2.5-VL-7B + CGRO . We compare different ways of filtering and aggregating reasoning tokens for pixel-level localization.

S_{\text{T}}

denotes the semantic relevance score measuring alignment with anomaly concepts, and

S_{\text{I}}

denotes the spatial entropy score reflecting attention concentration. Results are pixel-level metrics averaged across four datasets.

Figure 4: Visualization of different token selection used in ReAL. Each column shows anomaly maps generated by: (a) keeping all tokens (b) filtering with spatial entropy $S_{\text{I}}$ , (c) filtering with semantic relevance $S_{\text{T}}$ , (d) combining both scores with weighted aggregation. $S_{\text{T}}$ measures semantic alignment with anomaly-related concepts, while $S_{\text{I}}$ measures attention concentration. Methods that incorporate both criteria yield more accurate anomaly maps.

Figure 5: Qualitative comparison of Qwen2.5-VL-7B , Qwen2.5-VL-7B+R1 , and Qwen2.5-VL-7B+CGRO .

Figure 6: Comparison of GPT-4.1, Triad, AnomalyGPT, EIAD, and our method.

hyperparameter			Image-level			Pixel-level
$\delta_{1}$	$\delta_{2}$	$k$	AUROC	AUPR	ACC	AUROC	AUPR	ACC
0.3	0.1	2	76.0	76.0	75.6	67.9	5.6	97.5
0.3	0.1	4	76.1	76.6	75.8	67.0	5.4	97.6
0.3	0.3	3	77.8	77.7	77.1	68.7	5.9	97.6
0.5	0.1	3	73.3	74.4	72.1	69.5	6.3	97.5
0.3	0.1	3	79.9	78.8	79.8	70.5	7.4	97.8

Table 4: Ablation study of the CGRO hyperparameters, including the number of top-

k

reasoning tokens, the Jaccard threshold

\delta_{1}

and

\delta_{2}

. Base model is Qwen2.5-VL-7B + CGRO . Results are image-level and pixel-level metrics on the WFDD dataset.

$\tau_{\text{i}}$	$\tau_{\text{t}}$	AUROC	AUPR	ACC
Maximum Curvature	Maximum Curvature	70.7	6.5	97.5
Median	Maximum Curvature	70.4	6.3	97.5
Median	Median	70.4	7.5	97.6
Maximum Curvature	Median	70.5	7.4	97.8

Table 5: Ablation study comparing different threshold configurations for

\tau_{\text{i}}

and

\tau_{\text{t}}

. Base model is Qwen2.5-VL-7B + CGRO . Results are pixel-level metrics on the WFDD dataset.

Model	Image-level					Pixel-level
Model	AVG	SDD	DTD	WFDD	MVTec	AVG	SDD	DTD	WFDD	MVTec
Vanilla	63.4, 59.4	65.0, 13.7	68.3, 82.3	60.5, 63.7	59.6, 77.7	64.7, 2.6	60.6, 0.1	79.7, 4.7	58.3, 1.6	60.1, 4.1
Vanilla + ReAL	63.4, 59.4	65.0, 13.7	68.3, 82.3	60.5, 63.7	59.6, 77.7	61.7, 5.1	56.6, 0.1	76.0, 13.2	56.5, 2.0	57.5, 5.0
Vanilla + CGRO	83.9 , 76.7	81.4, 44.3	94.5, 96.4	79.9, 78.8	79.8, 87.1	72.7 , 5.9	72.5, 0.2	87.1, 13.1	63.3, 2.7	67.9, 7.5
Full	83.9 , 76.7	81.4, 44.3	94.5, 96.4	79.9, 78.8	79.8, 87.1	80.7 , 13.3	82.3, 2.6	94.2, 26.4	70.5, 7.4	75.6, 16.6

Table 6: Dataset-wise results image-level and pixel-level performance for the ReAL and CGRO ablation study. Ablation on ReAL and CGRO. Vanilla denotes the original Qwen2.5-VL-7B . All metrics are macro-averaged across four AD datasets. Detection results are reported as (AUROC, AUPR).

Spatial entropy $S_{\text{I}}$	Semantic relevant $S_{\text{T}}$	AVG			SDD			DTD			WFDD			MVTec
Spatial entropy $S_{\text{I}}$	Semantic relevant $S_{\text{T}}$	AUROC	AUPR	ACC	AUROC	AUPR	ACC	AUROC	AUPR	ACC	AUROC	AUPR	ACC	AUROC	AUPR	ACC
		72.7	5.9	92.6	72.5	0.2	97.0	87.1	13.1	97.9	63.3	2.7	95.6	67.9	7.5	80.0
✓		74.1	11.8	89.2	75.0	1.0	99.7	92.4	31.1	98.6	55.0	1.7	68.6	74.0	13.4	90.0
	✓	76.7	14.0	90.2	75.2	1.5	99.8	93.5	35.0	98.6	60.4	3.3	72.1	77.5	16.3	90.3
✓	✓	80.7	13.3	97.1	82.3	2.6	99.8	94.2	26.4	98.6	70.5	7.4	97.8	75.6	16.6	92.2

Table 7: Dataset-wise pixel-level results for the ablation on token selection in the ReAL module. Base model is Qwen2.5-VL-7B + CGRO . We compare four configurations: using all tokens, filtering with spatial entropy

S_{\text{I}}

, filtering with semantic relevance

S_{\text{T}}

, and combining both criteria. Results are pixel-level metrics reported on MVTec-AD, WFDD, SDD, and DTD, complementing Table 3 in the main paper.

Figure 7: Typical failure cases. (a) Attention distracted by salient but non-defective regions; (b) coarser predicted anomaly boundaries; (c) missed detections in multi-spot scenarios; and (d) overall localization failure caused by incorrect image-level predictions and the subsequent lack of valid reasoning.

Anomaly Localization

Anomaly localization identifies where unusual defects, events, or signals occur in images, videos, medical scans, and sensor data.

See all 147 papers

Latent Anomaly Knowledge Excavation: Unveiling Sparse Sensitive Neurons in Vision-Language Models
Apr 9, 2026Shaotian Li, Shangze Li, Chuancheng Shi +5Multimodal Anomaly Detection Zero-Shot Anomaly Detection
Macquarie University · Nanjing University of Science and Technology · The University of Sydney +1
Towards Explainable Industrial Anomaly Detection via Knowledge-Guided Latent Reasoning
Feb 10, 2026Peng Chen, Wenqiang Wang, Yunkang Cao +7Industrial Anomaly Detection Multimodal Anomaly Detection
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China · School of Artificial Intelligence and Robotics, Hunan University, Changsha, China · Department of Computer and Information Science, University of Macau, Macau, China
AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization
Aug 6, 2025Jingyi Liao, Yongyi Su, Rong-Cheng Tu +6VLM Adaptation Industrial Anomaly Detection
1Nanyang Technological University · Institute for Infocomm Research (I2R), A*STAR · 3South China University of Technology

Apr 9, 2026J/K move · Enter open · S save

Latent Anomaly Knowledge Excavation: Unveiling Sparse Sensitive Neurons in Vision-Language Models

Shaotian Li, Shangze Li, Chuancheng Shi, Wenhua Wu+4

Macquarie University · Nanjing University of Science and Technology · The University of Sydney+1

Multimodal Anomaly Detection Zero-Shot Anomaly Detection Interpretable Anomaly Detection VLM Interpretability Mechanistic Interpretability

Open paper AskPDF arXiv

Large-scale vision-language models (VLMs) exhibit remarkable zero-shot capabilities, yet the internal mechanisms driving their anomaly detection (AD) performance remain poorly understood. Current methods predominantly treat VLMs as black-box feature extractors, assuming that anomaly-specific knowledge must be acquired through external adapters or memory banks. In this paper, we challenge this assumption by arguing that anomaly knowledge is intrinsically embedded within pre-trained models but remains latent and under-activated. We hypothesize that this knowledge is concentrated within a sparse subset of anomaly-sensitive neurons. To validate this, we propose latent anomaly knowledge excavation (LAKE), a training-free framework that identifies and elicits these critical neuronal signals using only a minimal set of normal samples. By isolating these sensitive neurons, LAKE constructs a highly compact normality representation that integrates visual structural deviations with cross-modal semantic activations. Extensive experiments on industrial AD benchmarks demonstrate that LAKE achieves state-of-the-art performance while providing intrinsic, neuron-level interpretability. Ultimately, our work advocates for a paradigm shift: redefining anomaly detection as the targeted activation of latent pre-trained knowledge rather than the acquisition of a downstream task.

Abstract

Anomaly Localization

Explore similar work

Latent Anomaly Knowledge Excavation: Unveiling Sparse Sensitive Neurons in Vision-Language Models

Towards Explainable Industrial Anomaly Detection via Knowledge-Guided Latent Reasoning

AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization