Organizations: School of Information Science and Electronic Engineering, Shanghai Jiao Tong University · MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University
In the progress of industrial anomaly detection, general anomaly detection (GAD) is an emerging trend and also the ultimate goal. Unlike the conventional single- and multi-class AD, general AD aims to train a general AD model that can directly detect anomalies in diverse novel classes without any retraining or fine-tuning on the target data. Recently, Multimodal Large Language Models (MLLMs) have shown great promise in achieving general anomaly detection due to their revolutionary visual understanding and language reasoning capabilities. However, MLLM's general AD ability remains underexplored due to: (1) MLLMs are pretrained on amounts of data sourced from the Web, these data still have significant gaps with the data in AD scenarios. Moreover, the image-text pairs during pretraining are also not specifically for AD tasks. (2) The current mainstream AD datasets are image-based and not yet suitable for post-training MLLMs. To facilitate MLLM-based general AD research, we present MMR-AD, which is a comprehensive benchmark for both training and evaluating MLLM-based AD models. With MMR-AD, we reveal that the AD performance of current SOTA generalist MLLMs still falls far behind the industrial requirements. Based on MMR-AD, we also propose a baseline model, Anomaly-R1, which is a reasoning-based AD model that learns from the CoT data in MMR-AD and is further enhanced by reinforcement learning. Extensive experiments show that our Anomaly-R1 achieves remarkable improvements over generalist MLLMs in both anomaly detection and localization.
Figures & tables
Figure 1: (a) Overview of our MMR-AD dataset. (b) Visualization of the anomaly detection (AD) and anomaly localization (AL) accuracy comparison. Through post-training on our MMR-AD dataset, our finetuned Anomaly-R1-7B shows remarkable performance improvement, especially in anomaly localization.
Image Source
Sampled Images
Text Tokens
Object Categories
Anomaly Types
Num Anomalies
MVTecAD [ 6 ]
2485
503306
15
37
1588
VisA [ 68 ]
2287
501651
12
35
1568
MVTecLOCO [ 5 ]
1889
366251
6
47
1078
MVTec3D [ 7 ]
1176
254543
10
26
1132
MPDD [ 23 ]
540
112631
6
10
416
GoodsAD [ 61 ]
2028
381232
6
13
1705
Table 1: Statistics on the composition of our MMR-AD dataset.
Figure 2: The illustration of the text generation pipeline. We utilize the (reference, input) image pair and leverage Qwen2.5-VL-72B to automate the generation of reasoning-based texts. We prompt the model to generate the reasoning-based AD thinking process ( i.e. , based on comparison and analysis of the two images). To ensure that the model can correctly recognize abnormal areas, we plot red bboxes on the image as visual hints and also provide the anomaly types and bbox coordinates of the abnormal areas as text hints to the instruction. In the generated texts, the red parts mark the anomaly-related reasoning words.
System Prompt
You are a professional industrial image inspector, particularly skilled at identifying abnormal (defective) areas in images.
User Prompt
- < Reference Image >< Input Image > The first image is a normal sample, and you need to carefully inspect and analyze these two images to determine if there are any anomalies (defects) in the second image.
- If there are anomalies, return their locations in the form of coordinates (hint: the red box in the second image is our reminder, and the location and label of the abnormal area is: hint ) .
- You first need to think about the reasoning and analysis process in the mind and then provide us with the answer.
- The red box is our hint, you can’t generate any content related to it.
Table 2: Template for text data generation instruction. hint will be replaced with the specific coordinates and types of anomalous regions. The prompt template will also be used in subsequent training and testing, but the gray part in the template will be removed.
Figure 3: Data examples from our MMR-AD dataset and comparison with MMAD [ 24 ] and Anomaly-Instruct-125K [ 49 ] . In our MMR-AD, each data example is based on two images: the first is the normal reference sample, and the second is the input sample. The reasoning and analysis words are highlighted in blue and red , red especially marks the anomaly-related reasoning words. Finally, a summary of the reasoning and analysis contents is provided at the end of < think > , these words are highlighted in bold .
Dataset
Object
Image Number
Defect
Reasoning
Text
Text Tokens
Anomalous Region
Categories
Normal
Anomaly
All
Types
Text
Tokens
Per Sample
Location
MMAD [ 24 ]
38
3290
5076
8366
244
✘
0.58M
69
coarse
Anomaly-Instruct-125K [ 49 ]
463
63628
61285
125K
/
✔ ✗
10M
80
coarse
MMR-AD
188
28222
98915
127K
395
✔
26M
208
precise
Table 3: Comparison with previous multi-modal AD datasets on different attributes. ✘: no reasoning text. ✔ ✗ : short description text. ✔: detailed reasoning text. “coarse” means anomalous location is described in coarse-grained words, such as “top left”, “bottom right”, etc. “precise” means anomalous location is described in fine-grained coordinates, such as “[ xmin , ymin , xmax , ymax ]”. In Anomaly-Instruct-125K, there is no defect type label given for each sample, thus the “Defect Types” column is hard to count.
Model
MVTecAD
VisA
MVTec3D
MPDD
Accuracy
Recall
Precision
Accuracy
Recall
Precision
Accuracy
Recall
Precision
Accuracy
Recall
Precision
Commercial Model
Gemini-2.5-pro
79.4/34.4
98.4/46.0
79.0/48.8
65.7/10.5
97.2/17.1
62.7/21.3
75.7/10.9
90.1/18.3
83.2/19.9
62.1/16.3
92.2/23.2
60.8/26.7
GPT-4o
68.9/8.1
74.1/12.9
82.4/16.9
57.6/3.7
69.4/5.8
61.1/9.2
67.8/7.0
74.0/11.5
83.4/14.8
63.1/13.6
84.3/20.4
64.3/22.3
GPT-5
78.7/41.8
95.9/64.2
79.4/53.0
65.8/18.5
94.5/31.0
63.3/30.7
75.0/27.2
85.3/41.1
84.1/43.2
68.4/21.2
91.9/34.5
67.0/29.2
OpenAI-o4-mini
81.3/34.4
94.4/53.5
83.1/46.5
73.0/12.3
89.1/20.9
71.4/22.2
74.1/15.7
77.0/25.5
89.3/29.4
70.5/23.9
85.7/40.6
74.1/31.1
Table 4: Performance evaluation of anomaly detection and localization for both commercial and open-source MLLMs. All the MLLM-based models (except AnomalyGPT) are based on the (reference, input) image pair as the image input and use the instruction template in Tab. 2 for generation. ⋅ / ⋅ means anomaly detection/localization metrics, respectively. Anomaly-R1-7B † is a variant in which we add domain knowledge (see Sec. 4.1 ). Bold means the best performance (the full-shot AD models are not included in the comparison).
Model
MVTecAD
VisA
Accuracy
Recall
Precision
Accuracy
Recall
Precision
w/o Reasoning Data
81.3/53.5
89.7/63.4
84.8/70.6
66.1/18.8
68.0/24.8
68.6/39.5
w/o Normal Reference Image
81.7/62.1
87.2/71.4
88.3/80.1
62.5/24.9
52.2/29.1
80.8/70.5
w/o CoT Cold-Start
75.1/12.3
84.3/15.4
81.4/23.9
68.0/5.3
77.5/6.3
68.3/10.8
w/o Reinforcement learning
84.5/55.9
88.1/64.4
87.9/77.7
73.3/34.3
63.8/40.3
83.4/68.3
w/o Contrastive Sampling
87.7/62.2
85.1/69.2
96.9/84.6
73.7/35.8
56.8/39.3
91.0/80.7
Table 5: Further analysis experiments. We investigate the impact of Reasoning-based Text, Normal Reference Image, Cold Start Initialization, Contrastive Sampling, and Domain knowledge.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Green marks correct reasoning and correct bbox coordinates, red marks wrong reasoning and imprecise bbox coordinates. Qwen2.5-VL-72B shows severe hallucination, thinking that there is an abnormal dark spot, and locating to the white area at the bottom of the image.
Figure 5: Green marks correct reasoning and correct bbox coordinates, red marks wrong reasoning and imprecise bbox coordinates. Both GPT-4o and Qwen2.5-VL-72B show hallucination, thinking that there is the copper protrusion defect without observing the severe cut defect.
Figure 6: Green marks correct reasoning and correct bbox coordinates, red marks wrong reasoning and imprecise bbox coordinates. Although both GPT-4o and Qwen2.5-VL-72B generate correct reasoning, the anomaly localization results are still not precise enough.
Figure 7: Failure case. Red marks wrong reasoning.
Figure 8: Failure case. Red marks wrong reasoning and incorrect bbox coordinates.
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China · Macau University of Science and Technology, Macau, China · Tencent YouTu Lab, Shenzhen and Shanghai, China +3