AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison
Authors: Xi Jiang, Yue Guo, Jian Li, Yong Liu, Bin-Bin Gao, Hanqiu Deng, Jun Liu, Heng Zhao, +2 more
Organizations: Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China · Macau University of Science and Technology, Macau, China · Tencent YouTu Lab, Shenzhen and Shanghai, China · Nanjing University, Nanjing, China · Centre for Frontier AI Research, A*STAR, Singapore · Shanghai Jiao Tong University, Shanghai, China
Multimodal Large Language Models (MLLMs) have achieved impressive success in natural visual understanding, yet they consistently underperform in industrial anomaly detection (IAD). This is because MLLMs trained mostly on general web data differ significantly from industrial images. Moreover, they encode each image independently and can only compare images in the language space, making them insensitive to subtle visual differences that are key to IAD. To tackle these issues, we present AD-Copilot, an interactive MLLM specialized for IAD via visual in-context comparison. We first design a novel data curation pipeline to mine inspection knowledge from sparsely labeled industrial images and generate precise samples for captioning, VQA, and defect localization, yielding a large-scale multimodal dataset Chat-AD rich in semantic signals for IAD. On this foundation, AD-Copilot incorporates a novel Comparison Encoder that employs cross-attention between paired image features to enhance multi-image fine-grained perception, and is trained with a multi-stage strategy that incorporates domain knowledge and gradually enhances IAD skills. In addition, we introduce MMAD-BBox, an extended benchmark for anomaly localization with bounding-box-based evaluation. The experiments show that AD-Copilot achieves 82.3% accuracy on the MMAD benchmark, outperforming all other models without any data leakage. In the MMAD-BBox test, it achieves a maximum improvement of 3.35× over the baseline. AD-Copilot also exhibits excellent generalization of its performance gains across other specialized and general-purpose benchmarks. Remarkably, AD-Copilot surpasses human expert-level performance on several IAD tasks, demonstrating its potential as a reliable assistant for real-world industrial inspection. All datasets and models will be released for the broader benefit of the community.
Figures & tables
Fig. 1: AD-Copilot was trained on meticulously curated datasets, including high-quality synthetic IAD caption and VQA data, together with general-domain data aligned with IAD objectives. Experiment results show that AD-Copilot delivers strong performance across a range of IAD tasks. Case studies on downstream applications further highlight its practical utility and substantial potential in real-world manufacturing settings.
Fig. 2: The pipeline for Chat-AD synthesis. Each query image is paired with a normal template image, and the generated text explicitly describes visual differences between them, establishing comparison as the foundational data paradigm. ∗ Currently private, upcoming public dataset.
Fig. 3: The model architecture of AD-Copilot. The Comparison Encoder performs cross-attention between paired image features to generate comparison tokens, enabling visual in-context comparison at the encoding stage.
Stage
Training Data Composition
Amount
Stage 0: Learning to Compare Full Finetuning: Comparison Encoder
TABLE I: Overview of data mixture for each training stage. All datasets have been curated through our data pipeline (Sec. III ).
Fig. 4: Task and metric of MMAD-BBox.
Anomaly
Defect
Object
Model
Scale
Discrimination
Classification
Localization
Description
Analysis
Classification
Analysis
Average
Human (expert)
-
95.24
75.00
92.31
83.33
94.20
86.11
80.37
86.65
Human (ordinary)
-
86.90
66.25
85.58
71.25
81.52
89.58
69.72
78.69
Gemini-1.5-pro
-
68.63
60.12
58.56
70.38
82.46
89.20
82.25
73.09
GPT-4o
-
68.63
65.80
55.62
73.21
83.41
94.98
82.80
74.92
Seed1.5-VL
-
65.30
64.32
63.35
73.80
84.38
91.65
83.67
75.21
TABLE II: Performance comparison of both proprietary and open-source MLLMs in MMAD with the standard 1-shot setting. † Thinking model. ∗ Models trained with MMAD data and used domain knowledge.
ACC@IoU(%)
Model
Scale
mIoU (%)
0.1
0.2
0.3
0.5
Qwen2.5-VL
7B
10.47
28.39
18.83
13.39
5.15
Qwen3-VL
8B
9.30
25.14
17.39
12.30
5.74
InternVL3.5
8B
16.34
38.15
29.26
22.63
12.54
Seed1.5-VL
-
16.58
38.62
23.90
18.81
12.50
Qwen2.5-VL(Finetuned)
7B
23.30
54.08
41.44
32.39
17.35
TABLE III: Comparison on the MMAD-BBox benchmark (1-shot).
Table 8
Fig. 5: Data scaling behavior of AD-Copilot. Performance improves with more data, but removing the Comparison Encoder or using lower-quality annotations causes large drops.
Anomaly
Defect
Object
Model
Scale
Discrimination
Classification
Localization
Description
Analysis
Classification
Analysis
Average
AD-Copilot-Thinking
7B
73.95
74.29
76.40
84.92
86.93
91.86
87.67
82.29
w/o bbox reward
7B
65.87
71.44
71.37
84.46
87.82
90.14
87.05
79.74
w/o choice reward
7B
65.24
65.81
62.47
77.67
84.63
89.30
82.27
75.34
w/o RL
7B
73.63
67.12
64.89
79.58
86.15
92.10
87.70
78.74
TABLE VI: Ablation study of reward functions in the RL stage.
Comparison Encoder
Stage 0
Stage 1
Stage 2
Stage 3
ACC
F1
72.19
72.49
✓
77.28
65.65
✓
✓
76.92
73.27
✓
✓
✓
77.91
73.60
✓
✓
✓
✓
78.74
75.14
✓
✓
✓
✓
80.46
71.73
TABLE VII: Ablation study of AD-Copilot on the MMAD benchmark.
Fig. 6: Visualization of the attention ratio between base tokens and comparison tokens across different tasks.
Model
Accuracy
Recall
Precision
F1
Qwen2.5-VL-3B
61.51
53.19
74.87
60.88
+ Comparison Encoder
62.69
57.69
73.89
63.27
TABLE VIII: Transfer of the Comparison Encoder to Qwen2.5-VL-3B on the Anomaly Discrimination task of MMAD (without finetuning).
Model
0-shot
2-shot
4-shot
8-shot
Qwen2.5-VL-7B
68.91
72.20
72.29
71.63
AD-Copilot-7B
73.35
77.53
77.42
76.67
TABLE IX: Multi-shot experiment on MMAD (Average ACC).
Fig. 7: Qualitative evaluation of AD-Copilot compared with other models across various IAD tasks.
Model Setting
ACC
F1
AD-Copilot-Thinking
82.29
74.23
+ Domain Knowledge (Inference)
82.84
74.59
+ Domain Knowledge (Training & Inference)
83.15
77.60
TABLE X: Effect of incorporating domain knowledge.
Setting
ACC
F1
0-shot (No Template)
79.15
36.41
1-shot (Default, Random Template)
82.29
74.23
1-shot+ (Retrieved Template)
83.37
74.85
TABLE XI: Effect of template image settings on model performance.
In the progress of industrial anomaly detection, general anomaly detection (GAD) is an emerging trend and also the ultimate goal. Unlike the conventional single- and multi-class AD, general AD aims to train a general AD model that can directly detect anomalies in diverse novel classes without any retraining or fine-tuning on the target data. Recently, Multimodal Large Language Models (MLLMs) have shown great promise in achieving general anomaly detection due to their revolutionary visual understanding and language reasoning capabilities. However, MLLM's general AD ability remains underexplored due to: (1) MLLMs are pretrained on amounts of data sourced from the Web, these data still have significant gaps with the data in AD scenarios. Moreover, the image-text pairs during pretraining are also not specifically for AD tasks. (2) The current mainstream AD datasets are image-based and not yet suitable for post-training MLLMs. To facilitate MLLM-based general AD research, we present MMR-AD, which is a comprehensive benchmark for both training and evaluating MLLM-based AD models. With MMR-AD, we reveal that the AD performance of current SOTA generalist MLLMs still falls far behind the industrial requirements. Based on MMR-AD, we also propose a baseline model, Anomaly-R1, which is a reasoning-based AD model that learns from the CoT data in MMR-AD and is further enhanced by reinforcement learning. Extensive experiments show that our Anomaly-R1 achieves remarkable improvements over generalist MLLMs in both anomaly detection and localization.
Xincheng Yao, Zefeng Qian, Chao Shi +2
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University · MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University
Multimodal large language models (MLLMs) have shown remarkable capability in bridging visual perception and textual reasoning, enabling zero-shot understanding across diverse industrial scenarios. However, their performance in open-vocabulary industrial anomaly detection (IAD) is often limited by domain-misaligned reasoning and hallucinated structural inferences. To address these challenges, we propose IndusAgent, a tool-augmented agentic framework for open-vocabulary IAD. Specifically, we first construct Indus-CoT, a structured dataset that integrates global visual observations, high-resolution local patches, and expert normalcy priors, providing supervision for fine-tuning the model on rigorous industrial inspection trajectories. Building on this, IndusAgent dynamically orchestrates a set of external tools, including dynamic region cropping, high-frequency feature enhancement, and prior retrieval, thus enabling the agent to actively resolve visual ambiguities and disentangle subtle anomalies. Furthermore, we introduce a gated reinforcement learning objective that jointly optimizes anomaly classification, localization accuracy, anomaly type reasoning, and efficient tool usage, ensuring that tool invocation occurs only when beneficial. Extensive evaluations on five industrial anomaly benchmarks, including MVTec-AD, VisA, MPDD, DTD, and SDD, demonstrate that IndusAgent achieves state-of-the-art zero-shot performance among all existing methods, validating our robustness and generalization capacity.
Rongbin Tan, Fangfang Lin, Zhenlong Yuan +10
State Key Lab of Processors, Institute of Computing Technology, CAS
A single multimodal large language model (MLLM) struggles to excel simultaneously at detection, localization, description, and reasoning in multimodal industrial anomaly understanding (MM-IAU). We show that in-domain training does not close this gap. On MMAD, a widely adopted MM-IAU benchmark, trained specialists reach at most 75.5% accuracy in defect localization, against 92.3% for human experts, and even detect anomalies less accurately than their untrained base model. Meanwhile, different MLLMs offer complementary strengths but share this weakness in fine-grained perception, so combining them alone cannot remove it. We therefore propose SiGMA, a spatially grounded multi-agent framework that divides labor between heterogeneous MLLM agents and a dedicated visual defect expert. A multimodal searcher supplies industrial knowledge and normal references, the defect expert turns query-reference comparison into calibrated anomaly evidence, and a label-free reliability controller weighs each source by task-wise competence and query-level evidence quality. SiGMA reaches 85.2% average accuracy on MMAD, 4.0% above the strongest trained specialist and Gemini-2.5-Pro and within 1.5% of human experts. Even with three agents of at most 9B parameters, it reaches 84.4%, and new MLLMs join without retraining.
Xingwu Zhang, Duanyang Du, Huiling Zhu +5
Hunan University · University of Aberdeen · South China Normal University +1