AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison
Authors: Xi Jiang, Yue Guo, Jian Li, Yong Liu, Bin-Bin Gao, Hanqiu Deng, Jun Liu, Heng Zhao, +2 more
Organizations: Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China · Macau University of Science and Technology, Macau, China · Tencent YouTu Lab, Shenzhen and Shanghai, China · Nanjing University, Nanjing, China · Centre for Frontier AI Research, A*STAR, Singapore · Shanghai Jiao Tong University, Shanghai, China
Multimodal Large Language Models (MLLMs) have achieved impressive success in natural visual understanding, yet they consistently underperform in industrial anomaly detection (IAD). This is because MLLMs trained mostly on general web data differ significantly from industrial images. Moreover, they encode each image independently and can only compare images in the language space, making them insensitive to subtle visual differences that are key to IAD. To tackle these issues, we present AD-Copilot, an interactive MLLM specialized for IAD via visual in-context comparison. We first design a novel data curation pipeline to mine inspection knowledge from sparsely labeled industrial images and generate precise samples for captioning, VQA, and defect localization, yielding a large-scale multimodal dataset Chat-AD rich in semantic signals for IAD. On this foundation, AD-Copilot incorporates a novel Comparison Encoder that employs cross-attention between paired image features to enhance multi-image fine-grained perception, and is trained with a multi-stage strategy that incorporates domain knowledge and gradually enhances IAD skills. In addition, we introduce MMAD-BBox, an extended benchmark for anomaly localization with bounding-box-based evaluation. The experiments show that AD-Copilot achieves 82.3% accuracy on the MMAD benchmark, outperforming all other models without any data leakage. In the MMAD-BBox test, it achieves a maximum improvement of 3.35× over the baseline. AD-Copilot also exhibits excellent generalization of its performance gains across other specialized and general-purpose benchmarks. Remarkably, AD-Copilot surpasses human expert-level performance on several IAD tasks, demonstrating its potential as a reliable assistant for real-world industrial inspection. All datasets and models will be released for the broader benefit of the community.
Figures & tables
Fig. 1: AD-Copilot was trained on meticulously curated datasets, including high-quality synthetic IAD caption and VQA data, together with general-domain data aligned with IAD objectives. Experiment results show that AD-Copilot delivers strong performance across a range of IAD tasks. Case studies on downstream applications further highlight its practical utility and substantial potential in real-world manufacturing settings.
Fig. 2: The pipeline for Chat-AD synthesis. Each query image is paired with a normal template image, and the generated text explicitly describes visual differences between them, establishing comparison as the foundational data paradigm. ∗ Currently private, upcoming public dataset.
Fig. 3: The model architecture of AD-Copilot. The Comparison Encoder performs cross-attention between paired image features to generate comparison tokens, enabling visual in-context comparison at the encoding stage.
Stage
Training Data Composition
Amount
Stage 0: Learning to Compare Full Finetuning: Comparison Encoder
TABLE I: Overview of data mixture for each training stage. All datasets have been curated through our data pipeline (Sec. III ).
Fig. 4: Task and metric of MMAD-BBox.
Anomaly
Defect
Object
Model
Scale
Discrimination
Classification
Localization
Description
Analysis
Classification
Analysis
Average
Human (expert)
-
95.24
75.00
92.31
83.33
94.20
86.11
80.37
86.65
Human (ordinary)
-
86.90
66.25
85.58
71.25
81.52
89.58
69.72
78.69
Gemini-1.5-pro
-
68.63
60.12
58.56
70.38
82.46
89.20
82.25
73.09
GPT-4o
-
68.63
65.80
55.62
73.21
83.41
94.98
82.80
74.92
Seed1.5-VL
-
65.30
64.32
63.35
73.80
84.38
91.65
83.67
75.21
TABLE II: Performance comparison of both proprietary and open-source MLLMs in MMAD with the standard 1-shot setting. † Thinking model. ∗ Models trained with MMAD data and used domain knowledge.
ACC@IoU(%)
Model
Scale
mIoU (%)
0.1
0.2
0.3
0.5
Qwen2.5-VL
7B
10.47
28.39
18.83
13.39
5.15
Qwen3-VL
8B
9.30
25.14
17.39
12.30
5.74
InternVL3.5
8B
16.34
38.15
29.26
22.63
12.54
Seed1.5-VL
-
16.58
38.62
23.90
18.81
12.50
Qwen2.5-VL(Finetuned)
7B
23.30
54.08
41.44
32.39
17.35
TABLE III: Comparison on the MMAD-BBox benchmark (1-shot).
Table 8
Fig. 5: Data scaling behavior of AD-Copilot. Performance improves with more data, but removing the Comparison Encoder or using lower-quality annotations causes large drops.
Anomaly
Defect
Object
Model
Scale
Discrimination
Classification
Localization
Description
Analysis
Classification
Analysis
Average
AD-Copilot-Thinking
7B
73.95
74.29
76.40
84.92
86.93
91.86
87.67
82.29
w/o bbox reward
7B
65.87
71.44
71.37
84.46
87.82
90.14
87.05
79.74
w/o choice reward
7B
65.24
65.81
62.47
77.67
84.63
89.30
82.27
75.34
w/o RL
7B
73.63
67.12
64.89
79.58
86.15
92.10
87.70
78.74
TABLE VI: Ablation study of reward functions in the RL stage.
Comparison Encoder
Stage 0
Stage 1
Stage 2
Stage 3
ACC
F1
72.19
72.49
✓
77.28
65.65
✓
✓
76.92
73.27
✓
✓
✓
77.91
73.60
✓
✓
✓
✓
78.74
75.14
✓
✓
✓
✓
80.46
71.73
TABLE VII: Ablation study of AD-Copilot on the MMAD benchmark.
Fig. 6: Visualization of the attention ratio between base tokens and comparison tokens across different tasks.
Model
Accuracy
Recall
Precision
F1
Qwen2.5-VL-3B
61.51
53.19
74.87
60.88
+ Comparison Encoder
62.69
57.69
73.89
63.27
TABLE VIII: Transfer of the Comparison Encoder to Qwen2.5-VL-3B on the Anomaly Discrimination task of MMAD (without finetuning).
Model
0-shot
2-shot
4-shot
8-shot
Qwen2.5-VL-7B
68.91
72.20
72.29
71.63
AD-Copilot-7B
73.35
77.53
77.42
76.67
TABLE IX: Multi-shot experiment on MMAD (Average ACC).
Fig. 7: Qualitative evaluation of AD-Copilot compared with other models across various IAD tasks.
Model Setting
ACC
F1
AD-Copilot-Thinking
82.29
74.23
+ Domain Knowledge (Inference)
82.84
74.59
+ Domain Knowledge (Training & Inference)
83.15
77.60
TABLE X: Effect of incorporating domain knowledge.
Setting
ACC
F1
0-shot (No Template)
79.15
36.41
1-shot (Default, Random Template)
82.29
74.23
1-shot+ (Retrieved Template)
83.37
74.85
TABLE XI: Effect of template image settings on model performance.
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University · MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University