Organizations: Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Casivision · Weiqiao-UCAS Science and Technology Park
Large multimodal models (LMMs) exhibit strong task generalization capabilities, offering new opportunities for zero-shot visual anomaly segmentation (ZSAS). However, existing LMM-based segmentation approaches still face fundamental limitations: anomaly concepts are inherently abstract and context-dependent, lacking stable visual prototypes, and the weak alignment between high-level semantic embeddings and pixel-level spatial features hinders precise anomaly localization. To address these challenges, we present AG-VAS (Anchor-Guided Visual Anomaly Segmentation), a new framework that expands the LMM vocabulary with three learnable semantic anchor tokens-[SEG], [NOR], and [ANO], establishing a unified anchor-guided segmentation paradigm. Specifically, [SEG] serves as an absolute semantic anchor that translates abstract anomaly semantics into explicit, spatially grounded visual entities (e.g., holes or scratches), while [NOR] and [ANO] act as relative anchors that model the contextual contrast between normal and abnormal patterns across categories. To further enhance cross-modal alignment, we introduce a Semantic-Pixel Alignment Module (SPAM) that aligns language-level semantic embeddings with high-resolution visual features, along with an Anchor-Guided Mask Decoder (AGMD) that performs anchor-conditioned mask prediction for precise anomaly localization. In addition, we curate Anomaly-Instruct20K, a large-scale instruction dataset that organizes anomaly knowledge into structured descriptions of appearance, shape, and spatial attributes, facilitating effective learning and integration of the proposed semantic anchors. Extensive experiments on six industrial and medical benchmarks demonstrate that AG-VAS achieves consistent state-of-the-art performance in the zero-shot setting.
Figures & tables
Figure 1: Comparison of zero-shot anomaly segmentation capabilities between LISA [ 29 ] and our AG-VAS.
Figure 2: AG-VAS is an effective and efficient LMM for zero-shot anomaly segmentation and understanding. (Left: raw image; Right: segmentation result.) The visualization results are shown across the following scenarios: (1)–(4) direct segmentation, (5)–(6) describe-then-segment, (7) segment-then-explain, and (8) segmentation within dialogue.
Figure 3: The pipeline of AG-VAS. We design two types of semantic anchors: the absolute anchor [ SEG ], which injects world and contextual knowledge about the appearance, structure, and location of anomalies (e.g., holes, scratches) into the segmentation process, and the relative normal/anomalous anchors [ NOR ]/[ ANO ], which guide the model to understand anomalies in relation to normal patterns. All parameters except those for the semantic/pixel image encoder, visual projector and LLM are trainable.
Figure 4: (a) An example from our Anomaly-Instruct20K dataset. (b) An example of instruction-tuning with dynamic fusion of templates during the actual training process. Note that only one specific template is shown here, with further details provided in the Appendix.
Method
Base Model
Industrial Datasets
Medical Datasets
MVTec-AD
KSDD2
RSDD
ISIC
ColonDB
ClinicDB
CLIP-Based (AP, F1-Max, IoUano )
WinCLIP [ 25 ]
CLIP(ViT-B-16-240)
(18.0, 24.3, 0.0)
(17.1, 24.6, 0.0)
(2.1, 6.6, 0.1)
(62.4, 64.1, 0.24)
(14.3, 21.0, 0.0)
(19.4, 27.3, 0.0)
APRIL-GAN [ 15 ]
CLIP(ViT-L-14-336)
(40.8, 43.3, 25.1)
(61.6, 62.6, 41.8)
(30.6, 40.2, 24.3)
(69.8, 69.3, 12.8)
(23.2, 32.5, 14.4)
(38.8, 44.4, 25.3)
AnomalyCLIP [ 72 ]
CLIP(ViT-L-14-336)
(34.5, 39.1, 22.6)
(41.8, 50.3, 27.6)
(19.1, 30.6, 8.8)
(74.4, 70.8, 51.0)
(31.3, 39.8, 25.2)
(42.2, 45.9, 30.0)
AnomalyCLIP* [ 72 ]
CLIP(ViT-L-14-336)
(35.2, 39.0, 22.4)
(48.8, 53.6, 28.6)
(25.5, 34.4, 8.5)
(76.5, 71.4, 54.1)
(34.2, 39.8, 22.7)
(44.1, 47.3, 29.4)
Bayes-PFL [ 45 ]
CLIP(ViT-L-14-336)
(48.3, 49.1, 22.2)
(73.7, 63.5, 25.4)
(39.1, 45.2, 3.7)
(84.6, 76.8, 60.0)
(31.9, 39.2, 16.8)
(53.2, 51.7, 21.0)
Table 1: Comparison with existing state-of-the-art methods. The best results are marked in red , while the second-best methods are indicated in blue . The superscript * indicates that these methods were retrained on the anomaly segmentation dataset used in our AG-VAS.
Method
Base Model
IoUano
IoUnor
Average
PixelLM [ 52 ]
LLaVA-13B
11.9
2.4
7.2
PaDT [ 55 ]
Qwen2.5-VL-7B
16.4
0.0
8.2
LISA [ 29 ]
LLaVA-7B
10.2
1.9
6.1
LISA [ 29 ]
LLaVA-13B
13.7
6.3
10.0
LISA* [ 29 ]
LLaVA-13B
31.8
43.5
37.7
LISA [ 29 ]
LLaVA-OneVision-7B
32.2
4.0
18.1
Table 2: Comparison of the model’s ability to reject segmentation on normal samples on the MVTec-AD.
AP
F1-Max
IoUnor
IoUano
Module Ablation
w/o [SEG]
49.1
50.9
85.6
42.1
w/o [NOR][ANO]
47.2
49.7
52.1
39.5
w/o SPAM
46.5
48.0
70.6
41.4
Training Dataset Ablation
w/o Anomaly-Instruct20K
48.4
50.0
85.2
39.9
w/o Anomaly-Seg20K
49.3
51.5
53.5
42.4
w/o General Segmentation
36.1
36.5
70.2
34.9
Table 3: Ablation on different components.
Segmentation Mode
AP
F1-Max
IoUnor
IoUano
Describe-then-segment
49.3
51.3
58.8
43.6
Describe-then-segment-plus
51.9
53.0
90.2
45.1
Segment-then-explain
48.6
51.4
91.1
43.4
Direct Segmentation
51.0
52.7
87.7
44.8
Table 4: Ablation on different segmentation mode during inference.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: The Anomaly-Instruct20K generation pipeline for ZSAS.
Figure 6: Word cloud of Anomaly-Instruct20K. Note that the special anchor words [NOR], [ANO], and [SEG] have been removed to reflect the true data distribution.
Dataset
Name
Type
Task
Number
Sampling Rate
General Segmentation Datasets
ADE20K [ 71 ]
Semantic Segmentation
Direct Segmentation
∼ 200k
0.4
COCO-Stuff [ 12 ]
Direct Segmentation
PACO-LVIS [ 50 ]
Part Semantic Segmentation
Direct Segmentation
PartImageNet [ 23 ]
Direct Segmentation
PASCAL-Part [ 14 ]
Direct Segmentation
refCLEF [ 27 ]
Referring Segmentation
Direct Segmentation
Appendix
Table 5: Construction of the Training Dataset.
Figure 7: The proposed AGMD architecture.
Table 13
Figure 8: Failure Cases.
Figure 9: The design of system prompts.
Figure 10: The design of instruction and answer templates for direct segmentation.
Figure 11: The design of instruction templates for Describe-then-Segment and Rejection.
object
AnomalyCLIP
Bayes-PFL
PaDT
PixelLM
LISA-7B
LISA-13B
LISA-13B*
LISA-OV
LISA-OV*
AG-VAS
bottle
55.3
64.7
10.1
22.9
14.2
15.6
34.3
35.0
47.5
45.8
cable
12.3
15.1
6.1
9.7
8.8
22.6
10.8
25.3
27.8
27.1
capsule
27.7
34.4
4.0
7.0
15.1
15.7
14.4
14.9
17.1
38.4
carpet
56.6
82.2
1.4
40.8
1.1
11.3
59.4
30.4
54.5
72.8
grid
24.1
41.1
0.7
0.7
0.4
0.5
60.4
23.4
57.5
64.1
hazelnut
43.4
70.9
5.8
7.6
11.5
11.7
36.0
34.8
55.9
70.2
Appendix
Table 8: Comparison of different categories in terms of pixel-level AP on the MVTec-AD.
object
AnomalyCLIP
Bayes-PFL
PaDT
PixelLM
LISA-7B
LISA-13B
LISA-13B*
LISA-OV
LISA-OV*
AG-VAS
bottle
51.6
61.3
18.9
34.6
27.4
30.5
38.3
38.3
46.4
47.0
cable
18.9
23.5
14.8
16.4
13.8
27.3
17.1
36.9
38.0
34.5
capsule
31.0
40.5
7.9
13.2
27.2
26.3
27.8
26.7
27.8
46.5
carpet
57.1
74.9
3.7
43.7
3.2
18.6
57.2
37.7
52.4
66.3
grid
32.0
41.6
1.6
1.5
1.4
1.4
61.7
29.1
59.2
64.2
hazelnut
47.6
66.2
11.3
15.5
23.0
23.2
44.3
36.3
56.9
65.0
Appendix
Table 9: Comparison of different categories in terms of pixel-level F1-Max on the MVTec-AD.
object
AnomalyCLIP
Bayes-PFL
PaDT
PixelLM
LISA-7B
LISA-13B
LISA-13B*
LISA-OV
LISA-OV*
AG-VAS
bottle
27.3
25.2
20.0
26.7
20.0
25.4
30.7
36.7
23.8
41.4
cable
10.3
7.5
11.5
5.0
3.4
8.9
4.0
23.4
16.0
29.2
capsule
16.8
7.6
4.7
4.9
5.4
8.3
16.0
14.1
12.1
36.3
carpet
32.5
48.9
16.4
29.6
3.3
13.6
46.5
39.0
43.6
52.1
grid
20.5
24.4
12.9
2.0
0.8
0.5
43.7
32.8
47.9
41.7
hazelnut
28.0
9.4
14.5
13.2
17.6
23.0
45.8
44.7
45.7
64.4
Appendix
Table 10: Comparison of different categories in terms of IoUano on the MVTec-AD.
object
AnomalyCLIP
Bayes-PFL
PaDT
PixelLM
LISA-7B
LISA-13B
LISA-13B*
LISA-OV
LISA-OV*
AG-VAS
bottle
0.0
0.0
0.0
0.0
0.0
0.0
10.0
0.0
75.0
85.0
cable
17.2
0.0
0.0
34.5
25.9
17.2
0.0
0.0
56.9
55.2
capsule
73.9
0.0
0.0
0.0
0.0
0.0
0.0
0.0
91.3
95.7
carpet
78.6
96.4
0.0
0.0
0.0
3.6
100.0
57.1
89.3
100.0
grid
76.2
95.2
0.0
0.0
0.0
0.0
95.2
0.0
95.2
95.2
hazelnut
7.5
0.0
0.0
0.0
0.0
0.0
0.0
0.0
65.0
97.5
Appendix
Table 11: Comparison of different categories in terms of IoUnor on the MVTec-AD.
Figure 12: Comparison of the dialogue and segmentation capabilities between the proposed AG-VAS and LISA [ 29 ] .
Figure 13: Comparison of the dialogue and segmentation capabilities between the proposed AG-VAS and LISA [ 29 ] .
Figure 14: Some examples under the describe-then-segment task. Left: original images; Right: AG-VAS segmentation results.
Figure 15: Some examples under the segment-then-explain task. Left: original images; Right: AG-VAS segmentation results.
Figure 16: Generalization of AG-VAS to fine-grained segmentation instructions. Left: original images; Right: AG-VAS segmentation results. AG-VAS effectively captures both positional cues (e.g., “top-left”) and attribute-specific descriptions (e.g., “the smallest”), achieving precise localization of anomalous regions.
Figure 17: Examples showcasing AG-VAS’s ability to leverage world knowledge for implicit reasoning and direct segmentation. Left: original images; Right: AG-VAS segmentation results. AG-VAS exploits its learned world knowledge to conduct implicit reasoning and precisely localize anomalous regions.
Figure 18: Visualization of segmentation results for the bottle and tile classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5..
Figure 19: Visualization of segmentation results for the cable and grid classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.
Figure 20: Visualization of segmentation results for the capsule and carpet classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.
Figure 21: Visualization of segmentation results for the hazelnut and leather classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.
Figure 22: Visualization of segmentation results for the metal nut and pill classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.
Figure 23: Visualization of segmentation results for the toothbrush and transistor classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.
Figure 24: Visualization of segmentation results for the wood and zipper classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.
Figure 25: Visualization of segmentation results for the screw class on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.
Figure 26: Visualization of segmentation results on KSDD2 . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.
Figure 27: Visualization of segmentation results on RSDD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.
Figure 28: Visualization of segmentation results for the skin class on ISIC . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.
Figure 29: Visualization of segmentation results for the colon class on ClinicDB . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.
Figure 30: Visualization of segmentation results for the colon class on ColonDB . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.
Zero-shot anomaly detection (ZSAD) aims to detect and localize anomalies in unseen categories without access to target-specific training data. Although recent CLIP-based methods have demonstrated promising generalization through vision-language alignment, they remain limited in capturing diverse anomaly semantics and subtle local variations. To address these limitations, we propose VFAD, a unified framework that combines variational semantic prompting with frequency-adaptive representation learning. Specifically, we introduce a Variational Semantic Prompt Extractor (VSPE), which adaptively aggregates anomaly-relevant local semantics from dense patch tokens and regularizes them through a variational information bottleneck, thereby incorporating fine-grained visual cues and enabling more precise cross-modal alignment. Furthermore, we develop a Frequency-Adaptive Representation Aggregation (FARA) module that leverages wavelet-based frequency decomposition and frequency-specific expert aggregation to enhance anomaly-discriminative visual representations. By jointly strengthening semantic guidance and visual representation learning, VFAD improves both anomaly discrimination and fine-grained localization. Extensive experiments on 13 industrial and medical benchmarks demonstrate that VFAD consistently outperforms existing state-of-the-art ZSAD methods across diverse anomaly scenarios. The code will be publicly available upon publication.
Peng Chen, Kaige Li, Wei Wang +5
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China
Zero-shot anomaly detection (ZSAD) requires detecting and localizing anomalies without access to target-class anomaly samples. Mainstream methods rely on vision-language models (VLMs) such as CLIP: they build hand-crafted or learned prompt sets for normal and abnormal semantics, then compute image-text similarities for open-set discrimination. While effective, this paradigm depends on a text encoder and cross-modal alignment, which can lead to training instability and parameter redundancy. This work revisits the necessity of the text branch in ZSAD and presents VisualAD, a purely visual framework built on Vision Transformers. We introduce two learnable tokens within a frozen backbone to directly encode normality and abnormality. Through multi-layer self-attention, these tokens interact with patch tokens, gradually acquiring high-level notions of normality and anomaly while guiding patches to highlight anomaly-related cues. Additionally, we incorporate a Spatial-Aware Cross-Attention (SCA) module and a lightweight Self-Alignment Function (SAF): SCA injects fine-grained spatial information into the tokens, and SAF recalibrates patch features before anomaly scoring. VisualAD achieves state-of-the-art performance on 13 zero-shot anomaly detection benchmarks spanning industrial and medical domains, and adapts seamlessly to pretrained vision backbones such as the CLIP image encoder and DINOv2. Code: https://github.com/7HHHHH/VisualAD
Yanning Hou, Peiyuan Li, Zirui Liu +4
College of Intelligence Science and Technology, National University of Defense Technology, Changsha, China · School of Artificial Intelligence, Anhui University, Hefei, China · State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Anhui University, Hefei, China
Zero-shot anomaly detection (ZSAD) aims to identify anomalies in unseen domains, a setting that is particularly critical for industrial and medical applications where domain shifts are prevalent. However, most CLIP-based ZSAD methods anchor semantics solely on the text modality, making performance highly sensitive to prompt design and leading to weak visual grounding. To mitigate these limitations, we propose a Dual-Anchor framework that complements conventional text anchors with hierarchical image anchors constructed via a top-down grouping mechanism. This mechanism progressively aggregates local-to-global image features to form normal and abnormal group tokens, which serve as image anchors and act as gating signals in a Group-Gated Token Refiner to enhance the global representation. The refined image anchors are then fused with text prompts to construct dynamic state prompts. By jointly reinforcing visual and textual semantics, our framework stabilizes image-text alignment, reduces prompt dependency, and achieves strong generalization across 8 industrial and 6 medical benchmarks.