AG-VAS: Anchor-Guided Zero-Shot Visual Anomaly Segmentation with Large Multimodal Models

Zhen Qu; Xian Tao; Xiaoyi Bao; Dingrong Wang; ShiChen Qu; Zhengtao Zhang; Xingang Wang

cs.CVMar 1, 2026

AG-VAS: Anchor-Guided Zero-Shot Visual Anomaly Segmentation with Large Multimodal Models

Authors: Zhen Qu, Xian Tao, Xiaoyi Bao, Dingrong Wang, ShiChen Qu, Zhengtao Zhang, Xingang Wang

Organizations: Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Casivision · Weiqiao-UCAS Science and Technology Park

Abstract

Large multimodal models (LMMs) exhibit strong task generalization capabilities, offering new opportunities for zero-shot visual anomaly segmentation (ZSAS). However, existing LMM-based segmentation approaches still face fundamental limitations: anomaly concepts are inherently abstract and context-dependent, lacking stable visual prototypes, and the weak alignment between high-level semantic embeddings and pixel-level spatial features hinders precise anomaly localization. To address these challenges, we present AG-VAS (Anchor-Guided Visual Anomaly Segmentation), a new framework that expands the LMM vocabulary with three learnable semantic anchor tokens-[SEG], [NOR], and [ANO], establishing a unified anchor-guided segmentation paradigm. Specifically, [SEG] serves as an absolute semantic anchor that translates abstract anomaly semantics into explicit, spatially grounded visual entities (e.g., holes or scratches), while [NOR] and [ANO] act as relative anchors that model the contextual contrast between normal and abnormal patterns across categories. To further enhance cross-modal alignment, we introduce a Semantic-Pixel Alignment Module (SPAM) that aligns language-level semantic embeddings with high-resolution visual features, along with an Anchor-Guided Mask Decoder (AGMD) that performs anchor-conditioned mask prediction for precise anomaly localization. In addition, we curate Anomaly-Instruct20K, a large-scale instruction dataset that organizes anomaly knowledge into structured descriptions of appearance, shape, and spatial attributes, facilitating effective learning and integration of the proposed semantic anchors. Extensive experiments on six industrial and medical benchmarks demonstrate that AG-VAS achieves consistent state-of-the-art performance in the zero-shot setting.

Figures & tables

Figure 1: Comparison of zero-shot anomaly segmentation capabilities between LISA [ 29 ] and our AG-VAS.

Figure 2: AG-VAS is an effective and efficient LMM for zero-shot anomaly segmentation and understanding. (Left: raw image; Right: segmentation result.) The visualization results are shown across the following scenarios: (1)–(4) direct segmentation, (5)–(6) describe-then-segment, (7) segment-then-explain, and (8) segmentation within dialogue.

Figure 3: The pipeline of AG-VAS. We design two types of semantic anchors: the absolute anchor [ SEG ], which injects world and contextual knowledge about the appearance, structure, and location of anomalies (e.g., holes, scratches) into the segmentation process, and the relative normal/anomalous anchors [ NOR ]/[ ANO ], which guide the model to understand anomalies in relation to normal patterns. All parameters except those for the semantic/pixel image encoder, visual projector and LLM are trainable.

Figure 4: (a) An example from our Anomaly-Instruct20K dataset. (b) An example of instruction-tuning with dynamic fusion of templates during the actual training process. Note that only one specific template is shown here, with further details provided in the Appendix.

	Method	Base Model	Industrial Datasets			Medical Datasets
			MVTec-AD	KSDD2	RSDD	ISIC	ColonDB	ClinicDB
CLIP-Based (AP, F1-Max, $\text{IoU}_{\text{ano}}$ )	WinCLIP [ 25 ]	CLIP(ViT-B-16-240)	(18.0, 24.3, 0.0)	(17.1, 24.6, 0.0)	(2.1, 6.6, 0.1)	(62.4, 64.1, 0.24)	(14.3, 21.0, 0.0)	(19.4, 27.3, 0.0)
	APRIL-GAN [ 15 ]	CLIP(ViT-L-14-336)	(40.8, 43.3, 25.1)	(61.6, 62.6, 41.8)	(30.6, 40.2, 24.3)	(69.8, 69.3, 12.8)	(23.2, 32.5, 14.4)	(38.8, 44.4, 25.3)
	AnomalyCLIP [ 72 ]	CLIP(ViT-L-14-336)	(34.5, 39.1, 22.6)	(41.8, 50.3, 27.6)	(19.1, 30.6, 8.8)	(74.4, 70.8, 51.0)	(31.3, 39.8, 25.2)	(42.2, 45.9, 30.0)
	AnomalyCLIP* [ 72 ]	CLIP(ViT-L-14-336)	(35.2, 39.0, 22.4)	(48.8, 53.6, 28.6)	(25.5, 34.4, 8.5)	(76.5, 71.4, 54.1)	(34.2, 39.8, 22.7)	(44.1, 47.3, 29.4)
	Bayes-PFL [ 45 ]	CLIP(ViT-L-14-336)	(48.3, 49.1, 22.2)	(73.7, 63.5, 25.4)	(39.1, 45.2, 3.7)	(84.6, 76.8, 60.0)	(31.9, 39.2, 16.8)	(53.2, 51.7, 21.0)

Table 1: Comparison with existing state-of-the-art methods. The best results are marked in red , while the second-best methods are indicated in blue . The superscript * indicates that these methods were retrained on the anomaly segmentation dataset used in our AG-VAS.

Method	Base Model	$\text{IoU}_{\text{ano}}$	$\text{IoU}_{\text{nor}}$	Average
PixelLM [ 52 ]	LLaVA-13B	11.9	2.4	7.2
PaDT [ 55 ]	Qwen2.5-VL-7B	16.4	0.0	8.2
LISA [ 29 ]	LLaVA-7B	10.2	1.9	6.1
LISA [ 29 ]	LLaVA-13B	13.7	6.3	10.0
LISA* [ 29 ]	LLaVA-13B	31.8	43.5	37.7
LISA [ 29 ]	LLaVA-OneVision-7B	32.2	4.0	18.1

Table 2: Comparison of the model’s ability to reject segmentation on normal samples on the MVTec-AD.

		AP	F1-Max	$\text{IoU}_{\text{nor}}$	$\text{IoU}_{\text{ano}}$
Module Ablation	w/o [SEG]	49.1	50.9	85.6	42.1
	w/o [NOR][ANO]	47.2	49.7	52.1	39.5
	w/o SPAM	46.5	48.0	70.6	41.4
Training Dataset Ablation	w/o Anomaly-Instruct20K	48.4	50.0	85.2	39.9
	w/o Anomaly-Seg20K	49.3	51.5	53.5	42.4
	w/o General Segmentation	36.1	36.5	70.2	34.9

Table 3: Ablation on different components.

Segmentation Mode	AP	F1-Max	$\text{IoU}_{\text{nor}}$	$\text{IoU}_{\text{ano}}$
Describe-then-segment	49.3	51.3	58.8	43.6
Describe-then-segment-plus	51.9	53.0	90.2	45.1
Segment-then-explain	48.6	51.4	91.1	43.4
Direct Segmentation	51.0	52.7	87.7	44.8

Table 4: Ablation on different segmentation mode during inference.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The Anomaly-Instruct20K generation pipeline for ZSAS.

Figure 6: Word cloud of Anomaly-Instruct20K. Note that the special anchor words [NOR], [ANO], and [SEG] have been removed to reflect the true data distribution.

Dataset	Name	Type	Task	Number	Sampling Rate
General Segmentation Datasets	ADE20K [ 71 ]	Semantic Segmentation	Direct Segmentation	$\sim$ 200k	0.4
	COCO-Stuff [ 12 ]	Semantic Segmentation	Direct Segmentation
	PACO-LVIS [ 50 ]	Part Semantic Segmentation	Direct Segmentation
	PartImageNet [ 23 ]		Direct Segmentation
	PASCAL-Part [ 14 ]		Direct Segmentation
	refCLEF [ 27 ]	Referring Segmentation	Direct Segmentation

Appendix

Table 5: Construction of the Training Dataset.

Figure 7: The proposed AGMD architecture.

Table 13

Figure 9: The design of system prompts.

Figure 10: The design of instruction and answer templates for direct segmentation.

Figure 11: The design of instruction templates for Describe-then-Segment and Rejection.

object	AnomalyCLIP	Bayes-PFL	PaDT	PixelLM	LISA-7B	LISA-13B	LISA-13B*	LISA-OV	LISA-OV*	AG-VAS
bottle	55.3	64.7	10.1	22.9	14.2	15.6	34.3	35.0	47.5	45.8
cable	12.3	15.1	6.1	9.7	8.8	22.6	10.8	25.3	27.8	27.1
capsule	27.7	34.4	4.0	7.0	15.1	15.7	14.4	14.9	17.1	38.4
carpet	56.6	82.2	1.4	40.8	1.1	11.3	59.4	30.4	54.5	72.8
grid	24.1	41.1	0.7	0.7	0.4	0.5	60.4	23.4	57.5	64.1
hazelnut	43.4	70.9	5.8	7.6	11.5	11.7	36.0	34.8	55.9	70.2

Appendix

Table 8: Comparison of different categories in terms of pixel-level AP on the MVTec-AD.

object	AnomalyCLIP	Bayes-PFL	PaDT	PixelLM	LISA-7B	LISA-13B	LISA-13B*	LISA-OV	LISA-OV*	AG-VAS
bottle	51.6	61.3	18.9	34.6	27.4	30.5	38.3	38.3	46.4	47.0
cable	18.9	23.5	14.8	16.4	13.8	27.3	17.1	36.9	38.0	34.5
capsule	31.0	40.5	7.9	13.2	27.2	26.3	27.8	26.7	27.8	46.5
carpet	57.1	74.9	3.7	43.7	3.2	18.6	57.2	37.7	52.4	66.3
grid	32.0	41.6	1.6	1.5	1.4	1.4	61.7	29.1	59.2	64.2
hazelnut	47.6	66.2	11.3	15.5	23.0	23.2	44.3	36.3	56.9	65.0

Appendix

Table 9: Comparison of different categories in terms of pixel-level F1-Max on the MVTec-AD.

object	AnomalyCLIP	Bayes-PFL	PaDT	PixelLM	LISA-7B	LISA-13B	LISA-13B*	LISA-OV	LISA-OV*	AG-VAS
bottle	27.3	25.2	20.0	26.7	20.0	25.4	30.7	36.7	23.8	41.4
cable	10.3	7.5	11.5	5.0	3.4	8.9	4.0	23.4	16.0	29.2
capsule	16.8	7.6	4.7	4.9	5.4	8.3	16.0	14.1	12.1	36.3
carpet	32.5	48.9	16.4	29.6	3.3	13.6	46.5	39.0	43.6	52.1
grid	20.5	24.4	12.9	2.0	0.8	0.5	43.7	32.8	47.9	41.7
hazelnut	28.0	9.4	14.5	13.2	17.6	23.0	45.8	44.7	45.7	64.4

Appendix

Table 10: Comparison of different categories in terms of

\text{IoU}_{\text{ano}}

on the MVTec-AD.

object	AnomalyCLIP	Bayes-PFL	PaDT	PixelLM	LISA-7B	LISA-13B	LISA-13B*	LISA-OV	LISA-OV*	AG-VAS
bottle	0.0	0.0	0.0	0.0	0.0	0.0	10.0	0.0	75.0	85.0
cable	17.2	0.0	0.0	34.5	25.9	17.2	0.0	0.0	56.9	55.2
capsule	73.9	0.0	0.0	0.0	0.0	0.0	0.0	0.0	91.3	95.7
carpet	78.6	96.4	0.0	0.0	0.0	3.6	100.0	57.1	89.3	100.0
grid	76.2	95.2	0.0	0.0	0.0	0.0	95.2	0.0	95.2	95.2
hazelnut	7.5	0.0	0.0	0.0	0.0	0.0	0.0	0.0	65.0	97.5

Appendix

Table 11: Comparison of different categories in terms of

\text{IoU}_{\text{nor}}

on the MVTec-AD.

Figure 12: Comparison of the dialogue and segmentation capabilities between the proposed AG-VAS and LISA [ 29 ] .

Figure 13: Comparison of the dialogue and segmentation capabilities between the proposed AG-VAS and LISA [ 29 ] .

Figure 14: Some examples under the describe-then-segment task. Left: original images; Right: AG-VAS segmentation results.

Figure 15: Some examples under the segment-then-explain task. Left: original images; Right: AG-VAS segmentation results.

Figure 16: Generalization of AG-VAS to fine-grained segmentation instructions. Left: original images; Right: AG-VAS segmentation results. AG-VAS effectively captures both positional cues (e.g., “top-left”) and attribute-specific descriptions (e.g., “the smallest”), achieving precise localization of anomalous regions.

Figure 17: Examples showcasing AG-VAS’s ability to leverage world knowledge for implicit reasoning and direct segmentation. Left: original images; Right: AG-VAS segmentation results. AG-VAS exploits its learned world knowledge to conduct implicit reasoning and precisely localize anomalous regions.

Figure 18: Visualization of segmentation results for the bottle and tile classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5..

Figure 19: Visualization of segmentation results for the cable and grid classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.

Figure 20: Visualization of segmentation results for the capsule and carpet classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.

Figure 21: Visualization of segmentation results for the hazelnut and leather classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.

Figure 22: Visualization of segmentation results for the metal nut and pill classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.

Figure 23: Visualization of segmentation results for the toothbrush and transistor classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.

Figure 24: Visualization of segmentation results for the wood and zipper classes on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.

Figure 25: Visualization of segmentation results for the screw class on MVTecAD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.

Figure 26: Visualization of segmentation results on KSDD2 . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.

Figure 27: Visualization of segmentation results on RSDD . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.

Figure 28: Visualization of segmentation results for the skin class on ISIC . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.

Figure 29: Visualization of segmentation results for the colon class on ClinicDB . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.

Figure 30: Visualization of segmentation results for the colon class on ColonDB . The top row shows the original input images, the middle row presents the ground-truth masks, and the bottom row displays the binarized segmentation results using the default threshold of 0.5.

Zero-Shot Anomaly Detection

Zero-shot anomaly detection finds unusual patterns in unseen domains without task-specific labeled examples.

See all 85 papers

VFAD: Variational Semantic Prompting Meets Frequency-Adaptive Representation Learning for Zero-Shot Anomaly Detection
Jul 31, 2026Peng Chen, Kaige Li, Wei Wang +5VLM Adaptation Frequency-Domain Feature Learning
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China
VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer
Mar 9, 2026Yanning Hou, Peiyuan Li, Zirui Liu +4Zero-Shot Anomaly Detection Vision Foundation Model Adaptation
College of Intelligence Science and Technology, National University of Defense Technology, Changsha, China · School of Artificial Intelligence, Anhui University, Hefei, China · State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Anhui University, Hefei, China
Dual Anchors, Do It Better: Hierarchical Group Merging for Zero-Shot Anomaly Detection
Aug 12, 2026Jimin Roh, DongKyu Kim, Suk-Ju KangVLM Adaptation Multimodal Anomaly Detection
Sogang University · LG Electronics · NAVER Cloud

Jul 31, 2026J/K move · Enter open · S save

VFAD: Variational Semantic Prompting Meets Frequency-Adaptive Representation Learning for Zero-Shot Anomaly Detection

Peng Chen, Kaige Li, Wei Wang, Mingbo Yang+4

School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China

VLM Adaptation Frequency-Domain Feature Learning Medical Image Anomaly Detection Industrial Anomaly Detection Anomaly Localization Zero-Shot Anomaly Detection

Open paper AskPDF arXiv

Zero-shot anomaly detection (ZSAD) aims to detect and localize anomalies in unseen categories without access to target-specific training data. Although recent CLIP-based methods have demonstrated promising generalization through vision-language alignment, they remain limited in capturing diverse anomaly semantics and subtle local variations. To address these limitations, we propose VFAD, a unified framework that combines variational semantic prompting with frequency-adaptive representation learning. Specifically, we introduce a Variational Semantic Prompt Extractor (VSPE), which adaptively aggregates anomaly-relevant local semantics from dense patch tokens and regularizes them through a variational information bottleneck, thereby incorporating fine-grained visual cues and enabling more precise cross-modal alignment. Furthermore, we develop a Frequency-Adaptive Representation Aggregation (FARA) module that leverages wavelet-based frequency decomposition and frequency-specific expert aggregation to enhance anomaly-discriminative visual representations. By jointly strengthening semantic guidance and visual representation learning, VFAD improves both anomaly discrimination and fine-grained localization. Extensive experiments on 13 industrial and medical benchmarks demonstrate that VFAD consistently outperforms existing state-of-the-art ZSAD methods across diverse anomaly scenarios. The code will be publicly available upon publication.

Abstract

Zero-Shot Anomaly Detection

Explore similar work

VFAD: Variational Semantic Prompting Meets Frequency-Adaptive Representation Learning for Zero-Shot Anomaly Detection

VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer

Dual Anchors, Do It Better: Hierarchical Group Merging for Zero-Shot Anomaly Detection