The CLIP model's outstanding generalization has driven recent success in Zero-Shot Anomaly Detection (ZSAD) for detecting anomalies in unseen categories. The core challenge in ZSAD is to specialize the model for anomaly detection tasks while preserving CLIP's powerful generalization capability. Existing approaches attempting to solve this challenge share the fundamental limitation of a patch-agnostic design that processes all patches monolithically without regard for their unique characteristics. To address this limitation, we propose MoECLIP, a Mixture-of-Experts (MoE) architecture for the ZSAD task, which achieves patch-level adaptation by dynamically routing each image patch to a specialized Low-Rank Adaptation (LoRA) expert based on its unique characteristics. Furthermore, to prevent functional redundancy among the LoRA experts, we introduce (1) Frozen Orthogonal Feature Separation (FOFS), which orthogonally separates the input feature space to force experts to focus on distinct information, and (2) a simplex equiangular tight frame (ETF) loss to regulate the expert outputs to form maximally equiangular representations. Comprehensive experimental results across 14 benchmark datasets spanning industrial and medical domains demonstrate that MoECLIP outperforms existing state-of-the-art methods. The code is available at https://github.com/CoCoRessa/MoECLIP.
Figures & tables
Figure 1: Comparison between existing CLIP-Adapter based method and our MoECLIP. (a) The general CLIP-based ZSAD framework. (b) Existing methods apply a uniform adaptation to all patches, regardless of their unique characteristics. (c) In contrast, our MoECLIP utilizes a Mixture of Experts to achieve patch-specialized adaptation, dynamically routing each patch to experts that are differentiated by FOFS and an ETF loss.
Figure 2: The framework of MoECLIP. MoE is integrated into multiple layers of the CLIP Vision Encoder, enabling dynamic expert routing for each image patch to learn patch-specific representations for ZSAD. Within each MoE, FOFS enforces expert specialization by orthogonally separating the feature space and ETF loss further enhances expert diversity by maximizing the equiangular separation of expert outputs. PAA then aggregates the refined patch features across multiple scales to capture anomalies of different sizes.
metric
Domain
Dataset
WinCLIP [ 29 ] (CVPR 2023)
April-GAN [ 8 ] (CVPRw 2023)
AnomalyCLIP [ 55 ] (ICLR 2024)
†AdaCLIP [ 7 ] (ECCV 2024)
†AA-CLIP [ 38 ] (CVPR 2025)
Bayes-PFL [ 44 ] (CVPR 2025)
MoECLIP (Ours)
Image-level (AUROC, AP)
Industrial
MVTec-AD
(91.8, 95.1)
(86.1, 93.6)
(91.9, 96.2)
(91.5, 96.1 )
(90.9, 96.0)
( 92.2 , 96.1 )
( 93.9 , 96.8 )
VisA
(78.1, 77.5)
(77.5, 80.8)
(82.1, 85.4)
( 83.0 , 85.5)
(79.2, 83.7)
( 86.8 , 89.3 )
( 83.6 , 86.2 )
BTAD
(83.3, 84.1)
(73.4, 69.6)
(92.5, 94.2)
(91.6, 92.4)
( 94.8 , 97.5 )
(93.0, 96.7)
( 93.1 , 98.0 )
RSDD
(85.3, 65.3)
(72.7, 68.3)
(74.0, 73.2)
(83.8, 80.1)
( 94.9 , 94.2 )
(91.3, 89.7)
( 95.3 , 95.1 )
DTD-Synthetic
( 95.0 , 97.9 )
(85.5, 94.0)
(93.3, 97.7)
(91.5, 96.3)
(92.5, 97.7)
(93.5, 97.7)
( 95.5, 98.6 )
Medical
Brain MRI
(45.1, 80.3)
(58.8, 87.8)
(70.8, 90.6)
(79.5, 94.5 )
(79.6, 94.4)
( 81.9 , 94.5 )
( 88.5, 97.1 )
Table 1: Comparison with state-of-the-art methods across industrial and medical domains under the ZSAD setting. The symbol † indicates results obtained from models re-trained under our setting. The best performance is in bold and the second-best is underlined .
Figure 3: Visualization of Grad-CAM and Patch Selection Map for each expert at layer 18. The Ground Truth image is shown on the far left. The top row (Grad-CAM) highlights each expert’s focus region. The bottom row (Patch Selection) illustrates the patches where the corresponding expert was the router’s Top-1 choice (shown in green). The value in each subplot title represents the expert’s average renormalized routing weight based on the Top-1 setting for its Top-1 assigned patches.
Figure 4: Visualization of Anomaly Maps comparing MoECLIP with previous ZSAD methods across industrial and medical domains. The first column shows the Ground Truth, and the remaining columns show anomaly maps from each method.
Components of MoECLIP
Datasets
MVTec-AD
DTD-Synthetic
Head CT
ColonDB
Average
base
Vanilla CLIP
(38.4, 74.1)
(33.9, 71.6)
( − , 56.5)
(49.5, − )
(40.6, 67.4)
(a)
w/o FOFS & ETF Loss
(91.6, 91.7)
(97.8, 93.1)
( − , 94.4)
(84.1, − )
(91.2, 93.1)
(b)
w/o FOFS
(92.0, 92.8)
(98.3, 93.9)
( − , 95.0)
(85.3, − )
(91.9, 93.9)
(c)
w/o ETF Loss
(92.2, 92.7)
(98.2, 93.4)
( − , 96.1)
(84.6, − )
(91.7, 94.1)
(d)
w/o Depth-wise Adapter
(92.0, 92.5)
(98.1, 93.8)
( − , 94.5)
(85.0, − )
(91.7, 93.6)
Table 2: Ablation study on the different components of MoECLIP, evaluated on industrial datasets MVTec-AD and DTD-Synthetic, and medical datasets Head CT and ColonDB using (Pixel-level AUROC, Image-level AUROC) metrics. The best performance is in bold .
Figure 5: Ablation Study on Inter-Expert similarity heatmap at layer 18. The heatmap shows the average pairwise cosine similarity between expert features, computed on the MVTec-AD test set. Values approaching +1 (red) indicate high redundancy, while values approaching 0 (white) or negative values (blue) signify successful differentiation.
Figure 6: Ablation Study on the Number of Experts K (from 1 to 8) on the MVTec-AD dataset.
Domain
Dataset
#Classes
#Normal Images
#Anomalous Images
Data Types
Evaluation (Pixel / Image)
Industrial
MVTec
15
467
1,258
object & texture
(O, O)
VisA
12
962
1,200
object
(O, O)
BTAD
3
451
290
object & texture
(O, O)
RSDD
1
387
387
texture
(O, O)
DTD-synthetic
12
375
947
texture
(O, O)
Medical
Brain MRI
1
640
1,013
brain
(O, O)
Table 3: Summary of key statistics for the 14 publicly available industrial and medical datasets used in our study. It reports the number of classes, normal and anomalous images, and data types. The last column indicates the availability of evaluation (Pixel-level / Image-level), where ’X’ denotes cases not applicable due to the absence of ground truth masks or normal images.
State
Prompt
Prompt Template
{} a photo of a {}
Normal Prompt
[ class ] the [ class ] a [ class ]
Abnormal Prompt
[ class ] with damage [ class ] with defect [ class ] with flaw damaged [ class ] broken [ class ]
Table 4: Templates used for the Text Prompts
Figure 7: Ablation study of hyperparameters on the MVTec-AD dataset.
Backbone
Patch size
Image Resolution
Image-level
Pixel-level
AUROC
AP
AUROC
AP
ViT-B/32-224
32
448×448
82.7
91.8
88.4
28.5
ViT-B/16-224
16
448×448
87.9
94.7
90.9
39.3
ViT-L/14-224
14
518×518
91.8
95.8
91.3
42.7
ViT-L/14-336
14
448×448
93.1
96.7
91.0
45.2
ViT-L/14-336
14
518×518
93.9
96.8
92.5
45.7
Table 5: Ablation study on different backbones, image resolutions, and patch sizes, evaluated on the MVTec-AD dataset. The best performance is in bold .
Method
Prompt
Image-level
Pixel-level
AUROC
AP
AUROC
AP
MoECLIP (Fixed)
tn=a photo of a [class]ta=a photo of a [state][class]
Table 6: Comparison of fixed vs. learnable text prompt strategies. The table compares our default fixed-prompt ensemble against a learnable prompt tuning variant, evaluated on the MVTec-AD dataset. The best performance is in bold .
MoE Integration Layers
Image-Level
Pixel-Level
AUROC
AP
AUROC
AP
1, 2, 3, 4
91.0
96.2
92.2
43.6
20, 21, 22, 23, 24
91.8
96.6
91.6
43.4
3, 6, 9, 12, 15, 18, 21, 24
92.3
96.5
90.9
45.3
6, 12, 18, 24
93.9
96.8
92.5
45.7
Table 7: Ablation study on the layer placement of MoE modules. The best performance is in bold .
Layers
Image-Level
Pixel-Level
AUROC
AP
AUROC
AP
Layer 6
76.0
87.4
83.4
28.9
Layer 12
90.0
95.0
91.7
43.7
Layer 18
92.0
96.3
91.0
43.4
Layer 24
91.6
95.7
91.4
42.6
Ensemble
93.9
96.8
92.5
45.7
Table 8: Ablation study on the multi-layer feature ensemble. The table compares our ’Ensemble’ method (aggregating 4 layers) against configurations using features from only a single ViT layer (6, 12, 18, or 24). The best performance is in bold .
PAA Scale s
Image-Level
Pixel-Level
AUROC
AP
AUROC
AP
1
92.8
96.0
92.1
44.1
3
92.3
96.5
91.2
42.8
5
91.8
96.5
91.5
43.2
1, 3
90.7
95.2
92.4
45.2
1, 5
92.2
96.5
91.9
45.2
Table 9: Ablation study on the multi-scale configuration of PAA. The best performance is in bold .
Method
Image-Level
Pixel-Level
AUROC
AP
AUROC
AP
MLP Router
93.3
97.1
92.1
45.5
Full-Rank Expert
92.8
96.7
91.9
46.0
Contrastive Constraint
92.4
96.2
91.7
45.6
Similarity Constraint
92.7
96.7
91.9
45.5
MoECLIP
93.9
96.8
92.5
45.7
Table 10: Robustness analysis of expert configurations and router designs on the MVTec-AD dataset evaluated by (AUROC, AP). The best performance is in bold .
Perturbation ( α )
Image-Level
Pixel-Level
AUROC
AP
AUROC
AP
0.0
93.8
96.8
92.5
45.7
0.2
93.6
96.7
92.5
45.7
0.4
93.7
96.7
92.5
45.6
0.6
93.7
96.7
92.5
45.5
0.8
93.5
96.7
92.5
45.6
Table 11: Robustness of the routing mechanism to logit perturbation on the MVTec-AD dataset evaluated by AUROC and AP.
MoECLIP (Industrial)
MoECLIP (Medical)
Dataset
Image-level
Pixel-level
Image-level
Pixel-level
(AUROC, AP)
(AUROC, AP)
(AUROC, AP)
(AUROC, AP)
Head CT
(96.6, 94.5)
–
(98.8, 99.0)
–
Liver CT
(74.0, 64.6)
(97.2, 10.8)
(77.2, 70.8)
(97.7, 10.3)
Retina OCT
(85.5, 84.9)
(96.2, 66.3)
(85.7. 84.4)
(95.9, 60.1)
ColonDB
–
(85.4, 34.8)
–
(88.7, 52.1)
Table 12: Comparison of MoECLIP’s ZSAD performance on 8 unseen medical datasets, when trained on a same-domain medical auxiliary dataset (Brain MRI) versus an out-of-domain industrial dataset (VisA). The best performance is in bold .
Model
Params (M)
Inference Time (ms)
Peak GPU Memory (MB)
FLOPs (G)
MVTec-AD
Image-level
Pixel-level
AA-CLIP (Single Expert)
441.3
119.6
13,853.8
1120.6
(90.9, 96.0)
(91.6, 45.4)
MoECLIP
433.6
158.9
9,096.4
1120.8
( 93.9 , 96.8 )
( 92.5 , 45.7 )
Table 13: Evaluation of computation overhead and ZSAD performance (AUROC, AP).
Figure 8: Comparison of training loss curves between the Single Expert Model and MoECLIP.
Figure 9: Distribution of Top-2 expert utilization (%) at layer 18. The stacked bars illustrate the average proportion of routing assignments among the four experts for each dataset.
Expert
Gradient ( ∇ )
Contrast ( σ )
Entropy ( H )
Specialization
Expert 1
48.7
17.7
1.7
Low-variance regions
Expert 2
152.2
66.9
5.4
Structural edges
Expert 3
137.0
60.9
5.7
Complex patterns
Expert 4
0.7
0.2
0.03
Background
Table 14: Quantitative analysis of expert specialization using low-level visual features at layer 18 on BrainMRI (Top-1 selection).
Figure 10: Inter-expert cosine similarity across layers on the MVTec-AD dataset. The top row represents the Original MoE lacking FOFS & ETF Loss, and the bottom row represents our full MoECLIP model. Each column corresponds to a different ViT layer. Values approaching +1 (red) indicate high redundancy, while values approaching 0 (white) or negative values (blue) signify successful differentiation.
Figure 11: Visualizations from different scales ( s∈{1,3,5} ) and layers (6, 12, 18, 24) demonstrate complementary characteristics across depth and spatial context. The final ensemble (bottom-right) presents a unified anomaly map combining all scales and layers.
Figure 12: Visualization of Grad-CAM and patch selection maps for each expert at layer 6 on the hazelnut class of the MVTec-AD dataset. The Ground Truth image is shown on the far left. The first row (Grad-CAM) highlights each expert’s focus region. The second and third rows (Patch Selection) illustrate the patches where the corresponding expert was selected as the router’s Top-1 and Top-2 choices, respectively (shown in green). The value in each subplot title represents the expert’s average renormalized routing weight computed from its Top-1 and Top-2 assigned patches under their respective routing settings.
Figure 13: Visualization of Grad-CAM and patch selection maps for each expert at layer 12 on the hazelnut class of the MVTec-AD dataset. The Ground Truth image is shown on the far left. The first row (Grad-CAM) highlights each expert’s focus region. The second and third rows (Patch Selection) illustrate the patches where the corresponding expert was selected as the router’s Top-1 and Top-2 choices, respectively (shown in green). The value in each subplot title represents the expert’s average renormalized routing weight computed from its Top-1 and Top-2 assigned patches under their respective routing settings.
Figure 14: Visualization of Grad-CAM and patch selection maps for each expert at layer 18 on the hazelnut class of the MVTec-AD dataset. The Ground Truth image is shown on the far left. The first row (Grad-CAM) highlights each expert’s focus region. The second and third rows (Patch Selection) illustrate the patches where the corresponding expert was selected as the router’s Top-1 and Top-2 choices, respectively (shown in green). The value in each subplot title represents the expert’s average renormalized routing weight computed from its Top-1 and Top-2 assigned patches under their respective routing settings.
Figure 15: Visualization of Grad-CAM and patch selection maps for each expert at layer 24 on the hazelnut class of the MVTec-AD dataset. The Ground Truth image is shown on the far left. The first row (Grad-CAM) highlights each expert’s focus region. The second and third rows (Patch Selection) illustrate the patches where the corresponding expert was selected as the router’s Top-1 and Top-2 choices, respectively (shown in green). The value in each subplot title represents the expert’s average renormalized routing weight computed from its Top-1 and Top-2 assigned patches under their respective routing settings.
Figure 16: Anomaly Map results for the bottle in the MVTec-AD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 17: Anomaly Map results for the capsule in the MVTec-AD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 18: Anomaly Map results for the hazelnut in the MVTec-AD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 19: Anomaly Map results for the pill in the MVTec-AD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 20: Anomaly Map results for the screw in the MVTec-AD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 21: Anomaly Map results for the zipper in the MVTec-AD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 22: Anomaly Map results for the capsules in the VisA. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 23: Anomaly Map results for the chewinggum in the VisA. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 24: Anomaly Map results for the 02 in the BTAD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 25: Anomaly Map results for the Mesh_114 in the DTD-Synthetic. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 26: Anomaly Map results for the Marbled_079 in the DTD-Synthetic. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 27: Anomaly Map results on the RSDD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 28: Anomaly Map results on the Brain MRI. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 29: Anomaly Map results on the Liver CT. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 30: Anomaly Map results on the Head CT. The absence of pixel-level annotations restricts the use of this dataset to anomaly classification. The first row contains the original images. The second row shows the anomaly map results generated by MoECLIP.
Figure 31: Anomaly Map results on the CVC-ColonDB. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 32: Anomaly Map results on the CVC-ClinicDB. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 33: Anomaly Map results on the CVC-300. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 34: Anomaly Map results on the Endo. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 35: Anomaly Map results on the Kvasir. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Vision-language models have recently shown strong generalization in zero-shot anomaly detection (ZSAD), enabling the detection of unseen anomalies without task-specific supervision. However, existing approaches typically rely on fixed textual prompts, which struggle to capture complex semantics, and focus solely on spatial-domain features, limiting their ability to detect subtle anomalies. To address these challenges, we propose a wavelet-enhanced mixture-of-experts prompt learning method for ZSAD. Specifically, a variational autoencoder is employed to model global semantic representations and integrate them into prompts to enhance adaptability to diverse anomaly patterns. Wavelet decomposition extracts multi-frequency image features that dynamically refine textual embeddings through cross-modal interactions. Furthermore, a semantic-aware mixture-of-experts module is introduced to aggregate contextual information. Extensive experiments on 14 industrial and medical datasets demonstrate the effectiveness of the proposed method.
Peng Chen, Chao Huang
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen
Zero-Shot Anomaly Detection (ZSAD) aims to detect anomalies in unseen domains without target-domain adaptation. Recent CLIP-based methods have shown promising performance by leveraging prompt learning and visual-text alignment. However, most existing approaches rely on a single adaptation pathway, which may be insufficient for heterogeneous anomaly patterns across domains. In practice, anomalies exhibit vastly different characteristics, ranging from salient, localized structural disruptions to subtle, diffuse, and irregular variations. To address this challenge, we propose EntroAD, a structural entropy-guided zero-shot anomaly detection framework. Unlike previous methods, EntroAD introduces a dynamic routing mechanism to process different types of anomalies with specialized adaptation strategies. Specifically, we estimate patch-level structural entropy from self-attention-induced patch relations and use it as a proxy for relational uncertainty to guide anomaly-aware token routing. Based on this routing signal, we construct anomaly-aware routed tokens to better capture anomaly cues with different structural characteristics. We further introduce a confidence-aware dual-branch prompt adaptation module to stabilize visual-text alignment while preserving CLIP's transferable prior. Extensive experiments on 10 industrial and medical benchmarks show that EntroAD achieves state-of-the-art performance in challenging cross-dataset ZSAD settings.
Zero-shot anomaly detection (ZSAD) aims to detect and localize anomalies in unseen categories without access to target-specific training data. Although recent CLIP-based methods have demonstrated promising generalization through vision-language alignment, they remain limited in capturing diverse anomaly semantics and subtle local variations. To address these limitations, we propose VFAD, a unified framework that combines variational semantic prompting with frequency-adaptive representation learning. Specifically, we introduce a Variational Semantic Prompt Extractor (VSPE), which adaptively aggregates anomaly-relevant local semantics from dense patch tokens and regularizes them through a variational information bottleneck, thereby incorporating fine-grained visual cues and enabling more precise cross-modal alignment. Furthermore, we develop a Frequency-Adaptive Representation Aggregation (FARA) module that leverages wavelet-based frequency decomposition and frequency-specific expert aggregation to enhance anomaly-discriminative visual representations. By jointly strengthening semantic guidance and visual representation learning, VFAD improves both anomaly discrimination and fine-grained localization. Extensive experiments on 13 industrial and medical benchmarks demonstrate that VFAD consistently outperforms existing state-of-the-art ZSAD methods across diverse anomaly scenarios. The code will be publicly available upon publication.
Peng Chen, Kaige Li, Wei Wang +5
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China