cs.CVMar 3, 2026
SaveMoECLIP: Patch-Specialized Experts for Zero-shot Anomaly Detection
Organizations: Yonsei University, Seoul, Korea
Abstract
The CLIP model's outstanding generalization has driven recent success in Zero-Shot Anomaly Detection (ZSAD) for detecting anomalies in unseen categories. The core challenge in ZSAD is to specialize the model for anomaly detection tasks while preserving CLIP's powerful generalization capability. Existing approaches attempting to solve this challenge share the fundamental limitation of a patch-agnostic design that processes all patches monolithically without regard for their unique characteristics. To address this limitation, we propose MoECLIP, a Mixture-of-Experts (MoE) architecture for the ZSAD task, which achieves patch-level adaptation by dynamically routing each image patch to a specialized Low-Rank Adaptation (LoRA) expert based on its unique characteristics. Furthermore, to prevent functional redundancy among the LoRA experts, we introduce (1) Frozen Orthogonal Feature Separation (FOFS), which orthogonally separates the input feature space to force experts to focus on distinct information, and (2) a simplex equiangular tight frame (ETF) loss to regulate the expert outputs to form maximally equiangular representations. Comprehensive experimental results across 14 benchmark datasets spanning industrial and medical domains demonstrate that MoECLIP outperforms existing state-of-the-art methods. The code is available at https://github.com/CoCoRessa/MoECLIP.
Figures & tables
Figure 1: Comparison between existing CLIP-Adapter based method and our MoECLIP. (a) The general CLIP-based ZSAD framework. (b) Existing methods apply a uniform adaptation to all patches, regardless of their unique characteristics. (c) In contrast, our MoECLIP utilizes a Mixture of Experts to achieve patch-specialized adaptation, dynamically routing each patch to experts that are differentiated by FOFS and an ETF loss.
Figure 2: The framework of MoECLIP. MoE is integrated into multiple layers of the CLIP Vision Encoder, enabling dynamic expert routing for each image patch to learn patch-specific representations for ZSAD. Within each MoE, FOFS enforces expert specialization by orthogonally separating the feature space and ETF loss further enhances expert diversity by maximizing the equiangular separation of expert outputs. PAA then aggregates the refined patch features across multiple scales to capture anomalies of different sizes.
| metric | Domain | Dataset | WinCLIP [ 29 ] (CVPR 2023) | April-GAN [ 8 ] (CVPRw 2023) | AnomalyCLIP [ 55 ] (ICLR 2024) | †AdaCLIP [ 7 ] (ECCV 2024) | †AA-CLIP [ 38 ] (CVPR 2025) | Bayes-PFL [ 44 ] (CVPR 2025) | MoECLIP (Ours) |
| Image-level (AUROC, AP) | Industrial | MVTec-AD | (91.8, 95.1) | (86.1, 93.6) | (91.9, 96.2) | (91.5, 96.1 ) | (90.9, 96.0) | ( 92.2 , 96.1 ) | ( 93.9 , 96.8 ) |
| VisA | (78.1, 77.5) | (77.5, 80.8) | (82.1, 85.4) | ( 83.0 , 85.5) | (79.2, 83.7) | ( 86.8 , 89.3 ) | ( 83.6 , 86.2 ) | ||
| BTAD | (83.3, 84.1) | (73.4, 69.6) | (92.5, 94.2) | (91.6, 92.4) | ( 94.8 , 97.5 ) | (93.0, 96.7) | ( 93.1 , 98.0 ) | ||
| RSDD | (85.3, 65.3) | (72.7, 68.3) | (74.0, 73.2) | (83.8, 80.1) | ( 94.9 , 94.2 ) | (91.3, 89.7) | ( 95.3 , 95.1 ) | ||
| DTD-Synthetic | ( 95.0 , 97.9 ) | (85.5, 94.0) | (93.3, 97.7) | (91.5, 96.3) | (92.5, 97.7) | (93.5, 97.7) | ( 95.5, 98.6 ) | ||
| Medical | Brain MRI | (45.1, 80.3) | (58.8, 87.8) | (70.8, 90.6) | (79.5, 94.5 ) | (79.6, 94.4) | ( 81.9 , 94.5 ) | ( 88.5, 97.1 ) |
Table 1: Comparison with state-of-the-art methods across industrial and medical domains under the ZSAD setting. The symbol † indicates results obtained from models re-trained under our setting. The best performance is in bold and the second-best is underlined .
Figure 3: Visualization of Grad-CAM and Patch Selection Map for each expert at layer 18. The Ground Truth image is shown on the far left. The top row (Grad-CAM) highlights each expert’s focus region. The bottom row (Patch Selection) illustrates the patches where the corresponding expert was the router’s Top-1 choice (shown in green). The value in each subplot title represents the expert’s average renormalized routing weight based on the Top-1 setting for its Top-1 assigned patches.
Figure 4: Visualization of Anomaly Maps comparing MoECLIP with previous ZSAD methods across industrial and medical domains. The first column shows the Ground Truth, and the remaining columns show anomaly maps from each method.
| Components of MoECLIP | Datasets | |||||
| MVTec-AD | DTD-Synthetic | Head CT | ColonDB | Average | ||
| base | Vanilla CLIP | (38.4, 74.1) | (33.9, 71.6) | ( , 56.5) | (49.5, ) | (40.6, 67.4) |
| (a) | w/o FOFS & ETF Loss | (91.6, 91.7) | (97.8, 93.1) | ( , 94.4) | (84.1, ) | (91.2, 93.1) |
| (b) | w/o FOFS | (92.0, 92.8) | (98.3, 93.9) | ( , 95.0) | (85.3, ) | (91.9, 93.9) |
| (c) | w/o ETF Loss | (92.2, 92.7) | (98.2, 93.4) | ( , 96.1) | (84.6, ) | (91.7, 94.1) |
| (d) | w/o Depth-wise Adapter | (92.0, 92.5) | (98.1, 93.8) | ( , 94.5) | (85.0, ) | (91.7, 93.6) |
Table 2: Ablation study on the different components of MoECLIP, evaluated on industrial datasets MVTec-AD and DTD-Synthetic, and medical datasets Head CT and ColonDB using (Pixel-level AUROC, Image-level AUROC) metrics. The best performance is in bold .
Figure 5: Ablation Study on Inter-Expert similarity heatmap at layer 18. The heatmap shows the average pairwise cosine similarity between expert features, computed on the MVTec-AD test set. Values approaching +1 (red) indicate high redundancy, while values approaching 0 (white) or negative values (blue) signify successful differentiation.
Figure 6: Ablation Study on the Number of Experts (from 1 to 8) on the MVTec-AD dataset.
| Domain | Dataset | #Classes | #Normal Images | #Anomalous Images | Data Types | Evaluation (Pixel / Image) |
| Industrial | MVTec | 15 | 467 | 1,258 | object & texture | (O, O) |
| VisA | 12 | 962 | 1,200 | object | (O, O) | |
| BTAD | 3 | 451 | 290 | object & texture | (O, O) | |
| RSDD | 1 | 387 | 387 | texture | (O, O) | |
| DTD-synthetic | 12 | 375 | 947 | texture | (O, O) | |
| Medical | Brain MRI | 1 | 640 | 1,013 | brain | (O, O) |
Table 3: Summary of key statistics for the 14 publicly available industrial and medical datasets used in our study. It reports the number of classes, normal and anomalous images, and data types. The last column indicates the availability of evaluation (Pixel-level / Image-level), where ’X’ denotes cases not applicable due to the absence of ground truth masks or normal images.
| State | Prompt |
| Prompt Template | {} a photo of a {} |
| Normal Prompt | [ class ] the [ class ] a [ class ] |
| Abnormal Prompt | [ class ] with damage [ class ] with defect [ class ] with flaw damaged [ class ] broken [ class ] |
Table 4: Templates used for the Text Prompts
Figure 7: Ablation study of hyperparameters on the MVTec-AD dataset.
| Backbone | Patch size | Image Resolution | Image-level | Pixel-level | |||
| AUROC | AP | AUROC | AP | ||||
| ViT-B/32-224 | 32 | 82.7 | 91.8 | 88.4 | 28.5 | ||
| ViT-B/16-224 | 16 | 87.9 | 94.7 | 90.9 | 39.3 | ||
| ViT-L/14-224 | 14 | 91.8 | 95.8 | 91.3 | 42.7 | ||
| ViT-L/14-336 | 14 | 93.1 | 96.7 | 91.0 | 45.2 | ||
| ViT-L/14-336 | 14 | 93.9 | 96.8 | 92.5 | 45.7 | ||
Table 5: Ablation study on different backbones, image resolutions, and patch sizes, evaluated on the MVTec-AD dataset. The best performance is in bold .
| Method | Prompt | Image-level | Pixel-level | ||
| AUROC | AP | AUROC | AP | ||
| MoECLIP (Fixed) | 93.9 | 96.8 | 92.5 | 45.7 | |
| MoECLIP (Learnable) | 92.2 | 96.3 | 91.9 | 44.1 | |
Table 6: Comparison of fixed vs. learnable text prompt strategies. The table compares our default fixed-prompt ensemble against a learnable prompt tuning variant, evaluated on the MVTec-AD dataset. The best performance is in bold .
| MoE Integration Layers | Image-Level | Pixel-Level | ||
| AUROC | AP | AUROC | AP | |
| 1, 2, 3, 4 | 91.0 | 96.2 | 92.2 | 43.6 |
| 20, 21, 22, 23, 24 | 91.8 | 96.6 | 91.6 | 43.4 |
| 3, 6, 9, 12, 15, 18, 21, 24 | 92.3 | 96.5 | 90.9 | 45.3 |
| 6, 12, 18, 24 | 93.9 | 96.8 | 92.5 | 45.7 |
Table 7: Ablation study on the layer placement of MoE modules. The best performance is in bold .
| Layers | Image-Level | Pixel-Level | ||
| AUROC | AP | AUROC | AP | |
| Layer 6 | 76.0 | 87.4 | 83.4 | 28.9 |
| Layer 12 | 90.0 | 95.0 | 91.7 | 43.7 |
| Layer 18 | 92.0 | 96.3 | 91.0 | 43.4 |
| Layer 24 | 91.6 | 95.7 | 91.4 | 42.6 |
| Ensemble | 93.9 | 96.8 | 92.5 | 45.7 |
Table 8: Ablation study on the multi-layer feature ensemble. The table compares our ’Ensemble’ method (aggregating 4 layers) against configurations using features from only a single ViT layer (6, 12, 18, or 24). The best performance is in bold .
| PAA Scale | Image-Level | Pixel-Level | ||
| AUROC | AP | AUROC | AP | |
| 1 | 92.8 | 96.0 | 92.1 | 44.1 |
| 3 | 92.3 | 96.5 | 91.2 | 42.8 |
| 5 | 91.8 | 96.5 | 91.5 | 43.2 |
| 1, 3 | 90.7 | 95.2 | 92.4 | 45.2 |
| 1, 5 | 92.2 | 96.5 | 91.9 | 45.2 |
Table 9: Ablation study on the multi-scale configuration of PAA. The best performance is in bold .
| Method | Image-Level | Pixel-Level | ||
| AUROC | AP | AUROC | AP | |
| MLP Router | 93.3 | 97.1 | 92.1 | 45.5 |
| Full-Rank Expert | 92.8 | 96.7 | 91.9 | 46.0 |
| Contrastive Constraint | 92.4 | 96.2 | 91.7 | 45.6 |
| Similarity Constraint | 92.7 | 96.7 | 91.9 | 45.5 |
| MoECLIP | 93.9 | 96.8 | 92.5 | 45.7 |
Table 10: Robustness analysis of expert configurations and router designs on the MVTec-AD dataset evaluated by (AUROC, AP). The best performance is in bold .
| Perturbation ( ) | Image-Level | Pixel-Level | ||
| AUROC | AP | AUROC | AP | |
| 0.0 | 93.8 | 96.8 | 92.5 | 45.7 |
| 0.2 | 93.6 | 96.7 | 92.5 | 45.7 |
| 0.4 | 93.7 | 96.7 | 92.5 | 45.6 |
| 0.6 | 93.7 | 96.7 | 92.5 | 45.5 |
| 0.8 | 93.5 | 96.7 | 92.5 | 45.6 |
Table 11: Robustness of the routing mechanism to logit perturbation on the MVTec-AD dataset evaluated by AUROC and AP.
| MoECLIP (Industrial) | MoECLIP (Medical) | |||
| Dataset | Image-level | Pixel-level | Image-level | Pixel-level |
| (AUROC, AP) | (AUROC, AP) | (AUROC, AP) | (AUROC, AP) | |
| Head CT | (96.6, 94.5) | – | (98.8, 99.0) | – |
| Liver CT | (74.0, 64.6) | (97.2, 10.8) | (77.2, 70.8) | (97.7, 10.3) |
| Retina OCT | (85.5, 84.9) | (96.2, 66.3) | (85.7. 84.4) | (95.9, 60.1) |
| ColonDB | – | (85.4, 34.8) | – | (88.7, 52.1) |
Table 12: Comparison of MoECLIP’s ZSAD performance on 8 unseen medical datasets, when trained on a same-domain medical auxiliary dataset (Brain MRI) versus an out-of-domain industrial dataset (VisA). The best performance is in bold .
| Model | Params (M) | Inference Time (ms) | Peak GPU Memory (MB) | FLOPs (G) | MVTec-AD | |
| Image-level | Pixel-level | |||||
| AA-CLIP (Single Expert) | 441.3 | 119.6 | 13,853.8 | 1120.6 | (90.9, 96.0) | (91.6, 45.4) |
| MoECLIP | 433.6 | 158.9 | 9,096.4 | 1120.8 | ( 93.9 , 96.8 ) | ( 92.5 , 45.7 ) |
Table 13: Evaluation of computation overhead and ZSAD performance (AUROC, AP).
Figure 8: Comparison of training loss curves between the Single Expert Model and MoECLIP.
Figure 9: Distribution of Top-2 expert utilization (%) at layer 18. The stacked bars illustrate the average proportion of routing assignments among the four experts for each dataset.
| Expert | Gradient ( ) | Contrast ( ) | Entropy ( ) | Specialization |
| Expert 1 | 48.7 | 17.7 | 1.7 | Low-variance regions |
| Expert 2 | 152.2 | 66.9 | 5.4 | Structural edges |
| Expert 3 | 137.0 | 60.9 | 5.7 | Complex patterns |
| Expert 4 | 0.7 | 0.2 | 0.03 | Background |
Table 14: Quantitative analysis of expert specialization using low-level visual features at layer 18 on BrainMRI (Top-1 selection).
Figure 10: Inter-expert cosine similarity across layers on the MVTec-AD dataset. The top row represents the Original MoE lacking FOFS & ETF Loss, and the bottom row represents our full MoECLIP model. Each column corresponds to a different ViT layer. Values approaching +1 (red) indicate high redundancy, while values approaching 0 (white) or negative values (blue) signify successful differentiation.
Figure 11: Visualizations from different scales ( ) and layers (6, 12, 18, 24) demonstrate complementary characteristics across depth and spatial context. The final ensemble (bottom-right) presents a unified anomaly map combining all scales and layers.
Figure 12: Visualization of Grad-CAM and patch selection maps for each expert at layer 6 on the hazelnut class of the MVTec-AD dataset. The Ground Truth image is shown on the far left. The first row (Grad-CAM) highlights each expert’s focus region. The second and third rows (Patch Selection) illustrate the patches where the corresponding expert was selected as the router’s Top-1 and Top-2 choices, respectively (shown in green). The value in each subplot title represents the expert’s average renormalized routing weight computed from its Top-1 and Top-2 assigned patches under their respective routing settings.
Figure 13: Visualization of Grad-CAM and patch selection maps for each expert at layer 12 on the hazelnut class of the MVTec-AD dataset. The Ground Truth image is shown on the far left. The first row (Grad-CAM) highlights each expert’s focus region. The second and third rows (Patch Selection) illustrate the patches where the corresponding expert was selected as the router’s Top-1 and Top-2 choices, respectively (shown in green). The value in each subplot title represents the expert’s average renormalized routing weight computed from its Top-1 and Top-2 assigned patches under their respective routing settings.
Figure 14: Visualization of Grad-CAM and patch selection maps for each expert at layer 18 on the hazelnut class of the MVTec-AD dataset. The Ground Truth image is shown on the far left. The first row (Grad-CAM) highlights each expert’s focus region. The second and third rows (Patch Selection) illustrate the patches where the corresponding expert was selected as the router’s Top-1 and Top-2 choices, respectively (shown in green). The value in each subplot title represents the expert’s average renormalized routing weight computed from its Top-1 and Top-2 assigned patches under their respective routing settings.
Figure 15: Visualization of Grad-CAM and patch selection maps for each expert at layer 24 on the hazelnut class of the MVTec-AD dataset. The Ground Truth image is shown on the far left. The first row (Grad-CAM) highlights each expert’s focus region. The second and third rows (Patch Selection) illustrate the patches where the corresponding expert was selected as the router’s Top-1 and Top-2 choices, respectively (shown in green). The value in each subplot title represents the expert’s average renormalized routing weight computed from its Top-1 and Top-2 assigned patches under their respective routing settings.
Figure 16: Anomaly Map results for the bottle in the MVTec-AD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 17: Anomaly Map results for the capsule in the MVTec-AD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 18: Anomaly Map results for the hazelnut in the MVTec-AD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 19: Anomaly Map results for the pill in the MVTec-AD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 20: Anomaly Map results for the screw in the MVTec-AD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 21: Anomaly Map results for the zipper in the MVTec-AD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 22: Anomaly Map results for the capsules in the VisA. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 23: Anomaly Map results for the chewinggum in the VisA. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 24: Anomaly Map results for the 02 in the BTAD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 25: Anomaly Map results for the Mesh_114 in the DTD-Synthetic. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 26: Anomaly Map results for the Marbled_079 in the DTD-Synthetic. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 27: Anomaly Map results on the RSDD. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 28: Anomaly Map results on the Brain MRI. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 29: Anomaly Map results on the Liver CT. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 30: Anomaly Map results on the Head CT. The absence of pixel-level annotations restricts the use of this dataset to anomaly classification. The first row contains the original images. The second row shows the anomaly map results generated by MoECLIP.
Figure 31: Anomaly Map results on the CVC-ColonDB. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 32: Anomaly Map results on the CVC-ClinicDB. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 33: Anomaly Map results on the CVC-300. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 34: Anomaly Map results on the Endo. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
Figure 35: Anomaly Map results on the Kvasir. The first row contains the original images with red areas showing the ground truth. The second row shows the anomaly map results generated by MoECLIP.
More in this topic
Zero-Shot Anomaly Detection
Zero-shot anomaly detection finds unusual patterns in unseen domains without task-specific labeled examples.
See all 82 papers