Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · CASI Vision Technology Co., Ltd., Luoyang, Henan, China
Zero-Shot Multimodal Anomaly Detection (ZSMAD) aims to identify both appearance and geometric defects in novel categories without relying on target-specific training data, making it especially valuable in real-world scenarios constrained by data scarcity, privacy concerns, or high annotation costs. While recent approaches successfully adapt vision-language models like CLIP to achieve this open-vocabulary generalization, they still face significant challenges: (i) severe geometric information loss during 3D-to-2D point cloud rendering; (ii) spatially noisy 2D feature maps produced by raw CLIP patch embeddings, compounded by shallow late-stage modality fusion; and (iii) severe cross-modal semantic misalignment due to decoupled processing streams. To address these limitations, we propose AggCLIP, a novel framework that adapts frozen CLIP representations for ZSMAD by explicitly incorporating local geometric structure and visual spatial context. Specifically, to accurately localize fine-grained defects, we design a Geometry-Aware Spatial Aggregation (GSA) module that refines back-projected 3D point features via density-adaptive neighborhood weighting, alongside a Context-Aware Spatial Aggregation (CSA) module that smooths multi-scale 2D patch embeddings using a lightweight trainable adapter. Furthermore, to prevent semantic drift between modalities, we introduce a Dual-Stream Prompt Contrastive (DSPC) loss, which aligns the abstract concepts of normality and abnormality across the point cloud and RGB streams. Extensive experiments demonstrate that the proposed AggCLIP framework achieves state-of-the-art performance on the MVTec 3D-AD and Eyecandies benchmarks under both one-vs-rest and cross-dataset zero-shot settings, notably surpassing the strongest competitive methods by up to 2.9% in pixel-level AUPRO.
Figures & tables
Figure 1: Paradigm comparison for zero-shot multimodal anomaly detection. (a) Existing ZSMAD pipelines typically suffer from three fundamental limitations: Geometric Loss during 3D-to-2D rendering, Spatial Noise and Weak Integration due to raw CLIP embeddings and simplistic late-fusion, and Semantic Misalignment between prompts and multimodal features. (b) The proposed AggCLIP framework effectively addresses these gaps. It refines noisy back-projected point features via the Geometry-Aware Spatial Aggregation ( GSA ) module, suppresses 2D spatial noise using Context-Aware Spatial Aggregation ( CSA ) paired with a learnable Adapter , and directly enforces deep cross-modal semantic interaction via the Dual-Stream Prompt Contrastive ( DSPC ) loss.
Figure 2: Overview of AggCLIP. GSA refines back-projected point-cloud features, CSA and a lightweight adapter refine RGB patch features, and DSPC aligns modality-specific text prompt embeddings before multimodal anomaly scoring.
MVTec 3D-AD
Eyecandies
Methods
Object-level
Pixel-level
Object-level
Pixel-level
(O-R, O-A)
(P-R, P-P)
(O-R, O-A)
(P-R, P-P)
CLIP [ 32 ]
(60.4, 86.4)
(– , 56.0)
(73.0, 73.9)
(78.0, 31.8)
AnomalyCLIP [ 48 ]
(66.2, 87.6)
(91.6, 70.9)
(65.0, 67.5)
(85.0, 56.2)
PointAD [ 49 ]
(86.9, 96.1)
(97.2, 90.2)
(77.7, 80.4)
(95.3, 84.3)
ZUMA-FT [ 28 ]
( 88.2 , – )
( 97.8 , 93.1 )
(78.2, – )
( 97.5 , 89.4 )
Table 1: Performance comparison of zero-shot multimodal anomaly detection under the one-vs-rest setting on the MVTec 3D-AD and Eyecandies datasets. Best results are highlighted in red , and runners-up in blue . Throughout all tables, O-R, O-A, P-R, and P-P denote O-AUROC, O-AP, P-AUROC, and P-AUPRO, respectively.
Object-level
Pixel-level
Methods
(O-R, O-A)
(P-R, P-P)
AnomalyCLIP [ 48 ]
(65.7, 68.1)
(86.2, 61.3)
PointAD [ 49 ]
(78.6, 80.8)
(94.0, 80.7)
ZUMA-FT* [ 28 ]
(79.1, 81.2)
( 94.8 , 81.9)
GS-CLIP* [ 14 ]
( 80.2 , 82.2 )
(94.6, 82.4 )
AggCLIP (Ours)
( 81.4 , 84.2 )
( 95.8 , 85.3 )
Table 2: Performance comparison of zero-shot multimodal anomaly detection under the cross-dataset setting (trained on MVTec 3D-AD, tested on Eyecandies). * Indicates results obtained by running the official code.
Figure 3: Qualitative results on MVTec 3D-AD and Eyecandies. From top to bottom: input RGB images, 3D point clouds, ground truth, anomaly maps from GS-CLIP, and anomaly maps from AggCLIP.
Methods
GFLOPs
FPS
Time (s)
Memory (MB)
CLIP* [ 32 ]
2330.25
3.64
0.27
2270
PointAD* [ 49 ]
4252.45
3.20
0.31
2610
ZUMA-FT* [ 28 ]
4469.10
0.53
1.89
3856
GS-CLIP* [ 14 ]
9591.98
1.12
0.89
3677
AggCLIP (Ours)
4259.97
3.07
0.33
2645
Table 3: Efficiency comparison of different methods. * Indicates results obtained by running the official code.
Object-level
Pixel-level
Version
GSA
CSA
Adapter
Ldspc
(O-R, O-A)
(P-R, P-P)
V1
✗
✗
✗
✗
(81.2, 89.6)
(90.3, 88.9)
V2
✓
✗
✗
✗
(87.4, 96.7)
(97.8, 93.0)
V3
✗
✓
✗
✗
(85.5, 94.3)
(95.2, 90.4)
V4
✗
✓
✓
✗
(86.8, 95.7)
(96.2, 91.3)
V5
✓
✓
✓
✗
( 87.7 , 97.1 )
( 98.0 , 93.6 )
Table 4: Ablation on different components.
Figure 4: Ablation on key hyperparameters on MVTec 3D-AD under the one-vs-rest setting. (a) Varying the number of rendered views K . (b) Varying the GSA neighborhood size k .
Object-level
Pixel-level
Kernel Sizes
(O-R, O-A)
(P-R, P-P)
{1}
(88.0, 97.1)
(97.3, 92.9)
{3}
(87.9, 96.8)
(97.7, 92.6)
{5}
(87.7, 97.2)
(97.8, 93.0)
{1,3}
( 88.1 , 97.4 )
( 98.1 , 93.5 )
{1,3,5}
( 88.3 , 97.5 )
( 98.4 , 93.9 )
Table 5: Ablation study on the multi-scale Gaussian kernel sizes S in the CSA module.
Zero-shot 3D Anomaly Detection is an emerging task that aims to detect anomalies in a target dataset without any target training data, which is particularly important in scenarios constrained by sample scarcity and data privacy concerns. While current methods adapt CLIP by projecting 3D point clouds into 2D representations, they face challenges. The projection inherently loses some geometric details, and the reliance on a single 2D modality provides an incomplete visual understanding, limiting their ability to detect diverse anomaly types. To address these limitations, we propose the Geometry-Aware Prompt and Synergistic View Representation Learning (GS-CLIP) framework, which enables the model to identify geometric anomalies through a two-stage learning process. In stage 1, we dynamically generate text prompts embedded with 3D geometric priors. These prompts contain global shape context and local defect information distilled by our Geometric Defect Distillation Module (GDDM). In stage 2, we introduce Synergistic View Representation Learning architecture that processes rendered and depth images in parallel. A Synergistic Refinement Module (SRM) subsequently fuses the features of both streams, capitalizing on their complementary strengths. Comprehensive experimental results on four large-scale public datasets show that GS-CLIP achieves superior performance in detection. Code can be available at https://github.com/zhushengxinyue/GS-CLIP.
Zehao Deng, An Liu, Yan Wang
School of Computer Science and Technology, Soochow University · Institute for AI Industry Research (AIR), Tsinghua University
The CLIP model's outstanding generalization has driven recent success in Zero-Shot Anomaly Detection (ZSAD) for detecting anomalies in unseen categories. The core challenge in ZSAD is to specialize the model for anomaly detection tasks while preserving CLIP's powerful generalization capability. Existing approaches attempting to solve this challenge share the fundamental limitation of a patch-agnostic design that processes all patches monolithically without regard for their unique characteristics. To address this limitation, we propose MoECLIP, a Mixture-of-Experts (MoE) architecture for the ZSAD task, which achieves patch-level adaptation by dynamically routing each image patch to a specialized Low-Rank Adaptation (LoRA) expert based on its unique characteristics. Furthermore, to prevent functional redundancy among the LoRA experts, we introduce (1) Frozen Orthogonal Feature Separation (FOFS), which orthogonally separates the input feature space to force experts to focus on distinct information, and (2) a simplex equiangular tight frame (ETF) loss to regulate the expert outputs to form maximally equiangular representations. Comprehensive experimental results across 14 benchmark datasets spanning industrial and medical domains demonstrate that MoECLIP outperforms existing state-of-the-art methods. The code is available at https://github.com/CoCoRessa/MoECLIP.
Vision-language models offer a promising approach for zero-shot anomaly detection (ZSAD). However, due to object-centric bias, normal and anomalous text prototypes exhibit a high semantic overlap. While enforcing strict orthogonality between them improves discriminability, mapping highly contiguous visual inputs onto drastically orthogonal prototypes introduces a geometric dilemma, disrupting the pre-trained structural continuity. To address this problem, we propose Proximity-CLIP, a framework that visually calibrates the semantic margin to guide visual adaptation. First, we introduce a visually-calibrated semantic proximity learning mechanism that uses a bounded dynamic regularization to learn an appropriate semantic margin, ensuring discriminative separation while preserving structural alignment. Second, we design an Anomaly Query Module (AQM) driven by these text priors. Using the calibrated anomalous prototype as a semantic query, the AQM actively retrieves localized defect cues from contextual visual patches, mitigating the dilution of subtle anomalies during global pooling. Extensive experiments demonstrate that Proximity-CLIP outperforms current state-of-the-art methods across multiple ZSAD benchmarks with minimal architectural modifications.
Manwen Yang, Leqian Ding, Yu Guo +1
School of Software Engineering, Xi’an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University