Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · CASI Vision Technology Co., Ltd., Luoyang, Henan, China
Zero-Shot Multimodal Anomaly Detection (ZSMAD) aims to identify both appearance and geometric defects in novel categories without relying on target-specific training data, making it especially valuable in real-world scenarios constrained by data scarcity, privacy concerns, or high annotation costs. While recent approaches successfully adapt vision-language models like CLIP to achieve this open-vocabulary generalization, they still face significant challenges: (i) severe geometric information loss during 3D-to-2D point cloud rendering; (ii) spatially noisy 2D feature maps produced by raw CLIP patch embeddings, compounded by shallow late-stage modality fusion; and (iii) severe cross-modal semantic misalignment due to decoupled processing streams. To address these limitations, we propose AggCLIP, a novel framework that adapts frozen CLIP representations for ZSMAD by explicitly incorporating local geometric structure and visual spatial context. Specifically, to accurately localize fine-grained defects, we design a Geometry-Aware Spatial Aggregation (GSA) module that refines back-projected 3D point features via density-adaptive neighborhood weighting, alongside a Context-Aware Spatial Aggregation (CSA) module that smooths multi-scale 2D patch embeddings using a lightweight trainable adapter. Furthermore, to prevent semantic drift between modalities, we introduce a Dual-Stream Prompt Contrastive (DSPC) loss, which aligns the abstract concepts of normality and abnormality across the point cloud and RGB streams. Extensive experiments demonstrate that the proposed AggCLIP framework achieves state-of-the-art performance on the MVTec 3D-AD and Eyecandies benchmarks under both one-vs-rest and cross-dataset zero-shot settings, notably surpassing the strongest competitive methods by up to 2.9% in pixel-level AUPRO.
Figures & tables
Figure 1: Paradigm comparison for zero-shot multimodal anomaly detection. (a) Existing ZSMAD pipelines typically suffer from three fundamental limitations: Geometric Loss during 3D-to-2D rendering, Spatial Noise and Weak Integration due to raw CLIP embeddings and simplistic late-fusion, and Semantic Misalignment between prompts and multimodal features. (b) The proposed AggCLIP framework effectively addresses these gaps. It refines noisy back-projected point features via the Geometry-Aware Spatial Aggregation ( GSA ) module, suppresses 2D spatial noise using Context-Aware Spatial Aggregation ( CSA ) paired with a learnable Adapter , and directly enforces deep cross-modal semantic interaction via the Dual-Stream Prompt Contrastive ( DSPC ) loss.
Figure 2: Overview of AggCLIP. GSA refines back-projected point-cloud features, CSA and a lightweight adapter refine RGB patch features, and DSPC aligns modality-specific text prompt embeddings before multimodal anomaly scoring.
MVTec 3D-AD
Eyecandies
Methods
Object-level
Pixel-level
Object-level
Pixel-level
(O-R, O-A)
(P-R, P-P)
(O-R, O-A)
(P-R, P-P)
CLIP [ 32 ]
(60.4, 86.4)
(– , 56.0)
(73.0, 73.9)
(78.0, 31.8)
AnomalyCLIP [ 48 ]
(66.2, 87.6)
(91.6, 70.9)
(65.0, 67.5)
(85.0, 56.2)
PointAD [ 49 ]
(86.9, 96.1)
(97.2, 90.2)
(77.7, 80.4)
(95.3, 84.3)
ZUMA-FT [ 28 ]
( 88.2 , – )
( 97.8 , 93.1 )
(78.2, – )
( 97.5 , 89.4 )
Table 1: Performance comparison of zero-shot multimodal anomaly detection under the one-vs-rest setting on the MVTec 3D-AD and Eyecandies datasets. Best results are highlighted in red , and runners-up in blue . Throughout all tables, O-R, O-A, P-R, and P-P denote O-AUROC, O-AP, P-AUROC, and P-AUPRO, respectively.
Object-level
Pixel-level
Methods
(O-R, O-A)
(P-R, P-P)
AnomalyCLIP [ 48 ]
(65.7, 68.1)
(86.2, 61.3)
PointAD [ 49 ]
(78.6, 80.8)
(94.0, 80.7)
ZUMA-FT* [ 28 ]
(79.1, 81.2)
( 94.8 , 81.9)
GS-CLIP* [ 14 ]
( 80.2 , 82.2 )
(94.6, 82.4 )
AggCLIP (Ours)
( 81.4 , 84.2 )
( 95.8 , 85.3 )
Table 2: Performance comparison of zero-shot multimodal anomaly detection under the cross-dataset setting (trained on MVTec 3D-AD, tested on Eyecandies). * Indicates results obtained by running the official code.
Figure 3: Qualitative results on MVTec 3D-AD and Eyecandies. From top to bottom: input RGB images, 3D point clouds, ground truth, anomaly maps from GS-CLIP, and anomaly maps from AggCLIP.
Methods
GFLOPs
FPS
Time (s)
Memory (MB)
CLIP* [ 32 ]
2330.25
3.64
0.27
2270
PointAD* [ 49 ]
4252.45
3.20
0.31
2610
ZUMA-FT* [ 28 ]
4469.10
0.53
1.89
3856
GS-CLIP* [ 14 ]
9591.98
1.12
0.89
3677
AggCLIP (Ours)
4259.97
3.07
0.33
2645
Table 3: Efficiency comparison of different methods. * Indicates results obtained by running the official code.
Object-level
Pixel-level
Version
GSA
CSA
Adapter
Ldspc
(O-R, O-A)
(P-R, P-P)
V1
✗
✗
✗
✗
(81.2, 89.6)
(90.3, 88.9)
V2
✓
✗
✗
✗
(87.4, 96.7)
(97.8, 93.0)
V3
✗
✓
✗
✗
(85.5, 94.3)
(95.2, 90.4)
V4
✗
✓
✓
✗
(86.8, 95.7)
(96.2, 91.3)
V5
✓
✓
✓
✗
( 87.7 , 97.1 )
( 98.0 , 93.6 )
Table 4: Ablation on different components.
Figure 4: Ablation on key hyperparameters on MVTec 3D-AD under the one-vs-rest setting. (a) Varying the number of rendered views K . (b) Varying the GSA neighborhood size k .
Object-level
Pixel-level
Kernel Sizes
(O-R, O-A)
(P-R, P-P)
{1}
(88.0, 97.1)
(97.3, 92.9)
{3}
(87.9, 96.8)
(97.7, 92.6)
{5}
(87.7, 97.2)
(97.8, 93.0)
{1,3}
( 88.1 , 97.4 )
( 98.1 , 93.5 )
{1,3,5}
( 88.3 , 97.5 )
( 98.4 , 93.9 )
Table 5: Ablation study on the multi-scale Gaussian kernel sizes S in the CSA module.
School of Software Engineering, Xi’an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University