When Point Clouds Outperform Pixels: Rethinking Zero-Shot Multimodal Anomaly Detection

Chenglin Ye; Lupeng Liu; Dongbo Yu; Jun Xiao; Yunbiao Wang

cs.CVSep 22, 2026

When Point Clouds Outperform Pixels: Rethinking Zero-Shot Multimodal Anomaly Detection

Authors: Chenglin Ye, Lupeng Liu, Dongbo Yu, Jun Xiao, Yunbiao Wang

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences

Abstract

Zero-shot multimodal anomaly detection commonly assumes that RGB and point cloud modalities are equally reliable and can contribute uniformly to anomaly localization. We challenge this assumption. Using a set of recently proposed stringent metrics that penalize false anomaly responses in normal regions, we find that point clouds are substantially more reliable than RGB under zero-shot category shift. Motivated by this observation, we propose WOOPS (When Point Clouds Outperform Pixels), a reliability-aware zero-shot multimodal anomaly detection framework. To strengthen the more reliable geometric modality, we design a Multi-view Information Decoupling module to suppress heterogeneous information from multi-view point cloud projections and enhance point cloud feature quality. To avoid unconditional fusion, we further introduce a Modality Reliability Calibration module to adaptively calibrate modality contributions according to their reliability. Extensive experiments show that our method achieves the best or competitive performance under the new metrics in both unimodal and multimodal settings. Further analysis demonstrates that point cloud information also improves RGB-only inference, while ablations verify the effectiveness of both modules. Code will be released upon acceptance.

Figures & tables

Panel of Figure 1 — Figure 1: (a) We illustrate results using P-AUROC as a representative of conventional metrics ( Bergmann et al. 2022 ) and ${\mathrm{m}F_{\mathrm{1}}}^{.2}_{.8}$ as a representative of the new evaluation protocol ( Zhang et al. 2026 ) . Please zoom in for details. (b) Comparison of network architectures between our model and existing zero-shot multimodal anomaly detection methods.

Figure 2: The pipeline of WOOPS. WOOPS employs a frozen CLIP vision encoder $\Psi_{\mathrm{v}}$ to extract RGB and point cloud features from images and multi-view projections, respectively, while utilizing a frozen CLIP text encoder $\Psi_{\mathrm{t}}$ to derive text features from learnable normal and abnormal language vectors ( $\mathbf{P}_{n}$ and $\mathbf{P}_{a}$ ). Anomaly responses are computed via vision-language similarity. Furthermore, we propose a Multi-view Information Decoupling (MID) module to enhance point cloud features and a Modality Reliability Calibration (MRC) module to effectively calibrate anomaly responses across modalities.

Inference	Training	Method	Source	MVTec 3D-AD			Eyecandies
Inference	Training	Method	Source	${\mathrm{m}F_{\mathrm{1}}}^{.2}_{.8}$	$\mathrm{mAcc}^{.2}_{.8}$	$\mathrm{mIoU}^{.2}_{.8}$	${\mathrm{m}F_{\mathrm{1}}}^{.2}_{.8}$	$\mathrm{mAcc}^{.2}_{.8}$	$\mathrm{mIoU}^{.2}_{.8}$
RGB	RGB	AnomalyCLIP	ICLR’24	2.1	7.9	1.0	1.4	7.4	0.7
	RGB	FAPrompt	ICCV’25	2.8	9.7	1.5	2.5	8.8	1.2
	PC	PointAD (RGB)	NIPS’24	19.8	25.1	11.6	13.1	18.0	7.4
	PC	ZUMA-FT (RGB)	TPAMI’26	20.1	25.3	11.8	13.0	18.1	7.3
	MUL	Ours (RGB)	-	21.0	44.7	12.1	17.2	42.7	10.3

Table 1: Comparison with state-of-the-art methods on MVTec 3D-AD dataset and Eyecandies dataset. Results are ranked separately according to the inference modality configurations. The best and second-best results within each group are denoted by gray shading.

Panel of Figure 3 — Figure 3: Visualization results of WOOPS. From top to bottom: RGB input, RGB anomaly response, RGB reliability map, point cloud anomaly response, multimodal anomaly response, and ground truth. In the reliability map, warmer red tones indicate higher confidence, whereas cooler green tones denote lower reliability. As evident, regions exhibiting erroneous RGB responses correspond to low-reliability areas, demonstrating the effectiveness of our reliability calibration.

Setting	${\mathrm{m}F_{\mathrm{1}}}^{.2}_{.8}$
	MVTec 3D-AD			Eyecandies
	RGB	PC	MUL	RGB	PC	MUL
Full	21.0	22.6	19.3	17.2	22.4	18.9
w/o MID	18.5	17.4	18.1	13.1	13.8	13.3
w/o MRC	20.4	22.5	19.1	9.3	11.6	11.4
w/o $L_{fe}$	18.0	21.0	18.0	17.0	21.9	18.7

Table 2: Quantitative results of the ablation study. Top results are in bold, and the default setting is indicated by gray shading.

Panel of Figure 4 — Figure 4: t-SNE Visualization of Decoupled Features from the MID Module. "Sample" denotes the features of the projection views sampled by the MID module, while "Overall" refers to the point cloud features. Zoom in for details.

Multimodal Anomaly Detection

Multimodal anomaly detection identifies unusual patterns by combining complementary signals such as images, depth, video, text, and sensor data.

See all 97 papers

Rethinking Auxiliary Modalities in Multimodal Zero-shot Anomaly Detection: From Semantic Fusion to Conditional Modulation
Aug 14, 2026Peng Wu, Xin Ge, Yujia Sun +1Zero-Shot Anomaly Detection Multimodal Anomaly Detection
School of Computer Science, Northwestern Polytechnical University, China · School of Artificial Intelligence, Xidian University, China · School of Computing and Information Systems, Singapore Management University, Singapore
Bidirectional Adaptive Transformers for Multimodal Anomaly Detection
Oct 10, 2026Yuxin Jiang, Yunkang Cao, Weiming ShenMultimodal Anomaly Detection Multimodal Fusion
School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, 430000, Hubei, China
Align3D-AD: Cross-Modal Feature Alignment and Dual-Prompt Learning for Zero-shot 3D Anomaly Detection
May 7, 2026Letian Bai, Xuanming Cao, Juan Du +1Prompt Learning Cross-Modal Alignment
Smart Manufacturing Thrust, The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · College of Mechanical and Vehicle Engineering, Hunan University

Aug 14, 2026J/K move · Enter open · S save

Rethinking Auxiliary Modalities in Multimodal Zero-shot Anomaly Detection: From Semantic Fusion to Conditional Modulation

Peng Wu, Xin Ge, Yujia Sun, Guansong Pang

School of Computer Science, Northwestern Polytechnical University, China · School of Artificial Intelligence, Xidian University, China · School of Computing and Information Systems, Singapore Management University, Singapore

Zero-Shot Anomaly Detection Multimodal Anomaly Detection Uncertainty-Aware Multimodal Fusion

Open paper AskPDF arXiv

Recent foundation model-based methods have endowed RGB images with strong zero-shot anomaly detection (ZSAD) through vision-language pretraining. However, RGB observations alone remain limited in perceiving anomalies dominated by geometric deformation, depth variation, or subtle surface changes. Auxiliary modalities can provide complementary structural information, but existing multimodal methods typically fuse them directly into a shared semantic space, which may disturb the text-aligned anomaly semantics established by RGB foundation models and often requires modality-specific architectures. To address this issue, we propose a plug-and-play auxiliary-conditioned enhancement framework for zero-shot anomaly detection. Instead of reconstructing a joint multimodal anomaly semantic space, our framework preserves the original RGB image-text anomaly matching pathway and uses auxiliary observations as conditional signals for RGB feature refinement, allowing auxiliary modalities to seamlessly enhance existing RGB-based zero-shot anomaly detectors. Specifically, a lightweight meta-learning module takes global RGB and auxiliary representations as input and generates sample-adaptive low-rank residual updates to determine how RGB features should be refined. We further construct uncertainty-aware spatial modulation from the initial RGB anomaly response and auxiliary reliability, which determines where local residual updates are strengthened or suppressed. This global-to-local conditional modulation enables selective multimodal enhancement while preserving the original RGB anomaly semantics. Extensive experiments on MVTec 3D-AD and Eyecandies demonstrate that our framework consistently improves multiple popular RGB-based zero-shot anomaly detectors, achieving state-of-the-art performance for multimodal zero-shot anomaly detection.

Abstract

Multimodal Anomaly Detection

Explore similar work

Rethinking Auxiliary Modalities in Multimodal Zero-shot Anomaly Detection: From Semantic Fusion to Conditional Modulation

Bidirectional Adaptive Transformers for Multimodal Anomaly Detection

Align3D-AD: Cross-Modal Feature Alignment and Dual-Prompt Learning for Zero-shot 3D Anomaly Detection