cs.CVNov 18, 2026

AggCLIP: Aggregating CLIP for Zero-Shot Multimodal Anomaly Detection

Authors: Zhi Li, Hanyu Wang, Fei Shen, Zhengtao Zhang

Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · CASI Vision Technology Co., Ltd., Luoyang, Henan, China

Abstract

Zero-Shot Multimodal Anomaly Detection (ZSMAD) aims to identify both appearance and geometric defects in novel categories without relying on target-specific training data, making it especially valuable in real-world scenarios constrained by data scarcity, privacy concerns, or high annotation costs. While recent approaches successfully adapt vision-language models like CLIP to achieve this open-vocabulary generalization, they still face significant challenges: (i) severe geometric information loss during 3D-to-2D point cloud rendering; (ii) spatially noisy 2D feature maps produced by raw CLIP patch embeddings, compounded by shallow late-stage modality fusion; and (iii) severe cross-modal semantic misalignment due to decoupled processing streams. To address these limitations, we propose AggCLIP, a novel framework that adapts frozen CLIP representations for ZSMAD by explicitly incorporating local geometric structure and visual spatial context. Specifically, to accurately localize fine-grained defects, we design a Geometry-Aware Spatial Aggregation (GSA) module that refines back-projected 3D point features via density-adaptive neighborhood weighting, alongside a Context-Aware Spatial Aggregation (CSA) module that smooths multi-scale 2D patch embeddings using a lightweight trainable adapter. Furthermore, to prevent semantic drift between modalities, we introduce a Dual-Stream Prompt Contrastive (DSPC) loss, which aligns the abstract concepts of normality and abnormality across the point cloud and RGB streams. Extensive experiments demonstrate that the proposed AggCLIP framework achieves state-of-the-art performance on the MVTec 3D-AD and Eyecandies benchmarks under both one-vs-rest and cross-dataset zero-shot settings, notably surpassing the strongest competitive methods by up to 2.9% in pixel-level AUPRO.

Figures & tables

Explore similar work

CardsList
  1. GS-CLIP: Zero-shot 3D Anomaly Detection by Geometry-Aware Prompt and Synergistic View Representation Learning

    Feb 22, 2026Zehao Deng, An Liu, Yan WangZero-Shot Anomaly DetectionPoint Cloud Anomaly Detection

  2. MoECLIP: Patch-Specialized Experts for Zero-shot Anomaly Detection

    Mar 3, 2026Jun Yeong Park, JunYoung Seo, Minji Kang +1Zero-Shot Anomaly DetectionVision Foundation Model Adaptation

  3. Proximity-CLIP: Text-Guided Semantic Proximity Learning for Zero-Shot Anomaly Detection

    Sep 7, 2026Manwen Yang, Leqian Ding, Yu Guo +1VLM AdaptationAnomaly Localization