Bidirectional Adaptive Transformers for Multimodal Anomaly Detection

Yuxin Jiang; Yunkang Cao; Weiming Shen

cs.CVOct 10, 2026

Bidirectional Adaptive Transformers for Multimodal Anomaly Detection

Authors: Yuxin Jiang, Yunkang Cao, Weiming Shen

Organizations: School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, 430000, Hubei, China

Abstract

The effective integration of features derived from RGB images and 3D point clouds presents a substantial challenge in the field of multimodal anomaly detection (MAD). Existing MAD methods typically rely on static fusion paradigms built upon frozen pre-trained encoders. While these frameworks facilitate spatial alignment, they lack the mechanism to establish latent semantic correspondence, thereby inducing cross-modal interference that degrades anomaly discriminability. To mitigate these limitations, this study proposes a Bidirectional Adaptive Transformers (BAT) framework, which constructs comprehensive and semantically aligned multimodal representations by leveraging complementary attributes across modalities. The BAT framework introduces a Bidirectional Feature Alignment (BFA) module, which employs a mutual predictive mechanism to bridge the gap between 2D textural patterns and 3D geometric structures. This bidirectional interaction explicitly enhances the textural granularity of point cloud descriptors while augmenting the spatial contextual awareness of RGB features. To ensure that the latent representations possess sufficient informational density for alignment, BAT incorporates a parameter-efficient adaptation architecture. By utilizing lightweight image and point cloud adapters, the framework facilitates specialized fine-tuning of pre-trained models, mitigating the information loss inherent in frozen encoders. Extensive experiments conducted on MVTec 3D-AD and Eyecandies confirm that BAT achieves state-of-the-art anomaly detection performance, significantly outperforming existing MAD methods.

Figures & tables

Figure 1: Anomaly localization examples on MVTec 3D-AD. The inputs comprise RGB images (RGB), point clouds (PC), and ground truth (GT). Predictions are presented for RGB-based (2D), point cloud-based (3D), and multimodal methods, including M3DM and the proposed BAT. The BAT method demonstrates superior precision in anomaly localization compared to M3DM.

Figure 2: Comparison of multimodal anomaly detection (MAD) frameworks. (a): Method utilizes straightforward concatenation of images and point clouds with frozen pretrained encoders. (b): Method employs spatially aligned fusion with frozen pretrained encoders. (c): The proposed BAT framework introduces trainable adapters for both image and point cloud data and leverages mutual prediction to generate semantically aligned multimodal features.

Figure 3: Overall Architecture of the BAT Framework. The BAT framework integrates 2D and 3D processing pipelines through adaptive image and point transformers, generating modality-specific and fused features via bidirectional feature alignment. During testing, these features are stored in memory banks to enable precise multimodal anomaly detection.

Figure 4: Architecture of the PCR module.

Metric	Method	Bagel	Cable Gland	Carrot	Cookie	Dowel	Foam	Peach	Potato	Rope	Tire	Mean
I-AUROC	BTF ( Horwitz and Hoshen, 2023 )	0.918	0.748	0.967	0.883	0.932	0.582	0.896	0.912	0.921	0.886	0.865
	AST ( Rudolph et al., 2023 )	0.983	0.873	0.976	0.971	0.932	0.885	0.974	0.981	1.000	0.797	0.937
	M3DM ( Wang et al., 2023 )	0.994	0.909	0.972	0.976	0.960	0.942	0.973	0.899	0.972	0.850	0.945
	CFM ( Costanzino et al., 2024 )	0.994	0.888	0.984	0.993	0.980	0.888	0.941	0.943	0.980	0.953	0.954
	UCF ( Cheng et al., 2025 )	0.999	0.888	0.978	0.995	0.997	0.909	0.984	0.964	0.985	0.931	0.963
	BAT	0.996	0.896	0.992	0.997	0.976	0.914	0.985	0.971	0.998	0.970	0.970

Table 1: Performance of anomaly detection and localization on MVTec 3D-AD dataset measured by I-AUROC, P-AUROC, and AUPRO@30% scores. Best results are in bold .

Method	Bagel	Cable Gland	Carrot	Cookie	Dowel	Foam	Peach	Potato	Rope	Tire	Mean
AST	$0.388$	$0.322$	$0.470$	$0.411$	$0.328$	$0.275$	$0.474$	$0.487$	$0.360$	$0.474$	$0.398$
M3DM	$0.414$	$0.395$	$0.447$	$0.318$	0.422	$0.335$	$0.444$	$0.351$	$0.416$	$0.398$	$0.394$
CFM	$0.459$	0.431	0.485	0.469	$0.394$	$0.413$	$0.468$	$0.487$	0.464	$0.476$	$0.455$
BAT	0.492	0.431	0.485	0.469	$0.354$	0.490	0.495	0.493	0.464	0.493	0.465

Table 2: Performance of anomaly localization on MVTec 3D-AD dataset measured by AUPRO@1% scores. Best results are in bold .

Metric	Method	Can. C.	Cho. C.	Cho. P.	Conf.	Gum. B.	Haz. T.	Lic.S.	Lollip.	Marsh.	Pep. C.	Mean
I-AUROC	RGB-D ( Bonfiglioli et al., 2022 )	0.529	0.861	0.739	0.752	0.594	0.498	0.679	0.651	0.838	0.750	0.689
	RGB-cD-n ( Bonfiglioli et al., 2022 )	0.596	0.843	0.819	0.846	0.833	0.550	0.750	0.846	0.940	0.848	0.787
	AST ( Rudolph et al., 2023 )	0.574	0.747	0.747	0.889	0.596	0.617	0.816	0.841	0.987	0.987	0.780
	M3DM ( Wang et al., 2023 )	0.624	0.958	0.958	1.000	0.886	0.758	0.949	0.836	1.000	1.000	0.897
	CFM ( Costanzino et al., 2024 )	0.680	0.931	0.952	0.880	0.865	0.782	0.917	0.840	0.998	0.962	0.881
	BAT	0.605	0.970	0.950	0.995	0.902	0.654	0.930	0.853	0.998	1.000	0.886

Table 3: Performance of anomaly detection and localization on Eyecandies dataset measured by I-AUROC, P-AUROC, and AUPRO@30% scores. Best results are in bold .

Method	Can. C.	Cho. C.	Cho. P.	Conf.	Gum. B.	Haz. T.	Lic. S.	Lollip.	Marsh.	Pep. C.	Mean
AST	$0.035$	$0.230$	$0.129$	$0.234$	$0.092$	$0.069$	$0.139$	$0.090$	$0.255$	$0.224$	$0.149$
M3DM	$0.166$	$0.388$	$0.329$	$0.486$	$0.315$	$0.131$	$0.323$	$0.258$	$0.462$	$0.454$	$0.331$
CFM	0.229	$0.397$	0.345	$0.389$	0.353	0.188	$0.333$	$0.236$	$0.455$	$0.428$	$0.335$
BAT	$0.192$	0.403	0.345	0.491	0.353	0.188	0.345	0.266	0.467	0.458	0.348

Table 4: Performance of anomaly localization on Eyecandies dataset measured by AUPRO@1% scores. Best results are in bold .

Method	5-shot	10-shot	50-shot	Full	5-shot	10-shot	50-shot	Full	5-shot	10-shot	50-shot	Full
Method	I-AUROC				P-AUROC				AUPRO
BTF	0.671	0.695	0.806	0.865	0.980	0.983	0.989	0.992	0.920	0.928	0.947	0.959
AST	0.680	0.689	0.794	0.937	0.950	0.946	0.974	0.976	0.903	0.835	0.929	0.944
M3DM	0.796	0.821	0.903	0.945	0.984	0.986	0.989	0.992	0.927	0.939	0.952	0.964
BAT	0.798	0.842	0.908	0.966	0.987	0.989	0.991	0.995	0.950	0.956	0.959	0.973

Table 5: Performance of few-shot anomaly detection and localization on MVTec 3D-AD dataset measured by I-AUROC, P-AUROC, and AUPRO@30% scores. Best results are in bold .

Figure 5: Qualitative comparison of multimodal anomaly detection performance. Unlike M3DM, which produces broader and less precise anomaly maps often highlighting object outlines, BAT generates compact and accurate anomaly maps with minimal disruption, showcasing superior localization performance.

Structure	I-AUROC	P-AUROC	AUPRO@30%
Image Backbone	0.850	0.987	0.942
Linear Projection	0.843	0.986	0.943
MLP	0.537	0.908	0.722
Convolution	0.863	0.986	0.944
AIT	0.869	0.986	0.944

Table 6: Performance comparison of the AIT module across different image adaptor structures. Best results are in bold .

Structure	I-AUROC	P-AUROC	AUPRO@30%
Point Bakcbone	0.875	0.970	0.906
Full-Parameter Training	0.908	0.980	0.932
APT	0.921	0.980	0.933

Table 7: Performance comparison of the APT module against the point backbone and full fine-tuning approaches. Best results are in bold .

Alignment	I-AUROC	P-AUROC	AUPRO@30%
Concatenation	0.864	0.986	0.945
Spatial Alignment	0.945	0.992	0.971
2D-to-3D Alignment	0.922	0.988	0.959
3D-to-2D Alignment	0.945	0.991	0.966
Semantic Alignment (BFA)	0.970	0.995	0.976

Table 8: Performance comparison of different 2D and 3D feature alignment strategies. Best results are in bold .

Figure 6: Visualization of multimodal anomaly localization results with and without the BFA module. BFA reduces feature interference, significantly enhancing localization accuracy.

Figure 7: Visualization of different alignment strategies between 2D and 3D features. (a)–(d) depict non-aligned concatenation, unidirectional 2D-to-3D alignment, unidirectional 3D-to-2D alignment, and bidirectional alignment, respectively. (e) and (f) showcase their effects on semantic and geometric anomaly localization, with BFA achieving the most accurate results.

Figure 8: Visualization of 2D/3D single-modal anomaly detection results. Compared to the image and point backbones, both AIT and APT achieve superior precision in anomaly localization.

Modality	Module	Module size	I-AUROC	P-AUROC	AUPRO
2D	w/o AIT	327MB	0.850	0.987	0.942
2D	w/ AIT	343MB	0.869	0.986	0.944
3D	w/o APT	83MB	0.874	0.970	0.906
3D	w/ APT	85MB	0.921	0.980	0.933
Multimodal	w/ BFA	484MB	0.970	0.995	0.976

Table 9: Comprehensive ablation study results of single-modal adapters (AIT/APT) and multimodal bidirectional alignment within the BAT framework.

Figure 9: Visualization of multimodal anomaly detection results . Anomaly localization under the multimodal setting demonstrates superior performance compared to the single-modal settings.

Figure 10: t-SNE visualization of marginal group distributions for “carrot” and “tile” objects from MVTec 3D-AD.

	Method	Backbone	I-AUROC	AUPRO@30%
2D	M3DM	Supervised ImageNet 1K	0.670	0.808
		Supervised ImageNet 21K	0.662	0.837
		DINO ViT-S/8	0.819	0.937
		DINO ViT-B/8	0.850	0.942
	BAT	Supervised ImageNet 1K	0.697	0.822
		Supervised ImageNet 21K	0.679	0.843

Table 10: Performance comparison of different backbones. Best results are in bold .

Multimodal Anomaly Detection

Multimodal anomaly detection identifies unusual patterns by combining complementary signals such as images, depth, video, text, and sensor data.

See all 97 papers

CMDS-AD: Cross-Modal Dual-Stream Decoupling for Few-Shot Anomaly Detection
Jun 18, 2026Junhao Cai, Junyu Chen, Deyu Zeng +4Few-Shot Anomaly Detection Multimodal Anomaly Detection
Shenzhen University, Shenzhen, Guangdong 518060, China · Guangzhou Maritime University, Guangzhou, Guangdong 510725, China · Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong 511453, China
Align3D-AD: Cross-Modal Feature Alignment and Dual-Prompt Learning for Zero-shot 3D Anomaly Detection
May 7, 2026Letian Bai, Xuanming Cao, Juan Du +1Prompt Learning Cross-Modal Alignment
Smart Manufacturing Thrust, The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · College of Mechanical and Vehicle Engineering, Hunan University
Text-Guided Multimodal Unified Industrial Anomaly Detection
Apr 24, 2026Zewen Li, Shuo Ye, Zitong Yu +2Unified Multimodal Models Industrial Anomaly Detection
School of Computer Science & Software Engineering, Shenzhen University, China, 518060 · School of Computing and Information Technology, Great Bay University, Dongguan, 523000,China

Jun 18, 2026J/K move · Enter open · S save

CMDS-AD: Cross-Modal Dual-Stream Decoupling for Few-Shot Anomaly Detection

Junhao Cai, Junyu Chen, Deyu Zeng, Junhao Pang+3

Shenzhen University, Shenzhen, Guangdong 518060, China · Guangzhou Maritime University, Guangzhou, Guangdong 510725, China · Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong 511453, China

Few-Shot Anomaly Detection Multimodal Anomaly Detection Cross-Modal Feature Matching Anomaly Detection

Open paper AskPDF arXiv

Few-shot anomaly detection remains challenging due to limited training data. Multi-modal anomaly detection (MAD) offers a viable solution, leveraging 3D geometric cues to enrich 2D RGB representations and compensate for this scarcity. However, existing MAD methods apply spatially uniform feature processing, conflating stable macroscopic structures with high-frequency localized defect signals, exacerbating cross-modal misalignment and inflating false-positive rates. To overcome this, we present CMDS-AD, a Cross-Modal Dual-Stream Anomaly Detection framework. A LoRA-guided diffusion model generates diverse RGB samples to mitigate extreme data scarcity. For 3D normal augmentation, we employ a pre-trained diffusion model as a normal estimator. Crucially, this estimator inherently acts as a non-linear low-pass filter, directly extracting low-frequency normal representations from RGB inputs. This establishes an auxiliary estimated stream of purely low-frequency information, anchoring robust structural templates and assisting the uncompressed real stream, containing coupled high- and low-frequency components, to precisely isolate micro-defects. A Coordinate-Aware Hierarchical Feature Mapper adaptively aligns cross-modal semantics, while a multiplicative scoring mechanism filters modality-specific noise. Under the extreme 1-shot setting, CMDS-AD achieves absolute performance gains of 5.7% (I-AUROC) and 2.0% (AUPRO) on MVTec 3D-AD, alongside 7.7% and 5.6% improvements on EyeCandies, establishing a new state-of-the-art. Code is available at https://github.com/Junhaocai27/CMDS-AD

Abstract

Multimodal Anomaly Detection

Explore similar work

CMDS-AD: Cross-Modal Dual-Stream Decoupling for Few-Shot Anomaly Detection

Align3D-AD: Cross-Modal Feature Alignment and Dual-Prompt Learning for Zero-shot 3D Anomaly Detection

Text-Guided Multimodal Unified Industrial Anomaly Detection