Modulate-and-Map: Crossmodal Feature Mapping with Cross-View Modulation for 3D Anomaly Detection

Alex Costanzino; Pierluigi Zama Ramirez; Giuseppe Lisanti; Luigi Di Stefano

cs.CVApr 2, 2026

Modulate-and-Map: Crossmodal Feature Mapping with Cross-View Modulation for 3D Anomaly Detection

Authors: Alex Costanzino, Pierluigi Zama Ramirez, Giuseppe Lisanti, Luigi Di Stefano

Organizations: CVLab, University of Bologna · Ca’ Foscari University of Venice

Abstract

We present ModMap, a natively multiview and multimodal framework for 3D anomaly detection and segmentation. Unlike existing methods that process views independently, our method draws inspiration from the crossmodal feature mapping paradigm to learn to map features across both modalities and views, while explicitly modelling view-dependent relationships through feature-wise modulation. We introduce a cross-view training strategy that leverages all possible view combinations, enabling effective anomaly scoring through multiview ensembling and aggregation. To process high-resolution 3D data, we train and publicly release a foundational depth encoder tailored to industrial datasets. Experiments on SiM3D, a recent benchmark that introduces the first multiview and multimodal setup for 3D anomaly detection and segmentation, demonstrate that ModMap attains state-of-the-art performance by surpassing previous methods by wide margins.

Figures & tables

Panel of Figure 1 — Figure 1: View-dependent Artefacts. The first column shows acquisition artefacts observed in an image (top row: specular highlights, red box ) and a depth map (bottom row: missing depths, blue box ), dealing with two objects of SiM3D. As shown in the right column, other views of the same object are not affected by artefacts at the positions ( green boxes ) corresponding to those highlighted in the left column.

Figure 2: ModMap Training. Starting from the set of images $I$ and depths $D$ from a training sample, we select a source view $s$ and target view $t$ , and forward their images $I^{s},I^{t}$ and depths $D^{s},D^{t}$ to the image, $\mathcal{E}_{I}$ , and depth, $\mathcal{E}_{D}$ , encoders, respectively, so as to compute modality-specific features, $F^{s}_{I},F^{t}_{I}$ and $F^{s}_{D},F^{t}_{D}$ . Moreover, the one-hot encodings of the view indexes, $v_{s}$ and $v_{t}$ , are fed into the feature modulators, $\Phi_{I}$ and $\Phi_{D}$ , to generate modality-specific scale-and-shift parameters, $\gamma_{I},\beta_{I}$ and $\gamma_{D},\beta_{D}$ . Then, for both modalities, the source features are scale-and-shifted to obtain modulated source features that incorporate view conditioning: $F^{s\rightarrow t}_{I},F^{s\rightarrow t}_{D}$ . The modulated features are passed as inputs to the mapping networks $\mathcal{M}_{I\rightarrow D},\mathcal{M}_{D\rightarrow I}$ that predict the corresponding features from the other modality $\hat{F}^{s\rightarrow t}_{D},\hat{F}^{s\rightarrow t}_{I}$ . The predicted features are then compared to the actual target features $F^{t}_{D},F^{t}_{I}$ to optimise both the modulators and the mapping networks.

Figure 3: ModMap Inference. We process the set of $N$ images $I^{i}$ and $N$ depths $D^{i}$ , obtaining $N\times N$ anomaly maps for each of the two modalities. We ensemble the anomaly scores into $N$ refined 2D anomaly maps for each modality. Finally, we aggregate the 2D anomaly maps to obtain a 3D Anomaly Volume and an Instance-level Anomaly Score.

Figure 4: Rationale for ensembling. Squares represent features ( green : uncorrupted, red : corrupted by image artefacts, blue : corrupted by depth artefacts, grey : incorrect prediction), while arrows show mapping predictions.

Algorithm	Features		Detection									Segmentation
Algorithm	Image	Depth / Point Cloud	Pl. Stool	Rub. Bin	W. Vase	B. Furn.	Cont.	Pl. Vase	W. Stool	Sink Cab.	Mean	Pl. Stool	Rub. Bin	W. Vase	B. Furn.	Cont.	Pl. Vase	W. Stool	Sink Cab.	Mean
PatchCore	WRN-101	✗	0.740	0.987	0.636	0.777	0.774	0.551	1.000	0.566	0.754	0.710	0.461	0.761	0.675	0.694	0.747	0.380	0.609	0.630
	DINO-v2	✗	0.500	0.958	0.636	0.622	0.578	0.563	1.000	0.563	0.678	0.745	0.469	0.775	0.792	0.709	0.753	0.435	0.690	0.671
	✗	WRN-101	–	–	–	–	–	–	–	–	0.582	–	–	–	–	–	–	–	–	0.486
	✗	DINO-v2	–	–	–	–	–	–	–	–	0.600	–	–	–	–	–	–	–	–	0.469
	✗	FPFH	–	–	–	–	–	–	–	–	0.415	–	–	–	–	–	–	–	–	0.464

Table 1: Results on real-to-real setup of SiM3D. Best results in bold , runner-ups underlined . Missing entries (–) not reported in [ 11 ] .

Algorithm	Features		Detection									Segmentation
Algorithm	Image	Depth / Point Cloud	Pl. Stool	Rub. Bin	W. Vase	B. Furn.	Cont.	Pl. Vase	W. Stool	Sink Cab.	Mean	Pl. Stool	Rub. Bin	W. Vase	B. Furn.	Cont.	Pl. Vase	W. Stool	Sink Cab.	Mean
PatchCore	WRN-101	✗	0.462	0.235	0.537	0.353	0.678	0.576	0.517	0.250	0.451	0.734	0.437	0.751	0.454	0.678	0.741	0.386	0.618	0.600
	DINO-v2	✗	0.958	0.897	0.413	0.464	0.207	0.684	0.607	0.087	0.540	0.701	0.457	0.772	0.563	0.648	0.734	0.321	0.575	0.596
	✗	WRN-101	–	–	–	–	–	–	–	–	0.507	–	–	–	–	–	–	–	–	0.484
	✗	DINO-v2	–	–	–	–	–	–	–	–	0.246	–	–	–	–	–	–	–	–	0.471
	✗	FPFH	–	–	–	–	–	–	–	–	0.313	–	–	–	–	–	–	–	–	0.460

Table 2: Results on synthetic-to-real setup of SiM3D. Best results in bold , runner-ups underlined . Missing entries (–) not reported in [ 11 ] .

Figure 5: Qualitative results. Real-to-real (top) vs. Synthetic-to-real (bottom). Anomalies are highlighted by red boxes .

View Conditioning	Features		Detection	Segmentation
View Conditioning	Image	Point Cloud	Detection	Segmentation
✗	DINO-v2	FPFH	0.448	0.538
✓	DINO-v2	FPFH	0.548	0.663

Table 3: Impact of View Conditioning.

Features		Detection	Segmentation
Image	Depth / Point Cloud	Detection	Segmentation
DINO-v2	FPFH	0.548	0.663
	Point-MAE	0.543	0.545
	DINO-v2	0.575	0.684
	DINO-Depth	0.804	0.735

Table 4: Effects of 3D Feature Extractor.

Panel of Figure 6 — Figure 6: PCA of Depth Features.

Training Split	No. Samples	Detection	Segmentation
A	19647	0.568	0.667
B	23896	0.609	0.705
C	46528	0.804	0.735

Table 5: DINO-Depth Training Set.

Cross-View	Detection	Segmentation
Conditioning	Detection	Segmentation
✗	0.804	0.735
✓	0.844	0.804

Table 6: Cross-View Conditioning.

Model	Real-to-real		Synthetic-to-real
Model	Detection	Segmentation	Detection	Segmentation
Class-specific	0.844	0.804	0.623	0.755
Multi-class	0.850	0.801	0.618	0.758

Table 7: Class-specific vs. Multi-class.

No. Views	Inference Time	Detection	Segmentation
[#]	[ms]	Sink Cabinet
36	2886.768	1.000	0.785
18	1443.384	1.000	0.785
9	721.692	1.000	0.783
5	400.94	0.972	0.781

Table 8: No. Views vs. Inference Time vs. Performance Comparison on Sink Cabinet , i.e., the most computationally expensive class of SiM3D.

Features		Detection	Segmentation
Image	Depth	Detection	Segmentation
DINO-v3	DINO-v3	0.534	0.569
DINO-v2	DINO-v2	0.575	0.684

Table 9: Effects of Image Feature Extractor.

Aggregation	Detection									Segmentation
Aggregation	Pl. Stool	Rub. Bin	W. Vase	B. Furn.	Cont.	Pl. Vase	W. Stool	Sink Cab.	Mean	Pl. Stool	Rub. Bin	W. Vase	B. Furn.	Cont.	Pl. Vase	W. Stool	Sink Cab.	Mean
$\max$	0.909	0.990	0.945	0.647	0.740	0.607	0.916	1.000	0.844	0.855	0.707	0.863	0.791	0.831	0.885	0.711	0.785	0.804
$\min$	0.690	1.000	1.000	0.500	0.455	0.606	0.520	1.000	0.721	0.292	0.412	0.847	0.408	0.623	0.874	0.089	0.061	0.451
$\product$	0.700	1.000	0.972	0.568	0.520	0.618	0.583	1.000	0.745	0.563	0.634	0.881	0.635	0.710	0.869	0.365	0.471	0.641
$\mean$	0.736	1.000	0.972	0.556	0.669	0.613	0.708	1.000	0.782	0.775	0.677	0.878	0.749	0.794	0.865	0.569	0.740	0.756

Table 10: Aggregation Function.

Panel of Figure 8 — Figure 8: PCA of Depth Features.

Panel of Figure 9 — Figure 9: Image-to-Depth Cross-Views.

Panel of Figure 10 — Figure 10: Depth-to-Image Cross-Views.

Explore similar work

Cross-Modal Mapping and Dual-Branch Reconstruction for 2D-3D Multimodal Industrial Anomaly Detection
Mar 4, 2026Radia Daci, Vito Renò, Cosimo Patruno +4Industrial Anomaly Detection Multimodal Anomaly Detection
CNR-ISASI, Institute of Applied Sciences and Intelligent Systems / National Research Council of Italy, via Monteroni snc, Lecce, 73100, Italy · CNR-STIIMA, Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing, Via G. Amendola 122 D/O, Bari, 70126, Italy · Institute d’Electronique de Microelectronique et de Nanotechnologie (IEMN) / Universite Polytechnique Hauts de France; Universite de Lille; CNRS, UMR 8520, City, 59313, France
SGANet: Semantic and Geometric Alignment for Multimodal Multi-view Anomaly Detection
Apr 7, 2026Letian Bai, Chengyu Tao, Juan DuAnomaly Detection Multimodal Anomaly Detection
Smart Manufacturing Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511453, China · College of Mechanical and Vehicle Engineering, Hunan University, Changsha, 410082, China · The Hong Kong University of Science and Technology, Hong Kong SAR, 999077, China
Text-Guided Multimodal Unified Industrial Anomaly Detection
Apr 24, 2026Zewen Li, Shuo Ye, Zitong Yu +2Unified Multimodal Models Industrial Anomaly Detection
School of Computer Science & Software Engineering, Shenzhen University, China, 518060 · School of Computing and Information Technology, Great Bay University, Dongguan, 523000,China

Mar 4, 2026J/K move · Enter open · S save

Cross-Modal Mapping and Dual-Branch Reconstruction for 2D-3D Multimodal Industrial Anomaly Detection

Radia Daci, Vito Renò, Cosimo Patruno, Angelo Cardellicchio+3

CNR-ISASI, Institute of Applied Sciences and Intelligent Systems / National Research Council of Italy, via Monteroni snc, Lecce, 73100, Italy · CNR-STIIMA, Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing, Via G. Amendola 122 D/O, Bari, 70126, Italy · Institute d’Electronique de Microelectronique et de Nanotechnologie (IEMN) / Universite Polytechnique Hauts de France; Universite de Lille; CNRS, UMR 8520, City, 59313, France

Industrial Anomaly Detection Multimodal Anomaly Detection Unsupervised Anomaly Detection Reconstruction-Based Anomaly Detection Cross-Modal Learning

Open paper AskPDF arXiv

Multimodal industrial anomaly detection benefits from integrating RGB appearance with 3D surface geometry, yet existing unsupervised approaches commonly rely on memory banks, teacher-student architectures, or fragile fusion schemes, limiting robustness under noisy depth, weak texture, or missing modalities. This paper introduces CMDR-IAD, a lightweight and modality-flexible unsupervised framework for reliable anomaly detection in 2D+3D multimodal as well as single-modality (2D-only or 3D-only) settings. CMDR-IAD combines bidirectional 2D

\leftrightarrow

3D cross-modal mapping to model appearance-geometry consistency with dual-branch reconstruction that independently captures normal texture and geometric structure. A two-part fusion strategy integrates these cues: a reliability-gated mapping anomaly highlights spatially consistent texture-geometry discrepancies, while a confidence-weighted reconstruction anomaly adaptively balances appearance and geometric deviations, yielding stable and precise anomaly localization even in depth-sparse or low-texture regions. On the MVTec 3D-AD benchmark, CMDR-IAD achieves state-of-the-art performance while operating without memory banks, reaching 97.3% image-level AUROC (I-AUROC), 99.6% pixel-level AUROC (P-AUROC), and 97.6% AUPRO. On a real-world polyurethane cutting dataset, the 3D-only variant attains 92.6% I-AUROC and 92.5% P-AUROC, demonstrating strong effectiveness under practical industrial conditions. These results highlight the framework's robustness, modality flexibility, and the effectiveness of the proposed fusion strategies for industrial visual inspection. Our source code is available at https://github.com/ECGAI-Research/CMDR-IAD/

Abstract

Explore similar work

Cross-Modal Mapping and Dual-Branch Reconstruction for 2D-3D Multimodal Industrial Anomaly Detection

SGANet: Semantic and Geometric Alignment for Multimodal Multi-view Anomaly Detection

Text-Guided Multimodal Unified Industrial Anomaly Detection