SGANet: Semantic and Geometric Alignment for Multimodal Multi-view Anomaly Detection
Authors: Letian Bai, Chengyu Tao, Juan Du
Organizations: Smart Manufacturing Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511453, China · College of Mechanical and Vehicle Engineering, Hunan University, Changsha, 410082, China · The Hong Kong University of Science and Technology, Hong Kong SAR, 999077, China
Multi-view anomaly detection aims to identify surface defects on complex objects using observations captured from multiple viewpoints. However, existing unsupervised methods often suffer from feature inconsistency arising from viewpoint variations and modality discrepancies. To address these challenges, we propose a Semantic and Geometric Alignment Network (SGANet), a unified framework for multimodal multi-view anomaly detection that effectively combines semantic and geometric alignment to learn physically coherent feature representations across viewpoints and modalities. SGANet consists of three key components. The Selective Cross-view Feature Refinement Module (SCFRM) selectively aggregates informative patch features from adjacent views to enhance cross-view feature interaction. The Semantic-Structural Patch Alignment (SSPA) enforces semantic alignment across modalities while maintaining structural consistency under viewpoint transformations. The Multi-View Geometric Alignment (MVGA) further aligns geometrically corresponding patches across viewpoints. By jointly modeling feature interaction, semantic and structural consistency, and global geometric correspondence, SGANet effectively enhances anomaly detection performance in multimodal multi-view settings. Extensive experiments on the SiM3D and Eyecandies datasets demonstrate that SGANet achieves state-of-the-art performance in both anomaly detection and localization, validating its effectiveness in realistic industrial scenarios.
Figures & tables
Figure 1: Comparison of anomaly detection methods. Note that red solid circles denote correctly detected anomalies, while red dashed circles indicate undetected anomalies. (a) Traditional single-view anomaly detection methods process each view independently, making it difficult to leverage information from other viewpoints. (b) Multi-view anomaly detection methods integrate information across viewpoints via feature alignment. However, existing approaches suffer from semantic ambiguity due to reliance on similarity without geometric properties, inaccurate spatial modeling from approximate geometric correspondence, and the lack of effective consistency constraints. (c) Our method explicitly models relationships across views through feature refinement, semantic-structural alignment, and multi-view geometric alignment, enabling effective integration of complementary information and more reliable anomaly detection under viewpoint variations.
Figure 2: Overview of the proposed SGANet framework for multimodal multi-view anomaly detection. Multi-view images and depth maps are first encoded into patch-level features using frozen feature extractors F2D and F3D , respectively. The Selective Cross-view Feature Refinement Module (SCFRM, Section 3.3 ) first refines feature representations by aggregating semantically related features from adjacent viewpoints through modality-aware selection and cross-view feature aggregation. Based on the refined features, the Semantic-Structural Patch Alignment (SSPA, Section 3.4 ) enforces semantic alignment by matching corresponding patches between the 2D and 3D modalities within each view and further incorporates a structural alignment mechanism to maintain consistency under viewpoint transitions. To further ensure feature coherence, Multi-View Geometric Alignment (MVGA, Section 3.5 ) leverages explicit geometric correspondences to align features at spatially corresponding locations, thereby reducing feature discrepancies and promoting spatially consistent multi-view feature representations. During inference (Section 3.6 ), multimodal features are concatenated and compared with a memory bank constructed from normal samples to compute anomaly scores and generate anomaly maps.
Category
Nominal
Anomalous
Total
Views
Bathroom Furniture
8
11
19
36
Container
46
47
93
12
Plastic Stool
10
11
21
12
Plastic Vase
48
50
98
12
Rubbish Bin
20
20
40
12
Sink Cabinet
9
8
17
36
Table 1: Dataset statistics of the SiM3D dataset [ 12 ] .
Figure 3: Qualitative comparison on the SiM3D dataset [ 12 ] . Visualizations include defect regions highlighted in the input views and the corresponding anomaly score maps predicted by representative methods, including single-view baselines (PatchCore, AST), the multi-view method (MVAD), and the proposed SGANet framework.
Method
Modality
Pl. Stool
Rub. Bin
W. Vase
B. Furn.
Cont.
Pl. Vase
W. Stool
Sink Cab.
Mean
PatchCore (WRN-101) [ 28 ]
RGB
0.900
1.000
0.836
0.795
0.753
0.553
0.917
0.972
0.841
PatchCore (DINO-v2) [ 28 ]
RGB
0.736
1.000
0.745
0.795
0.860
0.865
1.000
0.931
0.867
EfficientAD [ 3 ]
RGB
0.280
0.732
0.000
0.878
0.424
0.730
0.928
0.712
0.586
AST [ 29 ]
RGB
0.900
0.260
0.618
0.625
0.547
0.423
0.625
0.403
0.550
MVAD [ 18 ]
RGB
0.573
0.990
0.427
0.727
0.669
0.567
0.938
0.875
0.721
BTF [ 19 ]
RGB + PC
0.421
0.217
0.504
0.565
0.545
0.471
0.678
0.424
0.478
Table 2: I-AUROC result on the SiM3D dataset. Best results in bold, runner-ups underlined .
Method
Modality
Pl. Stool
Rub. Bin
W. Vase
B. Furn.
Cont.
Pl. Vase
W. Stool
Sink Cab.
Mean
PatchCore (WRN-101) [ 28 ]
RGB
0.774
0.478
0.775
0.613
0.698
0.752
0.290
0.431
0.601
PatchCore (DINO-v2) [ 28 ]
RGB
0.790
0.489
0.792
0.609
0.715
0.756
0.290
0.459
0.613
EfficientAD [ 3 ]
RGB
0.682
0.462
0.763
0.534
0.680
0.743
0.407
0.488
0.595
AST [ 29 ]
RGB
0.595
0.354
0.754
0.338
0.594
0.748
0.231
0.083
0.462
MVAD [ 18 ]
RGB
0.754
0.481
0.760
0.359
0.655
0.745
0.256
0.326
0.542
BTF [ 19 ]
RGB + PC
0.551
0.402
0.750
0.377
0.614
0.741
0.092
0.030
0.445
Table 3: V-AUPRO@1% result on the SiM3D dataset. Best results in bold, runner-ups underlined .
Figure 4: I-AUROC and V-AUPRO@1% results on the SiM3D dataset using RGB modality, depth modality, and the proposed SGANet framework.
Category
Modality
Candy. Cane
Choco. Cook.
Choco. Pra.
Conf.
Gum. Bear
Hazel. Truf.
Lico. Sand.
Lolli.
Marsh.
Pep. Candy
Mean
PatchCore (WRN-101) [ 28 ]
RGB
0.381
0.592
0.534
0.677
0.409
0.707
0.448
0.621
0.651
0.562
0.558
PatchCore (DINO-v2) [ 28 ]
RGB
0.443
0.824
0.563
0.680
0.784
0.602
0.811
0.658
0.781
0.587
0.673
MVAD [ 18 ]
RGB
0.322
0.534
0.448
0.773
0.579
0.410
0.635
0.487
0.666
0.723
0.558
CFM (DINO-v2+FPFH) [ 11 ]
RGB + PC
0.390
0.542
0.514
0.643
0.572
0.426
0.613
0.618
0.688
0.533
0.554
AST [ 29 ]
RGB + Depth
0.448
0.424
0.574
0.370
0.548
0.595
0.613
0.662
0.498
0.325
0.506
SGANet (Ours)
RGB + Depth
0.451
0.925
0.787
0.757
0.692
0.534
0.795
0.770
0.842
0.874
0.743
Table 4: I-AUROC result on the Eyecandies dataset. Best results in bold, runner-ups underlined .
Category
Modality
Candy. Cane
Choco. Cook.
Choco. Pra.
Conf.
Gum. Bear
Hazel. Truf.
Lico. Sand.
Lolli.
Marsh.
Pep. Candy
Mean
PatchCore (WRN-101) [ 28 ]
RGB
0.911
0.823
0.651
0.787
0.777
0.712
0.847
0.960
0.891
0.772
0.813
PatchCore (DINO-v2) [ 28 ]
RGB
0.601
0.908
0.733
0.872
0.873
0.712
0.893
0.816
0.888
0.821
0.812
MVAD [ 18 ]
RGB
0.933
0.847
0.685
0.814
0.855
0.736
0.855
0.956
0.842
0.773
0.830
CFM (DINO-v2 + FPFH) [ 11 ]
RGB +PC
0.928
0.775
0.706
0.764
0.797
0.654
0.876
0.946
0.871
0.802
0.812
AST [ 29 ]
RGB + Depth
0.750
0.647
0.566
0.662
0.559
0.519
0.572
0.587
0.479
0.471
0.581
SGANet (Ours)
RGB + Depth
0.846
0.921
0.795
0.879
0.899
0.712
0.887
0.885
0.884
0.834
0.854
Table 5: P-AUROC result on the Eyecandies dataset. Best results in bold, runner-ups underlined .
Category
Modality
Candy. Cane
Choco. Cook.
Choco. Pra.
Conf.
Gum. Bear
Hazel. Truf.
Lico. Sand.
Lolli.
Marsh.
Pep. Candy
Mean
PatchCore (WRN-101) [ 28 ]
RGB
0.665
0.497
0.174
0.369
0.305
0.276
0.471
0.771
0.565
0.395
0.449
PatchCore (DINO-v2) [ 28 ]
RGB
0.177
0.715
0.325
0.660
0.614
0.322
0.695
0.328
0.626
0.491
0.495
MVAD [ 18 ]
RGB
0.717
0.508
0.196
0.489
0.388
0.210
0.454
0.770
0.523
0.399
0.465
CFM (DINO-v2 + FPFH) [ 11 ]
RGB +PC
0.742
0.395
0.278
0.387
0.286
0.185
0.544
0.744
0.524
0.405
0.449
AST [ 29 ]
RGB + Depth
0.431
0.293
0.202
0.242
0.214
0.130
0.130
0.163
0.041
0.021
0.187
SGANet (Ours)
RGB + Depth
0.490
0.750
0.520
0.665
0.591
0.310
0.645
0.470
0.583
0.527
0.555
Table 6: AUPRO@30% result on the Eyecandies dataset. Best results in bold, runner-ups underlined .
I-AUROC
V-AUPRO@1%
k=2
0.872
0.614
k=4
0.873
0.612
k=6
0.876
0.613
k=8
0.887
0.614
k=10
0.885
0.613
Table 7: Parameter analysis of hyperparameter k on the SiM3D dataset.
I-AUROC
V-AUPRO@1%
N=1
0.871
0.612
N=2
0.887
0.614
N=3
0.881
0.614
N=4
0.871
0.612
Table 8: Parameter analysis of the number of neighboring views N on the SiM3D dataset.
LSSPA
✔
✔
✔
✔
Lview
✗
✔
✗
✔
Ldiff
✗
✗
✔
✔
SiM3D
I-AUROC
0.870
0.874
0.874
0.887
V-AUPRO@1%
0.612
0.612
0.613
0.614
Eyecandies
I-AUROC
0.732
0.740
0.737
0.743
P-AUROC
0.845
0.851
0.853
0.854
Table 9: Ablation study of loss components on the SiM3D and Eyecandies datasets.
Figure 5: Qualitative comparison on the Eyecandies dataset [ 5 ] . Visualizations include RGB images (RGB), depth maps (Depth), binary masks (Binary Mask), and the corresponding anomaly score maps predicted by representative methods, including single-view baselines (PatchCore, AST), the multi-view method (MVAD), and the proposed SGANet framework.
Figure 6: I-AUROC, P-AUROC and AUPRO@30% results on the Eyecandies dataset using RGB modality, depth modality, and the proposed SGANet framework.
National Taiwan University Taipei, Taiwan · National Taiwan Normal University Taipei, Taiwan · Mitsubishi Electric Research Laboratories (MERL) Cambridge, MA, USA +3
School of Computer Science & Software Engineering, Shenzhen University, China, 518060 · School of Computing and Information Technology, Great Bay University, Dongguan, 523000,China