See and Detect: Image-Driven Open-Vocabulary Multi-View 3D Object Detection
Authors: Yating Xu, Na Zhao, Chenhan Jiang, Qingwen Zhang, Jiankang Deng, Jifei Song
Organizations: MVP Lab · Singapore University of Technology and Design · Hong Kong University of Science and Technology · KTH Royal Institute of Technology
Recent open-vocabulary multi-view indoor 3D detection methods typically follow a point-cloud-driven pipeline, where pseudo 3D boxes are generated from reconstructed or estimated geometry and semantic labels are inferred from projected regions. However, such a design makes box generation dependent on the completeness of point clouds and restricts semantic learning to local appearance, causing inferior 3D object localization and open-vocabulary classification. In this paper, we present See and Detect, an image-driven framework that uses multi-view images as the primary evidence for 3D box and semantic label generation. For box generation, we first discover object candidates from individual images, enabling objects missing in point clouds to be recovered from visual evidence. However, single-view predictions are often noisy and redundant. To address this, we propose multi-view consensus box consolidation to fuse spatially consistent predictions across views, and box quality assessment to reduce the impact of boxes with partial object coverage. For semantic label generation, we introduce context-aware semantic learning, where semantic labels are queried from full image context, enabling more accurate semantic prediction. Extensive experiments on ScanNet and ARKitScenes demonstrate that our method achieves state-of-the-art open-vocabulary multi-view 3D detection performance while running over 500× faster than the previous state-of-the-art method.
Figures & tables
Figure 1: Comparison of training pipelines for open-vocabulary multi-view 3D object detection. (a) Existing methods [ 12 , 7 ] propose point-cloud-driven approach for 3D box and semantic label generation, which leads to (i) missing object discovery and (ii) incorrect semantic label generation. (b) See and Detect (Ours) proposes a multi-view image-driven approach: objects are discovered through cross-view visual evidence, and semantic labels are generated by considering image contexts.
Figure 2: Our framework. During training, it is optimized by two main objectives: (1) a Weighted Localization Loss Lwloc , where generated pseudo 3D boxes are weighted by the quality scores Q (Eqn. 3 ) based on their geometric consistency (details shown in Fig. 3 ); and (2) a Text-3D Contrastive Loss Ltext-3D , which aligns 3D features f3D with semantic labels inferred from MLLM by jointly considering local appearance and global context. We apply Semantic-aware Negative Class Selection to generate informative negative classes.
Figure 3: Image-Driven Pseudo Box Generation. We perform single-view 3D object detection for each image using Cubify Anything. A scene-level undirected graph is constructed where edges represent spatial overlap (Eqn. 1 ). Multi-View Consensus Box Consolidation is applied to fuse consistent predictions and suppress spurious single-view detections. Finally, each box is assigned a quality score Q through Box Quality Assessment based on cross-view visibility I(vi) and SAM-based object coverage check S(vi) , which weights the localization loss during training.
Figure 4: Box quality assessment. (a) shows a pseudo box with incomplete object coverage due to detection from local image views. (b) shows a pseudo box with complete object coverage. We project the boxes to the nearby view vi for box quality assessment. The quality score Q can correctly reflect box quality.
Figure 5: Qualitative results of open-vocabulary multi-view 3D object detection on ScanNet60.
Method
Test Input
ScanNet60
ScanNet200
Inference Time(s)
mAP 25
mAP 50
mAP 25
mAP 50
CoDA [ 3 ]
pcd
9.0
2.2
4.4
0.9
1.8
OV-3DET [ 14 ]
pcd
8.1
1.4
2.7
0.5
2.6
OV-Uni3DETR [ 23 ]
pcd
19.4
-
-
-
-
OpenM3D [ 7 ]
imgs
-
-
4.2
-
0.4
Zoo3D [ 12 ]
imgs
23.9
10.8
16.5
6.3
378.3
Table 1: Open-vocabulary multi-view 3D object detection on ScanNet60 and ScanNet200 benchmarks. “Test Input” denotes the input at inference stage. Inference time is measured in seconds per scene. We report the best results of Zoo3D from their paper under the multi-view setting. Best results are marked bold.
Method
mAP 25
mAP 50
mAR 25
mAR 50
OpenM3D [ 7 ]
26.9
-
51.2
-
Zoo3D [ 12 ]
36.1
13.9
-
-
Ours
52.7
25.3
72.2
38.9
Table 2: Class-agnostic 3D object detection on ScanNet200. Best results are marked bold.
Method
ScanNet200
ARKitScenes
R 25
P 25
R 50
P 50
R 25
P 25
R 50
P 50
OpenM3D [ 7 ]
58.3
32.1
32.9
18.1
51.9
5.9
13.7
1.6
Zoo3D [ 12 ]
58.7
35.9
45.6
27.9
52.0
8.4
28.2
4.3
Ours
71.9
50.5
59.6
41.9
61.3
45.6
58.0
43.2
Table 3: 3D pseudo box evaluation on ScanNet200 and ARKitScenes dataset. We report recall (R) and precision (P) at IoU thresholds of 0.25 and 0.50. Best results are marked bold.
Table 9
Method
ScanNet60
ScanNet200
mAP 25
mAP 50
mAP 25
mAP 50
IMG Crop
20.6
11.4
7.8
4.4
IMG Crop+Text
21.0
11.7
9.9
4.5
Ours
28.5
15.1
16.4
7.7
Table 6: Ablation study on context-aware semantic learning. “IMG Crop“ and “IMG Crop+Text” are two semantic learning methods derived from image crops.
Range
ScanNet60
ScanNet200
mAP 25
mAP 50
mAP 25
mAP 50
[2,1203]
27.0
13.4
15.0
6.8
[150,400]
28.1
14.0
15.5
7.0
[400,800]
27.1
13.0
15.0
6.5
Ours
28.5
15.1
16.4
7.7
Table 7: Ablation study on the range [k1,k2] for the semantic-aware negative class selection on ScanNet60 and ScanNet200.