See and Detect: Image-Driven Open-Vocabulary Multi-View 3D Object Detection
Authors: Yating Xu, Na Zhao, Chenhan Jiang, Qingwen Zhang, Jiankang Deng, Jifei Song
Organizations: MVP Lab · Singapore University of Technology and Design · Hong Kong University of Science and Technology · KTH Royal Institute of Technology
Recent open-vocabulary multi-view indoor 3D detection methods typically follow a point-cloud-driven pipeline, where pseudo 3D boxes are generated from reconstructed or estimated geometry and semantic labels are inferred from projected regions. However, such a design makes box generation dependent on the completeness of point clouds and restricts semantic learning to local appearance, causing inferior 3D object localization and open-vocabulary classification. In this paper, we present See and Detect, an image-driven framework that uses multi-view images as the primary evidence for 3D box and semantic label generation. For box generation, we first discover object candidates from individual images, enabling objects missing in point clouds to be recovered from visual evidence. However, single-view predictions are often noisy and redundant. To address this, we propose multi-view consensus box consolidation to fuse spatially consistent predictions across views, and box quality assessment to reduce the impact of boxes with partial object coverage. For semantic label generation, we introduce context-aware semantic learning, where semantic labels are queried from full image context, enabling more accurate semantic prediction. Extensive experiments on ScanNet and ARKitScenes demonstrate that our method achieves state-of-the-art open-vocabulary multi-view 3D detection performance while running over 500× faster than the previous state-of-the-art method.
Figures & tables
Figure 1: Comparison of training pipelines for open-vocabulary multi-view 3D object detection. (a) Existing methods [ 12 , 7 ] propose point-cloud-driven approach for 3D box and semantic label generation, which leads to (i) missing object discovery and (ii) incorrect semantic label generation. (b) See and Detect (Ours) proposes a multi-view image-driven approach: objects are discovered through cross-view visual evidence, and semantic labels are generated by considering image contexts.
Figure 2: Our framework. During training, it is optimized by two main objectives: (1) a Weighted Localization Loss Lwloc , where generated pseudo 3D boxes are weighted by the quality scores Q (Eqn. 3 ) based on their geometric consistency (details shown in Fig. 3 ); and (2) a Text-3D Contrastive Loss Ltext-3D , which aligns 3D features f3D with semantic labels inferred from MLLM by jointly considering local appearance and global context. We apply Semantic-aware Negative Class Selection to generate informative negative classes.
Figure 3: Image-Driven Pseudo Box Generation. We perform single-view 3D object detection for each image using Cubify Anything. A scene-level undirected graph is constructed where edges represent spatial overlap (Eqn. 1 ). Multi-View Consensus Box Consolidation is applied to fuse consistent predictions and suppress spurious single-view detections. Finally, each box is assigned a quality score Q through Box Quality Assessment based on cross-view visibility I(vi) and SAM-based object coverage check S(vi) , which weights the localization loss during training.
Figure 4: Box quality assessment. (a) shows a pseudo box with incomplete object coverage due to detection from local image views. (b) shows a pseudo box with complete object coverage. We project the boxes to the nearby view vi for box quality assessment. The quality score Q can correctly reflect box quality.
Figure 5: Qualitative results of open-vocabulary multi-view 3D object detection on ScanNet60.
Method
Test Input
ScanNet60
ScanNet200
Inference Time(s)
mAP 25
mAP 50
mAP 25
mAP 50
CoDA [ 3 ]
pcd
9.0
2.2
4.4
0.9
1.8
OV-3DET [ 14 ]
pcd
8.1
1.4
2.7
0.5
2.6
OV-Uni3DETR [ 23 ]
pcd
19.4
-
-
-
-
OpenM3D [ 7 ]
imgs
-
-
4.2
-
0.4
Zoo3D [ 12 ]
imgs
23.9
10.8
16.5
6.3
378.3
Table 1: Open-vocabulary multi-view 3D object detection on ScanNet60 and ScanNet200 benchmarks. “Test Input” denotes the input at inference stage. Inference time is measured in seconds per scene. We report the best results of Zoo3D from their paper under the multi-view setting. Best results are marked bold.
Method
mAP 25
mAP 50
mAR 25
mAR 50
OpenM3D [ 7 ]
26.9
-
51.2
-
Zoo3D [ 12 ]
36.1
13.9
-
-
Ours
52.7
25.3
72.2
38.9
Table 2: Class-agnostic 3D object detection on ScanNet200. Best results are marked bold.
Method
ScanNet200
ARKitScenes
R 25
P 25
R 50
P 50
R 25
P 25
R 50
P 50
OpenM3D [ 7 ]
58.3
32.1
32.9
18.1
51.9
5.9
13.7
1.6
Zoo3D [ 12 ]
58.7
35.9
45.6
27.9
52.0
8.4
28.2
4.3
Ours
71.9
50.5
59.6
41.9
61.3
45.6
58.0
43.2
Table 3: 3D pseudo box evaluation on ScanNet200 and ARKitScenes dataset. We report recall (R) and precision (P) at IoU thresholds of 0.25 and 0.50. Best results are marked bold.
Table 9
Method
ScanNet60
ScanNet200
mAP 25
mAP 50
mAP 25
mAP 50
IMG Crop
20.6
11.4
7.8
4.4
IMG Crop+Text
21.0
11.7
9.9
4.5
Ours
28.5
15.1
16.4
7.7
Table 6: Ablation study on context-aware semantic learning. “IMG Crop“ and “IMG Crop+Text” are two semantic learning methods derived from image crops.
Range
ScanNet60
ScanNet200
mAP 25
mAP 50
mAP 25
mAP 50
[2,1203]
27.0
13.4
15.0
6.8
[150,400]
28.1
14.0
15.5
7.0
[400,800]
27.1
13.0
15.0
6.5
Ours
28.5
15.1
16.4
7.7
Table 7: Ablation study on the range [k1,k2] for the semantic-aware negative class selection on ScanNet60 and ScanNet200.
Open-vocabulary 3D object detection aims to localize and recognize objects beyond a fixed training taxonomy. In multi-view RGB settings, recent approaches often decouple geometry-based instance construction from semantic labeling, generating class-agnostic fragments and assigning open-vocabulary categories post hoc. While flexible, such decoupling leaves instance construction governed primarily by geometric consistency, without semantic constraints during merging. When geometric evidence is view-dependent and incomplete, this geometry-only merging can lead to irreversible association errors, including over-merging of distinct objects or fragmentation of a single instance. We propose Group3D, a multi-view open-vocabulary 3D detection framework that integrates semantic constraints directly into the instance construction process. Group3D maintains a scene-adaptive vocabulary derived from a multimodal large language model (MLLM) and organizes it into semantic compatibility groups that encode plausible cross-view category equivalence. These groups act as merge-time constraints: 3D fragments are associated only when they satisfy both semantic compatibility and geometric consistency. This semantically gated merging mitigates geometry-driven over-merging while absorbing multi-view category variability. Group3D supports both pose-known and pose-free settings, relying only on RGB observations. Experiments on ScanNet and ARKitScenes demonstrate that Group3D achieves state-of-the-art performance in multi-view open-vocabulary 3D detection, while exhibiting strong generalization in zero-shot scenarios. The project page is available at https://ubin108.github.io/Group3D/.
Youbin Kim, Jinho Park, Hogun Park +1
Department of Artificial Intelligence, Sungkyunkwan University · Department of Artificial Intelligence, Yonsei University
Three-dimensional object detection for autonomous driving is dominated by detectors trained on large corpora of human-annotated 3D boxes. Such a detector learns a fixed category list, and everything outside it is invisible. This paper asks whether the task can be solved training-free and open-vocabulary. A promptable segmentation model (SAM3), queried with class names as text prompts, supplies instance masks in the vehicle's six surround-view cameras, and the masks are turned into metric 3D boxes using the geometry of the scene. The core is a controlled three-stage comparison on nuScenes in which 2D detection is held fixed and only the source of 3D geometry changes. Geometry predicted from images alone reaches 0.183 mean average precision (mAP) under the official protocol; fitting boxes from raw LiDAR points inside the same masks with training-free rules reaches 0.298 mAP / 0.348 nuScenes detection score (NDS) at zero labeling cost; borrowing supervised box geometry at inference time lifts the same detections to 0.413 mAP / 0.555 NDS, which locates the pipeline's largest deficit in measurement precision rather than 2D detection, while class confusion and confidence calibration survive that substitution. Reversing the direction, a three-state camera-witness rule built from the same masks improves a supervised LiDAR-only detector from 0.596 to 0.630 mAP, roughly half the gain of fully supervised camera fusion, with no training. A coverage analysis shows that SAM3 finds 84% of in-range objects with a correctly named mask; the classes that fail in the official metric are misnamed or geometrically unforgiving, not unseen.
Ömer Faruk Deniz, Mustafa Taha Koçyiğit
Institute of Data Science and Artificial Intelligence, Boğaziçi University, South Campus, Bebek, 34342, Istanbul, Türkiye
Open-vocabulary monocular 3D detectors report strong in-domain performance, but each evaluates under a different protocol, several rely on per-image category oracles unavailable at deployment, and all collapse geometry and semantics into a single AP metric. To address this, we introduce OV3D-Bench, a diagnostic benchmark that compares open-vocabulary monocular 3D detectors under deployment-realistic conditions across seven indoor and outdoor datasets. Our benchmark replaces the per-image class name oracle with test-time dataset-level class name prompts, and decouples detection accuracy along three axes: localization, semantic robustness, and cross-domain transfer. We evaluate seven representative detectors and find that (i) they localize objects well yet often mislabel a correctly localized box as a semantically adjacent category; (ii) accuracy is highly sensitive to prompt phrasing (e.g. WildDet3D's performance collapses from 18.6 to 5.4 AP when prompted with "a detailed high-resolution photo of a car" rather than "car"); and (iii) the widely adopted target-aware protocol hides these errors (e.g. inflating DetAny3D's AP by 1.9 × on ScanNet). Lastly, we demonstrate that simply remapping a frozen closed-vocabulary detector's predictions using a contrastive vision-language encoder such as SigLIPv2 performs competitively against recent purpose-built open-vocabulary methods. This indicates that geometric localization is more mature, while open-vocabulary semantics remains the primary bottleneck.
Mariia Gladkova, Neehar Peri, Ishan Khatri +2
Technical University of Munich · Carnegie Mellon University · StackAV