cs.CVNov 13, 2026

See and Detect: Image-Driven Open-Vocabulary Multi-View 3D Object Detection

Authors: Yating Xu, Na Zhao, Chenhan Jiang, Qingwen Zhang, Jiankang Deng, Jifei Song

Organizations: MVP Lab · Singapore University of Technology and Design · Hong Kong University of Science and Technology · KTH Royal Institute of Technology

Abstract

Recent open-vocabulary multi-view indoor 3D detection methods typically follow a point-cloud-driven pipeline, where pseudo 3D boxes are generated from reconstructed or estimated geometry and semantic labels are inferred from projected regions. However, such a design makes box generation dependent on the completeness of point clouds and restricts semantic learning to local appearance, causing inferior 3D object localization and open-vocabulary classification. In this paper, we present See and Detect, an image-driven framework that uses multi-view images as the primary evidence for 3D box and semantic label generation. For box generation, we first discover object candidates from individual images, enabling objects missing in point clouds to be recovered from visual evidence. However, single-view predictions are often noisy and redundant. To address this, we propose multi-view consensus box consolidation to fuse spatially consistent predictions across views, and box quality assessment to reduce the impact of boxes with partial object coverage. For semantic label generation, we introduce context-aware semantic learning, where semantic labels are queried from full image context, enabling more accurate semantic prediction. Extensive experiments on ScanNet and ARKitScenes demonstrate that our method achieves state-of-the-art open-vocabulary multi-view 3D detection performance while running over 500×\times faster than the previous state-of-the-art method.

Figures & tables

Explore similar work

CardsList
  1. Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection

    Mar 23, 2026Youbin Kim, Jinho Park, Hogun Park +13D Object DetectionMulti-View 3D Object Detection

  2. Open-vocabulary 3D object detection with promptable segmentation

    Sep 16, 2026Ömer Faruk Deniz, Mustafa Taha KoçyiğitLidar-Based 3D Object DetectionMulti-View 3D Object Detection

  3. OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection

    Aug 17, 2026Mariia Gladkova, Neehar Peri, Ishan Khatri +2Open-Vocabulary Object DetectionMonocular 3D Object Detection