Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Training

Shangbo Yuan; Jie Xu; Xiaofeng Zhu; Na Zhao

cs.CVAug 20, 2026

Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Training

Authors: Shangbo Yuan, Jie Xu, Xiaofeng Zhu, Na Zhao

Organizations: School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore · School of Computer Science and Technology, Hainan University, Haikou, China

Abstract

Recently, open-vocabulary 3D object detection (3D-OVD) has gained increasing attention for its ability to detect unseen objects in 3D scenes. Existing approaches typically adopt a two-stage pipeline that first discovers novel objects using foundation models and then trains a 3D-OVD model based on these discovered objects. Although effective, this pipeline often suffers from inaccurate localization and mismatched classification during the discovery stage, which subsequently limits the performance of the model training stage. To address these limitations, we advocate for improving both the reliability of novel object discovery and the robustness of model training, and propose an innovative framework. Specifically, for reliable discovery, our co-distillation strategy distills high-quality novel objects by applying Hungarian matching over a comprehensive score that incorporates geometric consistency, structural objectness, and semantic certainty. To enhance robust model training, we further propose a dual-guidance learning scheme, incorporating a scene-awareness-guided uncertainty regularization for the regression head and an LLM-guided hierarchical alignment for the classification head, effectively mitigating the negative effects of imprecise 3D bounding boxes and semantic ambiguity. Extensive experiments on SUN RGB-D and ScanNetV2 demonstrate that our method achieves significant performance gains over state-of-the-art approaches. Code is available at https://github.com/shangboyuan/Co-3DGT

Figures & tables

Figure 1: Comparison between our Co-3DGT and two existing 3D-OVD paradigms. In the Novel Object Discovery stage: (a) 2D-Detection-based strategy [ 24 , 39 ] back-projects 2D-OVD results into 3D space, often leading to imprecise 3D bounding boxes; (b) 3D-Proposal-based methods [ 3 , 4 ] project 3D proposals from class-agnostic 3D detectors onto 2D plane and crop images for open-vocabulary labeling, which may introduce semantic ambiguity; (c) We propose Co-Distillation, jointly considering consider geometric constraints alongside 2D semantic and 3D localization inofrmation for more reliable novel objects. In the Training stage: (d) Previous methods typically employ equal supervision for both base and novel objects; while (e) We propose Dual Guidance Robust Training, integrating uncertainty regularization for the regression head and leveraging an LLM to achieve hierarchical semantic alignment.

Figure 2: Framework Overview of our Co-3DGT. In Co-Distillation Discovery stage: (a) a class-agnostic 3D detector generates 3D boxes, while a 2D open-vocabulary detector produces class-aware 2D boxes which are back-projected and to co-distill reliable class-aware 3D boxes. In Dual Guidance Robust Training stage: (b) Scene-Awareness-Guided Uncertainty Regularization exploits the scene-aware uncertainty from base classes to regularize the uncertainties for novel classes, dynamically weighting the 3D bbox regression. (c) LLM-Guided Hierarchical Alignment leverage LLM to form super-categories that increase structural semantic supervision beyond original categories, promoting the model to achieve robust classification.

Method	SUN RGB-D						ScanNetV2
Method	$\textup{AP}^{\text{novel}}_{25}$	$\textup{AP}^{\text{base}}_{25}$	$\textup{AP}^{\text{mean}}_{25}$	$\textup{AR}^{\text{novel}}_{25}$	$\textup{AR}^{\text{base}}_{25}$	$\textup{AR}^{\text{mean}}_{25}$	$\textup{AP}^{\text{novel}}_{25}$	$\textup{AP}^{\text{base}}_{25}$	$\textup{AP}^{\text{mean}}_{25}$	$\textup{AR}^{\text{novel}}_{25}$	$\textup{AR}^{\text{base}}_{25}$	$\textup{AR}^{\text{mean}}_{25}$
Det-PointCLIP [ 43 ]	0.09	5.04	1.17	21.98	65.03	31.33	0.13	2.38	0.50	33.38	54.88	36.96
Det-PointCLIPv2 [ 42 ]	0.12	4.82	1.14	21.33	63.74	30.55	0.13	1.75	0.40	32.60	54.52	36.25
Det-CLIP 2 [ 46 ]	0.88	22.74	5.63	22.21	65.04	31.52	0.14	1.76	0.40	34.26	56.22	37.92
3D-CLIP [ 28 ]	3.61	30.56	9.47	21.47	63.74	30.66	3.74	14.14	5.47	32.15	54.15	35.81
CoDA [ 3 ]	6.71	38.72	13.66	33.66	66.42	40.78	6.54	21.57	9.04	43.36	61.00	46.30

Table 1: Comparison of

\textup{AP}_{25}

(%) and

\textup{AR}_{25}

(%) for open-vocabulary 3D object detection on SUN RGB-D and ScanNetV2 datasets.

Method	Mean	toilet	bed	chair	sofa	dresser	table	cabinet	bookshelf	pillow	sink
OV-3DET [ 24 ]	18.02	57.29	42.26	27.06	31.50	8.21	14.17	2.98	5.56	23.00	31.60
CoDA [ 3 ]	19.32	68.09	44.04	28.72	44.57	3.41	20.23	5.32	0.03	27.95	45.26
CoDAv2 [ 4 ]	22.72	77.24	43.96	15.05	53.27	11.37	13.96	1.42	0.11	34.42	44.38
OV-Uni3DETR [ 39 ]	25.33	86.05	50.49	28.11	31.51	18.22	24.03	6.58	12.17	29.62	54.63
Co-3DGT (ours)	36.15	92.75	70.42	81.27	60.64	12.50	39.15	15.92	12.56	37.58	60.59
Method		bathtub	refrigerator	desk	nightstand	counter	door	curtain	box	lamp	bag

Table 2: Detailed comparison of per-category

\textup{AP}_{25}

(%) for open-vocabulary 3D object detection on ScanNetV2 dataset across 20 unseen categories.

Method	Novel Object Discovery				Training Result
	SUN RGB-D		ScanNetV2		SUN RGB-D		ScanNetV2
	$\textup{AP}^{\text{novel,disc}}_{25}$	$\textup{AR}^{\text{novel,disc}}_{25}$	$\textup{AP}^{\text{novel,disc}}_{25}$	$\textup{AR}^{\text{novel,disc}}_{25}$	$\textup{AP}^{\text{novel,det}}_{25}$	$\textup{AR}^{\text{novel,det}}_{25}$	$\textup{AP}^{\text{novel,det}}_{25}$	$\textup{AR}^{\text{novel,det}}_{25}$
2D-Detection-based	6.21	16.10	3.93	23.44	9.59	52.99	11.92	57.85
3D-Proposal-based	8.49	12.02	4.57	20.28	9.88	52.15	12.71	58.38
Co-distil(scratch)	10.45( +1.96 )	19.72( +7.70 )	9.35( +4.76 )	28.46( +8.18 )	11.76( +1.88 )	54.26( +2.11 )	16.68( +3.97 )	63.70( +5.32 )
Co-distil(CuTR)	10.97( +2.48 )	24.33( +12.31 )	10.46( +5.89 )	32.29( +12.01 )	12.69( +2.81 )	55.41( +3.26 )	18.74( +6.03 )	64.19( +5.81 )

Table 3: Ablation study of the Co-Distillation module on SUN RGB-D and ScanNetV2. Models are evaluated using

\textup{AP}^{\text{novel,disc}}_{25}

(%) and

\textup{AR}^{\text{novel,disc}}_{25}

(%) to measure pseudo-label quality on the training samples, and

\textup{AP}^{\text{novel,det}}_{25}

(%) and

\textup{AR}^{\text{novel,det}}_{25}

(%) to assess final detection performance.

Table 6

Panel of Figure 3 — Figure 3: Qualitative examples on SUN RGB-D (the first row) and ScanNetV2 (the second row) to visualize the 3D-OVD results by our method. The red and orange colored bboxes correspond to different base-class objects, while blue and green colored bboxes represent different novel-class objects.

3D Object Detection

3D object detection identifies objects and estimates their locations, sizes, and orientations from images, point clouds, radar, or fused sensors.

See all 181 papers

Open-vocabulary 3D object detection with promptable segmentation
Sep 16, 2026Ömer Faruk Deniz, Mustafa Taha KoçyiğitLidar-Based 3D Object Detection Multi-View 3D Object Detection
Institute of Data Science and Artificial Intelligence, Boğaziçi University, South Campus, Bebek, 34342, Istanbul, Türkiye
Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection
Mar 23, 2026Youbin Kim, Jinho Park, Hogun Park +13D Object Detection Multi-View 3D Object Detection
Department of Artificial Intelligence, Sungkyunkwan University · Department of Artificial Intelligence, Yonsei University
DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection
Apr 3, 2026Siheng Wang, Yanshu Li, Bohan Hu +12Open-Vocabulary Object Detection Detection Transformer
Jiangsu University · MBZUAI · Brown University +5

Sep 16, 2026J/K move · Enter open · S save

Open-vocabulary 3D object detection with promptable segmentation

Ömer Faruk Deniz, Mustafa Taha Koçyiğit

Institute of Data Science and Artificial Intelligence, Boğaziçi University, South Campus, Bebek, 34342, Istanbul, Türkiye

Lidar-Based 3D Object Detection Multi-View 3D Object Detection 3D Object Detection

Open paper AskPDF arXiv

Three-dimensional object detection for autonomous driving is dominated by detectors trained on large corpora of human-annotated 3D boxes. Such a detector learns a fixed category list, and everything outside it is invisible. This paper asks whether the task can be solved training-free and open-vocabulary. A promptable segmentation model (SAM3), queried with class names as text prompts, supplies instance masks in the vehicle's six surround-view cameras, and the masks are turned into metric 3D boxes using the geometry of the scene. The core is a controlled three-stage comparison on nuScenes in which 2D detection is held fixed and only the source of 3D geometry changes. Geometry predicted from images alone reaches 0.183 mean average precision (mAP) under the official protocol; fitting boxes from raw LiDAR points inside the same masks with training-free rules reaches 0.298 mAP / 0.348 nuScenes detection score (NDS) at zero labeling cost; borrowing supervised box geometry at inference time lifts the same detections to 0.413 mAP / 0.555 NDS, which locates the pipeline's largest deficit in measurement precision rather than 2D detection, while class confusion and confidence calibration survive that substitution. Reversing the direction, a three-state camera-witness rule built from the same masks improves a supervised LiDAR-only detector from 0.596 to 0.630 mAP, roughly half the gain of fully supervised camera fusion, with no training. A coverage analysis shows that SAM3 finds 84% of in-range objects with a correctly named mask; the classes that fail in the official metric are misnamed or geometrically unforgiving, not unseen.

Abstract

3D Object Detection

Explore similar work

Open-vocabulary 3D object detection with promptable segmentation

Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection

DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection