Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Training

Shangbo Yuan; Jie Xu; Xiaofeng Zhu; Na Zhao

cs.CVAug 20, 2026

Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Training

Authors: Shangbo Yuan, Jie Xu, Xiaofeng Zhu, Na Zhao

Organizations: School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore · School of Computer Science and Technology, Hainan University, Haikou, China

Abstract

Recently, open-vocabulary 3D object detection (3D-OVD) has gained increasing attention for its ability to detect unseen objects in 3D scenes. Existing approaches typically adopt a two-stage pipeline that first discovers novel objects using foundation models and then trains a 3D-OVD model based on these discovered objects. Although effective, this pipeline often suffers from inaccurate localization and mismatched classification during the discovery stage, which subsequently limits the performance of the model training stage. To address these limitations, we advocate for improving both the reliability of novel object discovery and the robustness of model training, and propose an innovative framework. Specifically, for reliable discovery, our co-distillation strategy distills high-quality novel objects by applying Hungarian matching over a comprehensive score that incorporates geometric consistency, structural objectness, and semantic certainty. To enhance robust model training, we further propose a dual-guidance learning scheme, incorporating a scene-awareness-guided uncertainty regularization for the regression head and an LLM-guided hierarchical alignment for the classification head, effectively mitigating the negative effects of imprecise 3D bounding boxes and semantic ambiguity. Extensive experiments on SUN RGB-D and ScanNetV2 demonstrate that our method achieves significant performance gains over state-of-the-art approaches. Code is available at https://github.com/shangboyuan/Co-3DGT

Figures & tables

Figure 1: Comparison between our Co-3DGT and two existing 3D-OVD paradigms. In the Novel Object Discovery stage: (a) 2D-Detection-based strategy [ 24 , 39 ] back-projects 2D-OVD results into 3D space, often leading to imprecise 3D bounding boxes; (b) 3D-Proposal-based methods [ 3 , 4 ] project 3D proposals from class-agnostic 3D detectors onto 2D plane and crop images for open-vocabulary labeling, which may introduce semantic ambiguity; (c) We propose Co-Distillation, jointly considering consider geometric constraints alongside 2D semantic and 3D localization inofrmation for more reliable novel objects. In the Training stage: (d) Previous methods typically employ equal supervision for both base and novel objects; while (e) We propose Dual Guidance Robust Training, integrating uncertainty regularization for the regression head and leveraging an LLM to achieve hierarchical semantic alignment.

Figure 2: Framework Overview of our Co-3DGT. In Co-Distillation Discovery stage: (a) a class-agnostic 3D detector generates 3D boxes, while a 2D open-vocabulary detector produces class-aware 2D boxes which are back-projected and to co-distill reliable class-aware 3D boxes. In Dual Guidance Robust Training stage: (b) Scene-Awareness-Guided Uncertainty Regularization exploits the scene-aware uncertainty from base classes to regularize the uncertainties for novel classes, dynamically weighting the 3D bbox regression. (c) LLM-Guided Hierarchical Alignment leverage LLM to form super-categories that increase structural semantic supervision beyond original categories, promoting the model to achieve robust classification.

Method	SUN RGB-D						ScanNetV2
Method	$\textup{AP}^{\text{novel}}_{25}$	$\textup{AP}^{\text{base}}_{25}$	$\textup{AP}^{\text{mean}}_{25}$	$\textup{AR}^{\text{novel}}_{25}$	$\textup{AR}^{\text{base}}_{25}$	$\textup{AR}^{\text{mean}}_{25}$	$\textup{AP}^{\text{novel}}_{25}$	$\textup{AP}^{\text{base}}_{25}$	$\textup{AP}^{\text{mean}}_{25}$	$\textup{AR}^{\text{novel}}_{25}$	$\textup{AR}^{\text{base}}_{25}$	$\textup{AR}^{\text{mean}}_{25}$
Det-PointCLIP [ 43 ]	0.09	5.04	1.17	21.98	65.03	31.33	0.13	2.38	0.50	33.38	54.88	36.96
Det-PointCLIPv2 [ 42 ]	0.12	4.82	1.14	21.33	63.74	30.55	0.13	1.75	0.40	32.60	54.52	36.25
Det-CLIP 2 [ 46 ]	0.88	22.74	5.63	22.21	65.04	31.52	0.14	1.76	0.40	34.26	56.22	37.92
3D-CLIP [ 28 ]	3.61	30.56	9.47	21.47	63.74	30.66	3.74	14.14	5.47	32.15	54.15	35.81
CoDA [ 3 ]	6.71	38.72	13.66	33.66	66.42	40.78	6.54	21.57	9.04	43.36	61.00	46.30

Table 1: Comparison of

\textup{AP}_{25}

(%) and

\textup{AR}_{25}

(%) for open-vocabulary 3D object detection on SUN RGB-D and ScanNetV2 datasets.

Method	Mean	toilet	bed	chair	sofa	dresser	table	cabinet	bookshelf	pillow	sink
OV-3DET [ 24 ]	18.02	57.29	42.26	27.06	31.50	8.21	14.17	2.98	5.56	23.00	31.60
CoDA [ 3 ]	19.32	68.09	44.04	28.72	44.57	3.41	20.23	5.32	0.03	27.95	45.26
CoDAv2 [ 4 ]	22.72	77.24	43.96	15.05	53.27	11.37	13.96	1.42	0.11	34.42	44.38
OV-Uni3DETR [ 39 ]	25.33	86.05	50.49	28.11	31.51	18.22	24.03	6.58	12.17	29.62	54.63
Co-3DGT (ours)	36.15	92.75	70.42	81.27	60.64	12.50	39.15	15.92	12.56	37.58	60.59
Method		bathtub	refrigerator	desk	nightstand	counter	door	curtain	box	lamp	bag

Table 2: Detailed comparison of per-category

\textup{AP}_{25}

(%) for open-vocabulary 3D object detection on ScanNetV2 dataset across 20 unseen categories.

Method	Novel Object Discovery				Training Result
	SUN RGB-D		ScanNetV2		SUN RGB-D		ScanNetV2
	$\textup{AP}^{\text{novel,disc}}_{25}$	$\textup{AR}^{\text{novel,disc}}_{25}$	$\textup{AP}^{\text{novel,disc}}_{25}$	$\textup{AR}^{\text{novel,disc}}_{25}$	$\textup{AP}^{\text{novel,det}}_{25}$	$\textup{AR}^{\text{novel,det}}_{25}$	$\textup{AP}^{\text{novel,det}}_{25}$	$\textup{AR}^{\text{novel,det}}_{25}$
2D-Detection-based	6.21	16.10	3.93	23.44	9.59	52.99	11.92	57.85
3D-Proposal-based	8.49	12.02	4.57	20.28	9.88	52.15	12.71	58.38
Co-distil(scratch)	10.45( +1.96 )	19.72( +7.70 )	9.35( +4.76 )	28.46( +8.18 )	11.76( +1.88 )	54.26( +2.11 )	16.68( +3.97 )	63.70( +5.32 )
Co-distil(CuTR)	10.97( +2.48 )	24.33( +12.31 )	10.46( +5.89 )	32.29( +12.01 )	12.69( +2.81 )	55.41( +3.26 )	18.74( +6.03 )	64.19( +5.81 )

Table 3: Ablation study of the Co-Distillation module on SUN RGB-D and ScanNetV2. Models are evaluated using

\textup{AP}^{\text{novel,disc}}_{25}

(%) and

\textup{AR}^{\text{novel,disc}}_{25}

(%) to measure pseudo-label quality on the training samples, and

\textup{AP}^{\text{novel,det}}_{25}

(%) and

\textup{AR}^{\text{novel,det}}_{25}

(%) to assess final detection performance.

Table 6

Panel of Figure 3 — Figure 3: Qualitative examples on SUN RGB-D (the first row) and ScanNetV2 (the second row) to visualize the 3D-OVD results by our method. The red and orange colored bboxes correspond to different base-class objects, while blue and green colored bboxes represent different novel-class objects.

3D Object Detection

3D object detection identifies objects and estimates their locations, sizes, and orientations from images, point clouds, radar, or fused sensors.

See all 181 papers

Sep 16, 2026

Open-vocabulary 3D object detection with promptable segmentation

Three-dimensional object detection for autonomous driving is dominated by detectors trained on large corpora of human-annotated 3D boxes. Such a detector learns a fixed category list, and everything outside it is invisible. This paper asks whether the task can be solved training-free and open-vocabulary. A promptable segmentation model (SAM3), queried with class names as text prompts, supplies instance masks in the vehicle's six surround-view cameras, and the masks are turned into metric 3D boxes using the geometry of the scene. The core is a controlled three-stage comparison on nuScenes in which 2D detection is held fixed and only the source of 3D geometry changes. Geometry predicted from images alone reaches 0.183 mean average precision (mAP) under the official protocol; fitting boxes from raw LiDAR points inside the same masks with training-free rules reaches 0.298 mAP / 0.348 nuScenes detection score (NDS) at zero labeling cost; borrowing supervised box geometry at inference time lifts the same detections to 0.413 mAP / 0.555 NDS, which locates the pipeline's largest deficit in measurement precision rather than 2D detection, while class confusion and confidence calibration survive that substitution. Reversing the direction, a three-state camera-witness rule built from the same masks improves a supervised LiDAR-only detector from 0.596 to 0.630 mAP, roughly half the gain of fully supervised camera fusion, with no training. A coverage analysis shows that SAM3 finds 84% of in-range objects with a correctly named mask; the classes that fail in the official metric are misnamed or geometrically unforgiving, not unseen.

Ömer Faruk Deniz, Mustafa Taha Koçyiğit

Institute of Data Science and Artificial Intelligence, Boğaziçi University, South Campus, Bebek, 34342, Istanbul, Türkiye

Lidar-Based 3D Object Detection Multi-View 3D Object Detection 3D Object Detection

Ask PDF

Mar 23, 2026

Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection

Open-vocabulary 3D object detection aims to localize and recognize objects beyond a fixed training taxonomy. In multi-view RGB settings, recent approaches often decouple geometry-based instance construction from semantic labeling, generating class-agnostic fragments and assigning open-vocabulary categories post hoc. While flexible, such decoupling leaves instance construction governed primarily by geometric consistency, without semantic constraints during merging. When geometric evidence is view-dependent and incomplete, this geometry-only merging can lead to irreversible association errors, including over-merging of distinct objects or fragmentation of a single instance. We propose Group3D, a multi-view open-vocabulary 3D detection framework that integrates semantic constraints directly into the instance construction process. Group3D maintains a scene-adaptive vocabulary derived from a multimodal large language model (MLLM) and organizes it into semantic compatibility groups that encode plausible cross-view category equivalence. These groups act as merge-time constraints: 3D fragments are associated only when they satisfy both semantic compatibility and geometric consistency. This semantically gated merging mitigates geometry-driven over-merging while absorbing multi-view category variability. Group3D supports both pose-known and pose-free settings, relying only on RGB observations. Experiments on ScanNet and ARKitScenes demonstrate that Group3D achieves state-of-the-art performance in multi-view open-vocabulary 3D detection, while exhibiting strong generalization in zero-shot scenarios. The project page is available at https://ubin108.github.io/Group3D/.

Youbin Kim, Jinho Park, Hogun Park +1

Department of Artificial Intelligence, Sungkyunkwan University · Department of Artificial Intelligence, Yonsei University

3D Object Detection Multi-View 3D Object Detection

Ask PDF

Apr 3, 2026

DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection

Open-vocabulary object detection (OVOD) enables models to recognize objects beyond predefined categories, but existing approaches remain limited in practical deployment. On the one hand, multimodal designs often incur substantial computational overhead due to their reliance on text encoders at inference time. On the other hand, tightly coupled training objectives introduce a trade-off between closed-set detection accuracy and open-world generalization. Thus, we propose Decoupled Cognition DETR (DeCo-DETR), a vision-centric framework that addresses these challenges through a unified decoupling paradigm. Instead of depending on online text encoding, DeCo-DETR constructs a hierarchical semantic prototype space from region-level descriptions generated by pre-trained LVLMs and aligned via CLIP, enabling efficient and reusable semantic representation. Building upon this representation, the framework further disentangles semantic reasoning from localization through a decoupled training strategy, which separates alignment and detection into parallel optimization streams. Extensive experiments on standard OVOD benchmarks demonstrate that DeCo-DETR achieves competitive zero-shot detection performance while significantly improving inference efficiency. These results highlight the effectiveness of decoupling semantic cognition from detection, offering a practical direction for scalable OVOD systems.

Siheng Wang, Yanshu Li, Bohan Hu +12

Jiangsu University · MBZUAI · Brown University +5

Open-Vocabulary Object Detection Detection Transformer Efficient Object Detection

Ask PDF

Abstract

3D Object Detection

Explore similar work

Open-vocabulary 3D object detection with promptable segmentation

Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection

DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection