Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection

Youbin Kim; Jinho Park; Hogun Park; Eunbyung Park

cs.CVMar 23, 2026

Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection

Authors: Youbin Kim, Jinho Park, Hogun Park, Eunbyung Park

Organizations: Department of Artificial Intelligence, Sungkyunkwan University · Department of Artificial Intelligence, Yonsei University

Abstract

Open-vocabulary 3D object detection aims to localize and recognize objects beyond a fixed training taxonomy. In multi-view RGB settings, recent approaches often decouple geometry-based instance construction from semantic labeling, generating class-agnostic fragments and assigning open-vocabulary categories post hoc. While flexible, such decoupling leaves instance construction governed primarily by geometric consistency, without semantic constraints during merging. When geometric evidence is view-dependent and incomplete, this geometry-only merging can lead to irreversible association errors, including over-merging of distinct objects or fragmentation of a single instance. We propose Group3D, a multi-view open-vocabulary 3D detection framework that integrates semantic constraints directly into the instance construction process. Group3D maintains a scene-adaptive vocabulary derived from a multimodal large language model (MLLM) and organizes it into semantic compatibility groups that encode plausible cross-view category equivalence. These groups act as merge-time constraints: 3D fragments are associated only when they satisfy both semantic compatibility and geometric consistency. This semantically gated merging mitigates geometry-driven over-merging while absorbing multi-view category variability. Group3D supports both pose-known and pose-free settings, relying only on RGB observations. Experiments on ScanNet and ARKitScenes demonstrate that Group3D achieves state-of-the-art performance in multi-view open-vocabulary 3D detection, while exhibiting strong generalization in zero-shot scenarios. The project page is available at https://ubin108.github.io/Group3D/.

Figures & tables

Figure 1: Left : Predicted 3D bounding boxes projected onto the input RGB images. Right : Comparison with the baseline under the multi-view, pose-free, zero-shot setting across different vocabulary sizes, where Group3D consistently achieves higher mAP 25 .

Figure 2: The overview of Group3D. Given multi-view RGB images, an MLLM predicts object categories across views, which are aggregated into a Scene Vocabulary Memory . Category-aware masks are lifted into 3D to construct a 3D Fragment Memory . The MLLM then organizes the vocabulary into semantic compatibility groups, which gate fragment merging together with geometric consistency to produce the final open-vocabulary 3D object instances. Finally, multi-view evidence is accumulated to determine the final open-vocabulary category and 3D bounding box for each object instance.

Point cloud-based
Method	Pose-free	Zero-shot	ScanNet20		ScanNet60
			mAP 25	mAP 50	mAP 25	mAP 50
Det-PointCLIPv2 † [ 60 ]	–	✗	–	–	0.2	–
3D-CLIP † [ 33 ]	–	✗	–	–	4.0	–
OV-3DET [ 26 ]	–	✗	18.0	–	–	–
CoDa † [ 3 ]	–	✗	19.3	–	9.0	–

Table 1: Quantitative results on the ScanNet benchmark under two category settings. Methods are grouped by input modality, including point cloud-based methods and multi-view image-based methods. For multi-view methods, results are further reported with and without ground-truth camera poses. † denotes methods that use 3D bounding boxes during training. Zoo3D 0 and Zoo3D 1 denote the zero-shot and self-supervised variants of Zoo3D, respectively.

	Inputs	toilet	bed	chair	sofa	dresser	table	cabinet	bookshelf	pillow	sink
OV-3DET	pc + pi	57.3	42.3	27.1	31.5	8.2	14.2	3.0	5.6	23.0	31.6
CoDA	pc	68.1	44.1	28.7	44.6	3.4	20.2	5.3	0.1	28.0	45.3
OV-Uni3DETR	pc	86.1	50.5	28.1	31.5	18.2	24.0	6.6	12.2	29.6	54.6
Zoo3D 1	pc + pi	78.4	54.4	74.4	65.5	33.6	19.1	14.1	32.3	46.1	27.3
Group3D	pi	91.3	80.5	61.1	78.0	55.4	56.2	24.7	41.3	43.7	62.3
(Ours)	ui	89.6	80.2	37.8	73.4	50.5	43.1	16.6	31.7	20.2	39.3

Table 2: Per-class AP 25 comparison on ScanNet20. ‘pc’ indicates the methods leverage the ground-truth point clouds, ‘pi’ denotes the use of posed images, and ‘ui’ means the use of unposed images.

Method	Pose-free	Zero-shot	ScanNet200		ARKitScenes
Method	Pose-free	Zero-shot	mAP 25	mAP 50	mAP 25	mAP 50
OpenM3D [ 11 ]	✗	✗	4.2	–	–	–
Zoo3D 0 [ 17 ]	✗	✓	14.3	6.2	–	–
Zoo3D 1 [ 17 ]	✗	✗	16.5	6.3	–	–
Ours	✗	✓	17.9	8.7	20.5	5.9
Zoo3D 0 [ 17 ]	✓	✓	8.3	2.9	13.0	2.6

Table 3: Quantitative results on ScanNet200 [ 35 ] and ARKitScenes [ 2 ] under the multi-view setting.

Figure 3: Qualitative results on ScanNet20 [ 26 ] under pose-known and pose-free settings.

Component	Method	mAP 25	mAP 50
Reconstruction	Depth Anything 3 [ 22 ]	41.2	18.5
Reconstruction	VGGT [ 45 ]	40.0	18.7
MLLM	GPT 5.1 [ 41 ]	41.2	18.5
MLLM	Qwen3-VL-8B [ 1 ]	38.5	16.9
Segmentation	SAM 3 [ 4 ]	41.2	18.5
Segmentation	Grounded SAM 2 [ 34 ]	39.7	17.6

Table 4: Ablation study on ScanNet20 with different components. Depth Anything 3 is trained on external datasets, while VGGT is pretrained on ScanNet.

Table 8

Figure 4: Qualitative results on ScanNet200 [ 35 ] under pose-known and pose-free settings.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Table 10

Figure 5: Qualitative results on ScanNet20 [ 26 ] under pose-known and pose-free settings.

Figure 6: Qualitative results on ScanNet200 [ 35 ] under pose-known and pose-free settings.

Figure 7: Qualitative results on ARKitScenes [ 2 ] under pose-known and pose-free settings.

Scene Vocabulary Memory	Semantic Compatibility Grouping
bin, book, box, cabinet, clock, copier, door, light switch, poster, printer, shelf, switch, table, trash can, washer, washing machine, wall, window	copier_printer : [copier, printer] washer_machine : [washer, washing machine] trash_container : [trash can, bin] switch_control : [light switch, switch]
blackboard, book, bookshelf, box, cabinet, chalkboard, chair, couch, desk, recliner, shelf, sofa, table, trash can	seating : [chair, sofa, couch, recliner] board : [chalkboard, blackboard] table_like : [table, desk] storage : [bookshelf, shelf, cabinet]
basket, bottle, box, can, carpet, container, cushion, door, jar, pillow, refrigerator, rug, tray, water bottle, window	small_containers : [box, container, basket, tray] drink_containers : [bottle, water bottle, can, jar] soft_items : [pillow, cushion] ground_cover : [carpet, rug]
bag, backpack, bathtub, blanket, computer, lamp, light, laptop, painting, phone, picture, pillow, shower, sheet, shoe, sneaker, suitcase, tablet, telephone, tissue, toilet paper, towel, trash can	computer_device : [laptop, computer, tablet] phone_device : [phone, telephone] bag_like : [bag, backpack, suitcase] footwear : [shoe, sneaker] wall_art : [painting, picture] bedding_cover : [blanket, sheet] lamp_light : [lamp, light] bath_fixture : [bathtub, shower] bath_tissue : [toilet paper, tissue]

Appendix

Table 9: Examples of semantic compatibility grouping predicted by the MLLM from the scene vocabulary memory.

Method	Pose-free	Zero-shot	ScanNet200
Method	Pose-free	Zero-shot	AP 25	AP 50
Group3D (Ours)	✗	✓	22.9	12.2
Group3D (Ours)	✓	✓	14.8	5.3

Appendix

Table 10: 3D instance segmentation results on ScanNet200 [ 35 ] .

3D Object Detection

3D object detection identifies objects and estimates their locations, sizes, and orientations from images, point clouds, radar, or fused sensors.

See all 181 papers

Aug 20, 2026

Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Training

Recently, open-vocabulary 3D object detection (3D-OVD) has gained increasing attention for its ability to detect unseen objects in 3D scenes. Existing approaches typically adopt a two-stage pipeline that first discovers novel objects using foundation models and then trains a 3D-OVD model based on these discovered objects. Although effective, this pipeline often suffers from inaccurate localization and mismatched classification during the discovery stage, which subsequently limits the performance of the model training stage. To address these limitations, we advocate for improving both the reliability of novel object discovery and the robustness of model training, and propose an innovative framework. Specifically, for reliable discovery, our co-distillation strategy distills high-quality novel objects by applying Hungarian matching over a comprehensive score that incorporates geometric consistency, structural objectness, and semantic certainty. To enhance robust model training, we further propose a dual-guidance learning scheme, incorporating a scene-awareness-guided uncertainty regularization for the regression head and an LLM-guided hierarchical alignment for the classification head, effectively mitigating the negative effects of imprecise 3D bounding boxes and semantic ambiguity. Extensive experiments on SUN RGB-D and ScanNetV2 demonstrate that our method achieves significant performance gains over state-of-the-art approaches. Code is available at https://github.com/shangboyuan/Co-3DGT

Shangbo Yuan, Jie Xu, Xiaofeng Zhu +1

School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore · School of Computer Science and Technology, Hainan University, Haikou, China

3D Object Detection Knowledge Distillation

Ask PDF

Mar 27, 2026

GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation

Open-vocabulary 3D semantic segmentation aims to segment arbitrary categories beyond the training set. Existing methods predominantly rely on distilling knowledge from 2D open-vocabulary models. However, aligning 3D features to the 2D representation space restricts intrinsic 3D geometric learning and inherits errors from 2D predictions. To address these limitations, we propose GeoGuide, a novel framework that leverages pretrained 3D models to integrate hierarchical geometry-semantic consistency for open-vocabulary 3D segmentation. Specifically, we introduce an Uncertainty-based Superpoint Distillation module to fuse geometric and semantic features for estimating per-point uncertainty, adaptively weighting 2D features within superpoints to suppress noise while preserving discriminative information to enhance local semantic consistency. Furthermore, our Instance-level Mask Reconstruction module leverages geometric priors to enforce semantic consistency within instances by reconstructing complete instance masks. Additionally, our Inter-Instance Relation Consistency module aligns geometric and semantic similarity matrices to calibrate cross-instance consistency for same-category objects, mitigating viewpoint-induced semantic drift. Extensive experiments on ScanNet v2, Matterport3D, and nuScenes demonstrate the superior performance of GeoGuide.

Xujing Tao, Chuxin Wang, Yubo Ai +8

University of Science and Technology of China · National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory

Open-Vocabulary 3D Segmentation 3D Semantic Segmentation 3D Point Cloud Segmentation Open-Vocabulary Semantic Segmentation Knowledge Distillation

Ask PDF

Aug 17, 2026

OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection

Open-vocabulary monocular 3D detectors report strong in-domain performance, but each evaluates under a different protocol, several rely on per-image category oracles unavailable at deployment, and all collapse geometry and semantics into a single AP metric. To address this, we introduce OV3D-Bench, a diagnostic benchmark that compares open-vocabulary monocular 3D detectors under deployment-realistic conditions across seven indoor and outdoor datasets. Our benchmark replaces the per-image class name oracle with test-time dataset-level class name prompts, and decouples detection accuracy along three axes: localization, semantic robustness, and cross-domain transfer. We evaluate seven representative detectors and find that (i) they localize objects well yet often mislabel a correctly localized box as a semantically adjacent category; (ii) accuracy is highly sensitive to prompt phrasing (e.g. WildDet3D's performance collapses from 18.6 to 5.4 AP when prompted with "a detailed high-resolution photo of a car" rather than "car"); and (iii) the widely adopted target-aware protocol hides these errors (e.g. inflating DetAny3D's AP by 1.9

\times

on ScanNet). Lastly, we demonstrate that simply remapping a frozen closed-vocabulary detector's predictions using a contrastive vision-language encoder such as SigLIPv2 performs competitively against recent purpose-built open-vocabulary methods. This indicates that geometric localization is more mature, while open-vocabulary semantics remains the primary bottleneck.

Mariia Gladkova, Neehar Peri, Ishan Khatri +2

Technical University of Munich · Carnegie Mellon University · StackAV

Open-Vocabulary Object Detection Monocular 3D Object Detection Prompt Sensitivity Domain Generalization

Ask PDF

Abstract

3D Object Detection

Explore similar work

Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Training

GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation

OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection