Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection

Youbin Kim; Jinho Park; Hogun Park; Eunbyung Park

cs.CVMar 23, 2026

Group3D: MLLM-Driven Semantic Grouping for Open-Vocabulary 3D Object Detection

Authors: Youbin Kim, Jinho Park, Hogun Park, Eunbyung Park

Organizations: Department of Artificial Intelligence, Sungkyunkwan University · Department of Artificial Intelligence, Yonsei University

Abstract

Open-vocabulary 3D object detection aims to localize and recognize objects beyond a fixed training taxonomy. In multi-view RGB settings, recent approaches often decouple geometry-based instance construction from semantic labeling, generating class-agnostic fragments and assigning open-vocabulary categories post hoc. While flexible, such decoupling leaves instance construction governed primarily by geometric consistency, without semantic constraints during merging. When geometric evidence is view-dependent and incomplete, this geometry-only merging can lead to irreversible association errors, including over-merging of distinct objects or fragmentation of a single instance. We propose Group3D, a multi-view open-vocabulary 3D detection framework that integrates semantic constraints directly into the instance construction process. Group3D maintains a scene-adaptive vocabulary derived from a multimodal large language model (MLLM) and organizes it into semantic compatibility groups that encode plausible cross-view category equivalence. These groups act as merge-time constraints: 3D fragments are associated only when they satisfy both semantic compatibility and geometric consistency. This semantically gated merging mitigates geometry-driven over-merging while absorbing multi-view category variability. Group3D supports both pose-known and pose-free settings, relying only on RGB observations. Experiments on ScanNet and ARKitScenes demonstrate that Group3D achieves state-of-the-art performance in multi-view open-vocabulary 3D detection, while exhibiting strong generalization in zero-shot scenarios. The project page is available at https://ubin108.github.io/Group3D/.

Figures & tables

Figure 1: Left : Predicted 3D bounding boxes projected onto the input RGB images. Right : Comparison with the baseline under the multi-view, pose-free, zero-shot setting across different vocabulary sizes, where Group3D consistently achieves higher mAP 25 .

Figure 2: The overview of Group3D. Given multi-view RGB images, an MLLM predicts object categories across views, which are aggregated into a Scene Vocabulary Memory . Category-aware masks are lifted into 3D to construct a 3D Fragment Memory . The MLLM then organizes the vocabulary into semantic compatibility groups, which gate fragment merging together with geometric consistency to produce the final open-vocabulary 3D object instances. Finally, multi-view evidence is accumulated to determine the final open-vocabulary category and 3D bounding box for each object instance.

Point cloud-based
Method	Pose-free	Zero-shot	ScanNet20		ScanNet60
			mAP 25	mAP 50	mAP 25	mAP 50
Det-PointCLIPv2 † [ 60 ]	–	✗	–	–	0.2	–
3D-CLIP † [ 33 ]	–	✗	–	–	4.0	–
OV-3DET [ 26 ]	–	✗	18.0	–	–	–
CoDa † [ 3 ]	–	✗	19.3	–	9.0	–

Table 1: Quantitative results on the ScanNet benchmark under two category settings. Methods are grouped by input modality, including point cloud-based methods and multi-view image-based methods. For multi-view methods, results are further reported with and without ground-truth camera poses. † denotes methods that use 3D bounding boxes during training. Zoo3D 0 and Zoo3D 1 denote the zero-shot and self-supervised variants of Zoo3D, respectively.

	Inputs	toilet	bed	chair	sofa	dresser	table	cabinet	bookshelf	pillow	sink
OV-3DET	pc + pi	57.3	42.3	27.1	31.5	8.2	14.2	3.0	5.6	23.0	31.6
CoDA	pc	68.1	44.1	28.7	44.6	3.4	20.2	5.3	0.1	28.0	45.3
OV-Uni3DETR	pc	86.1	50.5	28.1	31.5	18.2	24.0	6.6	12.2	29.6	54.6
Zoo3D 1	pc + pi	78.4	54.4	74.4	65.5	33.6	19.1	14.1	32.3	46.1	27.3
Group3D	pi	91.3	80.5	61.1	78.0	55.4	56.2	24.7	41.3	43.7	62.3
(Ours)	ui	89.6	80.2	37.8	73.4	50.5	43.1	16.6	31.7	20.2	39.3

Table 2: Per-class AP 25 comparison on ScanNet20. ‘pc’ indicates the methods leverage the ground-truth point clouds, ‘pi’ denotes the use of posed images, and ‘ui’ means the use of unposed images.

Method	Pose-free	Zero-shot	ScanNet200		ARKitScenes
Method	Pose-free	Zero-shot	mAP 25	mAP 50	mAP 25	mAP 50
OpenM3D [ 11 ]	✗	✗	4.2	–	–	–
Zoo3D 0 [ 17 ]	✗	✓	14.3	6.2	–	–
Zoo3D 1 [ 17 ]	✗	✗	16.5	6.3	–	–
Ours	✗	✓	17.9	8.7	20.5	5.9
Zoo3D 0 [ 17 ]	✓	✓	8.3	2.9	13.0	2.6

Table 3: Quantitative results on ScanNet200 [ 35 ] and ARKitScenes [ 2 ] under the multi-view setting.

Figure 3: Qualitative results on ScanNet20 [ 26 ] under pose-known and pose-free settings.

Component	Method	mAP 25	mAP 50
Reconstruction	Depth Anything 3 [ 22 ]	41.2	18.5
Reconstruction	VGGT [ 45 ]	40.0	18.7
MLLM	GPT 5.1 [ 41 ]	41.2	18.5
MLLM	Qwen3-VL-8B [ 1 ]	38.5	16.9
Segmentation	SAM 3 [ 4 ]	41.2	18.5
Segmentation	Grounded SAM 2 [ 34 ]	39.7	17.6

Table 4: Ablation study on ScanNet20 with different components. Depth Anything 3 is trained on external datasets, while VGGT is pretrained on ScanNet.

Table 8

Figure 4: Qualitative results on ScanNet200 [ 35 ] under pose-known and pose-free settings.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Table 10

Figure 5: Qualitative results on ScanNet20 [ 26 ] under pose-known and pose-free settings.

Figure 6: Qualitative results on ScanNet200 [ 35 ] under pose-known and pose-free settings.

Figure 7: Qualitative results on ARKitScenes [ 2 ] under pose-known and pose-free settings.

Scene Vocabulary Memory	Semantic Compatibility Grouping
bin, book, box, cabinet, clock, copier, door, light switch, poster, printer, shelf, switch, table, trash can, washer, washing machine, wall, window	copier_printer : [copier, printer] washer_machine : [washer, washing machine] trash_container : [trash can, bin] switch_control : [light switch, switch]
blackboard, book, bookshelf, box, cabinet, chalkboard, chair, couch, desk, recliner, shelf, sofa, table, trash can	seating : [chair, sofa, couch, recliner] board : [chalkboard, blackboard] table_like : [table, desk] storage : [bookshelf, shelf, cabinet]
basket, bottle, box, can, carpet, container, cushion, door, jar, pillow, refrigerator, rug, tray, water bottle, window	small_containers : [box, container, basket, tray] drink_containers : [bottle, water bottle, can, jar] soft_items : [pillow, cushion] ground_cover : [carpet, rug]
bag, backpack, bathtub, blanket, computer, lamp, light, laptop, painting, phone, picture, pillow, shower, sheet, shoe, sneaker, suitcase, tablet, telephone, tissue, toilet paper, towel, trash can	computer_device : [laptop, computer, tablet] phone_device : [phone, telephone] bag_like : [bag, backpack, suitcase] footwear : [shoe, sneaker] wall_art : [painting, picture] bedding_cover : [blanket, sheet] lamp_light : [lamp, light] bath_fixture : [bathtub, shower] bath_tissue : [toilet paper, tissue]

Appendix

Table 9: Examples of semantic compatibility grouping predicted by the MLLM from the scene vocabulary memory.

Method	Pose-free	Zero-shot	ScanNet200
Method	Pose-free	Zero-shot	AP 25	AP 50
Group3D (Ours)	✗	✓	22.9	12.2
Group3D (Ours)	✓	✓	14.8	5.3

Appendix

Table 10: 3D instance segmentation results on ScanNet200 [ 35 ] .

3D Object Detection

3D object detection identifies objects and estimates their locations, sizes, and orientations from images, point clouds, radar, or fused sensors.

See all 181 papers

Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Training
Aug 20, 2026Shangbo Yuan, Jie Xu, Xiaofeng Zhu +13D Object Detection Knowledge Distillation
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore · School of Computer Science and Technology, Hainan University, Haikou, China
GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation
Mar 27, 2026Xujing Tao, Chuxin Wang, Yubo Ai +8Open-Vocabulary 3D Segmentation 3D Semantic Segmentation
University of Science and Technology of China · National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory
OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection
Aug 17, 2026Mariia Gladkova, Neehar Peri, Ishan Khatri +2Open-Vocabulary Object Detection Monocular 3D Object Detection
Technical University of Munich · Carnegie Mellon University · StackAV

Aug 20, 2026J/K move · Enter open · S save

Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Training

Shangbo Yuan, Jie Xu, Xiaofeng Zhu, Na Zhao

School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, Singapore · School of Computer Science and Technology, Hainan University, Haikou, China

3D Object Detection Knowledge Distillation

Open paper AskPDF arXiv

Recently, open-vocabulary 3D object detection (3D-OVD) has gained increasing attention for its ability to detect unseen objects in 3D scenes. Existing approaches typically adopt a two-stage pipeline that first discovers novel objects using foundation models and then trains a 3D-OVD model based on these discovered objects. Although effective, this pipeline often suffers from inaccurate localization and mismatched classification during the discovery stage, which subsequently limits the performance of the model training stage. To address these limitations, we advocate for improving both the reliability of novel object discovery and the robustness of model training, and propose an innovative framework. Specifically, for reliable discovery, our co-distillation strategy distills high-quality novel objects by applying Hungarian matching over a comprehensive score that incorporates geometric consistency, structural objectness, and semantic certainty. To enhance robust model training, we further propose a dual-guidance learning scheme, incorporating a scene-awareness-guided uncertainty regularization for the regression head and an LLM-guided hierarchical alignment for the classification head, effectively mitigating the negative effects of imprecise 3D bounding boxes and semantic ambiguity. Extensive experiments on SUN RGB-D and ScanNetV2 demonstrate that our method achieves significant performance gains over state-of-the-art approaches. Code is available at https://github.com/shangboyuan/Co-3DGT

Abstract

3D Object Detection

Explore similar work

Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Training

GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation

OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection