Hierarchical Collaborative Fusion for 3D Instance-aware Referring Expression Segmentation

Keshen Zhou; Runnan Chen; Mingming Gong; Tongliang Liu

cs.CVMar 6, 2026

Hierarchical Collaborative Fusion for 3D Instance-aware Referring Expression Segmentation

Authors: Keshen Zhou, Runnan Chen, Mingming Gong, Tongliang Liu

Organizations: The University of Sydney · The University of Melbourne · Mohamed bin Zayed University of Artificial Intelligence

Abstract

Generalised 3D Referring Expression Segmentation (3D-GRES) localizes objects in 3D scenes based on natural language, even when descriptions match multiple or zero targets. Existing methods rely solely on sparse point clouds, lacking rich visual semantics for fine-grained descriptions. We propose HCF-RES, a multi-modal framework with two key innovations. First, Hierarchical Visual Semantic Decomposition leverages SAM instance masks to guide CLIP encoding at dual granularities -- pixel-level and instance-level features -- preserving object boundaries during 2D-to-3D projection. Second, Progressive Multi-level Fusion integrates representations through intra-modal collaboration, cross-modal adaptive weighting between 2D semantic and 3D geometric features, and language-guided refinement. HCF-RES achieves state-of-the-art results on both ScanRefer and Multi3DRefer.

Figures & tables

Fig. 1: (a) Language descriptions inherently encode hierarchical object semantics (e.g., main object ”chair” with attributes ”gray” and relationship ”below desk”). (b) Traditional methods rely solely on sparse point clouds, leading to wrong segmentation. (c) Our method leverages pre-trained vision models (CLIP [ 17 ] and SAM [ 10 ] ) on filtered multi-view images to extract rich semantic features, achieving better alignment across images, point clouds, and natural language.

Fig. 2: Overview of our Hierarchical Visual Semantic Decomposition : We employ the SAM [ 10 ] to segment the instances segmentation masks for each multi-view images without requiring annotations and each mask is then filtered by quality before encoding with CLIP [ 17 ] to obtain its instance-level and pixel-level features. These multi-granularity features are then subsequently projected to the point cloud and aggregated into superpoints representations.

Fig. 3: Pipeline of HCF-RES: The framework processes three input modalities: point clouds via 3D U-Net, text via RoBERTa, and multi-view images via CLIP guided by SAM * through our Hierarchical Visual Semantic Decomposition (Section III-C and Figure 2 ). After cross-modal feature integration and language-guided sampling, our Language-guided Instance Refinement enhances selected queries through scene context awareness and 2D semantic fusion. Enhanced queries are decoded via a 6-layer decoder [ 22 ] for final 3D referring expression segmentation.

Method	Venue	Overall
Method	Venue	[email protected]	[email protected]	mIoU
InstanceRefer [ 26 ]	ICCV2021	40.2	33.5	30.6
3D-STMN [ 23 ]	AAAI2024	54.6	39.8	39.5
SegPoint [ 5 ]	ECCV2024	-	-	41.7
Reason3D [ 8 ]	3DV2025	57.9	41.9	42.0
MCLN [ 16 ]	ECCV2024	58.7	50.7	44.7

TABLE I: 3D-RES results on ScanRefer.

	[email protected]				[email protected]
Method	ZT	ST	MT	All	ZT	ST	MT	All	mIoU
ReLA [ 13 ]	36.2 / 72.7	48.3 / 83.4	73.0	61.8	36.2 / 72.7	20.4 / 65.5	42.4	37.4	42.8
M3DRef-CLIP [ 27 ]	39.2 / 81.6	50.8 / 77.5	66.8	55.7	39.2 / 81.6	29.4 / 67.4	41.0	37.5	37.4
3D-STMN [ 23 ]	42.6 / 76.2	49.0 / 77.8	68.8	60.4	42.6 / 76.2	24.6 / 69.2	43.9	40.9	43.0
MDIN [ 22 ]	22.2 / 67.2	54.2 / 86.6	76.5	65.4	22.2 / 67.2	27.0 / 71.6	45.6	41.8	45.8
IPDN [ 2 ]	36.8 / 81.1	59.9 / 86.8	77.9	69.7	36.8 / 81.1	35.3 / 78.9	51.1	49.7	50.8

TABLE II: Comparison of the 3D-GRES methods on Multi3DRefer. ZT, ST, and MT represent zero target, single target, and multiple targets, respectively. The left and right sides of the “/” represent the situations with and without distractor objects, respectively.

Fig. 4: Qualitative Result to compare our method HCF-RES with the state-of-art MDIN [ 22 ] and IPDN [ 2 ] . Overall, our method produces more accurate segmentations. While IPDN achieves comparable results in general, it often shows noisy predictions near object boundaries.

VSD	MLF	[email protected]	[email protected]	mIoU
✗	✗	70.9	50.5	51.5
✗	✓	71.5	51.5	52.3
✓	✗	72.0	52.0	52.9
✓	✓	72.3	53.4	53.5

TABLE III: Ablation study on the proposed methods, evaluated on 3D-GRES.

Methods	Params (M)	Inference (ms) $\downarrow$	mIoU $\uparrow$
IPDN [ 2 ]	156.53	535	50.8
HCF-RES (ours)	157.69	523	53.5
$\Delta$ (ours $-$ IPDN)	+1.16 (+0.7%)	$-$ 12 ( $-$ 2.2%)	+2.7

TABLE IV: Inference speed and computational cost.

3D Referring Segmentation

See all 13 papers

3D-DRES: Detailed 3D Referring Expression Segmentation
Mar 3, 2026Qi Chen, Changli Wu, Jiayi Ji +23D Referring Segmentation Referring Expression Segmentation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Shanghai Innovation Institute · National University of Singapore
PC-CrossDiff: Point-Cluster Dual-Level Cross-Modal Differential Attention for Unified 3D Referring and Segmentation
Mar 18, 2026Wenbin Tan, Jiawen Lin, Fangyong Wang +43D Referring Segmentation 3D Visual Grounding
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, School of Informatics, Xiamen University, Xiamen, China · School of Big Data, Tongren University, Tongren, China · Hanjiang National Laboratory, Wuhan, China +2
TrackRef3D: Multi-View Consistent Track-then-Label for Open-World Referring Segmentation in 3D Gaussian Splatting
May 26, 2026Yuyang Tan, Renhe Zhang, Hang Zhang +2Multi-View Consistency 3D Gaussian Splatting
East China Normal University, Shanghai, China · Shanghai AI Laboratory · University of Electronic Science and Technology of China, Chengdu, China

Mar 3, 2026J/K move · Enter open · S save

3D-DRES: Detailed 3D Referring Expression Segmentation

Qi Chen, Changli Wu, Jiayi Ji, Yiwei Ma+1

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Shanghai Innovation Institute · National University of Singapore

3D Referring Segmentation Referring Expression Segmentation 3D Visual Grounding Visual Grounding

Open paper AskPDF arXiv

Current 3D visual grounding tasks only process sentence level detection or segmentation, which critically fails to leverage the rich compositional contextual reasonings within natural language expressions. To address this challenge, we introduce Detailed 3D Referring Expression Segmentation (3D-DRES), a new task that provides a phrase to 3D instance mapping, aiming at enhancing fine-grained 3D vision language understanding. To support 3D-DRES, we present DetailRefer, a new dataset comprising 54,432 descriptions spanning 11,054 distinct objects. Unlike previous datasets, DetailRefer implements a pioneering phrase-instance annotation paradigm where each referenced noun phrase is explicitly mapped to its corresponding 3D elements. Additionally, we introduce DetailBase, a purposefully streamlined yet effective baseline architecture that supports dual-mode segmentation at both sentence and phrase levels. Our experimental results demonstrate that models trained on DetailRefer not only excel at phrase-level segmentation but also show surprising improvements on traditional 3D-RES benchmarks.

Abstract

3D Referring Segmentation

Explore similar work

3D-DRES: Detailed 3D Referring Expression Segmentation

PC-CrossDiff: Point-Cluster Dual-Level Cross-Modal Differential Attention for Unified 3D Referring and Segmentation

TrackRef3D: Multi-View Consistent Track-then-Label for Open-World Referring Segmentation in 3D Gaussian Splatting