PC-CrossDiff: Point-Cluster Dual-Level Cross-Modal Differential Attention for Unified 3D Referring and Segmentation

Wenbin Tan; Jiawen Lin; Fangyong Wang; Yuan Xie; Yong Xie; Yachao Zhang; Yanyun Qu

cs.CVMar 18, 2026

PC-CrossDiff: Point-Cluster Dual-Level Cross-Modal Differential Attention for Unified 3D Referring and Segmentation

Authors: Wenbin Tan, Jiawen Lin, Fangyong Wang, Yuan Xie, Yong Xie, Yachao Zhang, Yanyun Qu

Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, School of Informatics, Xiamen University, Xiamen, China · School of Big Data, Tongren University, Tongren, China · Hanjiang National Laboratory, Wuhan, China · School of Computer Science and Technology, East China Normal University, Shanghai, China ·

Abstract

3D Visual Grounding (3DVG) aims to localize the referent of natural language referring expressions through two core tasks: Referring Expression Comprehension (3DREC) and Segmentation (3DRES). While existing methods achieve high accuracy in simple, single-object scenes, they suffer from severe performance degradation in complex, multi-object scenes that are common in real-world settings, hindering practical deployment. Existing methods face two key challenges in complex, multi-object scenes: inadequate parsing of implicit localization cues critical for disambiguating visually similar objects, and ineffective suppression of dynamic spatial interference from co-occurring objects, resulting in degraded grounding accuracy. To address these challenges, we propose PC-CrossDiff, a unified dual-task framework with a dual-level cross-modal differential attention architecture for 3DREC and 3DRES. Specifically, the framework introduces: (i) Point-Level Differential Attention (PLDA) modules that apply bidirectional differential attention between text and point clouds, adaptively extracting implicit localization cues via learnable weights to improve discriminative representation; (ii) Cluster-Level Differential Attention (CLDA) modules that establish a hierarchical attention mechanism to adaptively enhance localization-relevant spatial relationships while suppressing ambiguous or irrelevant spatial relations through a localization-aware differential attention block. Our method achieves state-of-the-art performance on the ScanRefer, NR3D, and SR3D benchmarks. Notably, on the Implicit subsets of ScanRefer, it improves the [email protected] score by +10.16% for the 3DREC task, highlighting its strong ability to parse implicit spatial cues.

Figures & tables

Figure 1: PC-CrossDiff vs. SOTA on 3DREC and 3DRES. ScanRefer (IoU@0.50), NR3D/SR3D (IoU@0.25). — Figure 1: PC-CrossDiff vs. SOTA on 3DREC and 3DRES. ScanRefer ([email protected]), NR3D/SR3D ([email protected]).

Figure 2: The framework of PC-CrossDiff. Text and visual features are input into the PLDA module to extract implicit localization cues and enhance point-level visual features. Simultaneously, the coordinates $F_{\mathrm{xyz}}$ are fed into the CLDA module for Spatial Relation Modeling (SRM), modeling cluster-level relations. These features are enhanced by LDA and integrated into the aligned features $F^{\prime}_{\text{v}}$ and $F^{\prime}_{\text{t}}$ , yielding $\tilde{F}_{\text{v}}^{\prime}$ and $\tilde{F}_{\text{t}}^{\prime}$ . The enhanced features are decoded by 3DREC and 3DRES heads to generate predictions, with $\mathcal{L}_{\text{DGTL}}$ facilitating task synergy.

Figure 3: LDA block. With $\mathit{F^{\prime}}_{\mathrm{v}}$ as query, DiffAtten ( Eq. 6 ) extracts relevant relations from $\mathit{F^{\prime}}_{\mathrm{oRel}}$ while suppressing irrelevant ones. Features then pass self-attention to yield $\mathit{\tilde{F}}_{\mathrm{v}}^{\prime}$ .

Method	Venue	Overall		mIoU	FPS
		0.25	0.50
TGNN	AAAI’21	37.50	31.40	27.80	2.85
X-RefSeg3D	AAAI’24	40.33	33.77	29.94	2.03
3D-STMN	AAAI’24	54.60	39.80	39.50	4.14
SegPoint	ECCV’24	-	-	41.70	-
MCLN	ECCV’24	58.70	50.70	44.72	4.81

Table 1: Comparison results of 3DRES on the ScanRefer dataset (inference: batchsize=1, on an NVIDIA 3090 GPU).

3DRES
Method	Multiple		Implicit
	0.25	0.50	0.25	0.50
X-RefSeg3D	28.16	23.61	43.71	37.33
3D-STMN	46.20	29.20	57.07	40.32
RefMask3D	48.09	40.77	49.30	44.71
MCLN	53.28	45.88	59.16	51.99

Table 2: Joint Evaluation of 3DRES and 3DREC across Multiple and Implicit Subsets in the ScanRefer Dataset.

Method	Venue	Multiple		Overall
		0.25	0.50	0.25	0.50
Two-stage Model
MVT	CVPR’22	31.92	25.26	40.80	33.26
3D-SPS	CVPR’22	40.32	29.82	48.82	36.98
BUTD-DETR	ECCV’22	44.73	33.97	50.42	38.60
ViL3DRel	NIPS’22	40.30	30.71	47.94	37.73

Table 3: Comparison results of single-stage and two-stage 3DREC on the ScanRefer dataset (*: Reproduction).

Method	Venue	SR3D	NR3D
3D-SPS	CVPR’22	62.60	51.50
MVT	CVPR’22	64.50	55.10
BUTD-DETR	ECCV’22	65.60	49.10
LAR	NIPS’22	59.60	48.90
EDA	CVPR’23	68.10	52.10
VPP-Net	CVPR’24	68.70	56.90

Table 4: 3DREC performance on SR3D and NR3D ([email protected]).

Component			3DREC		3DRES
$\bm{\mathcal{L_{\text{DGTL}}}}$	PLDA	CLDA	0.25	0.50	0.25	0.50	mIoU
w/o	w/o	w/o	56.76	44.86	58.43	49.66	43.85
$\checkmark$	w/o	w/o	56.95	45.50	58.49	50.86	44.78
$\checkmark$	$\checkmark$	w/o	57.56	46.20	59.19	50.99	45.12
$\checkmark$	w/o	$\checkmark$	57.69	46.18	59.61	51.88	45.68
$\checkmark$	$\checkmark$	$\checkmark$	58.47	47.89	60.41	52.52	46.39

Table 5: Ablation study of

\mathcal{L_{\text{DGTL}}}

, PLDA, and CLDA on 3DREC/3DRES (

\checkmark

: enabled; w/o: disabled).

CLDA		3DREC		3DRES
$N_{\mathrm{clust}}$	$S_{\mathrm{clust}}$	0.25	0.50	0.25	0.50	mIoU
256	4	56.11	46.37	57.83	50.88	44.66
128	8	58.15	47.09	60.21	52.40	46.16
64	16	58.47	47.89	60.41	52.52	46.39
32	32	57.36	47.01	59.25	51.53	45.54

Table 6: Ablation study of cluster configurations (

N_{\mathrm{clust}}

and

S_{\mathrm{clust}}

) on 3DREC and 3DRES tasks.

Component	3DREC		3DRES
	0.25	0.50	0.25	0.50	mIoU
$F^{(4)}_{\mathrm{v}}$	56.94	45.98	58.87	50.97	44.94
$\mathcal{K}^{\mathrm{t}2\mathrm{v}}_{\lambda}$	58.06	46.92	60.25	51.84	46.13
$F_{\mathrm{t}}$	57.93	46.71	60.02	52.42	46.27
$\mathcal{K}^{\mathrm{v}2\mathrm{t}}_{\lambda}$	58.47	47.89	60.41	52.52	46.39

Table 7: Comparison of localization accuracy when integrating different feature configurations from PLDA into CLDA.

Component	3DREC		3DRES
	0.25	0.50	0.25	0.50	mIoU
$\star$	56.64	45.81	58.54	50.63	44.85
PLDA+CLDA	58.47	47.89	60.41	52.52	46.39

Table 8: Performance comparison of our dual-level differential attention and standard cross-attention (

\star

: standard Transformer-based ( Vaswani et al. 2017 ) ).

Figure 4: Comparison of PC-CrossDiff (ours) and MCLN (baseline) for 3DREC and 3DRES with implicit cues in complex multi-object scenes. GT boxes are yellow.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

F^{\prime}_{\mathrm{oRel}}=\mathrm{DGCNN}(F_{\mathrm{iRel}},\mathbf{O})\in\mathbb{R}^{B\times N_{\mathrm{clust}}\times d_{\mathrm{mod}}}.

Appendix

Algorithm 1 The Working Process of CLDA.

Method	3DREC		3DRES
	0.25	0.50	0.25	0.50	mIoU
MCLN	68.43	57.30	64.70	57.88	49.84
Ours	70.95	61.27	68.70	62.04	53.43

Appendix

Table 9: Performance comparison of 3DREC and 3DRES on the SR3D dataset.

Method	3DREC		3DRES
	0.25	0.50	0.25	0.50	mIoU
MCLN †	53.75	45.82	51.79	46.58	40.47
Ours	59.91	50.48	57.56	52.64	46.00

Appendix

Table 10: Performance comparison of joint 3DREC and 3DRES tasks on the NR3D dataset ( † reproduced results; 200 epochs).

PLDA / CLDA		3DREC		3DRES
		0.25	0.50	0.25	0.50
w/o	w/o	51.96	40.76	53.28	45.88
$\checkmark$	w/o	52.35	41.51	53.84	46.67
w/o	$\checkmark$	52.74	41.50	54.57	47.56
$\checkmark$	$\checkmark$	53.59	43.12	55.48	47.98

Appendix

Table 11: Ablation study of PLDA and CLDA on ScanRefer-Multiple for 3DREC and 3DRES.

\checkmark

: included; w/o : no CLDA.

CLDA		3DREC		3DRES
$N_{\mathrm{clust}}$	$S_{\mathrm{clust}}$	0.25	0.50	0.25	0.50	mIoU
/	/	56.76	44.86	58.43	49.66	43.85
256	4	56.11	46.37	57.83	50.88	44.66
128	8	58.15	47.09	60.21	52.40	46.16
64	16	58.47	47.89	60.41	52.52	46.39
32	32	57.36	47.01	59.25	51.53	45.54

Appendix

Table 12: Ablation study of cluster configurations (

N_{\mathrm{clust}}

and

S_{\mathrm{clust}}

) on 3DREC and 3DRES tasks. / denotes no CLDA.

3D Referring Segmentation

See all 13 papers

Jun 30, 2026

PruneGround: Plug-and-play Spatial Pruning for 3D Visual Grounding

3D Visual Grounding (3DVG) aims to localize target objects in 3D scenes given natural language descriptions. Existing approaches typically perform reasoning over the entire scene, leading to ambiguous predictions and high computational cost, especially in cluttered environments. We observe that many referential expressions rely on local spatial context and often correspond to restricted spatial regions rather than the full scene. Motivated by this insight, we propose PruneGround, an effective plug-and-play framework for 3DVG built upon three key components. First, we introduce Language-Guided Spatial Pruning (LGSP), which leverages a frozen Vision Language Model (VLM) to identify language-relevant regions, thereby reducing spatial computation and grounding candidates in the narrower search space. Second, we propose MultiView-Conditioned Description Reformulation (MCDR), which decomposes complex expressions into simplified target--anchor relations and augments missing spatial cues through multi-view reasoning. Finally, we propose LLM-Grounder, which repurposes a detection-pretrained spatial LLM into a language-conditioned grounding model by aligning point cloud and linguistic representations within the pruned region. Extensive experiments on the three most popular point cloud benchmarks demonstrate that our method achieves state-of-the-art results on all three ScanRefer settings and on 9 out of 10 Nr3D/Sr3D settings. Code and models are publicly available: https://github.com/leduckhai/PruneGround

Duc Cao Dinh, Khai Le-Duc, Florent Draye +4

Knovel Engineering Lab · University of Toronto · Vector Institute +3

3D Spatial Reasoning Visually Grounded Reasoning 3D VLMs 3D Visual Grounding

Ask PDF

Mar 6, 2026

Hierarchical Collaborative Fusion for 3D Instance-aware Referring Expression Segmentation

Generalised 3D Referring Expression Segmentation (3D-GRES) localizes objects in 3D scenes based on natural language, even when descriptions match multiple or zero targets. Existing methods rely solely on sparse point clouds, lacking rich visual semantics for fine-grained descriptions. We propose HCF-RES, a multi-modal framework with two key innovations. First, Hierarchical Visual Semantic Decomposition leverages SAM instance masks to guide CLIP encoding at dual granularities -- pixel-level and instance-level features -- preserving object boundaries during 2D-to-3D projection. Second, Progressive Multi-level Fusion integrates representations through intra-modal collaboration, cross-modal adaptive weighting between 2D semantic and 3D geometric features, and language-guided refinement. HCF-RES achieves state-of-the-art results on both ScanRefer and Multi3DRefer.

Keshen Zhou, Runnan Chen, Mingming Gong +1

The University of Sydney · The University of Melbourne · Mohamed bin Zayed University of Artificial Intelligence

3D Referring Segmentation Referring Expression Segmentation 3D Visual Grounding 3D VLMs Multimodal Fusion

Ask PDF

Mar 3, 2026

3D-DRES: Detailed 3D Referring Expression Segmentation

Current 3D visual grounding tasks only process sentence level detection or segmentation, which critically fails to leverage the rich compositional contextual reasonings within natural language expressions. To address this challenge, we introduce Detailed 3D Referring Expression Segmentation (3D-DRES), a new task that provides a phrase to 3D instance mapping, aiming at enhancing fine-grained 3D vision language understanding. To support 3D-DRES, we present DetailRefer, a new dataset comprising 54,432 descriptions spanning 11,054 distinct objects. Unlike previous datasets, DetailRefer implements a pioneering phrase-instance annotation paradigm where each referenced noun phrase is explicitly mapped to its corresponding 3D elements. Additionally, we introduce DetailBase, a purposefully streamlined yet effective baseline architecture that supports dual-mode segmentation at both sentence and phrase levels. Our experimental results demonstrate that models trained on DetailRefer not only excel at phrase-level segmentation but also show surprising improvements on traditional 3D-RES benchmarks.

Qi Chen, Changli Wu, Jiayi Ji +2

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Shanghai Innovation Institute · National University of Singapore

3D Referring Segmentation Referring Expression Segmentation 3D Visual Grounding Visual Grounding

Ask PDF

Abstract

3D Referring Segmentation

Explore similar work

PruneGround: Plug-and-play Spatial Pruning for 3D Visual Grounding

Hierarchical Collaborative Fusion for 3D Instance-aware Referring Expression Segmentation

3D-DRES: Detailed 3D Referring Expression Segmentation