Part-Aware Open-Vocabulary 3D Affordance Grounding via Prototypical Semantic and Geometric Alignment

Dongqiang Gou; Xuming He

cs.CVMar 18, 2026

Part-Aware Open-Vocabulary 3D Affordance Grounding via Prototypical Semantic and Geometric Alignment

Authors: Dongqiang Gou, Xuming He

Organizations: ShanghaiTech University

Abstract

Grounding natural language questions to functionally relevant regions in 3D objects -- termed language-driven 3D affordance grounding -- is essential for embodied intelligence and human-AI interaction. Existing methods, while progressing from label-based to language-driven approaches, still face challenges in open-vocabulary generalization, fine-grained geometric alignment, and part-level semantic consistency. To address these issues, we propose a novel two-stage cross-modal framework that enhances both semantic and geometric representations for open-vocabulary 3D affordance grounding. In the first stage, large language models generate part-aware instructions to recover missing semantics, enabling the model to link semantically similar affordances. In the second stage, we introduce two key components: Affordance Prototype Aggregation (APA), which captures cross-object geometric consistency for each affordance, and Intra-Object Relational Modeling (IORM), which refines geometric differentiation within objects to support precise semantic alignment. We validate the effectiveness of our method through extensive experiments on a newly introduced benchmark, as well as two existing benchmarks, demonstrating superior performance in comparison with existing methods.

Figures & tables

Figure 1: Overview of motivations, illustrating (a) part-aware instruction generation and semantic–geometric alignment, (b) affordance prototype aggregation, (c) intra-object relational modeling.

Figure 2: Overview of our framework, comprising five stages: (1) Part-aware Instruction Representation (§ 3.2 ); (2) Object-level Geometric Modeling (§ 3.3 ); (3) Cross-modal Fusion (§ 3.4 ); (4) Decoding and Mask Prediction (§ 3.5 ); and (5) Prototype Aggregation (§ 3.6 ). Zoom-in views of IORM, CMFM, and MSSM are in the Appendix.

Dataset	#Point Cloud Inst.	#Question Inst.	#Image Inst.	OV‑Affordance	Probabilistic Mask
3DAffordanceNet	23K	✗	✗	✗	✓
PIAD	7K	✗	5162	✗	✓
LASO	8.4K	870	✗	✗	✓
Seq‑AFD	18K	— †	✗	✗	✓
Ours	23K	1840	✗	✓	✓

Table 1: Public 3D affordance datasets in recent literature. † Only a total of 180K points–text pairs is reported; instance‑level counts are not specified. ‡ Dataset provides 42119 points–text pairs without separate point cloud instance statistics.

Figure 3: Visualization comparison. Results under four evaluation splits with predictions from each method. Red: predicted regions.

		Method	aIoU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
Open-set	Full-view	XMF	11.37	76.33	0.529	0.117
		IAG	9.63	74.95	0.505	0.123
		LASO	13.86	76.12	0.540	0.109
		OpenAD	12.57	79.80	0.570	0.113
		Ours	18.38	82.33	0.606	0.110
	Partial-view	XMF	9.03	67.54	0.450	0.127

Table 2: Quantitative comparison across four evaluation splits using aIoU, AUC, SIM, and MAE. Best results are in bold.

Method	aIoU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
LASO	20.8	87.3	0.629	0.093
IAG	17.8	82.3	0.561	0.109
3D-SPS	11.4	76.2	0.433	0.138
ReLA	15.2	78.9	0.532	0.118
ReferTrans	13.7	79.8	0.497	0.124
Ours	20.8	88.8	0.637	0.090

Table 3: Quantitative results on the LASO dataset (Seen split). Results for all baselines are reported from the official LASO paper [ 22 ] .

	Method	mIoU $\uparrow$	Acc $\uparrow$	mAcc $\uparrow$
Full-view	ShapeLLM	0.88	0.28	0.99
	OpenAD-PointNet++	13.53	3.97	16.40
	OpenAD-DGCNN	11.15	3.84	13.86
	IAGNet	16.16	19.07	23.92
	LASO	22.41	15.90	30.22
	3D-ADLLM-Qwen	24.43	23.90	35.45

Table 4: Quantitative results on the 3D-AffordanceLLM dataset. Results for all baselines are reported from the official paper [ 7 ] .

		Method	aIoU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
Open-set	Full-view	w/o PIG	12.67	80.09	0.545	0.108
		w/o APA	14.31	82.30	0.585	0.105
		w/o PSGA	14.16	81.22	0.568	0.103
		w/o IORM	17.36	82.11	0.599	0.102
		Ours	18.38	82.33	0.606	0.110
	Partial-view	w/o PIG	11.34	75.99	0.532	0.113

Table 5: Ablation results under all four evaluation splits. We compare the full model with variants missing key components. Performance is measured using aIoU, AUC, SIM, and MAE.

Figure 4: Cross-dataset zero-shot predictions showing semantically consistent results across domain shifts.

Figure S1 : Model architecture details of three components: (a) Intra-Object Relational Modeling (IORM); (b) Cross-Modal Fusion Module (CMFM); (c) Multi-Scale Selection Module (MSSM).

Symbol	Shape	Description
$P$	$2048\times 3$	Input point cloud
$\mathbf{F}_{\text{T}}$	$512\times 40$	Token-level features of the instruction
$T_{i}$	$512$	Embedding of part-specific token
$\mathbf{F}_{P}^{l}$	$512\times 64$	Large-scale geometric features
$\tilde{\mathbf{F}}_{P}^{l}$	$512\times 64$	Enhanced large-scale features
$\tilde{\mathbf{F}}_{P}^{s}$	$512\times 128$	Enhanced small-scale features

Table S1 : Dimensions and semantic roles of principal tensors in our method. The prototype set

\mathbf{A}

is dynamic, initialized as

K=17

, and expands as new affordances are encountered during training.

Evaluation Setting	Split	#PC–AFD Pairs	#Qst. Inst.	#OV–AFD (I/C)	#Unseen OBJ (I/C)
Closed-set Seen	Train	32 709	742	—	—
Closed-set Seen	Test	4 645	53	—	—
Closed-set Unseen	Train	29 754	560	—	—
Closed-set Unseen	Test	4 645	53	—	414 / 6
Open-set Full-view	Train	32 709	742	—	—
Open-set Full-view	Test	4 645	69	975 / 16	—

Table S2: Statistics of the proposed dataset under four evaluation settings. PC–AFD: point cloud–affordance; Qst. Inst.: question instance; I/C: instance/category counts.

OBJ Error Rate	aIoU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
0%	18.38	82.33	0.606	0.110
10%	17.43	82.57	0.603	0.110
20%	17.41	82.57	0.603	0.111

Table S3 : Impact of part focus ambiguity rate on open-set full-view performance. Artificial ambiguity rates are introduced to test model robustness. OBJ denotes object.

AFD Error Rate	aIoU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
0%	18.38	82.33	0.606	0.110
10%	17.22	82.25	0.600	0.111
20%	17.03	81.46	0.587	0.113

Table S4 : Impact of affordance misclassification rate on open-set full-view performance. Artificial error rates are introduced for robustness analysis. AFD denotes affordance.

Figure S2 : Additional qualitative visualizations of affordance grounding results across four evaluation settings.

Figure S3 : More cross-dataset zero-shot predictions on ShapeNet objects, illustrating semantic consistency and domain robustness.

Figure S4 : Qualitative visualizations of affordance grounding results on LASO objects.

Figure S5 : Qualitative visualizations of affordance grounding results on IRAS objects.

$\beta_{1}$	$\beta_{2}$	aIoU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
0.2	0.6	18.38	82.33	0.606	0.110
0.3	0.5	17.55	82.30	0.605	0.101
0.5	0.3	17.77	82.78	0.596	0.104

Table S5 : Impact of loss balancing coefficients

\beta_{1}

and

\beta_{2}

on model performance in the open-set full-view setting.

3D Affordance Grounding

See all 15 papers

AffordAny: Open-World 3D Affordance Grounding from Monocular RGB Images via Vision-Language-Guided Geometric Reasoning
Aug 21, 2026Junqi Wu, Kaihua Tang, Xuanwen Chen +33D Affordance Grounding 3D Visual Grounding
Tongji University, Shanghai, China · Chinese Academy of Sciences, China
ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes
Aug 11, 2026Xinrui Lin, Sha Zhang, Shumin Wang +33D Affordance Grounding 3D Scene Understanding
University of Science and Technology of China, Hefei, China · The Chinese University of Hong Kong, Hong Kong SAR, China
BridgeAfford: Bridging Query-Target Gap for Generalizable 3D Affordance Grounding
Oct 15, 2026Yijie Su, Yakun Zhu, Yi Bin +53D Affordance Grounding 3D Visual Grounding
Tongji University · University of Electronic Science and Technology of China

Aug 21, 2026J/K move · Enter open · S save

AffordAny: Open-World 3D Affordance Grounding from Monocular RGB Images via Vision-Language-Guided Geometric Reasoning

Junqi Wu, Kaihua Tang, Xuanwen Chen, Hongzhi Li+2

Tongji University, Shanghai, China · Chinese Academy of Sciences, China

3D Affordance Grounding 3D Visual Grounding 3D Part Segmentation

Open paper AskPDF arXiv

Open-world 3D affordance grounding requires localizing functional object parts in 3D given free-form language queries. Existing methods typically assume pre-built object-centric 3D geometry and closed affordance ontologies, limiting deployment from raw RGB observations. We present AffordAny, an end-to-end framework that uses one monocular RGB image to construct large-scale text-conditioned 3D part supervision, ground affordances with a frozen vision-language model (VLM) guided decoder, and improve open-world generalization through pseudo-label self-training. Our automated pipeline produces a benchmark of 5,334 objects and 10,633 part-level samples spanning 473 categories, an order-of-magnitude increase in categorical diversity over prior work. The decoder progressively fuses frozen Cosmos-2B features with 3D geometry through spatial projection, instruction-conditioned semantic compression, and bidirectional geometry-semantics interaction. Minimal-perturbation pseudo-label self-training further adds new objects without human annotation. Under a systematic generalization protocol evaluating unseen objects, unseen categories, and unseen instruction paraphrases, our approach achieves 0.428 IoU on unseen objects and 0.315 IoU on unseen categories after self-training, with unseen-category mIoU improving by 6.3% relative (p<0.01) and an instruction sensitivity gap of only 0.105, demonstrating effectiveness and robustness of our method.

Abstract

3D Affordance Grounding

Explore similar work

AffordAny: Open-World 3D Affordance Grounding from Monocular RGB Images via Vision-Language-Guided Geometric Reasoning

ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes

BridgeAfford: Bridging Query-Target Gap for Generalizable 3D Affordance Grounding