AffordAny: Open-World 3D Affordance Grounding from Monocular RGB Images via Vision-Language-Guided Geometric Reasoning

Junqi Wu; Kaihua Tang; Xuanwen Chen; Hongzhi Li; Jianqiang Huang; Xian-Sheng Hua

cs.CVAug 21, 2026

AffordAny: Open-World 3D Affordance Grounding from Monocular RGB Images via Vision-Language-Guided Geometric Reasoning

Authors: Junqi Wu, Kaihua Tang, Xuanwen Chen, Hongzhi Li, Jianqiang Huang, Xian-Sheng Hua

Organizations: Tongji University, Shanghai, China · Chinese Academy of Sciences, China

Abstract

Open-world 3D affordance grounding requires localizing functional object parts in 3D given free-form language queries. Existing methods typically assume pre-built object-centric 3D geometry and closed affordance ontologies, limiting deployment from raw RGB observations. We present AffordAny, an end-to-end framework that uses one monocular RGB image to construct large-scale text-conditioned 3D part supervision, ground affordances with a frozen vision-language model (VLM) guided decoder, and improve open-world generalization through pseudo-label self-training. Our automated pipeline produces a benchmark of 5,334 objects and 10,633 part-level samples spanning 473 categories, an order-of-magnitude increase in categorical diversity over prior work. The decoder progressively fuses frozen Cosmos-2B features with 3D geometry through spatial projection, instruction-conditioned semantic compression, and bidirectional geometry-semantics interaction. Minimal-perturbation pseudo-label self-training further adds new objects without human annotation. Under a systematic generalization protocol evaluating unseen objects, unseen categories, and unseen instruction paraphrases, our approach achieves 0.428 IoU on unseen objects and 0.315 IoU on unseen categories after self-training, with unseen-category mIoU improving by 6.3% relative (p<0.01) and an instruction sensitivity gap of only 0.105, demonstrating effectiveness and robustness of our method.

Figures & tables

Figure 1: Overview of the proposed dataset. Given an RGB image and instruction, the pipeline reconstructs object-centric 3D geometry and grounds the target part. We evaluate unseen instructions, unseen objects, and unseen categories.

Figure 2: AffordAny data construction pipeline: LVIS images are reconstructed and rendered from six views; an LLM proposes parts and instructions, a segmentation model labels 2D views, and multi-view voting lifts labels to 3D.

					Taxonomy		Open
Dataset	Year	Source / Input	Annotation	Scale	Obj.	Part	Part	Obj.	Cat.
3D AffordanceNet ( Deng et al. 2021 )	2021	CAD / point cloud	3D heatmap	23K shapes	23	18	✗	✓	✗
GAPartNet ( Geng et al. 2023 )	2023	Articulated CAD / point cloud + mesh	3D part mask + pose	8,489 parts	27	9	✗	✓	✓
AffordPose ( Jian et al. 2023 )	2023	CAD / mesh + affordance	3D part + hand pose	26.7K inter.	13	8	✗	✗	✗
LASO ( Li et al. 2024 )	2024	CAD / point cloud + language	3D part mask	19,751 pairs	23	17	✗	✓	✓
AGPIL ( Zhu et al. 2025 )	2025	CAD + image / image + point cloud + language	3D heatmap	30,972 trip.	23	17	✗	✓	✓

Table 1: Comparison with related affordance and interactive-region datasets. Under “Open,” Part denotes an unrestricted part-level vocabulary, while Obj. and Cat. denote evaluation on unseen instances and categories; dashes indicate unavailable results.

Figure 3: Part–category distribution for the most frequent parts and categories.

Figure 4: Architecture overview of AffordAny. A frozen Cosmos-2B VLM encodes the instruction and image; projection injection, semantic compression, and GPBlock bidirectional fusion combine these features with 3D geometry to predict per-point affordance scores.

Model	Best Val	Unseen Instruction					Unseen Object					Unseen Category
Model	Best Val	IoU	mIoU	AUC	MAE $\downarrow$	SIM	IoU	mIoU	AUC	MAE $\downarrow$	SIM	IoU	mIoU	AUC	MAE $\downarrow$	SIM
OpenAD	.413	.484	.390	.878	.255	.360	.362	.272	.783	.304	.294	.266	.234	.683	.345	.257
LASO	.453	.451	.320	.853	.205	.374	.394	.282	.820	.222	.324	.266	.227	.723	.264	.262
LMAffordance3D	.454	.469	.330	.852	.188	.387	.418	.301	.820	.203	.323	.244	.211	.669	.278	.223
AffordAny (Ours)	.473	.680	.630	.956	.112	.604	.428	.306	.829	.212	.339	.305	.243	.716	.266	.268

Table 2: Decoder comparison after 300 epochs. “Unseen Instruction” uses held-out paraphrases. Bold is best;

\downarrow

is lower-is-better.

Model	$\Delta$ IoU	$\Delta$ mIoU	$\Delta$ AUC	$\Delta$ MAE	$\Delta$ SIM
OpenAD	-0.166	-0.166	-0.087	0.075	-0.065
LASO	-0.342	-0.429	-0.135	0.133	-0.323
LMAffordance3D	-0.360	-0.466	-0.140	0.129	-0.351
AffordAny	-0.105	-0.108	-0.031	0.037	-0.078

Table 3: Instruction robustness on matched object–part pairs (unseen

-

seen). Smaller absolute gaps are better.

Configuration	Unseen Object			Unseen Category
Configuration	IoU	AUC	MAE	IoU	AUC	MAE
Full model	.428	.829	.212	.305	.716	.266
w/o Proj. Inj.	.416	.803	.211	.275	.691	.277
w/o Sem. Compr.	.398	.793	.219	.245	.653	.276
w/o GPBlock	.403	.808	.218	.270	.686	.273
w/o Instr. dropout	.402	.807	.217	.288	.694	.273

Table 4: Ablation of AffordAny (200 epochs). Semantic compression, GPBlock, projection injection, and instruction dropout contribute

0.060

0.035

0.030

, and

0.017

unseen-category IoU, respectively.

Split	Configuration	IoU	mIoU	AUC	MAE
Unseen Object	Baseline	.428	.306	.829	.212
	+ Self-Training	.428	.315 ‡	.832 ‡	.223
	+ Platt cal.	.428	.306	.832 ‡	.211
Unseen Category	Baseline	.305	.243	.716	.266
	+ Self-Training	.315 ‡	.259 ‡	.720 ‡	.275
	+ Platt cal.	.309	.246	.720 ‡	.265 ∗

Table 5: Pseudo-label self-training before/after Platt calibration. Pre-calibration improves segmentation, especially unseen-category mIoU (+0.016, +6.3%), while calibration restores MAE but attenuates the gain.

{}^{\ddagger}p{<}0.01

;

{}^{*}p{<}0.05

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Instances/objects	Categories
LVIS training instances	1,270,141	1,203
Interactive-category pass	–	677
Image-filtered quota candidates	7,407	473
Completed object processing	7,398	473
Objects with validated parts	5,334	473
Validated part samples	10,633	473

Appendix

Table 6: Construction flow. Dashes denote a stage for which only the category-level count is used. The experimental manifests use a subset of the generated instruction pairs, as detailed in Table 7 .

Object split	Instruction use	Inst. rows	Objects	Cats.	Unique parts
Training	Used for optimization	12,134	3,049	331	511
Training	Unseen instruction	6,067	3,049	331	511
Validation	Unseen object	710	351	159	233
Validation	Unseen category	1,515	764	71	138
Test	Unseen object	789	391	209	275
Test	Unseen category	1,552	779	71	140

Appendix

Table 7: Experimental split statistics. “Inst. rows” counts object–part–instruction records; “Unique parts” counts normalized part-name types. For each training object–part pair, instruction indices 1 and 2 are used for optimization and index 0 is reserved for matched unseen-instruction evaluation. Validation and both test protocols also use only instruction index 0. For historical reasons, the optimization and unseen-instruction manifests are named train_unseen_instruction and train_seen_instruction , respectively. The final row reports unique objects, categories, and part-name types across manifests rather than summing those columns.

Panel of Figure 5 — Figure 5: Dataset vocabulary word clouds. Label size is proportional to frequency in the word-cloud counts; colors separate neighboring labels and do not encode semantic groups. Underscores preserve multi-word canonical names. The visualization is descriptive: exact split sizes and unique-vocabulary counts are reported in Table 7 .

Tensor	Shape per sample	Role
$\mathbf{X}$	$4{,}096\times 3$	normalized Gaussian centers
$\mathbf{F}$	$4{,}096\times 13$	cached point features
$\mathbf{h}_{\mathrm{aff}}$	$2{,}048$	instruction-conditioned global VLM state
$\mathbf{F}_{v}$	$144\times 2{,}048$	$12\times 12$ source-image token grid
$\mathbf{P}$	$4{,}096\times 256$	encoded point tokens
$\mathbf{Z}$	$16\times 256$	conditioned semantic prototypes

Appendix

Table 8: Principal tensors in the AffordAny decoder. Batch dimensions are omitted.

Cached-decoder quantity	Value
Trainable parameters	10.54M
BF16 parameter footprint	20.1 MiB
Profiled matrix operations	121.63 GFLOPs
Attention batched matmul	69.14 GFLOPs (56.8%)
Linear matmul	52.46 GFLOPs (43.1%)
BF16 latency, median / P95	16.5 / 18.6 ms

Appendix

Table 9: Batch-1 decoder cost for one 4,096-point object. Latency and allocated-memory measurements use BF16 autocast on one NVIDIA A100-SXM4-80GB after 20 warm-up iterations and over 100 timed forwards; inputs are already resident on the GPU.

Method	Frozen feature input	Adapted decoder	Trainable params.	Selected epoch
OpenAD	CLIP pooled text (512-D)	point–text cosine alignment	6.31M	130
LASO	77 CLIP text tokens	2 AFM/GPBlocks + 2-layer RPD	7.01M	120
LMAffordance3D	77 CLIP text tokens	2 fusion + 2 decoder layers	6.38M	140
AffordAny	Cosmos state + 144 visual tokens	prototypes + GPBlock + 4-layer decoder	10.54M	140

Appendix

Table 10: Controlled decoder implementations used in the main comparison. Parameter counts exclude frozen feature encoders. The comparison standardizes data, supervision, optimization, and model selection while preserving each method’s native feature representation and decoder design.

Analysis	Split	Model	$N$	IoU $\uparrow$	mIoU $\uparrow$	AUC $\uparrow$	MAE $\downarrow$
Human-verified	Unseen category	AffordAny	41	.3387	.2537	.7095	.3030
		LASO	41	.2628	.1953	.6879	.3193
		LMAffordance3D	41	.2596	.1988	.6452	.3151
		OpenAD	41	.2897	.2437	.6747	.3715
Human-verified	Unseen object	AffordAny	54	.4371	.3570	.8556	.1901
		LASO	54	.3656	.2766	.8299	.2165

Appendix

Table 11: Evaluation on the human-verified subsets, containing 41 unseen-category rows and 54 unseen-object rows. Bold denotes the best value within each split.

Split	Metric	Raw $\Delta$	95% CI
Unseen obj.	IoU	$+0.0002$	$[-0.0039,0.0042]$
	mIoU	$+0.0089$	$[0.0050,0.0118]$
Unseen cat.	IoU	$+0.0098$	$[0.0068,0.0131]$
	mIoU	$+0.0152$	$[0.0104,0.0183]$

Appendix

Table 12: Paired-bootstrap differences for raw self-training minus the supervised baseline. For compactness, the table lists the overlap metrics; AUC and MAE are computed from the same paired resamples and their significance markers are reported in the main paper. Self-training significantly improves category-macro mIoU on both splits and IoU on unseen categories (

p<0.001

), while maintaining unseen-object IoU (

p=0.461

Figure 6: Per-category IoU distributions. Violins summarize categories; dots are individual category values and black bars denote means. “Unseen instruction” evaluates unseen wording on the training objects.

Figure 7: Metric-complementarity examples on unseen categories. Each block shows ground truth and the prediction at threshold 0.5. The values motivate joint reporting of overlap, ranking, and probability-error metrics.

Figure 8: Qualitative comparison across unseen instructions, unseen objects, and unseen categories. Each group shows the source image/instruction, ground truth, and predictions. Red points denote the thresholded affordance region and the numbers are per-row IoU. These examples are selected for clear visualization and complement the aggregate results reported in the main tables.

Explore similar work

Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale
Jun 13, 2025Junha Lee, Eunha Park, Chunghyun Park +2Synthetic Data Generation for Robotics Embodied AI
Pohang University of Science and Engineering (POSTECH) · SqueezeBits · RLWLRD
Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances
May 12, 2026Qirui Wang, Jingyi He, Yining Pan +2Visually Grounded Reasoning Memory-Augmented Agents
TUM · A*STAR
Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning
Aug 3, 2025Hanqing Wang, Zhenhao Zhang, Kaiyang Ji +12Human-Object Interaction 3D Visual Grounding
The Hong Kong University of Science and Technology, Guangzhou, China · ShanghaiTech University, Shanghai, China · Zhejiang University, Hangzhou, China +3

Jun 13, 2025, cs.CVJ/K move · Enter open · S save

Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale

Junha Lee, Eunha Park, Chunghyun Park, Dahyun Kang+1

Pohang University of Science and Engineering (POSTECH) · SqueezeBits · RLWLRD

Synthetic Data Generation for Robotics Embodied AI 3D Visual Grounding

Open paper AskPDF arXiv

Affordance grounding aims to localize where to interact with an object, a fundamental capability for embodied agents. Yet progress is bottlenecked by data: manual annotation is prohibitively expensive and confines existing datasets to a narrow set of predefined object and affordance categories. We introduce Affogato, a framework for open-vocabulary affordance grounding centered on Affogato-750K, a large-scale dataset of 750K 3D affordance heatmaps paired with natural language queries. We build it with a fully automated pipeline that orchestrates foundation models to generate them at scale without human labeling. It covers significantly more diverse categories than any existing dataset. For reliable evaluation, we further provide 5K human-verified test pairs. We also present Espresso-3D and Espresso-2D, simple yet effective models with a unified architecture across both modalities. Pretraining on Affogato-750K improves both Espresso and prior methods and yields the largest gains on unseen object and affordance categories, showing that it provides broadly transferable supervision across architectures.

Abstract

Explore similar work

Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale

Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances

Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning