VAGNet: Grounding 3D Affordance from Human-Object Interactions in Videos

Aihua Mao; Kaihang Huang; Yong-Jin Liu; Chee Seng Chan; Ying He

cs.CVFeb 24, 2026

VAGNet: Grounding 3D Affordance from Human-Object Interactions in Videos

Authors: Aihua Mao, Kaihang Huang, Yong-Jin Liu, Chee Seng Chan, Ying He

Organizations: SCUT · THU · UM · NTU

Abstract

3D object affordance grounding aims to identify regions on 3D objects that support human-object interaction (HOI), a capability essential to embodied visual reasoning. However, most existing approaches rely on static visual or textual cues, neglecting that affordances are inherently defined by dynamic actions. As a result, they often struggle to localize the true contact regions involved in real interactions. We take a different perspective. Humans learn how to use objects by observing and imitating actions, not just by examining shapes. Motivated by this intuition, we introduce video-guided 3D affordance grounding, which leverages dynamic interaction sequences to provide functional supervision. To achieve this, we propose VAGNet, a framework that aligns video-derived interaction cues with 3D structure to resolve ambiguities that static cues cannot address. To support this new setting, we introduce PVAD, the first HOI video-3D pairing affordance dataset, providing functional supervision unavailable in prior works. Extensive experiments on PVAD show that VAGNet achieves state-of-the-art performance, significantly outperforming static-based baselines. The code and dataset will be open publicly.

Figures & tables

Figure 1: Motivation of Our Work. (a) Existing 3D affordance grounding methods rely mainly on static visual or textual cues, forcing the model to infer how an object might be used from its shape or static interaction context. Consequently, they struggle with perspective ambiguity, visually similar parts ( e.g . blade vs . handle), and complex multi-contact interactions. (b) Human-object interaction videos, in contrast, reveal affordance directly through use: they show how hands approach, contact, and move across object surfaces. Our approach leverages this insight by contextually aligning video observations with 3D object geometry and mapping the extracted interaction cues into 3D space. This paradigm shift grounds affordance by observing actual use rather than inferring from appearance, providing richer functional supervision and enabling more reliable 3D affordance grounding.

Figure 2: VAGNet Architecture. Our model takes a point cloud, its 2D projection, and a corresponding interaction video as input. These are first processed by three modality-specific encoders to extract point features ( $F_{p}$ ), image features ( $F_{i}$ ), and video features ( $F_{v}$ ). Then, the Multimodal Contextual Alignment Module (MCAM) aligns $F_{i}$ with $F_{v}$ to produce a joint 2D representation $F_{2d}$ , which is then fused with $F_{p}$ through a cross-attention layer and a point decoder to obtain the context-aligned 3D feature $F_{3d}$ . Subsequently, the Spatial-Temporal Fusion Module (STFM) integrates $F_{v}$ and $F_{3d}$ to produce the spatio-temporal feature $F_{f}$ , which is finally fed into a decoder to generate the 3D affordance mask.

Figure 3: The PVAD Dataset. (a) Overview with annotated affordance regions on point clouds highlighted in red. (b) Distribution of video samples across different affordance categories. (c) Statistics of video and point cloud counts for representative object-affordance pairs.

Image-3D Alignment Methods
	Seen				Unseen
Method	AUC ↑	aIOU ↑	SIM ↑	MAE ↓	AUC ↑	aIOU ↑	SIM ↑	MAE ↓
XMF [ 1 ]	90.83	35.85	0.668	0.074	57.22	8.05	0.268	0.315
IAGNet [ 48 ]	93.04	39.77	0.682	0.072	58.91	10.19	0.297	0.305
GREAT [ 39 ]	93.75	40.23	0.703	0.066	59.83	10.42	0.302	0.265
VAGNet-img (ours)	93.74	41.58	0.707	0.065	60.22	11.03	0.287	0.248

Table 1: Comparison on the PVAD Dataset . The best results are marked in bold . The second-best results are underlined . The dataset is divided into two partitions: Seen and Unseen . AUC and aIOU are shown in percentage.

Figure 4: Comparative visualization of affordance results generated by different methods, on selected test instances from Seen and Unseen settings. Each object is associated with its corresponding interaction video, represented by two sampled frames. The affordance probability for each 3D point is encoded in a heatmap (red means high probability). Ground truth (GT) are provided for reference.

Type	MCAM	STFM	Proj	AUC ↑	aIOU ↑	SIM ↑	MAE ↓
Seen		✓		92.97	39.94	0.682	0.072
	✓		✓	93.83	39.78	0.694	0.066
		✓	✓	93.87	41.86	0.711	0.064
	✓	✓	✓	94.33	42.96	0.723	0.061
Unseen		✓		59.26	11.64	0.288	0.279
	✓		✓	59.88	10.25	0.302	0.304

Table 2: Abalation Studies. We evaluate the contribution of each key module: the multimodal contextual alignment module MCAM , the spatial-temporal fusion module STFM , and the entire 2D branch ( Proj ). AUC and aIOU are shown in percentage. Best results are in bold .

Figure 5: Ablation of MCAM. (a) Contextual Attention Maps during the alignment between project image and video instruction. (b) Visualization results with/without MCAM.

Figure 6: Single Instruction with Multiple Affordances. A single video may capture different affordances of the same object, and the model should localize the most relevant one (“beat”).

Figure 7: Single Instruction with Multiple Objects. Grounding affordances on different objects with the same video that contains multiple interactions.

3D Affordance Grounding

See all 15 papers

ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes
Aug 11, 2026Xinrui Lin, Sha Zhang, Shumin Wang +33D Affordance Grounding 3D Scene Understanding
University of Science and Technology of China, Hefei, China · The Chinese University of Hong Kong, Hong Kong SAR, China
HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Grounding
Mar 2, 2026Lei Yao, Yong Chen, Yuejiao Su +33D Affordance Grounding 3D Visual Grounding
The Hong Kong Polytechnic University · Huazhong University of Science and Technology
Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning
Aug 3, 2025Hanqing Wang, Zhenhao Zhang, Kaiyang Ji +123D Affordance Grounding Human-Object Interaction
The Hong Kong University of Science and Technology, Guangzhou, China · ShanghaiTech University, Shanghai, China · Zhejiang University, Hangzhou, China +3

Aug 11, 2026J/K move · Enter open · S save

ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes

Xinrui Lin, Sha Zhang, Shumin Wang, Zenghuan Zhu+2

University of Science and Technology of China, Hefei, China · The Chinese University of Hong Kong, Hong Kong SAR, China

3D Affordance Grounding 3D Scene Understanding 3D VLMs 3D Visual Grounding

Open paper AskPDF arXiv

Task-driven 3D affordance grounding aims to localize the functional region in a cluttered 3D scene that enables an action specified by a natural-language instruction. Existing methods either predict 3D masks directly or construct them by selecting and fusing intermediate 2D/3D regions. However, they remain vulnerable to two intertwined failure modes: the predicted or selected regions may miss the target interaction area or have unsuitable granularity, while language grounding may confuse visually similar alternatives under relational instructions. To this end, we introduce ThinkAfford, which decouples high-recall affordance proposal generation from instruction-grounded reasoning. Specifically, the Affordance Proposal Generation module first uses learnable affordance prompts and multi-level visual features to predict interaction-conditioned heatmaps, extracting a variable number of fine-grained proposals without parsed object or part names as segmentation prompts. Visual-Prompted Affordance Reasoning then reasons over labeled proposal overlays using the full instruction, returning identifiers in a structured "think-then-answer" response. Moreover, Group Relative Policy Optimization uses proposal-level rewards from lifted 3D overlap to align VPAR selection with final 3D grounding. On the SceneFun3D validation split, ThinkAfford achieves 10.69% AP50 and 25.46% AP25 under the official evaluator, outperforming comparable 3D open-vocabulary and vision-language-model-based 2D-to-3D baselines. Module-level diagnostics further show that APG attains 77.5% recall at 25% intersection-over-union, while GRPO-trained VPAR achieves 72.1% selection accuracy on APG-covered queries, compared with 63.4% under supervised fine-tuning.

Abstract

3D Affordance Grounding

Explore similar work

ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes

HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Grounding

Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning