Organizations: School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Nanjing 211189, China · Southeast University-Monash University Joint Graduate School, Southeast University, Suzhou 215123, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
Understanding object affordances is essential for enabling robots to perform purposeful and fine-grained interactions in diverse and unstructured environments. However, existing approaches either rely on retrieval, which is fragile due to sparsity and coverage gaps, or on large-scale models, which frequently mislocalize contact points and mispredict post-contact actions when applied to unseen categories, thereby hindering robust generalization. We introduce Retrieval-Augmented Affordance Prediction (RAAP), a framework that unifies affordance retrieval with alignment-based learning. By decoupling static contact localization and dynamic action direction, RAAP transfers contact points via dense correspondence and predicts action directions through a retrieval-augmented alignment model that consolidates multiple references with dual-weighted attention. Trained on compact subsets of DROID and HOI4D with as few as tens of samples per task, RAAP achieves consistent performance across unseen objects and categories, and enables zero-shot robotic manipulation in both simulation and the real world. Project website: https://github.com/SEU-VIPGroup/RAAP.
Figures & tables
Fig. 2 : Overview of the Retrieval-Augmented Affordance Prediction (RAAP) framework. (a) Pipeline. Given an RGB-D input and a task label, RAAP retrieves top- K references from an affordance memory using CLIP-based similarity. It then predicts a 2D affordance (contact point and action direction) and lifts it to 3D for execution. (b) 2D Affordance Prediction. Static contact points are localized via dense correspondence using Stable Diffusion (SD) features, while dynamic action directions are inferred by a cross-image alignment module. Both query and reference images are encoded with a shared SigLIP-2 backbone; reference tokens are further modulated by their action vectors via FiLM, and fused with query tokens through gated cross-attention and a Transformer.
Tasks
RAM
A0-170M
RAAP K=0
RAAP K=3
Open microwave
49.82
119.64
51.75
37.00
Close microwave
55.74
103.10
61.38
30.88
Open drawer
51.73
112.89
15.45
16.23
Close drawer
83.04
85.16
52.11
48.25
Pickup bowl
53.85
34.37
36.76
31.45
Pickup bottle
82.62
33.86
38.70
37.67
TABLE I : MAE ↓ for task-aware dynamic affordance prediction. The first six rows correspond to individual manipulation tasks from the DROID dataset, the seventh row reports the averaged performance on HOI4D pickup tasks ( bowl , bottle , mug ). The final row shows the overall average across all tasks.
Fig. 3 : Qualitative comparison of 2D affordance predictions on the DROID dataset. The first row shows the input RGB image with ground-truth affordances (contact point and action direction), and the second row visualizes predictions from RAM, A0, and RAAP ( K=3 ).
Variant
Open/Close
Pickup
RAAP (Full)
33.09
34.56
w/o Gating
36.14
42.92
w/o Similarity
39.47
38.96
w/ Uniform Weights
37.63
36.54
TABLE II : Ablation on dual-weighted attention for dynamic affordance prediction (MAE ↓ , degrees). Results are reported on the DROID dataset for two task categories: open/close (e.g., microwave) and pickup (e.g., bowl, bottle).
Fig. 4 : MAE ↓ of RAAP as the number of retrieved references K varies from 0 to 4 on the DROID dataset.
Settings
Tasks
RAM
A0
RAAP (Ours)
Unseen Objects
Open drawer
70%
0%
85%
Close drawer
60%
5%
80%
Pickup bowl
85%
65%
90%
Pickup bottle
50%
35%
75%
Pickup mug
70%
25%
70%
Unseen Categories
Open cabinet
70%
0%
80%
TABLE III : Real-world manipulation success rates (20 trials per task). The top block shows unseen-object generalization, and the bottom block shows cross-category generalization.
Fig. 5 : Qualitative results on pickup kettle in MuJoCo with a UR5e manipulator. RAAP successfully transfers handle-oriented affordances to kettles.
Understanding spatial affordances -- comprising the contact regions of object interaction and the corresponding contact poses -- is essential for robots to effectively manipulate objects and accomplish diverse tasks. However, existing spatial affordance prediction methods mainly focus on locating the contact regions while delegating the pose to independent pose estimation approaches, which can lead to task failures due to inconsistencies between predicted contact regions and candidate poses. In this work, we propose RoboPCA, a pose-centered affordance prediction framework that jointly predicts task-appropriate contact regions and poses conditioned on instructions. To enable scalable data collection for pose-centered affordance learning, we devise Human2Afford, a data curation pipeline that automatically recovers scene-level 3D information and infers pose-centered affordance annotations from human demonstrations. With Human2Afford, scene depth and the interaction object's mask are extracted to provide 3D context and object localization, while pose-centered affordance annotations are obtained by tracking object points within the contact region and analyzing hand-object interaction patterns to establish a mapping from the 3D hand mesh to the robot end-effector orientation. By integrating geometry-appearance cues through an RGB-D encoder and incorporating mask-enhanced features to emphasize task-relevant object regions into the diffusion-based framework, RoboPCA outperforms baseline methods on image datasets, simulation, and real robots, and exhibits strong generalization across tasks and categories.
Zhanqi Xiao, Ruiping Wang, Xilin Chen
The authors are with the Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences (CAS), Beijing, 100190, China, and also with the University of Chinese Academy of Sciences, Beijing 100049, China.
Robotic manipulation systems commonly learn object affordances from human-annotated datasets, which reflect interaction possibilities for a human hand rather than the physical constraints of a specific robot. For example, a surface region of an object identified as graspable by a human annotator may be geometrically unreachable or mechanically infeasible for a robot, yet this mismatch is rarely addressed explicitly. To tackle this challenge, we proposed a framework that translates human affordance annotations into robot-executable affordance predictions conditioned on the physical properties of the robot. The network predicts which surface regions of an object are physically executable by the robot, given a 3D point cloud of an object and a descriptor encoding the robot's geometry and constraints. The model was trained using human-annotated affordances together with robotic execution outcomes from multiple sampled approach trials in simulation, where execution outcomes determined which regions are executable for the robot. The evaluation was performed on a household objects dataset using two parallel-jaw gripper configurations on a UR5e robot arm. The framework outperformed a model that does not receive information about the robot. The experimental results demonstrated that affordance prediction must be conditioned on the robot and that human annotations alone are insufficient for learning robot-executable affordance representations.
Sawera Yaseen, Serge Thill, Rekha Raja
School of Artificial Intelligence, Donders Centre for Cognition, Department of Human-Centred Intelligent Systems, Radboud University, Nijmegen, The Netherlands.
Affordance understanding bridges visual perception and physical action, serving as an explainable interface for robot manipulation in open and unstructured real-world environments. Yet, building an affordance foundation model that not only understands where and how the interaction should happen, but also generalizes across diverse environments, objects, and tasks, remains a long-standing research challenge. Existing methods typically address only part of this challenge, either localizing task-relevant regions without specifying executable motion, or predicting motion but with limited scalability. In this paper, we present ourmodel, a step towards an affordance foundation model for functionality understanding. From a single RGB-D observation and a language task description, ourmodel predicts a task-conditional functional mask (where to interact) and a 3D post-contact motion curve (how to interact). To support open-world generalization, we build a large-scale standardized data pipeline that converts heterogeneous robot, human, simulation, and real-world scan data into a shared affordance schema with language, masks, and object-centric 3D motion labels. We evaluate ourmodel from three aspects: for affordance segmentation, ourmodel outperforms all baselines by a large margin across 8 test sets from 4 benchmarks, improving mean gIoU/cIoU by +23.9/+26.3; for contact-point prediction, it predicts substantially more accurate points, with a 12.7--61.3% hit-rate gain over the best baseline; and for 3D motion, it achieves the best performance on all three test sets. ourmodel can be deployed for real-world robot manipulation without finetuning for robot embodiment or using task-specific heuristics, demonstrating the ability to adapt to open-world affordance tasks. Project page: https://www.zhaoningwang.com/AFUN
Zhaoning Wang, Yi Zhong, Jiawei Fu +2
University of Michigan · University of California, San Diego · NVIDIA