RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation
Authors: Zhanqi Xiao, Ruiping Wang, Xilin Chen
Organizations: The authors are with the Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences (CAS), Beijing, 100190, China, and also with the University of Chinese Academy of Sciences, Beijing 100049, China.
Understanding spatial affordances -- comprising the contact regions of object interaction and the corresponding contact poses -- is essential for robots to effectively manipulate objects and accomplish diverse tasks. However, existing spatial affordance prediction methods mainly focus on locating the contact regions while delegating the pose to independent pose estimation approaches, which can lead to task failures due to inconsistencies between predicted contact regions and candidate poses. In this work, we propose RoboPCA, a pose-centered affordance prediction framework that jointly predicts task-appropriate contact regions and poses conditioned on instructions. To enable scalable data collection for pose-centered affordance learning, we devise Human2Afford, a data curation pipeline that automatically recovers scene-level 3D information and infers pose-centered affordance annotations from human demonstrations. With Human2Afford, scene depth and the interaction object's mask are extracted to provide 3D context and object localization, while pose-centered affordance annotations are obtained by tracking object points within the contact region and analyzing hand-object interaction patterns to establish a mapping from the 3D hand mesh to the robot end-effector orientation. By integrating geometry-appearance cues through an RGB-D encoder and incorporating mask-enhanced features to emphasize task-relevant object regions into the diffusion-based framework, RoboPCA outperforms baseline methods on image datasets, simulation, and real robots, and exhibits strong generalization across tasks and categories.
Figures & tables
Fig. 1: Overview of our pipeline. (a) Pose-centered affordance annotations and complementary scene information are extracted from human demonstrations with Human2Afford for pose-centered affordance learning. (b) RoboPCA builds upon a diffusion framework to predict pose-centered affordances, an RGB-D encoder is used to effectively capture both geometry and appearance cues, and mask-enhanced features are incorporated to emphasize task-relevant object regions. (c) The predicted pose-centered affordances are transformed into 6-DoF poses using camera parameters, guiding the robot to complete the task.
Fig. 2: Overview of Human2Afford. (a) Given a human demonstration, we identify the demo description and extract key frames using a hand–object detector and VLMs. Depth and the interaction object mask are then obtained via metric depth estimation and segmentation. (b) Using the 3D hand mesh from a hand pose estimator, we extract the contact pose based on the inter-finger vector and palm normal. (c) Object points are tracked from the pre-contact to the contact frame, and points within the inter-finger contact region are modeled with GMM to extract the contact point.
Fig. 3: Qualitative results on AGD20K. The label ⋆⋆⋆⋆⋆ indicate the predicted contact points of different methods.
Models
SR ↑
NSS ↑
DTM ↓
RAM [ 24 ]
0.1824
0.1892
0.0689
RoboPoint [ 48 ]
0.2138
0.2188
0.0508
VRB [ 1 ]
0.2846
0.2557
0.2069
MOKA [ 26 ]
0.3711
0.3390
0.0331
RoboPCA (ours)
0.4403
0.4083
0.0445
TABLE I: Evaluation results on AGD20K dataset.
Fig. 4: Examples of task settings in simulation and the real world. We evaluate our method on 10 tasks in simulation (a; only 6 shown), and 9 tasks in the real world (b; only 6 shown) across various object categories to validate its effectiveness.
Models
Pick Cup
Stack Blocks
Take Umbrella
Water Plants
Sweep Dirt
Close Laptop
Open Drawer
Close Drawer
Open Microwave
Close Microwave
Avg.
RoboPoint [ 48 ]
84
4
24
12
44
36
24
56
4
72
36.0
VRB [ 1 ]
84
60
32
4
48
44
20
84
8
64
44.8
RAM [ 24 ]
84
40
40
36
64
44
16
60
8
60
45.2
MOKA [ 26 ]
84
36
32
20
48
52
28
88
8
72
46.8
RoboPCA (ours)
88
72
88
44
68
52
36
84
32
84
64.8
TABLE II: Success rate (%) on RLBench Multi-Task setting.
Fig. 5: Qualitative comparison of our model’s predicted contact points and poses with MOKA in real-world settings.
Models
Put Into
Tidy Table
Pour Water
Shake Bell
Play Drum
Open Drawer
Close Drawer
Open Microwave
Close Microwave
Avg.
MOKA [ 26 ]
90
100
80
50
60
50
70
10
80
65.6
RAM [ 24 ]
100
80
60
50
30
70
70
60
80
66.7
RoboPCA (ours)
100
90
70
80
100
70
80
60
100
83.3
TABLE III: Success rate (%) on 9 tasks in Real-world setting.
Robotic manipulation systems commonly learn object affordances from human-annotated datasets, which reflect interaction possibilities for a human hand rather than the physical constraints of a specific robot. For example, a surface region of an object identified as graspable by a human annotator may be geometrically unreachable or mechanically infeasible for a robot, yet this mismatch is rarely addressed explicitly. To tackle this challenge, we proposed a framework that translates human affordance annotations into robot-executable affordance predictions conditioned on the physical properties of the robot. The network predicts which surface regions of an object are physically executable by the robot, given a 3D point cloud of an object and a descriptor encoding the robot's geometry and constraints. The model was trained using human-annotated affordances together with robotic execution outcomes from multiple sampled approach trials in simulation, where execution outcomes determined which regions are executable for the robot. The evaluation was performed on a household objects dataset using two parallel-jaw gripper configurations on a UR5e robot arm. The framework outperformed a model that does not receive information about the robot. The experimental results demonstrated that affordance prediction must be conditioned on the robot and that human annotations alone are insufficient for learning robot-executable affordance representations.
Sawera Yaseen, Serge Thill, Rekha Raja
School of Artificial Intelligence, Donders Centre for Cognition, Department of Human-Centred Intelligent Systems, Radboud University, Nijmegen, The Netherlands.
Understanding object affordances is essential for enabling robots to perform purposeful and fine-grained interactions in diverse and unstructured environments. However, existing approaches either rely on retrieval, which is fragile due to sparsity and coverage gaps, or on large-scale models, which frequently mislocalize contact points and mispredict post-contact actions when applied to unseen categories, thereby hindering robust generalization. We introduce Retrieval-Augmented Affordance Prediction (RAAP), a framework that unifies affordance retrieval with alignment-based learning. By decoupling static contact localization and dynamic action direction, RAAP transfers contact points via dense correspondence and predicts action directions through a retrieval-augmented alignment model that consolidates multiple references with dual-weighted attention. Trained on compact subsets of DROID and HOI4D with as few as tens of samples per task, RAAP achieves consistent performance across unseen objects and categories, and enables zero-shot robotic manipulation in both simulation and the real world. Project website: https://github.com/SEU-VIPGroup/RAAP.
Qiyuan Zhuang, He-Yang Xu, Yijun Wang +3
School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Nanjing 211189, China · Southeast University-Monash University Joint Graduate School, Southeast University, Suzhou 215123, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
Affordance-guided imitation learning has shown impressive performance in robotic manipulation tasks by compressing visual perception into task-specific geometric constraints (e.g., fixed contact points). However, the commonly used static affordances can become inconsistent in precision-critical tasks or under object location perturbations, leading to post-contact trajectory drift. To address this issue, we propose AffordTrajDP, a dynamic framework that constructs affordance trajectories via object-centric temporal propagation to guide the progressive manipulation process. Specifically, given an RGB-D observation, our core insight is that a retrieved anchor affordance, which captures the desired contact point between the end-effector and the target object, can be propagated forward via affordance propagation, using the object's SE(3) pose as a natural propagation medium, to yield an affordance trajectory that provides temporally consistent, state-aware guidance throughout execution. AffordTrajDP achieves 70.0% average success rate on ManiSkill3, outperforming strong baselines by up to 17.8%. Real-world experiments on Galaxea A1 and UR7e robotic arms, covering StackCube, PickCup, AdapterInsertion, Ring-on-Peg, Put-in-Bowl, and USB Insertion, further validate robustness under object placement variations and appearance changes, with seen and unseen object instances evaluated on Galaxea A1, and ablations confirm the contribution of each proposed component.