RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignment

Qiyuan Zhuang; He-Yang Xu; Yijun Wang; Xin-Yang Zhao; Yang-Yang Li; Xiu-Shen Wei

cs.ROMar 31, 2026

RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignment

Authors: Qiyuan Zhuang, He-Yang Xu, Yijun Wang, Xin-Yang Zhao, Yang-Yang Li, Xiu-Shen Wei

Organizations: School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Nanjing 211189, China · Southeast University-Monash University Joint Graduate School, Southeast University, Suzhou 215123, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China

Abstract

Understanding object affordances is essential for enabling robots to perform purposeful and fine-grained interactions in diverse and unstructured environments. However, existing approaches either rely on retrieval, which is fragile due to sparsity and coverage gaps, or on large-scale models, which frequently mislocalize contact points and mispredict post-contact actions when applied to unseen categories, thereby hindering robust generalization. We introduce Retrieval-Augmented Affordance Prediction (RAAP), a framework that unifies affordance retrieval with alignment-based learning. By decoupling static contact localization and dynamic action direction, RAAP transfers contact points via dense correspondence and predicts action directions through a retrieval-augmented alignment model that consolidates multiple references with dual-weighted attention. Trained on compact subsets of DROID and HOI4D with as few as tens of samples per task, RAAP achieves consistent performance across unseen objects and categories, and enables zero-shot robotic manipulation in both simulation and the real world. Project website: https://github.com/SEU-VIPGroup/RAAP.

Figures & tables

Fig. 2 : Overview of the Retrieval-Augmented Affordance Prediction (RAAP) framework. (a) Pipeline. Given an RGB-D input and a task label, RAAP retrieves top- $K$ references from an affordance memory using CLIP-based similarity. It then predicts a 2D affordance (contact point and action direction) and lifts it to 3D for execution. (b) 2D Affordance Prediction. Static contact points are localized via dense correspondence using Stable Diffusion (SD) features, while dynamic action directions are inferred by a cross-image alignment module. Both query and reference images are encoded with a shared SigLIP-2 backbone; reference tokens are further modulated by their action vectors via FiLM, and fused with query tokens through gated cross-attention and a Transformer.

Tasks	RAM	A0-170M	RAAP K=0	RAAP K=3
Open microwave	49.82	119.64	51.75	37.00
Close microwave	55.74	103.10	61.38	30.88
Open drawer	51.73	112.89	15.45	16.23
Close drawer	83.04	85.16	52.11	48.25
Pickup bowl	53.85	34.37	36.76	31.45
Pickup bottle	82.62	33.86	38.70	37.67

TABLE I : MAE

\downarrow

for task-aware dynamic affordance prediction. The first six rows correspond to individual manipulation tasks from the DROID dataset, the seventh row reports the averaged performance on HOI4D pickup tasks ( bowl , bottle , mug ). The final row shows the overall average across all tasks.

Fig. 3 : Qualitative comparison of 2D affordance predictions on the DROID dataset. The first row shows the input RGB image with ground-truth affordances (contact point and action direction), and the second row visualizes predictions from RAM, A0, and RAAP ( $K=3$ ).

Variant	Open/Close	Pickup
RAAP (Full)	33.09	34.56
w/o Gating	36.14	42.92
w/o Similarity	39.47	38.96
w/ Uniform Weights	37.63	36.54

TABLE II : Ablation on dual-weighted attention for dynamic affordance prediction (MAE

\downarrow

, degrees). Results are reported on the DROID dataset for two task categories: open/close (e.g., microwave) and pickup (e.g., bowl, bottle).

Fig. 4 : MAE

\downarrow

of RAAP as the number of retrieved references

K

varies from 0 to 4 on the DROID dataset.

Settings	Tasks	RAM	A0	RAAP (Ours)
Unseen Objects	Open drawer	70%	0%	85%
	Close drawer	60%	5%	80%
	Pickup bowl	85%	65%	90%
	Pickup bottle	50%	35%	75%
	Pickup mug	70%	25%	70%
Unseen Categories	Open cabinet	70%	0%	80%

TABLE III : Real-world manipulation success rates (20 trials per task). The top block shows unseen-object generalization, and the bottom block shows cross-category generalization.

Fig. 5 : Qualitative results on pickup kettle in MuJoCo with a UR5e manipulator. RAAP successfully transfers handle-oriented affordances to kettles.

Explore similar work

RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation
Mar 8, 2026Zhanqi Xiao, Ruiping Wang, Xilin ChenRobotic Manipulation
The authors are with the Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences (CAS), Beijing, 100190, China, and also with the University of Chinese Academy of Sciences, Beijing 100049, China.
Cross-Embodiment Affordance Transfer for Robotic Manipulation
Oct 13, 2026Sawera Yaseen, Serge Thill, Rekha RajaRobotic Manipulation Robot Learning
School of Artificial Intelligence, Donders Centre for Cognition, Department of Human-Centred Intelligent Systems, Radboud University, Nijmegen, The Netherlands.
AFUN: Towards an Affordance Foundation Model for Functionality Understanding
Jun 1, 2026Zhaoning Wang, Yi Zhong, Jiawei Fu +2Language-Conditioned Robot Manipulation Robot Foundation Models
University of Michigan · University of California, San Diego · NVIDIA

Mar 8, 2026J/K move · Enter open · S save

RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation

Zhanqi Xiao, Ruiping Wang, Xilin Chen

The authors are with the Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences (CAS), Beijing, 100190, China, and also with the University of Chinese Academy of Sciences, Beijing 100049, China.

Robotic Manipulation

Open paper AskPDF arXiv

Understanding spatial affordances -- comprising the contact regions of object interaction and the corresponding contact poses -- is essential for robots to effectively manipulate objects and accomplish diverse tasks. However, existing spatial affordance prediction methods mainly focus on locating the contact regions while delegating the pose to independent pose estimation approaches, which can lead to task failures due to inconsistencies between predicted contact regions and candidate poses. In this work, we propose RoboPCA, a pose-centered affordance prediction framework that jointly predicts task-appropriate contact regions and poses conditioned on instructions. To enable scalable data collection for pose-centered affordance learning, we devise Human2Afford, a data curation pipeline that automatically recovers scene-level 3D information and infers pose-centered affordance annotations from human demonstrations. With Human2Afford, scene depth and the interaction object's mask are extracted to provide 3D context and object localization, while pose-centered affordance annotations are obtained by tracking object points within the contact region and analyzing hand-object interaction patterns to establish a mapping from the 3D hand mesh to the robot end-effector orientation. By integrating geometry-appearance cues through an RGB-D encoder and incorporating mask-enhanced features to emphasize task-relevant object regions into the diffusion-based framework, RoboPCA outperforms baseline methods on image datasets, simulation, and real robots, and exhibits strong generalization across tasks and categories.

Abstract

Explore similar work

RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation

Cross-Embodiment Affordance Transfer for Robotic Manipulation

AFUN: Towards an Affordance Foundation Model for Functionality Understanding