Affordance Transfer Across Object Instances via Semantically Anchored Functional Map
Authors: Xiaoxiang Dong, Weiming Zhi
Organizations: Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA. · School of Computer Science, The University of Sydney, Australia. · Australian Centre for Robotics, The University of Sydney, Australia. · College of Connected Computing, Vanderbilt University, TN, USA.
Traditional learning from demonstration (LfD) generally demands a cumbersome collection of physical demonstrations, which can be time-consuming and challenging to scale. Recent advances show that robots can instead learn from human videos by extracting interaction cues without direct robot involvement. However, a fundamental challenge remains: how to generalize demonstrated interactions across different object instances that share similar functionality but vary significantly in geometry. In this work, we propose Semantic Anchored Functional Maps (SemFM), a framework for transferring affordances across objects from a single visual demonstration. Starting from a coarse mesh reconstructed from an image, our method identifies semantically corresponding functional regions between objects, selects mutually exclusive semantic anchors, and propagates these constraints over the surface using a functional map to obtain a dense, semantically consistent correspondence. This enables demonstrated interaction regions to be transferred across geometrically diverse objects in a lightweight and interpretable manner. Experiments on synthetic object categories and real-world robotic manipulation tasks show that our approach enables accurate affordance transfer with modest computational cost, making it well-suited for practical robotic perception-to-action pipelines.
Figures & tables
Fig. 1 : Overview of the proposed Semantic Anchored Functional Map pipeline. Starting from a single RGB observation of a demonstrated interaction, we reconstruct a coarse object mesh and extract the demonstrated affordance region. Semantic features are used to identify corresponding anchor regions across objects, which constrain a functional map to produce a smooth dense correspondence. The transferred affordance region on the target object is then used to generate a feasible grasp, enabling execution on a real robot.
Fig. 2 : Illustration of functional maps. Given two related meshes (cat and tiger), functional maps recover a smooth correspondence that aligns semantically similar regions while maintaining spatial coherence over the surface.
Fig. 3 : Overview of the proposed Semantic Anchored Functional Map pipeline. Given a single RGB observation of a demonstrated hand-object interaction and another object, we first construct coarse object meshes and extract the demonstrated affordance region. Semantic features are extracted using pretrained embeddings, lifted into 3D, and then used to identify corresponding anchor regions across objects, which constrain a functional map to produce smooth, dense correspondences. This then enables the affordance region to be transferred.
Fig. 4 : Pipeline acquiring coarse 3D mesh and affordance from single-view image. The mesh is inferred from an RGB image and object mask using SAM3D. The hand mask is lifted to 3D and intersected with the mesh to obtain the affordance region.
Fig. 5 : Example of semantic anchor selection. Each mesh is clustered into five regions. The top two mutually exclusive cluster pairs are selected as semantic anchors based on cross-object similarity: the first anchor pairs cluster 2 of object 1 with cluster 1 of object 2 (similarity 0.89), and the second pairs cluster 3 of object 1 with cluster 4 of object 2 (similarity 0.85).
Fig. 6 : Qualitative results between our proposed SemFM against affordance transfer baselines.
Fig. 7 : Visualization of selected six categories. From top-left to bottom-right: Category 1 containing tools with handles; Category 2 containing shoes; Category 3 containing cutting tools; Category 4 containing cups, kettles, and water bottles; Category 5 containing controllers; Category 6 containing scissor-type tools.
Fig. 8 : GPT-4o comparison pipeline shown. Here, the results are slightly erroneous.
Fig. 9 : Accuracy-runtime trade-off of multi-view VLM prompting. Solid curves show GPT-4o performance as the number of rendered views increases, while dotted horizontal lines indicate the performance of our method for each category.
Fig. 10 : Illustration of Semantic-Anchored-Functional-Map performance in real-world. The leftmost image of the top row illustrates the setup used in real world; the rest of top row images show affordance demonstrations that are transferred from; the middle row images show the target real-world object to be transferred to; the last row images show the transferred affordance as a grasp executed by a robot arm with parallel-jaw gripper.
Category
C. 1
C. 2
C. 3
C. 4
C. 5
C. 6
FM-WKS
0.03
0.10
0.05
0.07
0.15
0.11
VLM (1 POV)
0.48
0.40
0.45
0.45
0.42
0.47
VLM (6 POV)
0.68
0.65
0.69
0.56
0.68
0.56
Ours
0.70
0.60
0.69
0.53
0.65
0.62
TABLE I : Quantitative comparison on synthetic categories.
Robotics Institute, Carnegie Mellon University, USA. · College of Connected Computing, Vanderbilt University, USA. · College of Engineering, University of California, Berkeley, USA. +1