Affordance Transfer Across Object Instances via Semantically Anchored Functional Map
Authors: Xiaoxiang Dong, Weiming Zhi
Organizations: Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA. · School of Computer Science, The University of Sydney, Australia. · Australian Centre for Robotics, The University of Sydney, Australia. · College of Connected Computing, Vanderbilt University, TN, USA.
Traditional learning from demonstration (LfD) generally demands a cumbersome collection of physical demonstrations, which can be time-consuming and challenging to scale. Recent advances show that robots can instead learn from human videos by extracting interaction cues without direct robot involvement. However, a fundamental challenge remains: how to generalize demonstrated interactions across different object instances that share similar functionality but vary significantly in geometry. In this work, we propose Semantic Anchored Functional Maps (SemFM), a framework for transferring affordances across objects from a single visual demonstration. Starting from a coarse mesh reconstructed from an image, our method identifies semantically corresponding functional regions between objects, selects mutually exclusive semantic anchors, and propagates these constraints over the surface using a functional map to obtain a dense, semantically consistent correspondence. This enables demonstrated interaction regions to be transferred across geometrically diverse objects in a lightweight and interpretable manner. Experiments on synthetic object categories and real-world robotic manipulation tasks show that our approach enables accurate affordance transfer with modest computational cost, making it well-suited for practical robotic perception-to-action pipelines.
Figures & tables
Fig. 1 : Overview of the proposed Semantic Anchored Functional Map pipeline. Starting from a single RGB observation of a demonstrated interaction, we reconstruct a coarse object mesh and extract the demonstrated affordance region. Semantic features are used to identify corresponding anchor regions across objects, which constrain a functional map to produce a smooth dense correspondence. The transferred affordance region on the target object is then used to generate a feasible grasp, enabling execution on a real robot.
Fig. 2 : Illustration of functional maps. Given two related meshes (cat and tiger), functional maps recover a smooth correspondence that aligns semantically similar regions while maintaining spatial coherence over the surface.
Fig. 3 : Overview of the proposed Semantic Anchored Functional Map pipeline. Given a single RGB observation of a demonstrated hand-object interaction and another object, we first construct coarse object meshes and extract the demonstrated affordance region. Semantic features are extracted using pretrained embeddings, lifted into 3D, and then used to identify corresponding anchor regions across objects, which constrain a functional map to produce smooth, dense correspondences. This then enables the affordance region to be transferred.
Fig. 4 : Pipeline acquiring coarse 3D mesh and affordance from single-view image. The mesh is inferred from an RGB image and object mask using SAM3D. The hand mask is lifted to 3D and intersected with the mesh to obtain the affordance region.
Fig. 5 : Example of semantic anchor selection. Each mesh is clustered into five regions. The top two mutually exclusive cluster pairs are selected as semantic anchors based on cross-object similarity: the first anchor pairs cluster 2 of object 1 with cluster 1 of object 2 (similarity 0.89), and the second pairs cluster 3 of object 1 with cluster 4 of object 2 (similarity 0.85).
Fig. 6 : Qualitative results between our proposed SemFM against affordance transfer baselines.
Fig. 7 : Visualization of selected six categories. From top-left to bottom-right: Category 1 containing tools with handles; Category 2 containing shoes; Category 3 containing cutting tools; Category 4 containing cups, kettles, and water bottles; Category 5 containing controllers; Category 6 containing scissor-type tools.
Fig. 8 : GPT-4o comparison pipeline shown. Here, the results are slightly erroneous.
Fig. 9 : Accuracy-runtime trade-off of multi-view VLM prompting. Solid curves show GPT-4o performance as the number of rendered views increases, while dotted horizontal lines indicate the performance of our method for each category.
Fig. 10 : Illustration of Semantic-Anchored-Functional-Map performance in real-world. The leftmost image of the top row illustrates the setup used in real world; the rest of top row images show affordance demonstrations that are transferred from; the middle row images show the target real-world object to be transferred to; the last row images show the transferred affordance as a grasp executed by a robot arm with parallel-jaw gripper.
Category
C. 1
C. 2
C. 3
C. 4
C. 5
C. 6
FM-WKS
0.03
0.10
0.05
0.07
0.15
0.11
VLM (1 POV)
0.48
0.40
0.45
0.45
0.42
0.47
VLM (6 POV)
0.68
0.65
0.69
0.56
0.68
0.56
Ours
0.70
0.60
0.69
0.53
0.65
0.62
TABLE I : Quantitative comparison on synthetic categories.
Transferring manipulation skills across object instances that share functionality but differ in geometry remains a fundamental challenge in robot learning. While recent correspondence methods leverage dense visual descriptors and 3D feature fields, nearest-neighbor feature matching often produces spatially incoherent correspondences that fail to recover the local geometric frames required for reliable skill transfer. We introduce SemAnCorr, a training-free framework that establishes dense correspondence by selecting semantically consistent anchor regions through joint pose-correspondence optimization and propagating these constraints over the object surface using functional maps. The resulting correspondences preserve both semantic consistency and geometric coherence, enabling object-centric manipulation skills to transfer across geometrically diverse instances. We evaluate SemAnCorr on a dense correspondence benchmark built on PartNet-Mobility, achieving 90.8% semantic accuracy in our benchmark evaluation while improving geometric coherence over recent state-of-the-art baselines. Finally, we show that these improvements translate directly into real-world manipulation performance: using a single demonstration, SemAnCorr enables substantially more reliable zero-shot manipulation skill transfer to previously unseen objects than existing correspondence methods. Videos and additional visualizations are available at https://semancorr.github.io .
Xiaoxiang Dong, William Baron, Hongyi Chen +3
Robotics Institute, Carnegie Mellon University, USA. · College of Connected Computing, Vanderbilt University, USA. · College of Engineering, University of California, Berkeley, USA. +1
Generalizing robotic manipulation to unseen objects remains challenging, as learning-based approaches require many demonstrations and fail in few-shot settings. Prior work transfers affordances through semantic retrieval, but semantics alone neglect geometric similarity, which is critical for manipulation. We propose GRAFT, a geometry-aware correspondence framework for zero-shot manipulation transfer using only one demonstration per object. Objects are represented as part-based graphs, where part-level descriptors support global instance retrieval and part correspondence, and vertex-level descriptors enable fine-grained contact point matching. For an unseen object, our method first retrieves the most functionally and geometrically similar instance from the demonstration buffer with aligned functional parts, and finally propagates the contact points through point-wise correspondence.
We present Actron3D, a framework that enables robots to acquire transferable 6-DoF manipulation skills from monocular, uncalibrated, RGB-only human demonstration videos. Our key idea is to represent manipulation knowledge within a video as a continuous neural function over object space. At the core of Actron3D lies the Neural Affordance Function, which distills geometry, visual features, contact priors, and action flows from diverse demonstration videos into a compact 3D neural representation. During deployment, we adopt a hierarchical pipeline that retrieves the matched affordance function and transfers encoded manipulation knowledge to novel objects through coarse-to-fine differentiable optimization. Leveraging the continuous nature of Neural Affordance Function, the framework performs spatial queries over multimodal features to align demonstrations with observations and generates precise 6-DoF manipulation policy. Experiments in both simulation and the real-world demonstrate that Actron3D significantly outperforms prior methods, achieving a 14.9 percentage point improvement in the average success rate across 13 tasks while requiring only 2-3 demonstration videos per task.