cs.ROJun 9, 2026

Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Open-World Contact Localization

Authors: Yitian ShiDi WenZhengqi HanZicheng GuoYu HuEdgar WelteKunyu PengRainer Stiefelhagen+1 more

Organizations: Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

Abstract

Learning from human video demonstrations remains challenging due to noisy hand-object interactions, unseen objects with partial observation, and cross-embodiment discrepancy. To address these challenges, we present \textit{HOWTransfer} (\emph{H}and-\emph{O}bject \emph{O}pen-\emph{W}orld Transfer), a hand-centric framework that distills human demonstrations into contact-aware, taxonomy-informed, and diverse robotic trajectories. Instead of relying on object-specific descriptions, vision-language queries, or explicit object-state tracking, \emph{HOWTransfer} recovers temporally consistent 3D hand motion and localizes temporal contact intervals by reasoning over observed hand-object interaction cues. The localized contact onsets are then used to retarget human grasp intent into multi-modal parallel-jaw grasp hypotheses, which are propagated along the recovered wrist trajectory to generate robot-executable motions. Finally, a trajectory editing stage refines contact alignment and produces diverse executable variants from a single demonstration. Experiments across diverse manipulation tasks show that \emph{HOWTransfer} enables accurate contact localization and high-quality robot motion retargeting with 86%86\% success, which is preferred over teleoperated trajectories in a blinded preference study.

Explore similar work

CardsList
  1. Touch2Robot: Robot Touch in the Human Demonstration Loop

    Sep 21, 2026Shengcheng Luo, Xiaoyang Cheng, Hong Ying +6