Learning dexterous manipulation from human videos remains an open challenge, particularly in the context of transferring a single demonstration into a spatially generalizable robotic skill and maintaining physically plausible human-like interactions. Unlike prior works that emphasize hand-motion imitation, we introduce an object-centric framework for acquiring dexterous skills from egocentric human RGB-D videos. Within our framework, we formulate the reinforcement learning (RL) objective in terms of object motion to decouple the underlying skill from the specific hand trajectory and impose constraints on hand-object interaction states to promote more anatomically natural hand configurations. Extensive experiments in simulation and on real robots demonstrate that our method enables efficient learning from human videos and generalizable skill transfer across novel spatial configurations.
Figures & tables
Fig. 2 : Overview of HandMimicX . (a1) Given an RGB-D egocentric video, we reconstruct the manipulated object and hand motion, and then align both reconstructions with the depth observations. (a2) We extract the hand-object interaction state at the first contact frame as the interaction prior ghand∗ for policy learning. (b1) We train an object-centric teacher policy πT using PPO with SAPG. A correspondence-aware encoder maps the current and goal object point clouds to a geometric latent representation, which is concatenated with otprop , at−1 and dh,o to form the observation otT and then fed into the actor and critic MLPs. (b2) During training, the teacher πT learns to manipulate the object and track arbitrary goal poses. (b3) We use DAgger to distill the teacher πT into a student πS under partial observations otS . Finally, (c) we deploy the distilled student policy on the real robot to execute diverse trajectories, extending beyond the original video demonstration.
Fig. 3 : Visualization of contact and hand pose retargeting. We use HACO to estimate contact regions on the MANO hand mesh and transfer them to the robotic hand for contact retargeting. Hand pose retargeting is performed using [ 41 ] .
Fig. 4 : Visualization of sampled objects for policy pretraining. The collection spans diverse categories, including everyday objects and tools, with varied geometries.
Policy
Nail Hammering
Umbrella Hanging
Debris Sweeping
Upright Sprayer
Cup Relocation
Bowl Stacking
Eggplant Placement
Pepper Lifting
Avg.
SimToolReal
39.2
48.2
42.3
64.6
64.4
69.0
77.1
53.6
57.3
Dex4D
57.8
42.7
75.3
78.4
69.9
58.0
89.6
62.5
66.8
HandMimicX
90.5
83.9
85.7
81.9
84.9
75.2
92.9
87.8
85.4
TABLE I : Success rates (%) across eight manipulation tasks in simulation, with averages in the final column.
Fig. 5 : Visualization of hand–object interaction poses from SimToolReal, Dex4D, and HandMimicX. Our HandMimicX produces the most anatomically natural hand poses.
Policy
Nail Hammering
Umbrella Hanging
Debris Sweeping
Upright Sprayer
Cup Relocation
Bowl Stacking
Eggplant Placement
Pepper Lifting
Avg.
w/o Pose Prior
78.3
64.1
79.4
64.5
69.7
76.6
92.5
75.9
75.1
w/o Correspondence
85.8
81.7
82.3
72.1
78.5
64.4
87.4
74.2
78.3
w/o Pretraining
79.7
71.6
62.2
71.3
75.0
61.8
90.6
74.1
73.3
HandMimicX
90.5
83.9
85.7
81.9
84.9
75.2
92.9
87.8
85.4
TABLE II : Ablation study evaluating the contributions of the interaction-pose prior, point-correspondence module, and policy pretraining to the overall manipulation performance of our framework. Results are reported as task success rates (%) across different ablation settings, with averages in the final column.
Nail Hammering
Umbrella Hanging
Debris Sweeping
Upright Sprayer
Cup Relocation
Bowl Stacking
Eggplant Placement
Pepper Lifting
Avg. (%)
8/10
7/10
7/10
6/10
7/10
6/10
8/10
8/10
71.3
TABLE III : Skill transfer evaluation across eight real-world manipulation tasks. Each cell reports successful trials / total trials. The final column shows the average success rate.
Fig. 6 : Visualization of real-world execution with HandMimicX. Object trajectories extracted from human demonstration videos serve as targets for robot execution (please refer to the accompanying video for further visualizations).
Fig. 7 : Real-world hardware setup and objects used in our experiments. The system comprises a 6-DoF UR5 robotic arm equipped with a 20-DoF Wuji hand and a single side-mounted Intel RealSense D435 stereo camera.
Fig. 8 : Skill transfer from a single object trajectory extracted from egocentric video. Panels (a) and (b) show executions from different initial object positions and orientations. Panel (c) visualizes the 128 canonical surface keypoints used to represent the object state for the actor.
Recent advances in 3D hand-object interaction (HOI) perception have opened new opportunities for learning dexterous robotic skills from human videos. However, existing dexterous imitation learning methods often rely on costly and manually teleoperated robot demonstrations, requiring substantial human effort. Therefore, we build DEXCAST, a framework that learns dexterous robotic hand control directly from monocular RGB-D videos by leveraging joint HOI reconstructions, without requiring any teleoperation. Furthermore, how to model high-dimensional robotic hand motion and extract temporal context to inform future action generation remains a key challenge. To this end, we propose a DexHand Motion Perception Module (DMPM) to capture inter-joint coordination and temporal dependencies in compact latent representations that condition our diffusion policy. Extensive real-world experiments demonstrate that DEXCAST achieves exceptional performance across diverse dexterous manipulation tasks.
Zihan Gao, Yuhong Zhang, Mengzhe Wang +7
Tsinghua University · TsingTec · Shenzhen University +2
While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.
Jusuk Lee, Sungha Kim, Yeonsoo Park +8
Seoul National University · University of Maryland, College Park · All Purpose AI +1
Human manipulation videos are a convenient and intuitive source for robot learning. However, directly transferring human dexterity to robots remains challenging due to perception errors and embodiment gap. To address this, we introduce Video2Sim2Real, a full-stack framework for autonomous skill acquisition from a single human manipulation video. Our framework first uses off-the-shelf foundation models to reconstruct a simulator-ready digital twin and extract robot and object motion priors. Rather than treating the extracted robot motion as a reliable reference throughout execution, our key idea is to recover and leverage the most fundamental sources of supervision from the demonstrated skill: We identify object-centric keyframes to optimize the corresponding robot configurations using object information from the simulator, and use these configurations as anchors that refine the robot motion such that it ultimately has the desired impact on the environment. To bridge the remaining sim-to-real gap, we introduce a sim-to-real strategy that decouples robustness to noisy and incomplete perception from variations in hand-object interaction dynamics. Specifically, we learn to recalibrate robot configurations from noisy real-world point clouds via IL, and leverage residual RL to perform local finger-level adaptations to ensure for robust and effective interactions. Finally, a collision-aware motion planning module enables spatial generalization to novel object configurations. Across several everyday manipulation tasks, Video2Sim2Real improves simulated task success, safety, and trajectory coherence over numerous baselines, and achieves better sim-to-real transfer than existing techniques. These results demonstrate a promising path toward autonomous dexterous skill acquisition from human videos.
Yunhai Han, Jianuo Qiu, Linhao Bai +14
1Georgia Institute of Technology · University of Pennsylvania · 3Toyota Research Institute +1