HandMimicX: Learning Human-Like Dexterous Manipulation via an Object-Centric Policy from Egocentric Videos

Zihan Gao; Yufei Liu; Yuhong Zhang; Zhen Yu; Xinhao Huang; Zhinan You; Kai W. Ho; Zipeng Ye

cs.ROOct 30, 2026

HandMimicX: Learning Human-Like Dexterous Manipulation via an Object-Centric Policy from Egocentric Videos

Authors: Zihan Gao, Yufei Liu, Yuhong Zhang, Zhen Yu, Xinhao Huang, Zhinan You, Kai W. Ho, Zipeng Ye,

Organizations: Tsinghua University · TsingTec · MeiTuan · The Chinese University of Hong Kong, Shenzhen ·

Abstract

Learning dexterous manipulation from human videos remains an open challenge, particularly in the context of transferring a single demonstration into a spatially generalizable robotic skill and maintaining physically plausible human-like interactions. Unlike prior works that emphasize hand-motion imitation, we introduce an object-centric framework for acquiring dexterous skills from egocentric human RGB-D videos. Within our framework, we formulate the reinforcement learning (RL) objective in terms of object motion to decouple the underlying skill from the specific hand trajectory and impose constraints on hand-object interaction states to promote more anatomically natural hand configurations. Extensive experiments in simulation and on real robots demonstrate that our method enables efficient learning from human videos and generalizable skill transfer across novel spatial configurations.

Figures & tables

Fig. 2 : Overview of HandMimicX . (a1) Given an RGB-D egocentric video, we reconstruct the manipulated object and hand motion, and then align both reconstructions with the depth observations. (a2) We extract the hand-object interaction state at the first contact frame as the interaction prior $\boldsymbol{g}_{\mathrm{hand}}^{*}$ for policy learning. (b1) We train an object-centric teacher policy $\boldsymbol{\pi}^{\mathrm{T}}$ using PPO with SAPG. A correspondence-aware encoder maps the current and goal object point clouds to a geometric latent representation, which is concatenated with $\boldsymbol{o}_{t}^{\mathrm{prop}}$ , $\boldsymbol{a}_{t-1}$ and $\boldsymbol{d}^{\text{h,o}}$ to form the observation $\boldsymbol{o}_{t}^{\mathrm{T}}$ and then fed into the actor and critic MLPs. (b2) During training, the teacher $\boldsymbol{\pi}^{\mathrm{T}}$ learns to manipulate the object and track arbitrary goal poses. (b3) We use DAgger to distill the teacher $\boldsymbol{\pi}^{\mathrm{T}}$ into a student $\boldsymbol{\pi}^{\mathrm{S}}$ under partial observations $\boldsymbol{o}_{t}^{\mathrm{S}}$ . Finally, (c) we deploy the distilled student policy on the real robot to execute diverse trajectories, extending beyond the original video demonstration.

Fig. 3 : Visualization of contact and hand pose retargeting. We use HACO to estimate contact regions on the MANO hand mesh and transfer them to the robotic hand for contact retargeting. Hand pose retargeting is performed using [ 41 ] .

Fig. 4 : Visualization of sampled objects for policy pretraining. The collection spans diverse categories, including everyday objects and tools, with varied geometries.

Policy	Nail Hammering	Umbrella Hanging	Debris Sweeping	Upright Sprayer	Cup Relocation	Bowl Stacking	Eggplant Placement	Pepper Lifting	Avg.
SimToolReal	39.2	48.2	42.3	64.6	64.4	69.0	77.1	53.6	57.3
Dex4D	57.8	42.7	75.3	78.4	69.9	58.0	89.6	62.5	66.8
HandMimicX	90.5	83.9	85.7	81.9	84.9	75.2	92.9	87.8	85.4

TABLE I : Success rates (%) across eight manipulation tasks in simulation, with averages in the final column.

Fig. 5 : Visualization of hand–object interaction poses from SimToolReal, Dex4D, and HandMimicX. Our HandMimicX produces the most anatomically natural hand poses.

Policy	Nail Hammering	Umbrella Hanging	Debris Sweeping	Upright Sprayer	Cup Relocation	Bowl Stacking	Eggplant Placement	Pepper Lifting	Avg.
w/o Pose Prior	78.3	64.1	79.4	64.5	69.7	76.6	92.5	75.9	75.1
w/o Correspondence	85.8	81.7	82.3	72.1	78.5	64.4	87.4	74.2	78.3
w/o Pretraining	79.7	71.6	62.2	71.3	75.0	61.8	90.6	74.1	73.3
HandMimicX	90.5	83.9	85.7	81.9	84.9	75.2	92.9	87.8	85.4

TABLE II : Ablation study evaluating the contributions of the interaction-pose prior, point-correspondence module, and policy pretraining to the overall manipulation performance of our framework. Results are reported as task success rates (%) across different ablation settings, with averages in the final column.

Nail Hammering	Umbrella Hanging	Debris Sweeping	Upright Sprayer	Cup Relocation	Bowl Stacking	Eggplant Placement	Pepper Lifting	Avg. (%)
8/10	7/10	7/10	6/10	7/10	6/10	8/10	8/10	71.3

TABLE III : Skill transfer evaluation across eight real-world manipulation tasks. Each cell reports successful trials / total trials. The final column shows the average success rate.

Fig. 6 : Visualization of real-world execution with HandMimicX. Object trajectories extracted from human demonstration videos serve as targets for robot execution (please refer to the accompanying video for further visualizations).

Fig. 7 : Real-world hardware setup and objects used in our experiments. The system comprises a 6-DoF UR5 robotic arm equipped with a 20-DoF Wuji hand and a single side-mounted Intel RealSense D435 stereo camera.

Fig. 8 : Skill transfer from a single object trajectory extracted from egocentric video. Panels (a) and (b) show executions from different initial object positions and orientations. Panel (c) visualizes the 128 canonical surface keypoints used to represent the object state for the actor.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,765 papers

DexCast: Learning Real-World Dexterous Control from Human Videos
Oct 10, 2026Zihan Gao, Yuhong Zhang, Mengzhe Wang +7Robotic Manipulation Robot Imitation Learning
Tsinghua University · TsingTec · Shenzhen University +2
Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration
Oct 8, 2026Jusuk Lee, Sungha Kim, Yeonsoo Park +8Robotic Manipulation Learning from Demonstration
Seoul National University · University of Maryland, College Park · All Purpose AI +1
Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video
Jun 7, 2026Yunhai Han, Jianuo Qiu, Linhao Bai +14Robot Skill Learning Sim-to-Real Transfer
1Georgia Institute of Technology · University of Pennsylvania · 3Toyota Research Institute +1

Oct 10, 2026J/K move · Enter open · S save

DexCast: Learning Real-World Dexterous Control from Human Videos

Zihan Gao, Yuhong Zhang, Mengzhe Wang, Kaisheng Liu+6

Tsinghua University · TsingTec · Shenzhen University+2

Robotic Manipulation Robot Imitation Learning Motion Representation Learning

Open paper AskPDF arXiv

Recent advances in 3D hand-object interaction (HOI) perception have opened new opportunities for learning dexterous robotic skills from human videos. However, existing dexterous imitation learning methods often rely on costly and manually teleoperated robot demonstrations, requiring substantial human effort. Therefore, we build DEXCAST, a framework that learns dexterous robotic hand control directly from monocular RGB-D videos by leveraging joint HOI reconstructions, without requiring any teleoperation. Furthermore, how to model high-dimensional robotic hand motion and extract temporal context to inform future action generation remains a key challenge. To this end, we propose a DexHand Motion Perception Module (DMPM) to capture inter-joint coordination and temporal dependencies in compact latent representations that condition our diffusion policy. Extensive real-world experiments demonstrate that DEXCAST achieves exceptional performance across diverse dexterous manipulation tasks.

Abstract

Robotic Manipulation

Explore similar work

DexCast: Learning Real-World Dexterous Control from Human Videos

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video