Gen2Dex: Learning Dexterous Manipulation from Simulation-Initialized Generated Video Demonstrations
Authors: Xiaowen Qiu, Yian Wang, Haoxiang You, Chuang Gan
Organizations: Xiaowen Qiu, Yian Wang, and Chuang Gan are with the University of Massachusetts Amherst, Amherst, MA 01003, USA. · Haoxiang You is with the Department of Mechanical Engineering, Yale University, New Haven, CT 06520, USA.
Human video demonstrations provide rich manipulation strategies for dexterous manipulation learning. We study a setting in which a fully specified simulated scene is used to generate videos that specify the demonstrated manipulation behavior. This setup provides access to scene information such as object geometry, articulation, and camera parameters, while allowing direct control over task and scene configuration. However, generated videos may contain geometric distortions, inaccurate contacts, and other physical inconsistencies, making the recovered motion difficult to execute directly on a robot. We present an end-to-end framework for converting such simulation-grounded videos into physically executable dexterous robot demonstrations. Our perception pipeline leverages the known scene through model-based object tracking with dense foundation-model features and depth-based hand-object alignment. The recovered human motion is then retargeted and physically grounded using residual reinforcement learning, where a backward curriculum and a cache of physically valid reset states enable the policy to correct imperfect video-derived references while preserving the demonstrated manipulation strategy. On clean ARCTIC demonstrations, our method achieves competitive performance with prior dexterous retargeting approaches and shows greater robustness when applied to references extracted from generated videos. We further demonstrate diverse rigid and articulated manipulation tasks in simulation and successful open-loop transfer of the optimized trajectories to a real dexterous robot.
Figures & tables
Fig. 2: Overview of our pipeline for generating physically executable dexterous robot demonstrations from generated videos.
Method
FoundationPose
Ours
IOU
0.562
0.785
Recall
0.741
0.953
TABLE I: Object-tracking performance on generated videos.
Perception
Trajectory optimizer
Tasks solved ( ↑ )
Do-as-I-Do
Do-as-I-Do MPC
1/6
Do-as-I-Do
Our residual RL
2/6
Ours
Our residual RL
6/6
TABLE II: Perception and trajectory optimization comparison on six rigid-object tasks.
Method
ARCTIC
Generated videos
DexMachina
62.40
38.20
SPIDER
63.51
67.06
CHORD
56.30
41.15
Ours
65.75
76.18
TABLE III: Mean ADD-AUC ( ↑ ) on clean ARCTIC and generated-video references, averaged over all evaluated sequences in each set.
Fig. 3: Qualitative comparison on our generated-video tasks. Each example shows the generated demonstration followed by the best rollout frame from DexMachina, SPIDER, CHORD, and ours.
Fig. 4: Qualitative simulation results. Each example shows a generated video demonstration and keyframes from the corresponding policy rollout.
Human demonstrations offer rich examples of precise dexterous manipulation and a promising source of robot training data. However, high-fidelity reproduction of demonstrated motions and hand-object interactions across robot embodiments remains challenging under physical constraints. We present DexForge, a differentiable physics-grounded framework for converting human video demonstrations into high-fidelity robot trajectories. We reconstruct spherical-Gaussian object models and hand-object motion from visual observations, then build a differentiable simulator combining efficient Gaussian collision detection with existing differentiable dynamics. Based on this simulator, DexForge combines contact-aware kinematic retargeting with force-aware dynamics retargeting: robot-adapted stable contacts guide kinematic reference construction and subsequent gradient-based control refinement for precise physical motion reproduction. Experiments on 130 DexYCB and HOT3D demonstrations across seven dexterous hands show success-rate gains of approximately 35-53 percentage points over the baseline, with object position and orientation tracking errors on successful trajectories reduced by approximately 34-67% and 71-78%, respectively. Further experiments demonstrate open-loop transfer to MuJoCo and real-robot execution. Our project page is available at https://wmz1226.github.io/DexForge/
Meizhong Wang, Kun Cao, Ruiqi Ni +2
Tongji University · Shanghai Research Institute for Intelligent Autonomous Systems · Purdue University +1
Human demonstrations are a scalable data source for learning dexterous manipulation, but the embodiment gap prevents human motion from being executed directly on robots. Inverse kinematics (IK) retargets human motion to robots efficiently but ignores dynamics, often producing infeasible motions. Reinforcement learning (RL) and sampling-based model predictive control (MPC) are commonly employed to yield dynamically feasible motions, but both are sample-inefficient and sensitive to hyperparameters. RL suffers from costly and unstable training and tedious reward engineering; MPC avoids policy optimization, yet retargets each trajectory in isolation, and solving one does not make the next easier. Sampling cost grows rapidly with dataset size and task difficulty. We hypothesize that dynamically feasible trajectories concentrate near a low-dimensional manifold shared across demonstrations, so that retargeting can be reduced to sampling from that manifold, conditioned on human motion, rather than solving a fresh optimization problem for every demonstration. We propose Generative Neural Retargeting (GNR), which uses a flow matching model to sample feasible trajectories. GNR outperforms MPC with only 8.5% of the samples required by MPC, achieving a success rate of 56.20% compared to 27.20% for MPC. GNR can be used for scalable and efficient retargeting of large-scale, long-horizon, and millimeter precision human demonstrations: by applying GNR within a real-to-sim data engine, we produce a dexterous manipulation dataset with dense contact-force labels, spanning 223k demonstrations and 3.3k object geometries.
Dechen Gao, Yue Yang, Ben Abbatematteo +8
Meta Reality Labs Research · UC Davis · UNC Chapel Hill +3
While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.
Jusuk Lee, Sungha Kim, Yeonsoo Park +8
Seoul National University · University of Maryland, College Park · All Purpose AI +1