DexForge: High-Fidelity Physics-Informed Dexterous Retargeting

Meizhong Wang; Kun Cao; Ruiqi Ni; Lihua Xie; Yiguang Hong

cs.ROOct 5, 2026

DexForge: High-Fidelity Physics-Informed Dexterous Retargeting

Authors: Meizhong Wang, Kun Cao, Ruiqi Ni, Lihua Xie, Yiguang Hong

Organizations: Tongji University · Shanghai Research Institute for Intelligent Autonomous Systems · Purdue University · Nanyang Technological University, Singapore

Abstract

Human demonstrations offer rich examples of precise dexterous manipulation and a promising source of robot training data. However, high-fidelity reproduction of demonstrated motions and hand-object interactions across robot embodiments remains challenging under physical constraints. We present DexForge, a differentiable physics-grounded framework for converting human video demonstrations into high-fidelity robot trajectories. We reconstruct spherical-Gaussian object models and hand-object motion from visual observations, then build a differentiable simulator combining efficient Gaussian collision detection with existing differentiable dynamics. Based on this simulator, DexForge combines contact-aware kinematic retargeting with force-aware dynamics retargeting: robot-adapted stable contacts guide kinematic reference construction and subsequent gradient-based control refinement for precise physical motion reproduction. Experiments on 130 DexYCB and HOT3D demonstrations across seven dexterous hands show success-rate gains of approximately 35-53 percentage points over the baseline, with object position and orientation tracking errors on successful trajectories reduced by approximately 34-67% and 71-78%, respectively. Further experiments demonstrate open-loop transfer to MuJoCo and real-robot execution. Our project page is available at https://wmz1226.github.io/DexForge/

Figures & tables

Figure 1: Overview of DexForge. (a) Hand and object reconstruction builds spherical Gaussian object models under RGB-D supervision and recovers hand–object motion from demonstration videos. (b) Contact-aware kinematic retargeting refines demonstrated contacts for robot reachability and contact stability, generating penetration-free references that track them. (c) Force-aware dynamics retargeting optimizes controls through simulation gradients in ComFree-GS, which combines efficient differentiable Gaussian collision detection with ComFree dynamics. System identification further reduces target-system mismatch to support high-fidelity physical reproduction.

Figure 2: Quantitative results on all 20 objects in DexYCB (top); front-view rendering and distance-field visualization for a representative object (bottom).

Figure 3: Dexterous hands and diverse objects used in evaluation. DexForge is evaluated on diverse objects and seven robotic hands that differ in size, degrees of freedom, and finger count.

Figure 4: Visualization of selected kinematic retargeting results from Contact-aware and IK.

Figure 5: Qualitative results of force-aware dynamics retargeting. (a) Motion reproduction by our method and baselines relative to the MANO reference. (b) Retargeting across seven robotic hands.

Figure 6: Sim-to-real open-loop transfer. Controls optimized in ComFree-GS are directly replayed on a Unitree G1 equipped with a LEAP Hand v1.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Method	Object pose		Hand accuracy		Geometric consistency
		Translation (mm)	Rotation ( ∘ )	World-frame MPJPE (mm)	Wrist-aligned MPJPE (mm)	Penetration depth (mm)	Penetrating frames (%)
DexYCB	GT	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$	$3.15\pm 1.79$	$59.5\pm 13.6$
DexYCB	Ours	$0.40\pm 0.16$	$0.46\pm 0.38$	$6.42\pm 1.18$	$4.82\pm 0.86$	$0.24\pm 0.15$	$3.9\pm 8.6$
HOT3D	GT	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.00\pm 0.00$	$6.13\pm 3.04$	$97.8\pm 7.3$
HOT3D	Ours	$12.23\pm 3.11$	$4.83\pm 0.78$	$23.47\pm 2.34$	$9.21\pm 1.21$	$1.38\pm 0.21$	$7.8\pm 4.7$

Appendix

Table 5: Hand–object reconstruction results on real multi-view videos from DexYCB and HOT3D (mean

\pm

SD across sequences). GT denotes dataset annotations. Lower is better for all metrics.

Figure 7: Visualization of the reconstructed hand motion and the estimated object pose on a DexYCB sequence, overlaid on the original video from one camera view. Columns show four evenly spaced frames in temporal order. Top: the reconstructed 21 hand keypoints projected into the image. Bottom: the reconstructed MANO hand mesh and the estimated object pose, visualized as the green 3D bounding box of the object’s Gaussian model.

Figure 8: Forward collision-query time with and without BVH for 1–64 parallel environments on an NVIDIA RTX 4090D GPU.

Panel of Figure 9 — Figure 9: Performance–runtime comparison between DexForge and Contact-aware+SPIDER. Object position (a) and rotation (b) tracking errors are plotted against optimization time, with runtime varied through the optimization iteration limit of each method. Curves show the mean over three runs and shaded regions indicate the standard deviation. Under the same runtime budget, DexForge consistently achieves lower tracking errors.

Panel of Figure 10 — Figure 10: Sensitivity to control-knot spacing under a matched $200\,\mathrm{s}$ optimization budget and $H=4$ control intervals. Errors are computed on a fixed subset of ablation sequences successfully retargeted by both methods under their default configurations; at other spacings, all runs on this subset are included regardless of success. Panels show object position (a) and rotation (b) tracking errors. Curves show mean errors over $3$ runs, with shaded regions indicating $\pm 1$ standard deviation. Contact-aware+SPIDER is largely insensitive to knot spacing, whereas DexForge achieves lower errors at finer spacings but degrades as the spacing and prediction-window duration increase, consistent with greater difficulty in gradient propagation through longer rollouts.

Explore similar work

FlashDexRetarget: Accelerating Dexterous Manipulation Data Generation through Multi-Motion Retargeting
Oct 1, 2026Kyungmin Lee, Sibeen Kim, Dongyoon Hwang +6Cross-Embodiment Retargeting Dexterous Manipulation
KAIST AI · Holiday Robotics
DexSynRefine: Synthesizing and Refining Human-Object Interaction Motion for Physically Feasible Dexterous Robot Actions
May 7, 2026Hyesung Lee, Hyunwoo Jung, Si-Hwan Heo +1Contact-Rich Dexterous Manipulation Dexterous Manipulation
Korea Institute of Science and Technology · KAIST · Hanyang University
C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video
Aug 7, 2026Jie Ren, Zhehao Jiang, Yinhong Yang +9Contact-Rich Dexterous Manipulation Dexterous Manipulation
1Nanjing University, Nanjing, China. · 2China Mobile Research Institute, Beijing, China.

Oct 1, 2026, cs.ROJ/K move · Enter open · S save

FlashDexRetarget: Accelerating Dexterous Manipulation Data Generation through Multi-Motion Retargeting

Kyungmin Lee, Sibeen Kim, Dongyoon Hwang, Yoonsang Oh+5

KAIST AI · Holiday Robotics

Cross-Embodiment Retargeting Dexterous Manipulation Critic Learning Flash

Open paper AskPDF arXiv

Human hand-object demonstrations provide a scalable source of data for dexterous robot learning, but transferring them across embodiments requires physically feasible retargeting. Existing physics-based methods typically optimize each demonstration independently, leading to either limited success under finite simulation budgets or training costs that grow with dataset size. We introduce FlashDexRetarget, an RL framework for multi-reference dexterous retargeting. We formulate retargeting as multi-reference tracking, jointly learning a single policy across many demonstrations with off-policy RL and geometric supervision of the demonstrated interactions. This shared training formulation amortizes optimization across references while enabling the policy to track diverse hand-object interactions. On a 50-motion benchmark from TACO, OakInk2, and HOT3D using XHand and Sharpa Wave Hand as target embodiments, FlashDexRetarget retargets 90% of demonstrations using about 30 GPU-hours, compared with about 46% at about 3,000 GPU-hours for CHORD. This corresponds to about 100 times lower training compute and a 44-percentage-point improvement in retargeting success. Ablations examine the key design choices, while experiments with up to 1,000 motions and real-world replay further demonstrate the scalability and practical applicability of our method.

Abstract

Explore similar work

FlashDexRetarget: Accelerating Dexterous Manipulation Data Generation through Multi-Motion Retargeting

DexSynRefine: Synthesizing and Refining Human-Object Interaction Motion for Physically Feasible Dexterous Robot Actions

C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video