Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking

Yuxuan Kuang; Jiashun Wang; Yixuan Sun; Yufei Ye; Sungjae Park; Umar Iqbal; Katerina Fragkiadaki; Shubham Tulsiani

cs.ROOct 20, 2026

Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking

Authors: Yuxuan Kuang, Jiashun Wang, Yixuan Sun, Yufei Ye, Sungjae Park, Umar Iqbal, Katerina Fragkiadaki, Shubham Tulsiani

Organizations: Carnegie Mellon University · NVIDIA

Abstract

Dexterous multi-finger hands hold promise to unlock diverse skills like functional grasps, in-hand reorientation, and dynamic handover that the human-designed world assumes, yet existing learning recipes force a trade-off. Goal-reaching reinforcement learning generalizes to arbitrary object poses but, lacking any human prior, converges to unnatural and unreliable finger strategies; motion tracking reproduces human-like behavior from a single clip but cannot deviate from it. We present AdaDex, a goal-conditioned RL framework that turns a single hand-object trajectory of a few seconds---from mocap data or HOI reconstruction from in-the-wild RGB(D) video---into an adaptable dexterous skill. At the core of AdaDex is Hybrid Dexterous Mimicking: a shared policy is trained on two populations of massively parallel environments: motion tracking environments trained with dense motion tracking objective, which instill the human prior from human motion; and generalization environments with noised initial states and goals and a sparse object-success reward, which force the skill to work far off the reference. From seconds of human motion per skill, the resulting policies can learn diverse adaptable dexterous skills, such as functional grasping, object reposing, in-hand reorientation with finger gaiting, and dynamic behaviors like throw-and-catch. Our method outperforms baselines in terms of generalization and functionality, and we demonstrate transfer across simulators and to the real world. Project page: https://yuxuank.com/AdaDex/.

Figures & tables

Figure 1: AdaDex learns diverse adaptable dexterous skills, such as functional grasping, object reposing, in-hand reorientation, and dynamic behaviors, from seconds of human motion.

Figure 2: Left: Hybrid Dexterous Mimicking is a goal-conditioned RL formulation in which dense reference tracking and sparse goal reaching are trained as parallel environment types under a single shared actor. Right: our method can learn adaptable dexterous skills with a single human motion reconstructed from video and transfer to an unseen simulator (MuJoCo) and the real world.

Setting (i): seen (trained) motions
Method	Progress (%) $\uparrow$	$\bar{e}^{o,\mathrm{p}}$ ( $\mathrm{cm}$ ) $\downarrow$	$\bar{e}^{o,\mathrm{R}}$ ( ∘ ) $\downarrow$	$\bar{e}^{w,\mathrm{p}}$ (cm) $\downarrow$	$\bar{e}^{w,\mathrm{R}}$ ( ∘ ) $\downarrow$
ManipTrans	$35.20\pm 29.71$	$13.69\pm 16.11$	$41.61\pm 40.97$	$45.55\pm 65.28$	$55.22\pm 44.03$
SimToolReal	$9.72\pm 5.44$	$22.15\pm 18.32$	$53.56\pm 26.77$	$21.43\pm 9.79$	$65.30\pm 21.25$
w/o TrackEnv	$8.45\pm 4.98$	$34.10\pm 20.38$	$76.64\pm 31.72$	$3.65\pm 1.81$	$42.52\pm 25.23$
w/o GenEnv	$\bm{97.47\pm 8.40}$	$\bm{0.62\pm 0.29}$	$\bm{2.24\pm 0.79}$	$\bm{2.76\pm 0.51}$	$\bm{8.16\pm 4.73}$
AdaDex	$\underline{96.74\pm 6.21}$	$\underline{0.72\pm 0.32}$	$\underline{4.16\pm 1.19}$	$\underline{2.82\pm 0.61}$	$\underline{11.45\pm 5.26}$

Table 1: Motion tracking performance on (i) seen motions and (ii) similar but unseen motions. Best and second-best mean values for each metric within each setting are bolded and underlined .

Figure 3: Qualitative comparisons. Trained on a single human motion, AdaDex can track seen and unseen motions naturally, while baselines struggle with generalization and functionality.

Figure 5: AdaDex learns diverse skills such as functional grasping, in-hand reorientation, dynamic throw-and-catch, and can achieve various downstream tasks with high dexterity. Our method also works on various robot hand embodiments and real arm-hand systems.

Figure 6: Sim-to-sim and sim-to-real transfer, in real time (1x speed, 30 Hz) , demonstrating physical feasibility of the learned dexterous behaviors.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Overview of our RGB-D hand-object reconstruction pipeline. Given an RGB-D video and camera intrinsics, we recover hand motion with Dyn-HaMR, reconstruct a metric object mesh with SAM 3D, and track its pose with FoundationPose++. A table plane is estimated from depth points using RANSAC. We align hand translations to the observed depth, then apply GeoCalib-based gravity alignment and table-height alignment to express the hand, object, and table in a shared world coordinate frame.

	DexYCB				HO3D				Runtime (s/frame) $\downarrow$
Method	MPJPE (mm) $\downarrow$	$\mathrm{CD}_{h}$ (cm) $\downarrow$	Seq. Success (%) $\uparrow$	Frame Success (%) $\uparrow$	MPJPE (mm) $\downarrow$	$\mathrm{CD}_{h}$ (cm) $\downarrow$	Seq. Success (%) $\uparrow$	Frame Success (%) $\uparrow$	Runtime (s/frame) $\downarrow$
EgoInfinity (RGB-D)	167.81	32.24	100.00	97.64	81.33	15.62	100.00	91.73	27.06
Do-As-I-Do (RGB-D)	42.11	19.11	72.59	73.41	40.88	4.77	92.31	92.03	76.04
Ours	25.03	7.64	100.00	99.53	27.01	2.89	100.00	99.73	0.97

Appendix

Table 3: Hand reconstruction results on DexYCB and HO3D.

Figure 8: Qualitative comparison of hand–object reconstruction. Each row shows four time steps from an interaction, with RGB inputs arranged in reading order and reconstructions ordered from left to right. Do-As-I-Do and EgoInfinity are adapted to use the provided depth maps and calibrated camera intrinsics. Our reconstructions more closely preserve the object orientations and grasp configurations observed in the input videos.

Hyperparameter	Stage I	Stage II
TrackEnv / GenEnv mixture	$1.0\,/\,0.0$	$0.5\,/\,0.5$
Actor–critic MLP	$[1024,1024,512]$ , ELU; separate AC networks
Action log-std at stage start	$-0.5$ ( $\sigma=e^{-0.5}\approx 0.607$ )	restored from Stage I actor
Learning rate (LR)	$1\times 10^{-4}$	$2\times 10^{-5}$
LR warmup	$10$ epochs: $10^{-6}\rightarrow 10^{-4}$	$10$ epochs: $10^{-6}\rightarrow 2\times 10^{-5}$
KL threshold	$0.008$

Appendix

Table 4: Network details and training hyperparameters.

Reward term	TrackEnv	GenEnv
Wrist pose	$e^{-40d_{w}}+6e^{-\theta_{w}}$	$2e^{-40d_{w}}+e^{-3\theta_{w}}$
Object pose	$10e^{-80d_{o}}+5e^{-3\theta_{o}}$	$0$
Finger pose	Weighted exponential matching	$0$
Velocity matching	Weighted exponential matching	$0$
Low power	$e^{-10P_{h}}+e^{-2P_{w}}$	$0$
Action smoothness	$-\operatorname{mean}[(a_{t}-a_{t-1})^{2}]$	$0$

Appendix

Table 5: Reward shaping for tracking and generalization environments.

Object	Motion name	Frames	Time (s)
Setting (i): GRAB seen (trained) motions
Airplane	s1_airplane_fly_1	463	7.72
Alarm clock	s1_alarmclock_pass_1	279	4.65
Apple	s1_apple_eat_1	490	8.17
Bowl	s1_bowl_pass_1	208	3.47
Cup	s1_cup_pour_1	387	6.45

Appendix

Table 6: Benchmark motions: 12 distinct GRAB objects with one seconds-long motion each, 24 unseen GRAB motions (2 for each object), and 9 real-to-sim motions. Frames refer to the frame length of the trimmed motions for training; for unseen GRAB motions, they refer to the evaluated reference movement windows. All motions are in 60 Hz.

Robot Skill Learning

Robot skill learning studies how robots acquire, adapt, and combine reusable behaviors from demonstrations, interaction, and other data.

See all 317 papers

DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation
Oct 13, 2026Zhexi Luo, Zhixuan Xu, Rui Yang +3Robotic Manipulation Robot Learning
National University of Singapore (NUS) · RoboScience
ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning
Aug 19, 2026Jayjun Lee, Jessica Yin, Asif Rana +8Robot Learning Robotic RL
NVIDIA Corporation · Robotics Department, University of Michigan
Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration
Oct 8, 2026Jusuk Lee, Sungha Kim, Yeonsoo Park +8Robotic Manipulation Learning from Demonstration
Seoul National University · University of Maryland, College Park · All Purpose AI +1

Oct 13, 2026J/K move · Enter open · S save

DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation

Zhexi Luo, Zhixuan Xu, Rui Yang, Zeyu Xiao+2

National University of Singapore (NUS) · RoboScience

Robotic Manipulation Robot Learning Robot Imitation Learning Generalization in Robotic Manipulation Robot Manipulation Benchmarks

Open paper AskPDF arXiv

Achieving human-like dexterous manipulation remains a central challenge in robotics. Learning from human hand-object interaction (HOI) datasets offers a promising pathway to diverse dexterous skills, yet existing physics-based approaches train specialist policies for individual trajectories, thereby limiting skill reuse at scale. We introduce DexMimic, a generalist dexterous hand-object tracking policy jointly trained across diverse HOI trajectories. Efficient joint training requires allocating each rollout to both an undertrained phase and a starting state that supports continued tracking. DexMimic realizes this insight through Value-Guided Reset Allocation, which adapts the initialization distribution online to prioritize difficult phases and select reset states with higher expected tracking returns. We further introduce a benchmark spanning training, fine-tuning, and generalization. Experiments show improved tracking success and training efficiency, faster fine-tuning, and the potential for zero-shot generalization to unseen trajectories and objects; larger training sets further improve efficiency and generalization. Finally, we validate DexMimic on a real-world dexterous manipulation platform.

Abstract

Robot Skill Learning

Explore similar work

DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation

ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration