Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking

Yuxuan Kuang; Jiashun Wang; Yixuan Sun; Yufei Ye; Sungjae Park; Umar Iqbal; Katerina Fragkiadaki; Shubham Tulsiani

cs.ROOct 20, 2026

Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking

Authors: Yuxuan Kuang, Jiashun Wang, Yixuan Sun, Yufei Ye, Sungjae Park, Umar Iqbal, Katerina Fragkiadaki, Shubham Tulsiani

Organizations: Carnegie Mellon University · NVIDIA

Abstract

Dexterous multi-finger hands hold promise to unlock diverse skills like functional grasps, in-hand reorientation, and dynamic handover that the human-designed world assumes, yet existing learning recipes force a trade-off. Goal-reaching reinforcement learning generalizes to arbitrary object poses but, lacking any human prior, converges to unnatural and unreliable finger strategies; motion tracking reproduces human-like behavior from a single clip but cannot deviate from it. We present AdaDex, a goal-conditioned RL framework that turns a single hand-object trajectory of a few seconds---from mocap data or HOI reconstruction from in-the-wild RGB(D) video---into an adaptable dexterous skill. At the core of AdaDex is Hybrid Dexterous Mimicking: a shared policy is trained on two populations of massively parallel environments: motion tracking environments trained with dense motion tracking objective, which instill the human prior from human motion; and generalization environments with noised initial states and goals and a sparse object-success reward, which force the skill to work far off the reference. From seconds of human motion per skill, the resulting policies can learn diverse adaptable dexterous skills, such as functional grasping, object reposing, in-hand reorientation with finger gaiting, and dynamic behaviors like throw-and-catch. Our method outperforms baselines in terms of generalization and functionality, and we demonstrate transfer across simulators and to the real world. Project page: https://yuxuank.com/AdaDex/.

Figures & tables

Figure 1: AdaDex learns diverse adaptable dexterous skills, such as functional grasping, object reposing, in-hand reorientation, and dynamic behaviors, from seconds of human motion.

Figure 2: Left: Hybrid Dexterous Mimicking is a goal-conditioned RL formulation in which dense reference tracking and sparse goal reaching are trained as parallel environment types under a single shared actor. Right: our method can learn adaptable dexterous skills with a single human motion reconstructed from video and transfer to an unseen simulator (MuJoCo) and the real world.

Setting (i): seen (trained) motions
Method	Progress (%) $\uparrow$	$\bar{e}^{o,\mathrm{p}}$ ( $\mathrm{cm}$ ) $\downarrow$	$\bar{e}^{o,\mathrm{R}}$ ( ∘ ) $\downarrow$	$\bar{e}^{w,\mathrm{p}}$ (cm) $\downarrow$	$\bar{e}^{w,\mathrm{R}}$ ( ∘ ) $\downarrow$
ManipTrans	$35.20\pm 29.71$	$13.69\pm 16.11$	$41.61\pm 40.97$	$45.55\pm 65.28$	$55.22\pm 44.03$
SimToolReal	$9.72\pm 5.44$	$22.15\pm 18.32$	$53.56\pm 26.77$	$21.43\pm 9.79$	$65.30\pm 21.25$
w/o TrackEnv	$8.45\pm 4.98$	$34.10\pm 20.38$	$76.64\pm 31.72$	$3.65\pm 1.81$	$42.52\pm 25.23$
w/o GenEnv	$\bm{97.47\pm 8.40}$	$\bm{0.62\pm 0.29}$	$\bm{2.24\pm 0.79}$	$\bm{2.76\pm 0.51}$	$\bm{8.16\pm 4.73}$
AdaDex	$\underline{96.74\pm 6.21}$	$\underline{0.72\pm 0.32}$	$\underline{4.16\pm 1.19}$	$\underline{2.82\pm 0.61}$	$\underline{11.45\pm 5.26}$

Table 1: Motion tracking performance on (i) seen motions and (ii) similar but unseen motions. Best and second-best mean values for each metric within each setting are bolded and underlined .

Figure 3: Qualitative comparisons. Trained on a single human motion, AdaDex can track seen and unseen motions naturally, while baselines struggle with generalization and functionality.

Figure 5: AdaDex learns diverse skills such as functional grasping, in-hand reorientation, dynamic throw-and-catch, and can achieve various downstream tasks with high dexterity. Our method also works on various robot hand embodiments and real arm-hand systems.

Figure 6: Sim-to-sim and sim-to-real transfer, in real time (1x speed, 30 Hz) , demonstrating physical feasibility of the learned dexterous behaviors.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Overview of our RGB-D hand-object reconstruction pipeline. Given an RGB-D video and camera intrinsics, we recover hand motion with Dyn-HaMR, reconstruct a metric object mesh with SAM 3D, and track its pose with FoundationPose++. A table plane is estimated from depth points using RANSAC. We align hand translations to the observed depth, then apply GeoCalib-based gravity alignment and table-height alignment to express the hand, object, and table in a shared world coordinate frame.

	DexYCB				HO3D				Runtime (s/frame) $\downarrow$
Method	MPJPE (mm) $\downarrow$	$\mathrm{CD}_{h}$ (cm) $\downarrow$	Seq. Success (%) $\uparrow$	Frame Success (%) $\uparrow$	MPJPE (mm) $\downarrow$	$\mathrm{CD}_{h}$ (cm) $\downarrow$	Seq. Success (%) $\uparrow$	Frame Success (%) $\uparrow$	Runtime (s/frame) $\downarrow$
EgoInfinity (RGB-D)	167.81	32.24	100.00	97.64	81.33	15.62	100.00	91.73	27.06
Do-As-I-Do (RGB-D)	42.11	19.11	72.59	73.41	40.88	4.77	92.31	92.03	76.04
Ours	25.03	7.64	100.00	99.53	27.01	2.89	100.00	99.73	0.97

Appendix

Table 3: Hand reconstruction results on DexYCB and HO3D.

Figure 8: Qualitative comparison of hand–object reconstruction. Each row shows four time steps from an interaction, with RGB inputs arranged in reading order and reconstructions ordered from left to right. Do-As-I-Do and EgoInfinity are adapted to use the provided depth maps and calibrated camera intrinsics. Our reconstructions more closely preserve the object orientations and grasp configurations observed in the input videos.

Hyperparameter	Stage I	Stage II
TrackEnv / GenEnv mixture	$1.0\,/\,0.0$	$0.5\,/\,0.5$
Actor–critic MLP	$[1024,1024,512]$ , ELU; separate AC networks
Action log-std at stage start	$-0.5$ ( $\sigma=e^{-0.5}\approx 0.607$ )	restored from Stage I actor
Learning rate (LR)	$1\times 10^{-4}$	$2\times 10^{-5}$
LR warmup	$10$ epochs: $10^{-6}\rightarrow 10^{-4}$	$10$ epochs: $10^{-6}\rightarrow 2\times 10^{-5}$
KL threshold	$0.008$

Appendix

Table 4: Network details and training hyperparameters.

Reward term	TrackEnv	GenEnv
Wrist pose	$e^{-40d_{w}}+6e^{-\theta_{w}}$	$2e^{-40d_{w}}+e^{-3\theta_{w}}$
Object pose	$10e^{-80d_{o}}+5e^{-3\theta_{o}}$	$0$
Finger pose	Weighted exponential matching	$0$
Velocity matching	Weighted exponential matching	$0$
Low power	$e^{-10P_{h}}+e^{-2P_{w}}$	$0$
Action smoothness	$-\operatorname{mean}[(a_{t}-a_{t-1})^{2}]$	$0$

Appendix

Table 5: Reward shaping for tracking and generalization environments.

Object	Motion name	Frames	Time (s)
Setting (i): GRAB seen (trained) motions
Airplane	s1_airplane_fly_1	463	7.72
Alarm clock	s1_alarmclock_pass_1	279	4.65
Apple	s1_apple_eat_1	490	8.17
Bowl	s1_bowl_pass_1	208	3.47
Cup	s1_cup_pour_1	387	6.45

Appendix

Table 6: Benchmark motions: 12 distinct GRAB objects with one seconds-long motion each, 24 unseen GRAB motions (2 for each object), and 9 real-to-sim motions. Frames refer to the frame length of the trimmed motions for training; for unseen GRAB motions, they refer to the evaluated reference movement windows. All motions are in 60 Hz.

Robot Skill Learning

Robot skill learning studies how robots acquire, adapt, and combine reusable behaviors from demonstrations, interaction, and other data.

See all 317 papers

Oct 13, 2026

DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation

Achieving human-like dexterous manipulation remains a central challenge in robotics. Learning from human hand-object interaction (HOI) datasets offers a promising pathway to diverse dexterous skills, yet existing physics-based approaches train specialist policies for individual trajectories, thereby limiting skill reuse at scale. We introduce DexMimic, a generalist dexterous hand-object tracking policy jointly trained across diverse HOI trajectories. Efficient joint training requires allocating each rollout to both an undertrained phase and a starting state that supports continued tracking. DexMimic realizes this insight through Value-Guided Reset Allocation, which adapts the initialization distribution online to prioritize difficult phases and select reset states with higher expected tracking returns. We further introduce a benchmark spanning training, fine-tuning, and generalization. Experiments show improved tracking success and training efficiency, faster fine-tuning, and the potential for zero-shot generalization to unseen trajectories and objects; larger training sets further improve efficiency and generalization. Finally, we validate DexMimic on a real-world dexterous manipulation platform.

Zhexi Luo, Zhixuan Xu, Rui Yang +3

National University of Singapore (NUS) · RoboScience

Robotic Manipulation Robot Learning Robot Imitation Learning Generalization in Robotic Manipulation Robot Manipulation Benchmarks

Ask PDF

Aug 19, 2026

ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning

We introduce Accelerating Dexterity via Pre-Training (ADEPT), a large-scale reinforcement learning (RL) framework for learning sim-to-real transferable dexterity across high degree-of-freedom (DoF) robot embodiments that can solve long-horizon tasks directly from raw visuo-tactile perception. ADEPT pretrains a dexterous policy on a generic object reposing task, then post-trains downstream policies with this pretrained behavior as a prior. ADEPT enables learning new behaviors that are otherwise difficult to discover from scratch on multi-fingered robots and avoids learning the same set of skills over again for every new downstream task. The pretrained policy zero-shots the reposing phase of downstream tasks, but naïve RL fine-tuning rapidly degrades this capability during transfer. We address this with a stable post-training recipe combining behavior-cloning distillation, critic warm-up, and conservative on-policy updates. To safely exploit the full kinematic dexterity, we introduce a joint-space Geometric Fabric that mediates between the RL policy and the robot. We distill post-trained teachers into perceptive students that zero-shot sim-to-real transfer on two embodiments: a 23 DoF Kuka-Allegro with two RGB cameras, and a 29 DoF Flexiv-Sharpa with two RGB cameras and five vision-based tactile sensors, and can solve long-horizon tasks from challenging initial states with dexterity at human-level speed.

Jayjun Lee, Jessica Yin, Asif Rana +8

NVIDIA Corporation · Robotics Department, University of Michigan

Robot Learning Robotic RL Robotic Manipulation Sim-to-Real Transfer Long-Horizon Robotic Manipulation Cross-Embodiment Robot Learning

Ask PDF

Oct 8, 2026

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.

Jusuk Lee, Sungha Kim, Yeonsoo Park +8

Seoul National University · University of Maryland, College Park · All Purpose AI +1

Robotic Manipulation Learning from Demonstration Zero-Shot Robotic Manipulation RL for Robotics Reinforcement Learning Sim-to-Real Transfer

Ask PDF

Abstract

Robot Skill Learning

Explore similar work

DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation

ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration