DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation

Zhexi Luo; Zhixuan Xu; Rui Yang; Zeyu Xiao; Yiwen Hou; Lin Shao

cs.ROOct 13, 2026

DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation

Authors: Zhexi Luo, Zhixuan Xu, Rui Yang, Zeyu Xiao, Yiwen Hou, Lin Shao

Organizations: National University of Singapore (NUS) · RoboScience

Abstract

Achieving human-like dexterous manipulation remains a central challenge in robotics. Learning from human hand-object interaction (HOI) datasets offers a promising pathway to diverse dexterous skills, yet existing physics-based approaches train specialist policies for individual trajectories, thereby limiting skill reuse at scale. We introduce DexMimic, a generalist dexterous hand-object tracking policy jointly trained across diverse HOI trajectories. Efficient joint training requires allocating each rollout to both an undertrained phase and a starting state that supports continued tracking. DexMimic realizes this insight through Value-Guided Reset Allocation, which adapts the initialization distribution online to prioritize difficult phases and select reset states with higher expected tracking returns. We further introduce a benchmark spanning training, fine-tuning, and generalization. Experiments show improved tracking success and training efficiency, faster fine-tuning, and the potential for zero-shot generalization to unseen trajectories and objects; larger training sets further improve efficiency and generalization. Finally, we validate DexMimic on a real-world dexterous manipulation platform.

Figures & tables

Fig. 2: Overview of Value-Guided Reset Allocation in DexMimic. The reset map stores candidate states and their critic values for each trajectory and reference phase. As training progresses, reset phase allocation prioritizes under-learned phases, while reset state allocation favors candidates with higher expected tracking returns within each selected phase. Physics rollouts from the sampled initializations update the policy and critic, replenish the reset map, and refine phase sampling probabilities, forming an adaptive training loop.

Fig. 3: Qualitative simulation results of DexMimic. The top two rows show four bimanual rollouts, with four temporally ordered frames per rollout from left to right. The bottom two rows show representative states from 16 single-hand trajectories.

Methods	SR $\uparrow$ (%)	Time $\downarrow$ (h)	$E_{r}\downarrow$ (rad)	$E_{t}\downarrow$ (mm)	$E_{j}\downarrow$ (mm)	$E_{ft}\downarrow$ (mm)
ManipTrans	56.64	1.86	0.12	6.13	19.39	14.42
SPIDER	39.68	0.78	0.35	12.14	38.97	27.48
DexMimic	88.40	0.12	0.07	3.76	17.25	14.52

TABLE I: Tracking performance and average training or optimization time per trajectory on 50 challenging bimanual tasks.

E_{r}

E_{t}

E_{j}

, and

E_{ft}

denote object rotation, object translation, finger keypoint, and fingertip errors, respectively. Best values are highlighted in green.

Panel of Figure 5 — Fig. 6: Benchmark for generalization and adaptation. (a) Grouped bars show zero-shot SR at three generalization levels (L1–L3). The red line shows that the mean SR across these levels increases with co-training scale. (b) Fine-tuning the pretrained DexMimic base policy accelerates joint adaptation to 20 unseen trajectories.

Method	SR $\uparrow$ (%)	Time $\downarrow$ (h)	$E_{r}\downarrow$ (rad)	$E_{t}\downarrow$ (mm)	$E_{j}\downarrow$ (mm)	$E_{ft}\downarrow$ (mm)	FastFail $\downarrow$ (%)	WorstTraj SR $\uparrow$
RSI	66.51	0.150	0.28	5.17	22.64	20.26	96.34	$21/50$
Uniform	82.59	0.310	0.18	3.81	20.89	18.39	35.65	$0/50$
DexMimic	84.73	0.096	0.17	3.55	20.50	17.38	41.70	$38/50$

TABLE II: Ablations validate both reset allocation designs. FastFail and WorstTraj SR show that DexMimic addresses two key bottlenecks in generalist training: fast-fail resets and learning imbalance across trajectories.

Method	SR (%, $\uparrow$ ) / Time (min, $\downarrow$ )
	20	50	200	500
RSI	74.20 / 30.46	72.40 / 15.68	78.46 / 14.32	66.51 / 9.00
Uniform	76.90 / 24.00	67.64 / 17.43	77.79 / 25.89	82.59 / 18.60
DexMimic	85.80 / 20.39	75.44 / 10.25	88.86 / 8.63	84.73 / 5.76

TABLE III: Reset allocation variants trained on 20–500 trajectories. Each entry reports SR (%,

\uparrow

) / average training time per trajectory (min,

\downarrow

), using the same timing protocol as the main comparison.

Panel of Table 9 — TABLE IV: DexMimic across dexterous hands.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. S1: Parallel-task visualization of 28 recorded DexMimic trajectories: 23 single-hand and five bimanual. Each cell shows one representative state; the environments are arranged for visualization.

Term (unit)	Component reward	Weight
Hand pose tracking
Wrist position (m)	$\exp(-40E_{w,p})$	$0.10$
Wrist orientation (rad)	$\exp(-E_{w,R})$	$0.60$
Thumb (m)	$\exp(-100E_{T})$	$0.90$
Index (m)	$\exp(-90E_{I})$	$0.80$
Middle (m)	$\exp(-80E_{M})$	$0.75$

Appendix

TABLE S1: Single-hand reward components. Units refer to the underlying errors or physical quantities.

Fig. S3: Per-trajectory success rates versus reference trajectory length. Colors distinguish right-hand (RH) and left-hand (LH) trajectories.

Parameter	Value
PPO rollout / mini-epochs / minibatch	$32$ / $5$ / $1024$
Discount $\gamma$ / GAE $\lambda_{\mathrm{GAE}}$	$0.99$ / $0.95$
Adam learning rate / PPO clip	$5\!\times\!10^{-4}$ / $0.2$
Reset slots $K$ / anchor probability $\alpha$	$3$ / $0.1$
Survival threshold $c_{\min}$ / score decay $\delta$	$0.1$ / $10^{-5}$
Bin width / lookback $n$ / factor $\lambda$	$15$ / $5$ / $0.8$

Appendix

TABLE S2: Training and reset allocation hyperparameters.

Panel of Figure 14 — Fig. S4: Evaluation outcomes and estimated failure-category proportions. Failure labels denote tracking errors exceeding their respective thresholds. All percentages are relative to the 1,017 evaluation rollouts. Failure-category proportions are estimated by proportionally allocating unclassified failures across the five categories.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,628 papers

Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking
Oct 20, 2026Yuxuan Kuang, Jiashun Wang, Yixuan Sun +5Robot Skill Learning Robotic Manipulation
Carnegie Mellon University · NVIDIA
Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration
Oct 8, 2026Jusuk Lee, Sungha Kim, Yeonsoo Park +8Robotic Manipulation Learning from Demonstration
Seoul National University · University of Maryland, College Park · All Purpose AI +1
OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting
Oct 9, 2026Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon +3Motion Retargeting Robotic Manipulation
Multiply Labs, San Francisco, CA 94107, USA · Department of Control and Dynamical Systems, California Institute of Technology, Pasadena, CA 91125, USA · Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA

Oct 20, 2026J/K move · Enter open · S save

Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking

Yuxuan Kuang, Jiashun Wang, Yixuan Sun, Yufei Ye+4

Carnegie Mellon University · NVIDIA

Robot Skill Learning Robotic Manipulation RL for Robotics Goal-Conditioned Robotic Manipulation Human-to-Robot Motion Imitation

Open paper AskPDF arXiv

Dexterous multi-finger hands hold promise to unlock diverse skills like functional grasps, in-hand reorientation, and dynamic handover that the human-designed world assumes, yet existing learning recipes force a trade-off. Goal-reaching reinforcement learning generalizes to arbitrary object poses but, lacking any human prior, converges to unnatural and unreliable finger strategies; motion tracking reproduces human-like behavior from a single clip but cannot deviate from it. We present AdaDex, a goal-conditioned RL framework that turns a single hand-object trajectory of a few seconds---from mocap data or HOI reconstruction from in-the-wild RGB(D) video---into an adaptable dexterous skill. At the core of AdaDex is Hybrid Dexterous Mimicking: a shared policy is trained on two populations of massively parallel environments: motion tracking environments trained with dense motion tracking objective, which instill the human prior from human motion; and generalization environments with noised initial states and goals and a sparse object-success reward, which force the skill to work far off the reference. From seconds of human motion per skill, the resulting policies can learn diverse adaptable dexterous skills, such as functional grasping, object reposing, in-hand reorientation with finger gaiting, and dynamic behaviors like throw-and-catch. Our method outperforms baselines in terms of generalization and functionality, and we demonstrate transfer across simulators and to the real world. Project page: https://yuxuank.com/AdaDex/.

Abstract

Robotic Manipulation

Explore similar work

Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting