OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting
Authors: Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon, Joel Burdick, Dingqi Zhang, Jiuguang Wang
Organizations: Multiply Labs, San Francisco, CA 94107, USA · Department of Control and Dynamical Systems, California Institute of Technology, Pasadena, CA 91125, USA · Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA
Retargeting human demonstrations into dynamically feasible dexterous hand trajectories requires resolving complex contact physics, yet current methods either omit physical constraints or rely on computationally expensive sampling-based optimization in simulation. Reinforcement learning has previously been used to achieve planning and control through contact dynamics; however, standard policies require object-specific training and fail to generalize across arbitrary geometries. This work reframes physics-based retargeting as a residual learning problem, where we demonstrate that zero-shot generalization across objects relies on structured state representations. Specifically, formulating the observation space to include a localized contact-error signal guiding each robot link to its target coordinate on the object surface enables a single, unified residual policy to successfully grasp novel, unseen object geometries. Rolling out this policy in simulation on a kinematic reference trajectory yields a physics-based retargeter, OATH. Operating at 0.4 seconds of compute per second of reference, OATH achieves 2.4x faster-than-real-time performance and a 60-350x reduction in computational cost compared to sampling-based methods. The system recovers 58% of held-out and 54% of unseen-object DexYCB trajectories while maintaining a sub-2 cm tracking error.
Figures & tables
Fig. 1: OATH trains a generalist object-agnostic controller for dexterous grasping using the DexYCB dataset. Evaluation on entirely held-out objects from the dataset demonstrates zero-shot tracking and retargeting in real time using a Wuji Hand.
Fig. 2: The reference generation pipeline: (clockwise) 1) MANO meshes are fit overtop of a kinematic trajectory extracted from video or given by the DexYCB dataset, 2) TopoRetarget is applied to convert the MANO mesh to the Wuji Hand, 3) Contact locations gˉk,i , hˉk,i , and contact presence mˉk,i are computed and stored for training, 4) Differential IK is applied to determine a robot arm trajectory that follows the wrist, 5) TOPP is used to slow the trajectory to a safe speed on hardware.
PPO: clip 0.2, clipped value loss (coef. 1.0), GAE γ 0.99 / λ 0.95
Optimizer
Adam, lr 10−3 , KL-adaptive: ÷1.5 if KL >0.02 , ×1.5 if KL <0.005 , clamped to [10−5,10−2]
TABLE I: Network, training, simulation, and reference hyperparameters
Fig. 3: (Left) Reference contact points gˉk,i , hˉk,i are extracted from the kinematic trajectory. (Right) During policy training, the deviation between the reference contact points ek,i(qk) is computed and passed to the network as an observation.
Fig. 4: The station: a UR-5e and Wuji 2 hand holding the mug object from DexYCB dataset
Baselines
OATH
Metric
SPIDER [ 11 ]
DAID [ 10 ]
ReGrind [ 7 ]
CHORD [ 6 ]
Train
Test
Unseen
( ↑ ) Success rate (%)
22.0
100
5.4
2.0
90
58
54
( ↓ ) Object tracking error (mm)
29.9
20.4
43.2
7.9
11.7
15.3
13.8
( ↓ ) Compute time per second of reference
25.2
138
0.93
0.88
0.40
( ↓ ) Compute per successful second of reference
124
138
15.8
64.9
0.46
0.69
0.71
TABLE II: Controller and Baseline Performance on DexYCB Tasks
Fig. 5: Closed-loop policy rollouts across challenging DexYCB trajectories. Ghosted poses illustrate motion progression, with opacity increasing over time ( t=0→T ). From left to right: unseen objects—mug (narrow handle) and bowl (inverted trajectory); held-out test trajectories—pitcher (oblique contact surface) and banana (thin cross-section).
Fig. 6: Common failures of the kinematic retargeting: (left to right) a) fingers penetrate the tabletop, b) spurious contact on object approach, c) missing contact references in whole prehensile grasps, d) floating object references, which fall in simulation.
Fig. 7: Open-loop hardware execution on held-out DexYCB trajectories . Sequential stills ( t=0→T ) of open-loop joint playback on a UR5e manipulator equipped with a Wuji Hand 2. All four evaluations follow held-out trajectory references, with the mug testing an entirely unseen object geometry. When compounding tracking errors remain bounded, open-loop execution successfully manipulates the objects, demonstrating the physical plausibility of the retargeted references.
Training
Testing
Unseen
Reference/Observation input
SR (%)
Err. (mm)
SR (%)
Err. (mm)
SR (%)
Err. (mm)
All terms: qˉk:k+Hobj,qˉk:k+Hrobot,mˉk,i,ek,i
We introduce DexOAK, a learning-based object-aware kinematic retargeting framework for generating contact-consistent robot hand references from human motion-capture data. DexOAK first minimizes an interaction-graph Laplacian objective over a joint hand-object graph, so the robot reproduces the demonstrated object-relative interaction geometry rather than the human hand pose alone. These per-frame solutions form seed retargeted references, from which a feed-forward network conditioned on human keypoints and object point clouds learns the retargeting function. Object-aware finetuning then refines the network under contact, non-penetration, and self-collision losses that position the fingers on the demonstrated contacts and reduce artifacts destabilizing physics-based refinement. Once trained, the network transfers to unseen data by inference alone, without per-frame optimization. DexOAK references improve dynamic retargeting over the state-of-the-art across bimanual tasks with rigid and articulated objects. On the Inspire hand, success rises from 39.5% to 48.4% on OakInk-v2, and ADD-AUC rises from 62.7% to 77.9% on ARCTIC. Ablations trace the gains to object-aware kinematic retargeting and the amortized network refined with contact losses. Trained on OakInk-v2, it transfers to ARCTIC zero-shot and nears in-domain performance after fine-tuning. On hardware, 46/56 open-loop replays succeed across five tasks, enabling dexterous robot data generation from human data. Visualizations and interactive results are at: https://fspinola.github.io/dexoak/.
Federica Spinola, Robert Katzschmann, Cordelia Schmid
Inria and the Departement d’Informatique of Ecole Normale Supérieure, PSL University, France · Soft Robotics Lab, ETH Zürich, Switzerland
Human hand-object demonstrations provide a scalable source of data for dexterous robot learning, but transferring them across embodiments requires physically feasible retargeting. Existing physics-based methods typically optimize each demonstration independently, leading to either limited success under finite simulation budgets or training costs that grow with dataset size. We introduce FlashDexRetarget, an RL framework for multi-reference dexterous retargeting. We formulate retargeting as multi-reference tracking, jointly learning a single policy across many demonstrations with off-policy RL and geometric supervision of the demonstrated interactions. This shared training formulation amortizes optimization across references while enabling the policy to track diverse hand-object interactions. On a 50-motion benchmark from TACO, OakInk2, and HOT3D using XHand and Sharpa Wave Hand as target embodiments, FlashDexRetarget retargets 90% of demonstrations using about 30 GPU-hours, compared with about 46% at about 3,000 GPU-hours for CHORD. This corresponds to about 100 times lower training compute and a 44-percentage-point improvement in retargeting success. Ablations examine the key design choices, while experiments with up to 1,000 motions and real-world replay further demonstrate the scalability and practical applicability of our method.
Human hand-object demonstrations provide dense reference motions for training dexterous manipulation reinforcement learning (RL) policies through reference tracking. However, to use such demonstrations for RL policy learning, retargeting must preserve hand pose and task-relevant hand-object contact structure. Otherwise, contact and feasibility artifacts can degrade downstream RL policy performance. We introduce TopoRetarget, an interaction-preserving retargeting framework that uses a single set of parameters across diverse retargeting conditions while maintaining task-relevant hand-object interaction and adapting human demonstrations to dexterous robot hands. The method constructs a sparse interaction graph over hand and object keypoints and optimizes distance-weighted Laplacian deformation with directional consistency, kinematic constraints, and penetration handling. Evaluations show that the generated references improve both interaction fidelity and policy learning: TopoRetarget achieves the best contact precision and alignment over all baselines on the ContactPose Dataset, improves Pen-Spin training success by 40.6 percentage points over the existing baseline methods, and enables zero-shot transfer to Wuji Hand hardware on cube reorientation and pen spinning.