cs.ROOct 19, 2026

DexOAK: Object-Aware Kinematic Retargeting

Authors: Federica Spinola, Robert Katzschmann, Cordelia Schmid

Organizations: Inria and the Departement d’Informatique of Ecole Normale Supérieure, PSL University, France · Soft Robotics Lab, ETH Zürich, Switzerland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce DexOAK, a learning-based object-aware kinematic retargeting framework for generating contact-consistent robot hand references from human motion-capture data. DexOAK first minimizes an interaction-graph Laplacian objective over a joint hand-object graph, so the robot reproduces the demonstrated object-relative interaction geometry rather than the human hand pose alone. These per-frame solutions form seed retargeted references, from which a feed-forward network conditioned on human keypoints and object point clouds learns the retargeting function. Object-aware finetuning then refines the network under contact, non-penetration, and self-collision losses that position the fingers on the demonstrated contacts and reduce artifacts destabilizing physics-based refinement. Once trained, the network transfers to unseen data by inference alone, without per-frame optimization. DexOAK references improve dynamic retargeting over the state-of-the-art across bimanual tasks with rigid and articulated objects. On the Inspire hand, success rises from 39.5% to 48.4% on OakInk-v2, and ADD-AUC rises from 62.7% to 77.9% on ARCTIC. Ablations trace the gains to object-aware kinematic retargeting and the amortized network refined with contact losses. Trained on OakInk-v2, it transfers to ARCTIC zero-shot and nears in-domain performance after fine-tuning. On hardware, 46/56 open-loop replays succeed across five tasks, enabling dexterous robot data generation from human data. Visualizations and interactive results are at: https://fspinola.github.io/dexoak/.

Figures & tables

Fig. 1: DexOAK enables dexterous real-robot data generation from human motion-capture. Human demonstrations of Unplug , Pour , and Close Laptop (held out from retargeter training) are successfully replayed open-loop on a UR5 + right Allegro using DexOAK references refined by RL.
Fig. 2: Overview of DexOAK . From a human hand–object demonstration, our interaction-graph Laplacian optimizer matches human and robot Laplacian coordinates to produce a per-frame seed retargeted reference. A learned retargeter amortizes this optimization, autoregressively mapping the object cloud and human keypoints to the robot wrist and joints. It is trained on the seed references and further refined with object-aware losses. The resulting kinematic trajectory is later refined by an established residual RL stage ( Sec. IV , omitted from the figure). Colors: object (green), human (blue), robot (orange).
Fig. 3: DexOAK stages and downstream RL recover fingertip contacts missed by prior methods. Two OakInk-v2 [ 6 ] task frames (top: ignite, Allegro hand [ 18 ] ; bottom: unplug, Inspire hand [ 19 ] ). Hand-only AnyTeleop-Vector [ 13 ] , ManipTrans imitator [ 11 ] , and IG (w/o object) optimizer leave fingertips off or inside the object. IG (+ object) optimizer, Object-Aware Fine-Tuning (FT), and RL ( Sec. IV ) progressively improve fingertip placement.
 Kinematic metricsRL metric
Kinematic referenceContact compl. ↑\uparrowTPE ↓\downarrowSR MT ↑\uparrow
 (%)(mm)(%)
Without Object (Hand-Only Reference)
AnyTeleop-Vector [ 13 ]3.559.518.3
AnyTeleop-Position [ 13 ]0.577.911.1
AnyTeleop-DexPilot [ 13 , 14 ]0.353.80.0
TABLE I: Kinematic-reference comparison on the Allegro hand (right-hand OakInk-v2). All rows use ManipTrans residual RL. Results are our reproductions. Best in bold .
VariantContact completeness ↑\uparrowTPE ↓\downarrow
(%)(mm)
Interaction-Graph Laplacian Optimizer
without object anchors2.841.5
without wrist node19.63.51
Architecture and Training
without object cloud20.03.83
TABLE II: Component ablations of DexOAK on the Allegro hand (right-hand OakInk-v2). Kinematic metrics only, without RL. Each row changes one design choice from the full model, with all else fixed.
MethodSpeed (Hz) ↑\uparrowADD-AUC (%) ↑\uparrow
In-domain on ARCTIC
DexMachina [ 12 ]13.483.4
Ours: IG optimizer15.588.9
Ours: DexOAK1.491.3
Ours: generalization OakInk-v2 →\rightarrow ARCTIC
Zero-shot108.987.9
TABLE III: Cross-dataset transfer of DexOAK on the Allegro hand (bimanual OakInk-v2 →\rightarrow ARCTIC). All rows use DexMachina RL. Speed is evaluated frames divided by total target-data kinematic-stage time, including any optimization, training, or fine-tuning but excluding RL. ADD-AUC measures dynamic success. Best in bold , second-best underlined .
MethodSR MT (%) ↑\uparrowSR obj (%) ↑\uparrowADD-AUC (%) ↑\uparrow
OakInk-v2 (bimanual), Inspire hand
ManipTrans [ 11 ]39.5––
SPIDER [ 10 ]–47.9 ∗–
DexOAK +RL48.458.755.1
OakInk-v2 (bimanual), Allegro hand
SPIDER [ 10 ]–45.9 ∗–
TABLE IV: DexOAK +RL vs. SOTA retargeting. Methods use their own dynamic refinement, while ours uses the residual RL of Sec. IV . –: not reported. ∗ SPIDER uses more OakInk-v2 sequences, making its comparison indicative. Best in bold .
Fig. 4: Qualitative results across embodiments and datasets. Kinematic references (left) and dynamically refined trajectories (right). The top three rows compare ManipTrans [ 11 ] , SPIDER [ 10 ] , and DexOAK (+RL) on uncap alcohol lamp (OakInk-v2 [ 6 ] , Inspire [ 19 ] ). The remaining rows show DexOAK (+RL) on use box (ARCTIC [ 5 ] , Allegro [ 18 ] ) and a failure on shake lab container (OakInk-v2, Allegro), where the grasp breaks during shaking.
MethodTake outClose gatePourUnplugCut
DexOAK +RL12/12 [90%]12/12 [41%]10/12 [95%]8/12 [93%]4/8 [3%]
ManipTrans [ 11 ]6/12 [83%]12/12 [12%]8/12 [16%]0/4 [1%]– [0%]
TABLE V: Real-robot results. The first successful simulation rollout per policy (3 seeds) is replayed 4 times on a UR5 + right Allegro. Entries show hardware successes/trials [simulation SRMTSR_{\mathrm{MT}} ]. Denominators << 12: fewer than 3 policies yielded a successful rollout. –: no successful rollout.
FingerMANO kptAllegro tip frameInspire tip frame
Thumb4link_15.0_tipthumb_tip
Index8link_3.0_tipindex_tip
Middle12link_7.0_tipmiddle_tip
Ring16link_11.0_tipring_tip
Pinky20—pinky_tip
Wrist0wristhand_base_link
TABLE S1: Interaction-graph nodes per embodiment. Robot tip frames matched to MANO fingertip keypoints, plus a shared wrist node. Allegro drops the human pinky.
Interaction-graph optimizer (SLSQP)
Object anchors MoM_{o}64
Object cloud size1024
AdjacencyDelaunay tetra (kNN fallback, k=20k{=}20 )
Joint velocity limit vmax⁡v_{\max}12 rad/s
Frame step dt\mathrm{d}t1/301/30 s
Max iterations / tol64 / 10−610^{-6}
TABLE S2: Kinematic-stage hyperparameters. Optimizer, network, and the two training stages. Decoder width and depth differ by embodiment.
Parallel environments4096
Simulation step1/601/60 s
Episode length1200 steps
Finger controlPD position
Action dimension6+166+16 (wrist + fingers)
Discount γ\gamma0.99
GAE λ\lambda0.95
TABLE S3: Dynamic-retargeting (RL) hyperparameters. Residual PPO on the Allegro hand.
ClipSequenceFrames
box30-230-s01-u01box_use_0130–230
ketchup30-130-s01-u01ketchup_use_0130–130
ketchup40-340-s01-u02ketchup_use_0240–340
mixer30-200-s01-u01mixer_use_0130–200
mixer40-340-s01-u01mixer_use_0140–340
notebook40-340-s02-u02notebook_use_0240–340
TABLE S4: ARCTIC subset used (7 clips, 5 objects). Clip string is object,start,end,subject,use , over seeds {42,24,66,15,113}\{42,24,66,15,113\} .
s01__box_use_01s07__mixer_use_01s04__notebook_use_01
s02__box_use_01s01__mixer_grab_01s06__notebook_use_01
s04__box_use_01s01__ketchup_use_01s07__notebook_use_01
s06__box_use_01s01__ketchup_use_02s04__notebook_grab_01
s07__box_use_01s02__ketchup_use_01s01__waffleiron_use_01
s01__box_grab_01s04__ketchup_use_02s02__waffleiron_use_01
s01__mixer_use_01s06__ketchup_use_01s04__waffleiron_use_01
TABLE S5: ARCTIC training sequences (30 clips, 5 objects), used only for training. Evaluation uses the 7-clip subset of Table S4 . Sequence name is subject__object_action_index .
Bimanual (Inspire), 39Right-hand (Allegro), 27Left-hand (Allegro), 11, train
IDActionObjectIDActionObjectIDActionObject
c9cd3@0scoopspoonc9cd3@1spreadspoone76b2@1holdapple
d78fa@0open gatemicrowaved7c33@1rearrangespoon155cb@4place racktest tube
e76b2@0cutknifee76b2@2place inapple81a95@3rearrangetripod
e76b2@12open gatemicrowavee76b2@4pourbottle81a95@4place meshmesh
e76b2@13take outbowl5ced9@5remove lidbox lid81a95@5heat beakerbeaker
TABLE S6: OakInk-v2 tasks used. Identifier is seqhash@segment . Bimanual tasks are evaluated on Inspire, right-hand on Allegro, and the left-hand tasks are used for training only. Object and action labels are abbreviated.
More in this topic

Motion Retargeting

Motion retargeting transfers movement across different bodies, skeletons, or robots while preserving its intent and physical feasibility.

See all 96 papers

Explore similar work

CardsList
Oct 1, 2026

FlashDexRetarget: Accelerating Dexterous Manipulation Data Generation through Multi-Motion Retargeting

Human hand-object demonstrations provide a scalable source of data for dexterous robot learning, but transferring them across embodiments requires physically feasible retargeting. Existing physics-based methods typically optimize each demonstration independently, leading to either limited success under finite simulation budgets or training costs that grow with dataset size. We introduce FlashDexRetarget, an RL framework for multi-reference dexterous retargeting. We formulate retargeting as multi-reference tracking, jointly learning a single policy across many demonstrations with off-policy RL and geometric supervision of the demonstrated interactions. This shared training formulation amortizes optimization across references while enabling the policy to track diverse hand-object interactions. On a 50-motion benchmark from TACO, OakInk2, and HOT3D using XHand and Sharpa Wave Hand as target embodiments, FlashDexRetarget retargets 90% of demonstrations using about 30 GPU-hours, compared with about 46% at about 3,000 GPU-hours for CHORD. This corresponds to about 100 times lower training compute and a 44-percentage-point improvement in retargeting success. Ablations examine the key design choices, while experiments with up to 1,000 motions and real-world replay further demonstrate the scalability and practical applicability of our method.

Kyungmin Lee, Sibeen Kim, Dongyoon Hwang +6

KAIST AI · Holiday Robotics

RL for RoboticsCross-Embodiment Robot LearningMotion RetargetingRobotic ManipulationRobot Imitation Learning
AskPDF
Oct 9, 2026

OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting

Retargeting human demonstrations into dynamically feasible dexterous hand trajectories requires resolving complex contact physics, yet current methods either omit physical constraints or rely on computationally expensive sampling-based optimization in simulation. Reinforcement learning has previously been used to achieve planning and control through contact dynamics; however, standard policies require object-specific training and fail to generalize across arbitrary geometries. This work reframes physics-based retargeting as a residual learning problem, where we demonstrate that zero-shot generalization across objects relies on structured state representations. Specifically, formulating the observation space to include a localized contact-error signal guiding each robot link to its target coordinate on the object surface enables a single, unified residual policy to successfully grasp novel, unseen object geometries. Rolling out this policy in simulation on a kinematic reference trajectory yields a physics-based retargeter, OATH. Operating at 0.4 seconds of compute per second of reference, OATH achieves 2.4x faster-than-real-time performance and a 60-350x reduction in computational cost compared to sampling-based methods. The system recovers 58% of held-out and 54% of unseen-object DexYCB trajectories while maintaining a sub-2 cm tracking error.

Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon +3

Multiply Labs, San Francisco, CA 94107, USA · Department of Control and Dynamical Systems, California Institute of Technology, Pasadena, CA 91125, USA · Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA

Motion RetargetingRobotic ManipulationGeneralization in Robotic ManipulationHand-Object InteractionRL for RoboticsResidual Policy Learning
AskPDF
Jul 4, 2026

ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling

Learning robot dexterous manipulation from human manipulation videos requires reliably retargeting human intent to executable robot actions while maintaining stable hand-object contact, which remains a key challenge in embodied intelligence. Existing retargeting methods often ignore explicit contact modeling or rely on reinforcement learning, resulting in limited accuracy and generalization. To address this, we propose ObjRetarget, a human-to-robot motion retargeting framework for learning robot dexterous manipulation from human videos, which integrates anthropomorphic arm trajectory constraints with structured hand-object geometric modeling. For arm motion, reference trajectories extracted from human videos are used for initialization, followed by anthropomorphic constraints and redundancy-aware optimization to generate natural and accurate movements. For hand manipulation, ObjRetarget represents multi-finger contacts using polytope clusters and preserves contact structure through geometric invariants to improve stability. Experiments on real robots show that ObjRetarget improves manipulation success rates and contact stability across multiple dexterous tasks, and generalizes well to different demonstrations, object poses, and task settings.

Yuanchuan Lai, Qing Gao, Ziyan Liang +2

School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China. · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Shenzhen 518172, China. · School of Computing, University of Portsmouth, Portsmouth PO1 3HE, UK.

Contact-Rich Robotic ManipulationMotion RetargetingRobotic ManipulationRobot Imitation Learning
AskPDF