DexOAK: Object-Aware Kinematic Retargeting

Federica Spinola; Robert Katzschmann; Cordelia Schmid

cs.ROOct 19, 2026

DexOAK: Object-Aware Kinematic Retargeting

Authors: Federica Spinola, Robert Katzschmann, Cordelia Schmid

Organizations: Inria and the Departement d’Informatique of Ecole Normale Supérieure, PSL University, France · Soft Robotics Lab, ETH Zürich, Switzerland

Abstract

We introduce DexOAK, a learning-based object-aware kinematic retargeting framework for generating contact-consistent robot hand references from human motion-capture data. DexOAK first minimizes an interaction-graph Laplacian objective over a joint hand-object graph, so the robot reproduces the demonstrated object-relative interaction geometry rather than the human hand pose alone. These per-frame solutions form seed retargeted references, from which a feed-forward network conditioned on human keypoints and object point clouds learns the retargeting function. Object-aware finetuning then refines the network under contact, non-penetration, and self-collision losses that position the fingers on the demonstrated contacts and reduce artifacts destabilizing physics-based refinement. Once trained, the network transfers to unseen data by inference alone, without per-frame optimization. DexOAK references improve dynamic retargeting over the state-of-the-art across bimanual tasks with rigid and articulated objects. On the Inspire hand, success rises from 39.5% to 48.4% on OakInk-v2, and ADD-AUC rises from 62.7% to 77.9% on ARCTIC. Ablations trace the gains to object-aware kinematic retargeting and the amortized network refined with contact losses. Trained on OakInk-v2, it transfers to ARCTIC zero-shot and nears in-domain performance after fine-tuning. On hardware, 46/56 open-loop replays succeed across five tasks, enabling dexterous robot data generation from human data. Visualizations and interactive results are at: https://fspinola.github.io/dexoak/.

Figures & tables

Fig. 1: DexOAK enables dexterous real-robot data generation from human motion-capture. Human demonstrations of Unplug , Pour , and Close Laptop (held out from retargeter training) are successfully replayed open-loop on a UR5 + right Allegro using DexOAK references refined by RL.

Fig. 2: Overview of DexOAK . From a human hand–object demonstration, our interaction-graph Laplacian optimizer matches human and robot Laplacian coordinates to produce a per-frame seed retargeted reference. A learned retargeter amortizes this optimization, autoregressively mapping the object cloud and human keypoints to the robot wrist and joints. It is trained on the seed references and further refined with object-aware losses. The resulting kinematic trajectory is later refined by an established residual RL stage ( Sec. IV , omitted from the figure). Colors: object (green), human (blue), robot (orange).

Fig. 3: DexOAK stages and downstream RL recover fingertip contacts missed by prior methods. Two OakInk-v2 [ 6 ] task frames (top: ignite, Allegro hand [ 18 ] ; bottom: unplug, Inspire hand [ 19 ] ). Hand-only AnyTeleop-Vector [ 13 ] , ManipTrans imitator [ 11 ] , and IG (w/o object) optimizer leave fingertips off or inside the object. IG (+ object) optimizer, Object-Aware Fine-Tuning (FT), and RL ( Sec. IV ) progressively improve fingertip placement.

	Kinematic metrics		RL metric
Kinematic reference	Contact compl. $\uparrow$	TPE $\downarrow$	SR MT $\uparrow$
	(%)	(mm)	(%)
Without Object (Hand-Only Reference)
AnyTeleop-Vector [ 13 ]	3.5	59.5	18.3
AnyTeleop-Position [ 13 ]	0.5	77.9	11.1
AnyTeleop-DexPilot [ 13 , 14 ]	0.3	53.8	0.0

TABLE I: Kinematic-reference comparison on the Allegro hand (right-hand OakInk-v2). All rows use ManipTrans residual RL. Results are our reproductions. Best in bold .

Variant	Contact completeness $\uparrow$	TPE $\downarrow$
Variant	(%)	(mm)
Interaction-Graph Laplacian Optimizer
without object anchors	2.8	41.5
without wrist node	19.6	3.51
Architecture and Training
without object cloud	20.0	3.83

TABLE II: Component ablations of DexOAK on the Allegro hand (right-hand OakInk-v2). Kinematic metrics only, without RL. Each row changes one design choice from the full model, with all else fixed.

Method	Speed (Hz) $\uparrow$	ADD-AUC (%) $\uparrow$
In-domain on ARCTIC
DexMachina [ 12 ]	13.4	83.4
Ours: IG optimizer	15.5	88.9
Ours: DexOAK	1.4	91.3
Ours: generalization OakInk-v2 $\rightarrow$ ARCTIC
Zero-shot	108.9	87.9

TABLE III: Cross-dataset transfer of DexOAK on the Allegro hand (bimanual OakInk-v2

\rightarrow

ARCTIC). All rows use DexMachina RL. Speed is evaluated frames divided by total target-data kinematic-stage time, including any optimization, training, or fine-tuning but excluding RL. ADD-AUC measures dynamic success. Best in bold , second-best underlined .

Method	SR MT (%) $\uparrow$	SR obj (%) $\uparrow$	ADD-AUC (%) $\uparrow$
OakInk-v2 (bimanual), Inspire hand
ManipTrans [ 11 ]	39.5	–	–
SPIDER [ 10 ]	–	47.9 ∗	–
DexOAK +RL	48.4	58.7	55.1
OakInk-v2 (bimanual), Allegro hand
SPIDER [ 10 ]	–	45.9 ∗	–

TABLE IV: DexOAK +RL vs. SOTA retargeting. Methods use their own dynamic refinement, while ours uses the residual RL of Sec. IV . –: not reported. ∗ SPIDER uses more OakInk-v2 sequences, making its comparison indicative. Best in bold .

Fig. 4: Qualitative results across embodiments and datasets. Kinematic references (left) and dynamically refined trajectories (right). The top three rows compare ManipTrans [ 11 ] , SPIDER [ 10 ] , and DexOAK (+RL) on uncap alcohol lamp (OakInk-v2 [ 6 ] , Inspire [ 19 ] ). The remaining rows show DexOAK (+RL) on use box (ARCTIC [ 5 ] , Allegro [ 18 ] ) and a failure on shake lab container (OakInk-v2, Allegro), where the grasp breaks during shaking.

Method	Take out	Close gate	Pour	Unplug	Cut
DexOAK +RL	12/12 [90%]	12/12 [41%]	10/12 [95%]	8/12 [93%]	4/8 [3%]
ManipTrans [ 11 ]	6/12 [83%]	12/12 [12%]	8/12 [16%]	0/4 [1%]	– [0%]

TABLE V: Real-robot results. The first successful simulation rollout per policy (3 seeds) is replayed 4 times on a UR5 + right Allegro. Entries show hardware successes/trials [simulation

SR_{\mathrm{MT}}

]. Denominators

<

12: fewer than 3 policies yielded a successful rollout. –: no successful rollout.

Finger	MANO kpt	Allegro tip frame	Inspire tip frame
Thumb	4	link_15.0_tip	thumb_tip
Index	8	link_3.0_tip	index_tip
Middle	12	link_7.0_tip	middle_tip
Ring	16	link_11.0_tip	ring_tip
Pinky	20	—	pinky_tip
Wrist	0	wrist	hand_base_link

TABLE S1: Interaction-graph nodes per embodiment. Robot tip frames matched to MANO fingertip keypoints, plus a shared wrist node. Allegro drops the human pinky.

Interaction-graph optimizer (SLSQP)
Object anchors $M_{o}$	64
Object cloud size	1024
Adjacency	Delaunay tetra (kNN fallback, $k{=}20$ )
Joint velocity limit $v_{\max}$	12 rad/s
Frame step $\mathrm{d}t$	$1/30$ s
Max iterations / tol	64 / $10^{-6}$

TABLE S2: Kinematic-stage hyperparameters. Optimizer, network, and the two training stages. Decoder width and depth differ by embodiment.

Parallel environments	4096
Simulation step	$1/60$ s
Episode length	1200 steps
Finger control	PD position
Action dimension	$6+16$ (wrist + fingers)
Discount $\gamma$	0.99
GAE $\lambda$	0.95

TABLE S3: Dynamic-retargeting (RL) hyperparameters. Residual PPO on the Allegro hand.

Clip	Sequence	Frames
box30-230-s01-u01	box_use_01	30–230
ketchup30-130-s01-u01	ketchup_use_01	30–130
ketchup40-340-s01-u02	ketchup_use_02	40–340
mixer30-200-s01-u01	mixer_use_01	30–200
mixer40-340-s01-u01	mixer_use_01	40–340
notebook40-340-s02-u02	notebook_use_02	40–340

TABLE S4: ARCTIC subset used (7 clips, 5 objects). Clip string is object,start,end,subject,use , over seeds

\{42,24,66,15,113\}

s01__box_use_01	s07__mixer_use_01	s04__notebook_use_01
s02__box_use_01	s01__mixer_grab_01	s06__notebook_use_01
s04__box_use_01	s01__ketchup_use_01	s07__notebook_use_01
s06__box_use_01	s01__ketchup_use_02	s04__notebook_grab_01
s07__box_use_01	s02__ketchup_use_01	s01__waffleiron_use_01
s01__box_grab_01	s04__ketchup_use_02	s02__waffleiron_use_01
s01__mixer_use_01	s06__ketchup_use_01	s04__waffleiron_use_01

TABLE S5: ARCTIC training sequences (30 clips, 5 objects), used only for training. Evaluation uses the 7-clip subset of Table S4 . Sequence name is subject__object_action_index .

Bimanual (Inspire), 39			Right-hand (Allegro), 27			Left-hand (Allegro), 11, train
ID	Action	Object	ID	Action	Object	ID	Action	Object
c9cd3@0	scoop	spoon	c9cd3@1	spread	spoon	e76b2@1	hold	apple
d78fa@0	open gate	microwave	d7c33@1	rearrange	spoon	155cb@4	place rack	test tube
e76b2@0	cut	knife	e76b2@2	place in	apple	81a95@3	rearrange	tripod
e76b2@12	open gate	microwave	e76b2@4	pour	bottle	81a95@4	place mesh	mesh
e76b2@13	take out	bowl	5ced9@5	remove lid	box lid	81a95@5	heat beaker	beaker

TABLE S6: OakInk-v2 tasks used. Identifier is seqhash@segment . Bimanual tasks are evaluated on Inspire, right-hand on Allegro, and the left-hand tasks are used for training only. Object and action labels are abbreviated.

Motion Retargeting

Motion retargeting transfers movement across different bodies, skeletons, or robots while preserving its intent and physical feasibility.

See all 96 papers

FlashDexRetarget: Accelerating Dexterous Manipulation Data Generation through Multi-Motion Retargeting
Oct 1, 2026Kyungmin Lee, Sibeen Kim, Dongyoon Hwang +6RL for Robotics Cross-Embodiment Robot Learning
KAIST AI · Holiday Robotics
OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting
Oct 9, 2026Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon +3Motion Retargeting Robotic Manipulation
Multiply Labs, San Francisco, CA 94107, USA · Department of Control and Dynamical Systems, California Institute of Technology, Pasadena, CA 91125, USA · Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA
ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling
Jul 4, 2026Yuanchuan Lai, Qing Gao, Ziyan Liang +2Contact-Rich Robotic Manipulation Motion Retargeting
School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China. · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Shenzhen 518172, China. · School of Computing, University of Portsmouth, Portsmouth PO1 3HE, UK.

Oct 1, 2026J/K move · Enter open · S save

FlashDexRetarget: Accelerating Dexterous Manipulation Data Generation through Multi-Motion Retargeting

Kyungmin Lee, Sibeen Kim, Dongyoon Hwang, Yoonsang Oh+5

KAIST AI · Holiday Robotics

RL for Robotics Cross-Embodiment Robot Learning Motion Retargeting Robotic Manipulation Robot Imitation Learning

Open paper AskPDF arXiv

Human hand-object demonstrations provide a scalable source of data for dexterous robot learning, but transferring them across embodiments requires physically feasible retargeting. Existing physics-based methods typically optimize each demonstration independently, leading to either limited success under finite simulation budgets or training costs that grow with dataset size. We introduce FlashDexRetarget, an RL framework for multi-reference dexterous retargeting. We formulate retargeting as multi-reference tracking, jointly learning a single policy across many demonstrations with off-policy RL and geometric supervision of the demonstrated interactions. This shared training formulation amortizes optimization across references while enabling the policy to track diverse hand-object interactions. On a 50-motion benchmark from TACO, OakInk2, and HOT3D using XHand and Sharpa Wave Hand as target embodiments, FlashDexRetarget retargets 90% of demonstrations using about 30 GPU-hours, compared with about 46% at about 3,000 GPU-hours for CHORD. This corresponds to about 100 times lower training compute and a 44-percentage-point improvement in retargeting success. Ablations examine the key design choices, while experiments with up to 1,000 motions and real-world replay further demonstrate the scalability and practical applicability of our method.

Abstract

Motion Retargeting

Explore similar work

FlashDexRetarget: Accelerating Dexterous Manipulation Data Generation through Multi-Motion Retargeting

OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting

ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling