Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy

Tara Sadjadpour; Siming He; C. K. Wolfe; Haozhi Qi; Lea Wilken; S. Shankar Sastry; Claire Tomlin; Jitendra Malik

cs.ROSep 23, 2026

Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy

Authors: Tara Sadjadpour, Siming He, C. K. Wolfe, Haozhi Qi, Lea Wilken, S. Shankar Sastry, Claire Tomlin, Jitendra Malik

Organizations: Department of Electrical Engineering and Computer Science at the University of California, Berkeley, CA, USA

Abstract

Human hand-object interactions (HOIs) provide a rich source of demonstrations for dexterous manipulation, but learning directly from them presents challenges in bridging morphology gaps, ensuring dynamical feasibility, and sim-to-real deployment. We present Morphometric Imitation, a three-stage framework that transforms reconstructed HOIs into zero-shot sim-to-real visuomotor policies. First, morphometric optimization (MMO) kinematically retargets human motion across hand morphologies while preserving demonstrated contacts. Second, residual reinforcement learning (RL) refines the kinematic reference using object pose and contact information from the human motion to produce dynamically feasible robot demonstrations. Third, these demonstrations are distilled into visuomotor policies. Across three robot hands and ten HOIs, MMO improves contact F1 over the strongest of five baselines by at least 8 points for every hand, while also improving the success rate of downstream dynamic retargeting by as much as 35 points. Ablations on the residual RL show complementary benefits from using object pose and contact information. Finally, the visuomotor policies achieve 89.3% zero-shot success in 300 real-world trials on 30 objects. Project page: $\href{https://morphometricimitation.github.io}{\text{this https URL}}$

Figures & tables

Fig. 2: Visuomotor Policy Rollouts. Zero-shot sim-to-real rollouts for six objects: flashlight, cube, apple, hammer, lightbulb, and bowl. For each rollout, the first frame shows a time lapse of the reach from the home position to the pre-grasp, followed by the grasp in the second frame. The cube, apple, lightbulb, and bowl trajectories conclude with a lift. The flashlight trajectory additionally performs in-hand reorientation to point the flashlight forward, while the hammer trajectory reorients the hammer head toward the table before lifting.

Method	Kinematic Retargeting	Dynamic Retargeting	Zero-Shot Sim-to-Real Visuomotor Policy	Natural Human Motion Data
DexPilot [ 3 ]	${\color[rgb]{0,0.6,0}\checkmark}$	${\color[rgb]{0.75,0,0}\times}$	${\color[rgb]{0.75,0,0}\times}$	$--$
AnyTeleop [ 4 ]	${\color[rgb]{0,0.6,0}\checkmark}$	${\color[rgb]{0.75,0,0}\times}$	${\color[rgb]{0.75,0,0}\times}$	$--$
Position [ 4 ]	${\color[rgb]{0,0.6,0}\checkmark}$	${\color[rgb]{0.75,0,0}\times}$	${\color[rgb]{0.75,0,0}\times}$	$--$
Contact-Aware PyRoki [ 5 ]	${\color[rgb]{0,0.6,0}\checkmark}$	${\color[rgb]{0.75,0,0}\times}$	${\color[rgb]{0.75,0,0}\times}$	$--$
SPIDER [ 9 ]	${\color[rgb]{0.75,0,0}\times}$	${\color[rgb]{0,0.6,0}\checkmark}$	${\color[rgb]{0.75,0,0}\times}$	${\color[rgb]{0,0.6,0}\checkmark}$
Chen et al. [ 10 ]	${\color[rgb]{0.75,0,0}\times}$	${\color[rgb]{0,0.6,0}\checkmark}$	${\color[rgb]{0,0.6,0}\checkmark}$	${\color[rgb]{0,0.6,0}\checkmark}$

TABLE I: Comparison of kinematic and dynamic retargeting methods. Natural human motion, marked only for dynamic retargeting methods, denotes learning solely from human demonstrations collected without constraints that facilitate robot control.

Fig. 3: Morphometric Optimization. We illustrate the three stages of morphometric optimization across three diverse robot hand embodiments: Dex3, Allegro, and Sharpa. (1) Morphology matching initializes Scaled MANO using robot-to-human palm and finger-length ratios, then optimizes its morphology to match the target robot hand. (2) Contact matching optimizes the morphology-aligned hand to recover the contacts from the reference human hand-object interaction. (3) Robot pose recovery applies linear blend retargeting to transfer the resulting morphology- and contact-aligned MANO skeleton to the robot skeleton, followed by inverse kinematics to recover the robot joint configuration.

Fig. 4: Dynamic Retargeting Rollouts. We show time lapses of residual RL policy rollouts for three robot hand embodiments (Dex3, Allegro, and Sharpa) across three object categories (hammer, apple, and cube). All policies use the same training hyperparameters, demonstrating that the residual RL formulation transfers across distinct hand morphologies and grasp configurations without embodiment- or task-specific tuning.

	Dex3		Allegro		Sharpa
Method	Patch $\downarrow$	F1 $\uparrow$	Patch $\downarrow$	F1 $\uparrow$	Patch $\downarrow$	F1 $\uparrow$
	(mm)	(%)	(mm)	(%)	(mm)	(%)
OmniRetarget [ 6 ]	$26.9$ $\pm$ $10.9$	$7.6$ $\pm$ $11.3$	$21.2$ $\pm$ $13.1$	$12.8$ $\pm$ $16.0$	$21.1$ $\pm$ $15.5$	$14.5$ $\pm$ $18.1$
DexPilot [ 3 ]	$42.6$ $\pm$ $7.4$	$2.5$ $\pm$ $4.0$	$\underline{18.7}$ $\pm$ $4.9$	$13.1$ $\pm$ $6.8$	$12.6$ $\pm$ $3.7$	$21.9$ $\pm$ $9.8$
Position [ 4 ]	$\underline{18.9}$ $\pm$ $6.4$	$\underline{10.2}$ $\pm$ $7.6$	$26.3$ $\pm$ $39.4$	$\underline{22.0}$ $\pm$ $10.8$	$25.4$ $\pm$ $39.0$	$23.9$ $\pm$ $17.7$
AnyTeleop [ 4 ]	$42.1$ $\pm$ $7.2$	$2.7$ $\pm$ $3.9$	$19.0$ $\pm$ $5.1$	$13.2$ $\pm$ $7.1$	$13.3$ $\pm$ $3.7$	$21.0$ $\pm$ $9.8$

TABLE II: Kinematic retargeting contact preservation at

5\,\mathrm{mm}

tolerance (protocol in Sec. VII-B ). Mean

\pm

standard deviation over ten demonstrations per hand. Bold and underline: best and second-best per hand and metric.

Fig. 5: Qualitative kinematic retargeting at the end frames of the wineglass with Sharpa (first row) and the bowl with Allegro (second row). Each row shows the human reference and the four methods with the highest F1 at the same frame for direct comparisons. Results for all methods are provided in the Supplemental Material.

	Dex3			Allegro			Sharpa
Method	C-ADD $\downarrow$	SR $\uparrow$	Patch $\downarrow$	C-ADD $\downarrow$	SR $\uparrow$	Patch $\downarrow$	C-ADD $\downarrow$	SR $\uparrow$	Patch $\downarrow$
	( $10^{-2}$ m)	(%)	(mm)	( $10^{-2}$ m)	(%)	(mm)	( $10^{-2}$ m)	(%)	(mm)
OmniRetarget [ 6 ]	$\underline{5.36}$ $\pm$ $6.40$	$\underline{53.2}$ $\pm$ $38.9$	$103.1$ $\pm$ $137.8$	$6.98$ $\pm$ $7.11$	$61.3$ $\pm$ $41.8$	$56.6$ $\pm$ $50.7$	$6.59$ $\pm$ $6.25$	$\underline{56.5}$ $\pm$ $41.4$	$\underline{75.5}$ $\pm$ $101.7$
DexPilot [ 3 ]	$10.7$ $\pm$ $8.22$	$17.3$ $\pm$ $36.5$	$126.0$ $\pm$ $118.4$	$9.12$ $\pm$ $7.20$	$39.1$ $\pm$ $41.6$	$108.2$ $\pm$ $125.1$	$10.7$ $\pm$ $7.85$	$27.3$ $\pm$ $40.2$	$118.7$ $\pm$ $115.8$
Position [ 4 ]	$7.32$ $\pm$ $4.96$	$39.8$ $\pm$ $44.9$	$\underline{58.4}$ $\pm$ $60.7$	$\underline{6.26}$ $\pm$ $6.94$	$\underline{68.2}$ $\pm$ $34.5$	$\mathbf{41.5}$ $\pm$ $21.8$	$9.32$ $\pm$ $9.38$	$33.7$ $\pm$ $46.6$	$134.3$ $\pm$ $197.8$
AnyTeleop [ 4 ]	$11.2$ $\pm$ $7.99$	$19.1$ $\pm$ $31.8$	$121.7$ $\pm$ $109.3$	$8.06$ $\pm$ $5.76$	$37.2$ $\pm$ $42.8$	$98.3$ $\pm$ $105.4$	$8.11$ $\pm$ $5.35$	$36.7$ $\pm$ $38.8$	$95.0$ $\pm$ $87.9$

TABLE III: Residual RL dynamic retargeting from different kinematic references. Mean

\pm

standard deviation over ten categories, 2048 rollouts each (protocol in Sec. VII-B ). Bold and underline: best and second-best per hand and metric.

		Dex3			Allegro			Sharpa
Contact	Object Pose	C-ADD $\downarrow$	SR $\uparrow$	Patch $\downarrow$	C-ADD $\downarrow$	SR $\uparrow$	Patch $\downarrow$	C-ADD $\downarrow$	SR $\uparrow$	Patch $\downarrow$
		( $10^{-2}$ m)	(%)	(mm)	( $10^{-2}$ m)	(%)	(mm)	( $10^{-2}$ m)	(%)	(mm)
$\times$	$\checkmark$	$6.50$ $\pm$ $4.94$	$51.2$ $\pm$ $44.1$	$91.9$ $\pm$ $74.6$	$8.53$ $\pm$ $8.42$	$47.6$ $\pm$ $42.7$	$122.0$ $\pm$ $142.0$	$6.20$ $\pm$ $8.84$	$76.6$ $\pm$ $31.3$	$74.6$ $\pm$ $103.2$
$\checkmark$	$\times$	$8.04$ $\pm$ $4.21$	$42.5$ $\pm$ $34.0$	$72.1$ $\pm$ $48.3$	$8.61$ $\pm$ $3.65$	$41.6$ $\pm$ $35.7$	$53.1$ $\pm$ $44.0$	$7.42$ $\pm$ $2.27$	$54.3$ $\pm$ $41.3$	$47.1$ $\pm$ $25.7$
$\checkmark$	$\checkmark$	$\mathbf{4.10}$ $\pm$ $1.76$	$\mathbf{82.5}$ $\pm$ $9.5$	$\mathbf{47.2}$ $\pm$ $23.4$	$\mathbf{5.01}$ $\pm$ $3.38$	$\mathbf{69.9}$ $\pm$ $30.0$	$\mathbf{53.0}$ $\pm$ $25.2$	$\mathbf{3.41}$ $\pm$ $1.14$	$\mathbf{91.8}$ $\pm$ $3.6$	$\mathbf{25.1}$ $\pm$ $9.3$

TABLE IV: Ablation of the contact and object-pose information used by residual RL dynamic retargeting, starting from the same MMO kinematic references. Same protocol and units as Table III ; bold marks the best mean within each hand and metric.

	Alarm	Apple	Bowl	Cube	Cup	Flash.	Hammer	Bulb	Torus	Wine	Overall
Teacher (sim)	90.38	93.99	88.48	92.14	92.58	95.85	83.84	96.19	92.19	92.19	91.78
Student (sim)	100.00	100.00	87.50	76.56	100.00	100.00	92.19	98.44	96.88	85.94	93.75
Student (real)	86.67	86.67	90.00	86.67	93.33	86.67	90.00	100.00	80.00	93.33	89.33

TABLE V: Sharpa task success rates (%). Teacher and student simulation use 2,048 and 64 evaluation rollouts per category. Hardware uses 30 trials per category across three objects (10 poses each). Overall is the average across categories.

Fig. 6: Real-World Object Set. 30 objects used for zero-shot sim-to-real evaluation, comprising three instances from each of the ten object categories.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	$\tau=1\,\mathrm{mm}$			$\tau=5\,\mathrm{mm}$			$\tau=10\,\mathrm{mm}$
Method	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$
Allegro
OmniRetarget	$\mathbf{9.4}$ $\pm$ $12.3$	$1.1$ $\pm$ $1.9$	$1.8$ $\pm$ $3.1$	$\mathbf{35.6}$ $\pm$ $31.1$	$8.5$ $\pm$ $11.4$	$12.8$ $\pm$ $16.0$	$\mathbf{53.1}$ $\pm$ $30.6$	$18.0$ $\pm$ $16.8$	$25.2$ $\pm$ $20.8$
DexPilot	$2.3$ $\pm$ $1.6$	$1.9$ $\pm$ $1.5$	$2.0$ $\pm$ $1.5$	$15.3$ $\pm$ $7.7$	$12.5$ $\pm$ $7.3$	$13.1$ $\pm$ $6.8$	$26.8$ $\pm$ $11.2$	$24.4$ $\pm$ $10.3$	$24.3$ $\pm$ $9.4$
Position	$5.3$ $\pm$ $3.3$	$\underline{5.6}$ $\pm$ $7.0$	$4.4$ $\pm$ $3.2$	$26.6$ $\pm$ $11.8$	$\underline{23.1}$ $\pm$ $18.5$	$\underline{22.0}$ $\pm$ $10.8$	$40.5$ $\pm$ $12.9$	$\underline{37.2}$ $\pm$ $19.7$	$\underline{36.0}$ $\pm$ $11.8$
AnyTeleop	$2.2$ $\pm$ $1.7$	$1.9$ $\pm$ $1.7$	$2.0$ $\pm$ $1.7$	$15.4$ $\pm$ $7.5$	$12.5$ $\pm$ $7.7$	$13.2$ $\pm$ $7.1$	$26.4$ $\pm$ $10.9$	$24.0$ $\pm$ $10.8$	$24.0$ $\pm$ $9.5$

Appendix

TABLE VI: Location-aware contact across matched tolerances

\tau_{H}=\tau_{B}=\tau

. Failure-adjusted macro percentages (mean

\pm

population SD) over ten attempts per hand and method; coverage equals recall. The

5\,\mathrm{mm}

F1 repeats Table II , and location-unaware F1 is shown in Fig. 8 . Bold and underlining mark the highest and second-highest distinct unrounded means within each hand, tolerance, and metric; ties share formatting.

Method	$D_{\rm patch}\downarrow$	$D_{95}\downarrow$	Pen. rate (%) $\downarrow$	Max. depth $\downarrow$
Allegro ( $n_{\mathcal{I}}=9$ )
OmniRetarget	$21.2$ $\pm$ $13.1$	$39.2$ $\pm$ $22.7$	$\mathbf{0.0}$ $\pm$ $0.0$	$\mathbf{0.6}$ $\pm$ $0.4$
DexPilot	$\underline{18.7}$ $\pm$ $4.9$	$\underline{38.2}$ $\pm$ $7.6$	$47.1$ $\pm$ $25.7$	$10.3$ $\pm$ $2.6$
Position	$26.3$ $\pm$ $39.4$	$52.4$ $\pm$ $69.1$	$35.2$ $\pm$ $28.6$	$11.1$ $\pm$ $7.6$
AnyTeleop	$19.0$ $\pm$ $5.1$	$38.8$ $\pm$ $7.8$	$44.8$ $\pm$ $24.6$	$\underline{10.3}$ $\pm$ $2.6$
Contact PyRoki	$21.5$ $\pm$ $6.1$	$43.7$ $\pm$ $14.8$	$\underline{35.1}$ $\pm$ $27.6$	$10.5$ $\pm$ $6.4$

Appendix

TABLE VII: Geometry on matched demonstration sequences with valid retargeting outputs from every method. The human patch uses

5\,\mathrm{mm}

. Distances and sequence-maximum penetration depths are in mm; penetration rate is the percentage of frames deeper than

5\,\mathrm{mm}

. Entries are mean

\pm

population SD of sequence statistics. Failed runs have no geometric measurement. Bold and underlining mark the lowest and second-lowest distinct unrounded means within each hand and column; ties share formatting.

Fig. 7: Qualitative kinematic retargeting with all methods at the frames of Fig. 5 : (a) wineglass with Sharpa and (b) bowl with Allegro.

Comparison	$\Delta$ F1	95% interval
Allegro
Ours $-$ OmniRetarget	$17.4$	$[7.7,\,26.6]$
Ours $-$ DexPilot	$17.2$	$[11.6,\,23.3]$
Ours $-$ Position	$8.3$	$[1.2,\,14.7]$
Ours $-$ AnyTeleop	$17.1$	$[11.4,\,23.4]$
Ours $-$ Contact PyRoki	$14.9$	$[4.8,\,24.4]$

Appendix

TABLE VIII: Paired differences in location-aware F1 at

5\,\mathrm{mm}

, in percentage points. Positive values favor Ours. Intervals use 20,000 paired demonstration resamples (pointwise 95% percentile intervals). Scores are failure-adjusted with

N_{\rm att}=10

. On the matched sequences

\mathcal{I}

, every Allegro and Dex3 interval also excludes zero; Sharpa has no failures, so its matched intervals are identical. These intervals describe variation across the selected demonstrations, not independent policy training seeds.

	All rollouts				Successful rollouts
Method	F1 $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Max depth $\downarrow$	$n_{\rm cat}$	F1 $\uparrow$	Patch $\downarrow$
	(%)	(%)	(%)	(mm)		(%)	(mm)
Dex3 $N_{\rm att}=10,\hskip 8.50012ptn_{\mathcal{I}}=8$
OmniRetarget [ 6 ]	$5.5$ $\pm$ $6.2$	$6.8$ $\pm$ $6.9$	$5.2$ $\pm$ $5.9$	$4.0$ $\pm$ $1.5$	9/7	$7.3$ $\pm$ $7.3$	$25.3$ $\pm$ $7.1$
DexPilot [ 3 ]	$5.5$ $\pm$ $8.1$	$8.4$ $\pm$ $11.3$	$4.9$ $\pm$ $7.5$	$4.3$ $\pm$ $2.4$	2/2	$6.5$ $\pm$ $8.7$	$28.6$ $\pm$ $15.4$
Position [ 4 ]	$\underline{11.3}$ $\pm$ $7.2$	$\underline{15.2}$ $\pm$ $10.4$	$\underline{10.6}$ $\pm$ $8.1$	$\underline{3.5}$ $\pm$ $2.3$	5/4	$11.9$ $\pm$ $6.0$	$21.1$ $\pm$ $10.4$

Appendix

TABLE IX: Contact metrics of the achieved final grasps. SR and patch distance on all rollouts are in Tables III and IV . For successful rollouts only,

n_{\rm cat}

counts the categories with at least ten successful rollouts behind F1 / behind patch distance. Bold and underlining mark the best and second-best distinct means within each hand and metric.

Fig. 9: Real-World Policy Inputs. For ten of the 30 physical instances, the RGB image from the L515 camera (for reference only; the policy does not observe RGB), the raw depth image (darker is closer; red marks pixels with no LiDAR return), and the policy input: the 512-point cloud after table removal (magenta), drawn over the color-coded depth image.

Category	Instance	Photo	Physical instance				Reference object dimensions (cm)	Success rate
Category	Instance	Photo	Mass (g)	L (cm)	W (cm)	H (cm)	Reference object dimensions (cm)	Instance	Category
Alarm clock	Red		225	11.7	5.6	13.4	$12.6\times 4.5\times 10.8$	7/10	26/30
	Black		240	9.7	5.3	13.6		9/10
	Swivel		257	12.1	4.3	12.7		10/10
Cup	Short beige		47	7.9	7.9	10.6	$8.5\times 8.5\times 10.0$	10/10	28/30
	Tall beige		62	7.8	7.8	14.5		9/10

Appendix

TABLE X: Real-World Object Instances (I). The three physical instances of the first five object categories used in the hardware evaluation. For each instance, we report its photo, hand-measured mass and dimensions (length

\times

width

\times

height), and pick-up success rate over the 10 test poses. For each category, we additionally report the dimensions of the GRAB object mesh used to train the policy in simulation (before anisotropic scale randomization) and the success rate over all 30 trials. Simulated object masses are sampled in

[50,400]

Category	Instance	Photo	Physical instance				Reference object dimensions (cm)	Success rate
Category	Instance	Photo	Mass (g)	L (cm)	W (cm)	H (cm)	Reference object dimensions (cm)	Instance	Category
Light bulb	IKEA		26	6.0	6.0	10.5	$6.2\times 6.2\times 11.0$	10/10	30/30
	Triangle		42	6.4	6.4	9.7		10/10
	Teeth		41	6.0	6.0	10.0		10/10
Wine glass	Safeway		31	4.6	4.6	17.8	$7.1\times 7.1\times 17.2$	10/10	28/30
	Champagne flute		47	3.1	3.1	19.4		10/10

Appendix

TABLE XI: Real-World Object Instances (II). The three physical instances of the remaining five object categories; columns are as in Table X . The hammer, flashlight, and light bulb instances lie flat on the table; the light bulb dimensions are reported with its long axis as the height.

Fig. 10: Real-World Success Rate by Initial Object Location. Each panel shows one object category. The nine circles are the grid locations spanning the $10\,\text{cm}\times 10$ cm randomization region, viewed from above. The near row is closest to the robot arm base, and the fingers of the hand point in the far direction. Each circle is colored and labeled by the success rate over the category’s three physical instances at that location (3 trials per location, 27 of the 30 trials per category). The tenth test pose of each instance is not a grid location and is omitted.

Explore similar work

Jun 15, 2026cs.RO

TopoRetarget: Interaction-Preserving Retargeting for Dexterous Manipulation

Human hand-object demonstrations provide dense reference motions for training dexterous manipulation reinforcement learning (RL) policies through reference tracking. However, to use such demonstrations for RL policy learning, retargeting must preserve hand pose and task-relevant hand-object contact structure. Otherwise, contact and feasibility artifacts can degrade downstream RL policy performance. We introduce TopoRetarget, an interaction-preserving retargeting framework that uses a single set of parameters across diverse retargeting conditions while maintaining task-relevant hand-object interaction and adapting human demonstrations to dexterous robot hands. The method constructs a sparse interaction graph over hand and object keypoints and optimizes distance-weighted Laplacian deformation with directional consistency, kinematic constraints, and penetration handling. Evaluations show that the generated references improve both interaction fidelity and policy learning: TopoRetarget achieves the best contact precision and alignment over all baselines on the ContactPose Dataset, improves Pen-Spin training success by 40.6 percentage points over the existing baseline methods, and enables zero-shot transfer to Wuji Hand hardware on cube reorientation and pen spinning.

Jielin Wu, Shenzhe Yao, Guanqi He +6

IIIS, Tsinghua University

Cross-Embodiment Retargeting Dexterous Manipulation

Ask PDF

Sep 21, 2026cs.RO

Dexterous Robot Manipulation from Human Demonstrations via Contact-Anchored Retargeting and Residual Policy Learning

Learning dexterous manipulation from demonstrations is bottlenecked by data: the contact forces that determine whether a grasp succeeds are absent from every scalable source of human demonstrations. This paper builds on two observations. First, what survives the change from a human hand to a robot hand is the contact structure of a demonstration - which finger regions touch which object locations, and in what order - rather than its joint motion. Second, physical consistency need not be engineered per task: a single residual reinforcement learning (RL) policy, trained once across diverse demonstrations, can repair kinematic recordings into physically consistent, contact-annotated trajectories, and the same residual formulation restores dynamic feasibility after retargeting. These observations yield a three-stage pipeline that converts human motion-capture recordings into dexterous robot policies with no real-robot training data: physics refinement with a simulated MANO hand recovers contacts and forces, contact-anchored retargeting transfers the demonstrated contact structure through an objective independent of hand morphology, and residual policy learning adapts the result to robot actuation. The pipeline reconstructs 25,454 single-hand trajectories (success 7.3% -> 59.3%) and 25 dual-hand tasks (16.0% -> 62.4%) with one shared policy per setting, transfers one human dataset to four morphologically distinct robot hands (+62.4 pp), and executes four contact-rich bimanual tasks on physical hardware with zero real-robot training data.

Zihao Yang, Chengyuan Liu, Yu Zhou +6

DexGEM Lab · Shanghai Jiao Tong University · Tongji University +1

Cross-Embodiment Retargeting Bimanual Manipulation Robotic Manipulation Residual Policy Robot Policies Large-Scale Robot Demonstration Datasets

Ask PDF

Jul 13, 2026cs.RO

A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulation

Recent work in humanoid whole-body control has found success with a simple recipe: retarget human motion to robot kinematic references, then train policies via reinforcement learning (RL) to track them. But how does this recipe transfer to dexterous manipulation? The answer is not obvious, as manipulation involves complex, contact-rich dynamics and requires delicate regulation of contact modes and forces. We present REGRIND, a minimalist retargeting-guided RL pipeline that learns dexterous manipulation policies from a single human demonstration. REGRIND retargets human hand-object motion to a robot reference that preserves hand-object spatial and contact relationships, trains a residual RL policy in simulation to track object-centric keypoints along that reference, and transfers the resulting policy zero-shot to hardware with careful system identification. The resulting policies produce fluid, human-like behavior on two different multi-fingered hands across contact-rich tool-use tasks, including operating a pair of scissors and turning a screwdriver. Through systematic hardware experiments, we identify and analyze the key factors that govern sim-to-real transfer in dexterous manipulation, offering practical guidance for retargeting-based learning in contact-rich settings. Videos and code are available at https://yunhaifeng.com/REGRIND.

Yunhai Feng, Natalie Leung, Jiaxuan Wang +3

Cornell University · Amazon FAR

Robotic Manipulation Policies Cross-Embodiment Retargeting Dexterous Manipulation Whole-Body Control Offline Reinforcement Learning

Ask PDF