GRASP: Gradient-Guided Sampling for Contact-Rich Physics-Based Motion Retargeting

Yang Liu; Tong Wang; Yiming Bao; Heng Yang

cs.RONov 3, 2026

GRASP: Gradient-Guided Sampling for Contact-Rich Physics-Based Motion Retargeting

Authors: Yang Liu, Tong Wang, Yiming Bao, Heng Yang

Organizations: School of Engineering and Applied Sciences, Harvard University

Abstract

We study physics-based retargeting of human demonstrations for robot learning: finding robot controls that reproduce reference motions under contact dynamics. We propose GRASP (Gradient Recovery from Archived Samples for Proposals), which combines sampling-based exploration with gradient-guided refinement without requiring a differentiable simulator. GRASP-TO extends tempered sequential Monte Carlo by fitting local quadratic cost models to archived rollouts and using their gradients to guide Markov chain Monte Carlo rejuvenation. Metropolis-Hastings correction preserves the target distribution for each fixed proposal kernel, and a population analysis connects the recovered gradients to Gaussian smoothing. GRASP-MPC scales this optimizer to long demonstrations through warm-started receding-horizon planning. Experiments on dexterous hands and humanoids show accurate retargeting with a shared tracking objective and no task-specific contact rewards, including 92.8% success on 1352 bimanual OakInk2 demonstrations. The refined references improve PPO tracking in pen spinning and humanoid-object interaction. We also demonstrate dexterous manipulation and whole-body humanoid motion on real hardware.

Figures & tables

Figure 2: Control-knot parameterization for one control channel, with

K=5

and illustrative values. (a) Triangular weights interpolate between neighboring knots; the shaded interval highlights two overlapping weights. (b) The optimized offsets

z_{k}

(large dots) define an offset at every simulation step by linear interpolation. (c) Adding this offset to the reference command produces

u_{t}

. The interpolation applies componentwise to all

m

control channels. The plot ends at the last knot; its offset is held over the final interval.

Figure 3: Recovering a smooth proposal direction from nonsmooth rollout costs, illustrated with one reduced coordinate. (a) Current particles (filled crimson) and earlier evaluations (open gray) both belong to the archive; the cost has a kink and a jump. (b) Illustrative Gaussian proximity weights with width

\sigma_{\rm fit}

favor samples near the current population center

y=0

; shading marks

|y|\leq\sigma_{\rm fit}

, and marker area indicates regression weight. (c) The weighted quadratic fit is differentiable despite these cost transitions. The Gaussian-smoothed cost is shifted vertically to match the fitted value at zero, allowing their local slopes and curvatures to be compared. Proposition 1 gives the exact derivative identities for the ideal Gaussian population fit. All values are illustrative.

Figure 4: GRASP-MPC. (a) Each window plans over a limited horizon and executes only a prefix (crimson), specified by the commit count

C

. The next window starts from the simulated state reached by that prefix. Full GRASP-TO searches alternate with local refinement, reusing overlapping plans while exposing a new tail. (b) For a full search after the first window, shifted unexecuted controls center the prior, with the last knot repeated to fill the new tail. The shared boundary offset is fixed, while the prior widens toward the tail. Dots mark control knots; controls interpolate linearly and hold the final knot over the last interval. Both panels illustrate

K=8

C=2

; control values and prior widths are illustrative. Refinement windows instead reuse the candidate library, as detailed in Algorithm 2 .

Method	Succ. (%) $\uparrow$	$E_{\mathrm{pos}}$ (m) $\downarrow$	$E_{\mathrm{rot}}$ (rad) $\downarrow$
Dex-retargeting [ 40 ]	28.6	0.08	0.62
SPIDER [ 35 ] mjwp	71.4	0.04	0.57
SPIDER [ 35 ] mjwp_act	77.1	0.04	0.42
Do-as-I-Do [ 34 ] (Sharpa)	81.0	0.03	0.15
w/o transition reward	79.0	0.03	0.14
annealed sampling only	72.0	0.08	0.32

Table 1: Bimanual retargeting with Sharpa hands. GRASP is evaluated on 1352 OakInk2 demonstrations. All sampling-based methods use

N=1{,}024

particles.

E_{\mathrm{pos}}

and

E_{\mathrm{rot}}

denote mean object position and rotation errors.

Hand	Finger DoF	Tasks	Succ. (%) $\uparrow$	$E_{\mathrm{pos}}$ (cm) $\downarrow$	$E_{\mathrm{rot}}$ (rad) $\downarrow$
Sharpa	22	1352	94.2	2.87 / 2.17	0.117 / 0.090
Wuji	20	1352	92.4	2.87 / 2.02	0.117 / 0.086
XHand	12	1352	95.0	2.87 / 2.08	0.112 / 0.088

Table 2: Cross-embodiment retargeting on OakInk2 with one optimizer configuration with

N=2{,}000

particles for all three hands on 1,352 tasks per hand. Execution failures count as failures in the success rate. Finger DoF are per hand; every hand also has a 6-DoF base.

E_{\mathrm{pos}}

and

E_{\mathrm{rot}}

are mean / median object position and rotation errors over completed trajectories with finite metrics.

Figure 5: Simulated dexterous manipulation across three hand embodiments. Rows show (a) Sharpa wiping a whiteboard, (b) Wuji capping a pen, and (c) XHand stirring in a cup. Each row shows six frames from one retargeted OakInk2 trajectory, ordered from left to right, with a fixed crop.

				ADD-AUC (%) $\uparrow$
Method	Split	# Demos	# Objects	Sharpa
SPIDER [ 35 ]	Right hand	2,718	33	40.74
GRASP	All	5,400	33	81.54
	Right hand	2,117	33	84.87
	Left hand	1,213	33	85.44
	Bimanual	2,070	33	75.83

Table 3: Object tracking on HOT3D. ADD-AUC integrates the fraction of frames whose mean object-surface point error falls below a threshold over thresholds of 0–10 cm, normalized to a percentage. We use 512 area-uniform surface samples and average trajectories equally. GRASP uses Sharpa hands on the evaluated Aria and Quest3 clips.

Figure 6: PPO learning on nine pen-spinning clips using kinematic references (TopoRetarget) or GRASP-refined references (ours). Panels show (a) episode reward, (b) episode length, (c) object position error, and (d) shaft-angle error. Bold curves are 100-iteration moving averages of the training logs; faint curves are the unsmoothed values.

Figure 7: Open-loop trajectory playback with real dexterous hands. (a) Lifting a pumpkin-shaped container by its handle. (b) Pouring from a can into a cup. Frames progress from left to right.

			Robot			Object
Robot	Data	Method	$e_{j}\downarrow$	$e_{t}\downarrow$	$e_{r}\downarrow$	$e_{t}\downarrow$	$e_{r}\downarrow$
G1	LAFAN1	GMR [ 1 ]	1.08	2.01	2.40	–	–
		SPIDER [ 35 ]	0.58	0.11	0.07	–	–
		GRASP	0.30	0.02	0.06	–	–
G1	AMASS	GMR [ 1 ]	6.2	4.1	18.7	–	–
		SPIDER [ 35 ]	0.75	0.23	0.08	–	–

Table 4: Whole-body tracking on the Unitree G1.

e_{j}

denotes joint error;

e_{t}

and

e_{r}

denote translation and orientation errors for the pelvis and, on OMOMO, the object.

Figure 8: Simulated humanoid–object interactions generated by GRASP. Rows show (a) the Unitree G1 lifting a box, (b) the Unitree H2 lifting and turning a table, and (c) the Booster T1 lifting a chair. Each row shows six frames from one trajectory, ordered from left to right.

Figure 9: Humanoid PPO learning with three reference motions from OmniRetarget. (a) Object position and (b) orientation tracking rewards are compared for the original kinematic trajectories (red) and GRASP-refined trajectories (blue). Higher rewards indicate better tracking of the corresponding reference.

Figure 10: Closed-loop execution of learned policies on real humanoids. Rows show (a) crawling, (b) lifting, carrying, and placing a box, and (c) pushing a box along the floor. Each row shows six frames from one execution, ordered from left to right. Backgrounds are blurred for clarity, preserving the robot, manipulated object, and immediate contact regions.

Setting	Value
Timing	0.005 s simulation step; 0.2 s knot spacing
MPC window	Up to 15 knots; commit count $C=2$
Particles	$N=1{,}024$ in Table 1 ; $N=2{,}000$ for all hands in Table 2 ; ESS target $0.9N$
SMC moves	5 Markov moves per stage; at most 64 tempering stages
Temperature	$\beta=T_{\mathrm{clip}}/\tau_{0}$ ; $\tau_{0}=0.006$ s
Prior	Appendix 9

Table 5: GRASP-MPC settings for dexterous-hand retargeting.

Setting	Value
Simulator	MJWarp; direct PD control; no domain randomization
Timing	$1/60$ s simulation step; 0.1 s knot spacing
MPC window	8 knots; commit count $C=1$ ; first knot pinned
Particles	$N=1{,}000$ ; ESS target $0.9N$
SMC moves	Surrogate MALA; full-prior preconditioner; 5 Markov moves per stage; no fixed stage cap
Surrogate	Rank 2; ridge $10^{-3}$ ; window parameter 3

Table 6: GRASP-MPC settings for humanoid tracking. The LAFAN1/AMASS weight tuple is (pelvis translation, pelvis rotation, joint angles, contact); the OMOMO tuple is (pelvis translation, pelvis rotation, joint angles, object translation, object rotation).

Setting	Value
Problem sizes	Both tasks: $H=200$ , $N=100$ ; control dimension 1 (acrobot) or 4 (pusher-slider), giving 200 or 800 variables
Gradient TSMC	3 NUTS transitions per stage; at most 8 tree doublings and 200 adaptive tempering stages
GRASP-TO	99 proposals per particle and stage; at most 200 adaptive tempering stages
RWM-TSMC	99 proposals per particle and stage; at most 200 adaptive tempering stages
Parallel-MPPI	100 independent particles; 64 rollouts per update; 15 pusher-slider updates; acrobot updates matched to the per-particle rollout budget of rollout-based TSMC
Hardware	NVIDIA A100 80 GB GPU

Table 7: Settings for the trajectory optimization benchmarks.

Method	Cost $\downarrow$	Time (s)	Cost $\downarrow$	Success	Time (s)
	Acrobot		Pusher-slider
Gradient-based TSMC	$2534\pm 119$	2069	$63.7\pm 4.4$	11/11	3075
GRASP-TO	$\mathbf{2504\pm 98}$	46	$\mathbf{54.3\pm 2.6}$	11/11	354
RWM-TSMC	$2787\pm 210$	39	$117.3\pm 9.7$	1/11	71
Parallel-MPPI	$3117\pm 150$	2	$188.6\pm 15.2$	0/11	2

Table 8: Smooth and nonsmooth trajectory optimization benchmarks. Cost reports the mean

\pm

standard deviation, across seeds, of the lowest-cost particle in each final population. We use seven seeds for acrobot and eleven for the pusher-slider. A pusher-slider run is solved when its lowest-cost final box pose is within 10 cm and 0.5 rad of the goal; time is measured in wall-clock seconds.

Figure 11: Cost distributions of the final particle populations for representative runs on (a) acrobot and (b) pusher-slider; diamonds mark the best particles. (c) Pusher-slider trajectories, colored by time, with dashed goal poses. The gradient-based TSMC comparator uses simulator derivatives for acrobot and a contact relaxation for the pusher-slider.

$N$	$e_{\mathrm{pos}}$ (cm)		$e_{\mathrm{rot}}$ (rad)		Wall time (min)	Success (%)
	Mean	Median	Mean	Median	Median
512	3.274	2.504	0.1346	0.1068	30.98	93.12
1024	3.164	2.301	0.1263	0.0982	39.43	92.75
2000	2.848	2.162	0.1165	0.0905	57.49	94.16

Table 9: Particle count ablation on Sharpa. Position errors

e_{\mathrm{pos}}

are in centimeters and rotation errors

e_{\mathrm{rot}}

in radians. Wall time measures the MPC loop from entry to return, excluding initialization and final evaluation.

Figure 12: Sharpa tracking errors for three particle counts: (a) position and (b) rotation. Boxes span the interquartile range, black lines mark medians, whiskers extend to observations within 1.5 interquartile ranges, and diamonds mark minimum task errors.

Motion Retargeting

Motion retargeting transfers movement across different bodies, skeletons, or robots while preserving its intent and physical feasibility.

See all 110 papers

Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation
Oct 8, 2026Dechen Gao, Yue Yang, Ben Abbatematteo +8Motion Retargeting Robotic Manipulation
Meta Reality Labs Research · UC Davis · UNC Chapel Hill +3
OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting
Oct 9, 2026Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon +3Motion Retargeting Robotic Manipulation
Multiply Labs, San Francisco, CA 94107, USA · Department of Control and Dynamical Systems, California Institute of Technology, Pasadena, CA 91125, USA · Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA
A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulation
Jul 13, 2026Yunhai Feng, Natalie Leung, Jiaxuan Wang +3Robot Skill Learning Contact-Rich Robotic Manipulation
Cornell University · Amazon FAR

Oct 8, 2026J/K move · Enter open · S save

Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation

Dechen Gao, Yue Yang, Ben Abbatematteo, Nathan Godwin+7

Meta Reality Labs Research · UC Davis · UNC Chapel Hill+3

Motion Retargeting Robotic Manipulation Human-to-Robot Motion Imitation Robot Motion Generation

Open paper AskPDF arXiv

Human demonstrations are a scalable data source for learning dexterous manipulation, but the embodiment gap prevents human motion from being executed directly on robots. Inverse kinematics (IK) retargets human motion to robots efficiently but ignores dynamics, often producing infeasible motions. Reinforcement learning (RL) and sampling-based model predictive control (MPC) are commonly employed to yield dynamically feasible motions, but both are sample-inefficient and sensitive to hyperparameters. RL suffers from costly and unstable training and tedious reward engineering; MPC avoids policy optimization, yet retargets each trajectory in isolation, and solving one does not make the next easier. Sampling cost grows rapidly with dataset size and task difficulty. We hypothesize that dynamically feasible trajectories concentrate near a low-dimensional manifold shared across demonstrations, so that retargeting can be reduced to sampling from that manifold, conditioned on human motion, rather than solving a fresh optimization problem for every demonstration. We propose Generative Neural Retargeting (GNR), which uses a flow matching model to sample feasible trajectories. GNR outperforms MPC with only

8.5\%

of the samples required by MPC, achieving a success rate of

56.20\%

compared to

27.20\%

for MPC. GNR can be used for scalable and efficient retargeting of large-scale, long-horizon, and millimeter precision human demonstrations: by applying GNR within a real-to-sim data engine, we produce a dexterous manipulation dataset with dense contact-force labels, spanning

223

k demonstrations and

3.3

k object geometries.

Abstract

Motion Retargeting

Explore similar work

Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation

OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting

A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulation