Whole-Body Dexterity

Conor Mc Gartoll; An Dang; Pranay Thangeda; Arjun Gupta; Carolina Higuera; Nima Fazeli; Mustafa Mukadam; Manikantan Nambi

cs.ROOct 15, 2026

Whole-Body Dexterity

Authors: Conor Mc Gartoll, An Dang, Pranay Thangeda, Arjun Gupta, Carolina Higuera, Nima Fazeli, Mustafa Mukadam, Manikantan Nambi

Organizations: Amazon · University of California, San Diego · University of Michigan

Abstract

Current humanoid dexterous manipulation remains largely confined to end-effectors and simple grasps, falling short of humans' ability to engage the entire body by bracing and sliding bulky objects against the arms and torso. We introduce Whole-Body Dexterity (WBD), a general framework that uses the robot's body and appendages as contact surfaces to grasp and manipulate objects. Our sim-to-real pipeline first generates a dataset of feasible whole-body grasps, then trains reinforcement-learning experts on a general any-to-any repositioning objective by sampling initial and goal grasps from the dataset, and distills these experts into a single policy that uses proprioception and object geometry. We show that the distilled policy transfers zero-shot to a real Unitree G1, where it repositions a basketball and a cardboard box to regions around the torso and completes a long-horizon task that stows a box under one arm, carries it from a shelf to a table, and places it. The controller operates interactively, steered by coarse motion references from a human teleoperator. Across real-world whole-body manipulation tasks, the distilled controller completes 90% of trials, compared with 18% for direct retargeting and 4% for a generalist whole-body controller.

Figures & tables

Fig. 2: Whole-Body Dexterity Pipeline. (1) Generate a voxelized dataset of stable whole-body grasps. (2) Pre-train sphere WBD-EX experts on any-to-any repositioning using the grasp dataset. (3) Warm-start WBD-EX experts for other objects from the sphere experts. (4) Collect WBD-EX trajectories and distill them into a general dexterity controller for simulation and real-world tasks.

Fig. 3: Whole-Body Grasp Generation. Energy-based optimization [ 15 , 16 , 17 ] generates grasp candidates from specified arm, leg, and torso contact links (blue), followed by simulation validation under perturbations. The voxelized training set contains left-arm, right-arm, and both-arm grasps; leg grasps are excluded.

Fig. 4: Simulated object repositioning under noisy commands. Trials sample transfers among six body regions ( Section IV-B ). Training-object mean (top) and held-out sizes (bottom). Solid, dashed, and dotted lines show WBD refinement averaged over three training seeds, noisy experts, and clean experts, respectively. Bands denote pointwise

95\%

paired bootstrap confidence intervals clustered by source grasp over

256

matched trials per object, with training objects weighted equally. Trials are capped at

600

Fig. 5: Simulated WBD controller experiment. Examples of regional transfers in which the start grasp is shown on the fully visible G1, while the sampled far goal grasp in another region is shown as the blue ghost.

Fig. 6: Real-world whole-body dexterous manipulation. On-body repositioning (bottom) and shelf pickup, stowing, and carrying (top) using the WBD controller, guided by the teleoperator (right).

Task	Controller	Basketball				Cardboard box
Task	Controller	Reached $\uparrow$	Round trip $\uparrow$	Region (s) $\downarrow$	Total (s) $\downarrow$	Reached $\uparrow$	Round trip $\uparrow$	Region (s) $\downarrow$	Total (s) $\downarrow$
Shoulder tap	SONIC	2/10	0/10	15.2	—	2/10	2/10	15.9	24.9
	GMR	3/10	1/10	26.0	30.1	6/10	4/10	8.0	8.5
	WBD (ours)	10/10	9/10	7.7	17.3	10/10	10/10	4.1	6.3
Underarm stow	SONIC	1/10	0/10	28.0	—	2/10	0/10	26.1	—
	GMR	4/10	2/10	31.5	49.8	2/10	1/10	9.3	17.8

TABLE I: Real-world on-body repositioning. Ten trials per controller, object, and task, five per side, with a size 3 basketball and a cardboard box (15 cm cube): region-reaching and full round-trip success. Times are measured from trial start and averaged over trials reaching the respective endpoint.

Controller	Pick $\uparrow$	Stow $\uparrow$	Carry $\uparrow$	Un-stow $\uparrow$	Place $\uparrow$
SONIC	10/10	0/10	0/10	0/10	0/10
GMR	10/10	3/10	2/10	1/10	1/10
WBD (ours)	10/10	9/10	9/10	9/10	7/10

TABLE II: Real-world shelf task. Completed stages over ten trials per controller with the cardboard box.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 7: Torso contact regions for the WBD grasp dataset. One accepted arm–torso grasp is shown per region, each on a different object.

Parameter	Value
Closure weight $w_{\mathrm{closure}}$	$1$
Contact-distance weight $w_{\mathrm{dist}}$	$300$
Object-penetration weight $w_{\mathrm{link}}$	$200$
Joint-limit weight $w_{\mathrm{joint}}$	$1$
Self-collision weight $w_{\mathrm{self}}$	$10$
Contact-distance tolerance $d^{\star}$	$2\,\mathrm{mm}$

Appendix

TABLE III: WBD grasp-generation parameters. Optimization weights and tolerances.

Group	Term	Dim.
Proprioception	Arm joint positions, minus default	14
	Arm joint velocities	14
	Servo error, commanded $-$ measured	14
	Previous action	14
Command	Hold flag, commanded and max. goal distance	3
Command	Goal position relative to the object	3

Appendix

TABLE IV: WBD-EX observation terms. Current-timestep inputs, without history. The actor uses the first five groups with training noise; the critic uses their uncorrupted values plus the sampled dynamics parameters.

Term	Coefficient
Progress	$100$
Goal-region arrival	$0.15$
Transition completion	$20$
Hold quality	$0.2$
Arm neutrality	$0.5$
Failure penalty	$-20$

Appendix

TABLE V: Reward coefficients. Negative values denote penalties.

Parameter	Range and sampling schedule
Robot friction	Per-geometry multiplier $\mathcal{U}(0.75,1.25)$ , sampled on the first reset and fixed per environment.
Object mass and inertia	Mass $0.4$ – $1.2\,\mathrm{kg}$ for every object, with inertia co-scaled. Sampled at startup and fixed per environment.
Object center of mass	Offset $\pm 5\,\mathrm{mm}$ independently per axis. Sampled at startup and fixed per environment.
Perturbation force	Magnitude $\mathcal{U}(0,F_{\max})$ and isotropic direction, applied at the object center. No applied torque.
Force duration	$0.05$ – $0.20\,\mathrm{s}$ per perturbation.
Cooldown	$0.75$ – $2.0\,\mathrm{s}$ between perturbations.

Appendix

TABLE VI: WBD-EX domain randomization. Dynamics parameters are fixed per environment, while perturbation forces ramp up during training.

Fig. 8: Evaluation regions. Six torso-frame goal regions shown from two viewpoints. Required grasp modes are both arms for front and back, and the corresponding single arm for the underarms and shoulders.

Route	Region sequence
Underarm
F_LA_F	front $\to$ left underarm $\to$ front
F_RA_F	front $\to$ right underarm $\to$ front
LA_F_LA	left underarm $\to$ front $\to$ left underarm
RA_F_RA	right underarm $\to$ front $\to$ right underarm
Shoulder

Appendix

TABLE VII: Evaluation routes. Fourteen routes grouped by region. Each sequence includes the initial region; subsequent regions are goals.

Fig. 9: Training progress for one expert per geometry class. (a) Mean episode reward during training, averaged over up to

1{,}000

logged updates; bands are approximate

95\%

confidence intervals (CIs) from within-window block resampling. (b) Regional route completion, averaged equally over fourteen routes at

2{,}048

trials per route; bands are pointwise

95\%

evaluation CIs. One training seed per object.

Fig. 10: Regional route completion. Results at

50{,}000

iterations for one expert per geometry class, grouped by the body region the route visits. Error bars are pointwise

95\%

CIs from

2{,}048

trials per route. One training seed per object.

	Completion by region group (%)				All 14
Object (mm)	Underarm	Shoulder	Back	Loop	Mean (%)
Sphere $\varnothing$ 120	75.9	39.8	69.0	61.5	61.57
Sphere $\varnothing$ 140	80.8	60.6	73.1	71.5	71.50
Sphere $\varnothing$ 160	85.6	61.3	80.9	80.2	76.56
Sphere $\varnothing$ 180	86.1	76.0	85.4	83.2	82.59
Sphere $\varnothing$ 200	88.5	77.8	85.6	86.6	84.33

Appendix

TABLE VIII: WBD-EX expert results per object. Equal-weight route means with

2{,}048

trials per route.

	Underarm		Shoulder		Back		Loop		All 14 routes
Object (mm)	Scr.	Warm	Scr.	Warm	Scr.	Warm	Scr.	Warm	Scr.	Warm
Prism $100\!\times\!100\!\times\!200$	35.3	70.7	27.0	47.1	18.9	48.5	0.8	18.2	23.30	50.12
Cube 140	67.5	72.2	55.2	57.4	46.2	45.4	15.6	19.7	50.48	52.81
Cube 200	55.1	61.3	68.9	69.7	0.3	12.5	0.0	0.0	35.53	40.98
Cylinder $\varnothing 140\!\times\!160$	68.2	69.2	29.6	45.5	56.7	49.1	25.2	21.3	47.74	49.83
Sphere $\varnothing$ 140	81.1	80.8	39.1	60.6	72.7	73.1	71.8	71.5	65.37	71.50

Appendix

TABLE IX: Warm-start versus from-scratch experts by body region , at 50,000 matched total iterations with

2{,}048

trials per route. Entries are mean regional route completion (%), equally weighted over each group’s routes. Bold marks the higher score when the

95\%

evaluation CI for the difference excludes zero. One training seed per condition.

Fig. 11: Warm-started and from-scratch learning curves. Warm-started curves include

15{,}000

sphere pre-training iterations (shaded); their first point precedes target-object training. Markers show mean regional route completion, weighting all fourteen routes equally. Bars show pointwise

95\%

evaluation CIs (binomial trial uncertainty or evaluation-batch bootstrap), often smaller than the markers. One training seed per condition.

Fig. 12: Grasp escapability distributions. Bars give the percentage of that object’s grasps falling in each ten-point escapability band, colored by geometry class. Error bars are pointwise

95\%

bootstrap CIs over grasps, with measured scores held fixed; counts are exact for the scored pools.

Fig. 13: Regional route completion after grasp filtering. Mean completion over twelve equally weighted routes, excluding loops. Dashed lines show unfiltered baselines; stars mark selected caps and gains in percentage points. Bands are pointwise

95\%

paired bootstrap CIs over evaluation trials. Vertical axes span twenty percentage points.

$\alpha$	Method	Goals / trial	Time to failure (s)
$2.5$	Noisy expert	$8.47$	$29.1$
	Noisy + EMA	$7.82$	$20.5$
	Noisy + WBD	$\mathbf{23.79}$	$\mathbf{86.1}$
$4$	Noisy expert	$6.24$	$24.2$
	Noisy + EMA	$6.32$	$17.6$
	Noisy + WBD	$\mathbf{21.79}$	$\mathbf{64.6}$

Appendix

TABLE X: Command smoothing versus WBD refinement. Means weight the

13

training objects equally, and WBD values are averaged over three training seeds.

Whole-Body Robotic Manipulation

Whole-Body Robotic Manipulation coordinates a robot’s arms, hands, body, and mobility to manipulate objects while maintaining balance and contact.

See all 88 papers

A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulation
Jul 13, 2026Yunhai Feng, Natalie Leung, Jiaxuan Wang +3Robot Skill Learning Contact-Rich Robotic Manipulation
Cornell University · Amazon FAR
PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface
Oct 8, 2026Xiaomeng Xu, Han Zhang, Zhian Ruan +6Whole-Body Robotic Manipulation Contact-Rich Robotic Manipulation
Stanford University · Amazon FAR · Carnegie Mellon University
HumDex: Humanoid Dexterous Manipulation Made Easy
Mar 12, 2026Liang Heng, Yihe Tang, Jiajun Xu +3Robotic Manipulation Whole-Body Robotic Manipulation
USC Physical Superintelligence (PSI) Lab · WorldEngine AI

Jul 13, 2026J/K move · Enter open · S save

A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulation

Yunhai Feng, Natalie Leung, Jiaxuan Wang, Lujie Yang+2

Cornell University · Amazon FAR

Robot Skill Learning Contact-Rich Robotic Manipulation Sim-to-Real Transfer Motion Retargeting Robotic Manipulation Robot Imitation Learning+1

Open paper AskPDF arXiv

Recent work in humanoid whole-body control has found success with a simple recipe: retarget human motion to robot kinematic references, then train policies via reinforcement learning (RL) to track them. But how does this recipe transfer to dexterous manipulation? The answer is not obvious, as manipulation involves complex, contact-rich dynamics and requires delicate regulation of contact modes and forces. We present REGRIND, a minimalist retargeting-guided RL pipeline that learns dexterous manipulation policies from a single human demonstration. REGRIND retargets human hand-object motion to a robot reference that preserves hand-object spatial and contact relationships, trains a residual RL policy in simulation to track object-centric keypoints along that reference, and transfers the resulting policy zero-shot to hardware with careful system identification. The resulting policies produce fluid, human-like behavior on two different multi-fingered hands across contact-rich tool-use tasks, including operating a pair of scissors and turning a screwdriver. Through systematic hardware experiments, we identify and analyze the key factors that govern sim-to-real transfer in dexterous manipulation, offering practical guidance for retargeting-based learning in contact-rich settings. Videos and code are available at https://yunhaifeng.com/REGRIND.

Abstract

Whole-Body Robotic Manipulation

Explore similar work

A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulation

PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface

HumDex: Humanoid Dexterous Manipulation Made Easy