Current humanoid dexterous manipulation remains largely confined to end-effectors and simple grasps, falling short of humans' ability to engage the entire body by bracing and sliding bulky objects against the arms and torso. We introduce Whole-Body Dexterity (WBD), a general framework that uses the robot's body and appendages as contact surfaces to grasp and manipulate objects. Our sim-to-real pipeline first generates a dataset of feasible whole-body grasps, then trains reinforcement-learning experts on a general any-to-any repositioning objective by sampling initial and goal grasps from the dataset, and distills these experts into a single policy that uses proprioception and object geometry. We show that the distilled policy transfers zero-shot to a real Unitree G1, where it repositions a basketball and a cardboard box to regions around the torso and completes a long-horizon task that stows a box under one arm, carries it from a shelf to a table, and places it. The controller operates interactively, steered by coarse motion references from a human teleoperator. Across real-world whole-body manipulation tasks, the distilled controller completes 90% of trials, compared with 18% for direct retargeting and 4% for a generalist whole-body controller.
Figures & tables
Fig. 2: Whole-Body Dexterity Pipeline. (1) Generate a voxelized dataset of stable whole-body grasps. (2) Pre-train sphere WBD-EX experts on any-to-any repositioning using the grasp dataset. (3) Warm-start WBD-EX experts for other objects from the sphere experts. (4) Collect WBD-EX trajectories and distill them into a general dexterity controller for simulation and real-world tasks.
Fig. 3: Whole-Body Grasp Generation. Energy-based optimization [ 15 , 16 , 17 ] generates grasp candidates from specified arm, leg, and torso contact links (blue), followed by simulation validation under perturbations. The voxelized training set contains left-arm, right-arm, and both-arm grasps; leg grasps are excluded.
Fig. 4: Simulated object repositioning under noisy commands. Trials sample transfers among six body regions ( Section IV-B ). Training-object mean (top) and held-out sizes (bottom). Solid, dashed, and dotted lines show WBD refinement averaged over three training seeds, noisy experts, and clean experts, respectively. Bands denote pointwise 95% paired bootstrap confidence intervals clustered by source grasp over 256 matched trials per object, with training objects weighted equally. Trials are capped at 600 s.
Fig. 5: Simulated WBD controller experiment. Examples of regional transfers in which the start grasp is shown on the fully visible G1, while the sampled far goal grasp in another region is shown as the blue ghost.
Fig. 6: Real-world whole-body dexterous manipulation. On-body repositioning (bottom) and shelf pickup, stowing, and carrying (top) using the WBD controller, guided by the teleoperator (right).
Task
Controller
Basketball
Cardboard box
Reached ↑
Round trip ↑
Region (s) ↓
Total (s) ↓
Reached ↑
Round trip ↑
Region (s) ↓
Total (s) ↓
Shoulder tap
SONIC
2/10
0/10
15.2
—
2/10
2/10
15.9
24.9
GMR
3/10
1/10
26.0
30.1
6/10
4/10
8.0
8.5
WBD (ours)
10/10
9/10
7.7
17.3
10/10
10/10
4.1
6.3
Underarm stow
SONIC
1/10
0/10
28.0
—
2/10
0/10
26.1
—
GMR
4/10
2/10
31.5
49.8
2/10
1/10
9.3
17.8
TABLE I: Real-world on-body repositioning. Ten trials per controller, object, and task, five per side, with a size 3 basketball and a cardboard box (15 cm cube): region-reaching and full round-trip success. Times are measured from trial start and averaged over trials reaching the respective endpoint.
Controller
Pick ↑
Stow ↑
Carry ↑
Un-stow ↑
Place ↑
SONIC
10/10
0/10
0/10
0/10
0/10
GMR
10/10
3/10
2/10
1/10
1/10
WBD (ours)
10/10
9/10
9/10
9/10
7/10
TABLE II: Real-world shelf task. Completed stages over ten trials per controller with the cardboard box.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 7: Torso contact regions for the WBD grasp dataset. One accepted arm–torso grasp is shown per region, each on a different object.
Parameter
Value
Closure weight wclosure
1
Contact-distance weight wdist
300
Object-penetration weight wlink
200
Joint-limit weight wjoint
1
Self-collision weight wself
10
Contact-distance tolerance d⋆
2mm
Appendix
TABLE III: WBD grasp-generation parameters. Optimization weights and tolerances.
Group
Term
Dim.
Proprioception
Arm joint positions, minus default
14
Arm joint velocities
14
Servo error, commanded − measured
14
Previous action
14
Command
Hold flag, commanded and max. goal distance
3
Goal position relative to the object
3
Appendix
TABLE IV: WBD-EX observation terms. Current-timestep inputs, without history. The actor uses the first five groups with training noise; the critic uses their uncorrupted values plus the sampled dynamics parameters.
Per-geometry multiplier U(0.75,1.25) , sampled on the first reset and fixed per environment.
Object mass and inertia
Mass 0.4 – 1.2kg for every object, with inertia co-scaled. Sampled at startup and fixed per environment.
Object center of mass
Offset ±5mm independently per axis. Sampled at startup and fixed per environment.
Perturbation force
Magnitude U(0,Fmax) and isotropic direction, applied at the object center. No applied torque.
Force duration
0.05 – 0.20s per perturbation.
Cooldown
0.75 – 2.0s between perturbations.
Appendix
TABLE VI: WBD-EX domain randomization. Dynamics parameters are fixed per environment, while perturbation forces ramp up during training.
Fig. 8: Evaluation regions. Six torso-frame goal regions shown from two viewpoints. Required grasp modes are both arms for front and back, and the corresponding single arm for the underarms and shoulders.
Route
Region sequence
Underarm
F_LA_F
front → left underarm → front
F_RA_F
front → right underarm → front
LA_F_LA
left underarm → front → left underarm
RA_F_RA
right underarm → front → right underarm
Shoulder
Appendix
TABLE VII: Evaluation routes. Fourteen routes grouped by region. Each sequence includes the initial region; subsequent regions are goals.
Fig. 9: Training progress for one expert per geometry class. (a) Mean episode reward during training, averaged over up to 1,000 logged updates; bands are approximate 95% confidence intervals (CIs) from within-window block resampling. (b) Regional route completion, averaged equally over fourteen routes at 2,048 trials per route; bands are pointwise 95% evaluation CIs. One training seed per object.
Fig. 10: Regional route completion. Results at 50,000 iterations for one expert per geometry class, grouped by the body region the route visits. Error bars are pointwise 95% CIs from 2,048 trials per route. One training seed per object.
Completion by region group (%)
All 14
Object (mm)
Underarm
Shoulder
Back
Loop
Mean (%)
Sphere ∅ 120
75.9
39.8
69.0
61.5
61.57
Sphere ∅ 140
80.8
60.6
73.1
71.5
71.50
Sphere ∅ 160
85.6
61.3
80.9
80.2
76.56
Sphere ∅ 180
86.1
76.0
85.4
83.2
82.59
Sphere ∅ 200
88.5
77.8
85.6
86.6
84.33
Appendix
TABLE VIII: WBD-EX expert results per object. Equal-weight route means with 2,048 trials per route.
Underarm
Shoulder
Back
Loop
All 14 routes
Object (mm)
Scr.
Warm
Scr.
Warm
Scr.
Warm
Scr.
Warm
Scr.
Warm
Prism 100×100×200
35.3
70.7
27.0
47.1
18.9
48.5
0.8
18.2
23.30
50.12
Cube 140
67.5
72.2
55.2
57.4
46.2
45.4
15.6
19.7
50.48
52.81
Cube 200
55.1
61.3
68.9
69.7
0.3
12.5
0.0
0.0
35.53
40.98
Cylinder ∅140×160
68.2
69.2
29.6
45.5
56.7
49.1
25.2
21.3
47.74
49.83
Sphere ∅ 140
81.1
80.8
39.1
60.6
72.7
73.1
71.8
71.5
65.37
71.50
Appendix
TABLE IX: Warm-start versus from-scratch experts by body region , at 50,000 matched total iterations with 2,048 trials per route. Entries are mean regional route completion (%), equally weighted over each group’s routes. Bold marks the higher score when the 95% evaluation CI for the difference excludes zero. One training seed per condition.
Fig. 11: Warm-started and from-scratch learning curves. Warm-started curves include 15,000 sphere pre-training iterations (shaded); their first point precedes target-object training. Markers show mean regional route completion, weighting all fourteen routes equally. Bars show pointwise 95% evaluation CIs (binomial trial uncertainty or evaluation-batch bootstrap), often smaller than the markers. One training seed per condition.
Fig. 12: Grasp escapability distributions. Bars give the percentage of that object’s grasps falling in each ten-point escapability band, colored by geometry class. Error bars are pointwise 95% bootstrap CIs over grasps, with measured scores held fixed; counts are exact for the scored pools.
Fig. 13: Regional route completion after grasp filtering. Mean completion over twelve equally weighted routes, excluding loops. Dashed lines show unfiltered baselines; stars mark selected caps and gains in percentage points. Bands are pointwise 95% paired bootstrap CIs over evaluation trials. Vertical axes span twenty percentage points.
α
Method
Goals / trial
Time to failure (s)
2.5
Noisy expert
8.47
29.1
Noisy + EMA
7.82
20.5
Noisy + WBD
23.79
86.1
4
Noisy expert
6.24
24.2
Noisy + EMA
6.32
17.6
Noisy + WBD
21.79
64.6
Appendix
TABLE X: Command smoothing versus WBD refinement. Means weight the 13 training objects equally, and WBD values are averaged over three training seeds.
Recent work in humanoid whole-body control has found success with a simple recipe: retarget human motion to robot kinematic references, then train policies via reinforcement learning (RL) to track them. But how does this recipe transfer to dexterous manipulation? The answer is not obvious, as manipulation involves complex, contact-rich dynamics and requires delicate regulation of contact modes and forces. We present REGRIND, a minimalist retargeting-guided RL pipeline that learns dexterous manipulation policies from a single human demonstration. REGRIND retargets human hand-object motion to a robot reference that preserves hand-object spatial and contact relationships, trains a residual RL policy in simulation to track object-centric keypoints along that reference, and transfers the resulting policy zero-shot to hardware with careful system identification. The resulting policies produce fluid, human-like behavior on two different multi-fingered hands across contact-rich tool-use tasks, including operating a pair of scissors and turning a screwdriver. Through systematic hardware experiments, we identify and analyze the key factors that govern sim-to-real transfer in dexterous manipulation, offering practical guidance for retargeting-based learning in contact-rich settings. Videos and code are available at https://yunhaifeng.com/REGRIND.
Humanoid robots have many degrees of freedom and contact surfaces, yet most loco-manipulation research restricts manipulation to the hands and avoids other body contacts with the environment. To extend manipulation beyond the hands, we present PanTact, a system that learns humanoid whole-body dexterity, the capability to manipulate with any suitable body surface, directly from robot-free human demonstrations. Our key idea is to use contact as manipulation-centric information that transfers across embodiments. Contact locations and timing specify the interactions needed to accomplish a task, while the robot chooses how to coordinate its body to realize them within its own physical constraints. Specifically, PanTact captures, learns, and executes whole-body contact via: 1) a wearable, robot-free data collection interface that augments egocentric vision and bimanual UMI grippers with whole-body modular contact patches to record body contact locations and states; 2) a visuomotor policy that mitigates the egocentric observation gap and predicts whole-body contact goals; and 3) contact-aware whole-body control formulations that realize active contact goals using suitable robot surfaces. We deploy PanTact on a Unitree G1 humanoid without additional body contact sensors, demonstrating whole-body dexterity involving body support, pushing, sweeping, and load-sharing. Website: https://pantact.github.io
Xiaomeng Xu, Han Zhang, Zhian Ruan +6
Stanford University · Amazon FAR · Carnegie Mellon University
This paper investigates humanoid whole-body dexterous manipulation, where the efficient collection of high-quality demonstration data remains a central bottleneck. Existing teleoperation systems often suffer from limited portability, occlusion, or insufficient precision, which hinders their applicability to complex whole-body tasks. To address these challenges, we introduce HumDex, a portable teleoperation system designed for humanoid whole-body dexterous manipulation. Our system leverages IMU-based motion tracking to address the portability-precision trade-off, enabling accurate full-body tracking while remaining easy to deploy. For dexterous hand control, we further introduce a learning-based retargeting method that generates smooth and natural hand motions without manual parameter tuning. Beyond teleoperation, HumDex enables efficient collection of human motion data. Building on this capability, we propose a two-stage imitation learning framework that first pre-trains on diverse human motion data to learn generalizable priors, and then fine-tunes on robot data to bridge the embodiment gap for precise execution. We demonstrate that this approach significantly improves generalization to new configurations, objects, and backgrounds with minimal data acquisition costs. The entire system is fully reproducible and open-sourced at https://github.com/physical-superintelligence-lab/humdex.
Liang Heng, Yihe Tang, Jiajun Xu +3
USC Physical Superintelligence (PSI) Lab · WorldEngine AI