Humanoid robots have many degrees of freedom and contact surfaces, yet most loco-manipulation research restricts manipulation to the hands and avoids other body contacts with the environment. To extend manipulation beyond the hands, we present PanTact, a system that learns humanoid whole-body dexterity, the capability to manipulate with any suitable body surface, directly from robot-free human demonstrations. Our key idea is to use contact as manipulation-centric information that transfers across embodiments. Contact locations and timing specify the interactions needed to accomplish a task, while the robot chooses how to coordinate its body to realize them within its own physical constraints. Specifically, PanTact captures, learns, and executes whole-body contact via: 1) a wearable, robot-free data collection interface that augments egocentric vision and bimanual UMI grippers with whole-body modular contact patches to record body contact locations and states; 2) a visuomotor policy that mitigates the egocentric observation gap and predicts whole-body contact goals; and 3) contact-aware whole-body control formulations that realize active contact goals using suitable robot surfaces. We deploy PanTact on a Unitree G1 humanoid without additional body contact sensors, demonstrating whole-body dexterity involving body support, pushing, sweeping, and load-sharing. Website: https://pantact.github.io
Figures & tables
Figure 1: PanTact gives a humanoid whole-body dexterity, learned from robot-free human demonstrations. (a) The data collection interface captures whole-body contacts alongside vision and motion. (b) Contact provides an interface for human-to-robot transfer. The visuomotor policy predicts contact goals, and the whole-body controller realizes them using suitable robot surfaces. (c) PanTact enables diverse whole-body dexterous skills on a humanoid.
Figure 2: PanTact modular contact patch. Sensor, connector, and strap modules snap together into patches of arbitrary shape and extent. We showcase arm and knee patches.
Figure 3: Cross-Embodiment Contact Interface. The Contact-Predictive Whole-Body Manipulation Policy uses body-masked 3D egocentric observations, wrist images, and proprioception to predict EEF trajectories and contact goals (position and binary state). The Contact-Aware Whole-Body Controller coordinates whole-body motion to track EEF commands and realize active contact goals using suitable robot surfaces.
Figure 4: Grasp-and-Embrace. (a) Autonomous rollout of our policy deployed with Upper-Body IK , demonstrating capability to support a bulky object with arms while grasping small object with a gripper. (b) Overlayed test configurations with varied object placements and initial robot configurations. (c) Typical failure case of Humanoid UMI is insufficient arm contact to support the bulky object. (d) Task success rates. (e) Real-world tracking plots.
Figure 5: Sweep-and-Stow. (a) Our policy rollout with WBIK+SONIC tracker, demonstrating arm sweeping and lower-body pushing and locomotion at the same time. (b) Test configurations with seen and unseen object combinations, varied cabinet openings, and different robot-to-cabinet distances. (c) Typical failure case of the Humanoid UMI++ baseline include the gripper passes above an object or knocks it outside the bin, and the robot fails to advance a leg for cabinet closure as its arms approach joint limits. (d) Stage-wise and end-to-end success rates.
Figure 6: Carry-and-Push. (a) Our policy with WBIK+SONIC tracker, demonstrating arms load-sharing while walking and pushing with lower body. (b) Test scenarios with varied box placements and robot postures. (c) Failure case of Humanoid UMI++ , dropping the carried box because of insufficient contact support. (d) Task success rates. (e) Tracking plots of real-world rollouts.
Figure 7: Controller Comparisons. We replay the demonstrations in simulation with our three whole-body control formulations, with and without contact goals.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Transformer depth / width / heads
10 / 768 / 12
Training diffusion steps / inference DDIM steps
100 / 16
Training epochs
500
Effective batch size
48
Base learning rate
7.5×10−5
Encoder learning rate
7.5×10−6
Appendix
Table 1: Policy architecture and training hyperparameters.