Current approaches for humanoid whole-body manipulation, primarily relying on teleoperation or visual sim-to-real reinforcement learning, are hindered by hardware logistics and complex reward engineering. Consequently, demonstrated autonomous skills remain limited and are typically restricted to controlled environments. In this paper, we present the Humanoid Manipulation Interface (HuMI), a portable and efficient framework for learning diverse whole-body manipulation tasks across various environments. HuMI enables humanoid-free data collection by capturing rich whole-body motion using portable hardware. This data drives a hierarchical learning pipeline that translates human motions into dexterous and feasible humanoid skills. Extensive experiments across five whole-body tasks--including kneeling, squatting, tossing, walking, and bimanual manipulation--demonstrate that HuMI achieves a 3x increase in data collection efficiency compared to teleoperation and attains a 70% success rate in unseen environments.
Figures & tables
Figure 1: Humanoid Manipulation Interface (HuMI) . Left: A portable, humanoid-free data collection setup for transferring skills from humans to humanoids across varied environments. Right: Whole-body behaviors learned with the framework.
Figure 2: HuMI data collection. (a) Challenges : Gripper-only data under-specify whole-body motion, while naive scaling breaks scene alignment. (b) Hardware : Portable grippers and trackers with real-time IK preview to ensure feasibility. (c) Data Processing : Task-space SE(3) trajectories train the high-level policy, while whole-body IK solutions supervise the low-level controller.
Figure 3: HuMI control hierarchy. A 5Hz Diffusion Policy maps images and proprioception to task-space keypoint chunks; a 50Hz whole-body controller converts these targets into joint actions.
Figure 4: Impact of reference frame selection on tracking-induced chunk boundary offsets.
Figure 5: Within-chunk relative tracking mitigates drift in non-vision-grounded keypoints.
Figure 6: Whole-body capability results. (a) HuMI performance on four tasks. (b) Baseline failure modes ; red boxes highlight the failure behavior details. (c) Success rates for each task.
Figure 7: Data collection efficiency comparison.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: HuMI’s hardware setup .
Figure 9: IK preview interface .
Figure 10: Effect of online IK preview . Without online preview, unscaled human motions frequently exceed the target humanoid’s kinematic workspace or cause self-collisions.
Figure 11: Additional TWIST2 comparisons. Left: time required to collect 100 demonstrations for bimanual pick-and-place. Right: illustration of the single-knee pick-up task.
High-quality demonstration data are essential for humanoid robot skill learning, especially for whole-body behaviors that require coordinated perception, locomotion, and manipulation. Existing data-collection methods largely rely on robot teleoperation, which is constrained by hardware accessibility, operator expertise, and limited efficiency. Inspired by the Universal Manipulation Interface (UMI), we propose HumanoidUMI, a portable and robot-free framework for humanoid whole-body data collection. HumanoidUMI uses lightweight VR devices and UMI-inspired grippers to collect sparse human keypoint trajectories, wrist-view observations, and gripper actions. These demonstrations train a high-level policy to predict future keypoints, which are retargeted to robot-native whole-body references and executed by a whole-body controller. Experiments in five real-world scenarios demonstrate the effectiveness of the proposed framework and validate the collected demonstrations for transferable humanoid whole-body skill learning.
We present Whole-Body Mobile Manipulation Interface (HoMMI), a data collection and policy learning framework that learns whole-body mobile manipulation directly from robot-free human demonstrations. We augment UMI interfaces with egocentric sensing to capture the global context required for mobile manipulation, enabling portable, robot-free, and scalable data collection. However, naively incorporating egocentric sensing introduces a larger human-to-robot embodiment gap in both observation and action spaces, making policy transfer difficult. We explicitly bridge this gap with a cross-embodiment hand-eye policy design, including an embodiment agnostic visual representation; a relaxed head action representation; and a whole-body controller that realizes hand-eye trajectories through coordinated whole-body motion under robot-specific physical constraints. Together, these enable long-horizon mobile manipulation tasks requiring bimanual and whole-body coordination, navigation, and active perception. Results are best viewed on: https://hommi-robot.github.io
High-quality demonstration data are essential for humanoid robot skill learning, especially for whole-body behaviors that require coordinated perception, locomotion, and manipulation. Existing data-collection methods largely rely on robot teleoperation, which is constrained by hardware accessibility, operator expertise, and limited efficiency. Inspired by the Universal Manipulation Interface (UMI), we propose BifrostUMI, a portable and robot-free framework for humanoid whole-body data collection. BifrostUMI uses lightweight VR devices and UMI-inspired grippers to collect sparse human keypoint trajectories, wrist-view observations, and gripper actions. These demonstrations train a high-level policy to predict future keypoints, which are retargeted to robot-native whole-body references and executed by a whole-body controller. Experiments in five real-world scenarios demonstrate the effectiveness of the proposed framework and validate the collected demonstrations for transferable humanoid whole-body skill learning.