We present Whole-Body Mobile Manipulation Interface (HoMMI), a data collection and policy learning framework that learns whole-body mobile manipulation directly from robot-free human demonstrations. We augment UMI interfaces with egocentric sensing to capture the global context required for mobile manipulation, enabling portable, robot-free, and scalable data collection. However, naively incorporating egocentric sensing introduces a larger human-to-robot embodiment gap in both observation and action spaces, making policy transfer difficult. We explicitly bridge this gap with a cross-embodiment hand-eye policy design, including an embodiment agnostic visual representation; a relaxed head action representation; and a whole-body controller that realizes hand-eye trajectories through coordinated whole-body motion under robot-specific physical constraints. Together, these enable long-horizon mobile manipulation tasks requiring bimanual and whole-body coordination, navigation, and active perception. Results are best viewed on: https://hommi-robot.github.io
Figures & tables
Fig. 2 : HoMMI System Overview. We learn whole-body mobile manipulation from human demonstrations with an intuitive data collection interface (§ IV ), a cross-embodiment policy design with an embodiment-agnostic visual representation and a relaxed head action representation (§ V ), and a whole-body controller that achieves hand-eye tracking through whole-body motions respecting physical constraints (§ VI-B ).
Fig. 3 : Embodiment-Agnostic Visual Representation. We use a 3D representation for egocentric observations that allows using an embodiment-agnostic gripper coordinate frame, and masking out embodiment-specific arms and body observations.
Fig. 4 : Look-at Point Action Representation. To bridge the kinematic gap (e.g., height and neck DoF), we relax the head action constraint by representing the robot gaze as a “3D look-at point”. This representation allows effective active perception for gathering task-relevant information without over-constraining the robot to mimic human head motions exactly.
Fig. 5 : HoMMI Whole-Body Controller is designed to achieve precise end-effector tracking for accurate manipulation and effective active perception for information gathering. To do so, it uses (a) a relaxed head look-at point action representation that allows accurate bimanual end-effectors SE(3) tracking, circumventing the infeasibility and increased error associated with simultaneous 6-DoF head-hand tracking. In addition, we also apply (b) constraints and regularization to ensure stability and prevent the disastrous behaviors that would otherwise occur.
Fig. 6 : HoMMI Robot Hardware features a high DoF bimanual mobile manipulator with customized cameras and fingers that match the HoMMI data collection hardware.
Fig. 7 : Laundry Task. (a) Our cross-embodiment hand-eye policy rollout, highlighting our system’s capability of whole-body coordination and active perception. (b) Different test scenarios with different objects and bin locations. (c) Typical failure cases of the baselines.
Fig. 8 : Quantitative Results. Ours consistently outperforms baselines across all three long-horizon mobile manipulation tasks.
Fig. 9 : Delivery Task. (a) Our policy rollout, demonstrating long-horizon navigation over a large workspace and active perception. (b) Different test scenarios with different trolley locations and initial base positions and orientations. (c) Typical failure cases of the baselines.
Fig. 10 : Tablescape Task. (a) Our policy rollout, demonstrating precise bimanual and whole-body coordination. (b) Different test scenarios with different initial base positions and mat placement. (c) Typical failure cases of the baselines.
Fig. 11 : Egocentric Attention Comparison. We visualize attention maps for egocentric observations with yellow representing higher attention values. Ours exhibits clean attention highlighted around task-relevant objects, while baselines’ attentions are less informative.
Symbol
Laundry
Delivery
Tablescape
wp
10000
10000
10000
wo
10000
10000
10000
wnom,torso
50
1000
200
wnom,arm
50
1000
10
wcurr
50
1000
10
wbase,pos
50
50
5000
TABLE I : Whole-body IK parameters for the three evaluation tasks
Fig. 12 : Hardware Schematic.
Fig. 13 : High quality stereo pointcloud and informative attention.