Ego2Sim2Humanoid: Agentic Real-to-Sim for Learning Loco-Manipulation VLAs without Robot Demonstrations
Organizations: University of Notre Dame
Abstract
Imitation learning for visuomotor policies typically relies on teleoperated robot demonstrations, which are costly because every episode requires a trained operator controlling a physical robot in real time through teleoperation interfaces and wearable body-tracking hardware. Egocentric glasses capture human demonstrations without a robot or handheld grippers, yet they track only the head and in-view human hands, and directly copying human motion to a robot with different stature, arm reach and hand morphology fails to produce feasible contacts. \method{} converts egocentric glasses recordings into physically verified robot training trajectories. An agentic real-to-sim stage reconstructs collision and textured visual assets from the recorded RGB-D and head streams, completes unseen surfaces when needed, and predicts target contact bands on the manipulated objects. During motion adaptation, a grasp planner turns each contact band into a feasible robot wrist pose and finger closure, bilevel stance optimization plans collision-free standing postures that keep hand targets reachable, and a whole-body motion generator synthesizes lower-body locomotion and out-of-view arm motion. Two-stage motion correction then blends visibility transitions kinematically and trains a reinforcement-learning tracking policy in physics simulation under agent-designed semantic rewards, completing manipulation phases even when the object leaves the camera view. Rendering the reconstructed scene through the robot's onboard camera along the optimized rollout yields synchronized robot-view images, proprioception and verified action labels. Fine-tuned exclusively on this converted dataset, without real-robot demonstrations, GR00T N1.7 performed loco-manipulation tasks autonomously on a Unitree G1. Project page: https://robonana.github.io/ego2sim2humanoid/
Figures & tables
| Human | Planner | |
|---|---|---|
| Penetration (cm) | 1.8 [1.7, 2.0] | 0.8 [0.4, 1.4] |
| In-hand shift (cm) | 6.6 [5.3, 7.4] | 3.7 [2.6, 5.4] |
| In-hand rotation ( ) | 33.0 [29.1, 41.0] | 22.9 [18.1, 26.6] |
| Path | Hand (cm) | Grasps | ||
|---|---|---|---|---|
| EgoHumanoid | 71 | 0.5 | 70.8 | 0 |
| EgoHumanoid-V2, kinematic | 71 | 0.6 | 93.9 | 0 |
| EgoHumanoid-V2, full | 6 | 0.6 | 112.9 | 0 |
| E2S2H | 71 | 0.9 | 0.8 | 57 |