Organizations: Beijing Institute of Technology · State Key Laboratory of General Artificial Intelligence, BIGAI · Tsinghua University · Delta Intelligence
Learning generalizable humanoid loco-manipulation remains bottlenecked by costly real-world demonstrations. Simulation provides a scalable alternative, but requires visually faithful, physics-ready scenes and executable whole-body motions. We present Zero-HLM, a real-to-sim-to-real framework that transforms smartphone captures and simulated source demonstrations into scalable training data for humanoid loco-manipulation. Gaussian-guided agentic real-to-sim reconstruction aligns visual and physical scenes; motion-environment co-adaptation uses rollout feedback to correct whole-body references and interaction geometry. We generate 4,800 physically validated demonstrations for zero-shot transfer across three Unitree G1 tasks. Simulation-trained GR00T N1.7 achieves 66.7% mean full-task success, versus 46.7% with real-data training. On generalized configurations, expanded simulation training raises pooled success from 23% for base-trained real-data policies to 69%, without additional real demonstrations. Project page: https://young-bit.github.io/zero-hlm/
Figures & tables
Figure 1: Top: Zero-HLM generates humanoid loco-manipulation data covering diverse whole-body skills in simulation. Bottom: Policies trained solely on these data achieve direct sim-to-real transfer on a Unitree G1 under Base and generalize across Expanded configurations.
Figure 2: Overview of Zero-HLM. Left: Gaussian-guided agentic real-to-sim reconstruction aligns the physical scene with captured appearance and geometry. Middle: Motion–environment co-adaptation generates whole-body data from source demonstrations. Right: successful rollouts are paired with egocentric renderings to train policies for zero-shot real-world deployment.
Task
Entity / configuration
ΔX (cm)
ΔY (cm)
Δ Yaw ( ∘ )
A
Robot start: Base / Expanded [i,j] (a)
±15
±15
±15
Picking location: Base / Expanded [i,j] (c)
±2.5
±2.5
±15
Placement location: Base / Expanded [i,j] (c)
±2.5
±2.5
±20
Robot interaction pose
±10
±15
±15
B
Robot start: Base / Expanded [i,j] (b)
±15
±15
±15
Robot pickup pose
±10
±10
±15
Table 2: Simulation Configuration Sampling. Offsets are relative to the corresponding Base or Expanded [i,j] center. Endpoint widths describe a sampling region, rather than an independent object perturbation.
Table 4
Figure 5: Sim-Ours Generation Outcomes and Failure Breakdown. Ribbons partition rollout attempts by task segment and outcome. Generation success is 33%, 34%, and 25% for Tasks A, B, and C; ribbon percentages are rounded.
Figure 6: GPT-6 Astra Generation Outcomes After Stabilization. Success rates are 21.38%, 24.70%, and 0% for Tasks A–C. Task C failures occur at Bi-Pick (87%) and Kick (13%); its total attempt count is unavailable. Place* groups placement, hand-withdrawal, and acceptance failures.
Table 7
Figure 7: Mesh, Gaussian-Splatting, and Real-World Observations. Tasks A and B. Mesh and GS render the same simulation sequence; Real shows comparable stages from a separate physical rollout. Timestamps are independent across domains, not frame-aligned.
Figure 8: Sim-Data Scaling on Task A in Real-World Evaluation.
Table 10
Figure 9: Tasks and Generalization Evaluation. (a) Task A–C stage sequences. (b) Robot starts for Tasks A and B, picking and placement locations, and an alternative object–goal pair (top to bottom). Triangles/pins mark Base / Expanded ; colors show real-world successes per five trials, from yellow (0/5) to blue (5/5).
Figure 12
Figure 11: Co-Training Performance on Task A.
Figure 12: Benchmark Setup. Top-down layouts and sampled robot and object configurations for Tasks A–C, with enlarged views of the interaction regions.