Scaling up visual humanoid loco-manipulation skill acquisition requires reducing the human effort needed as tasks and spatial configurations expand, particularly in demonstration collection and task-specific training design. We present SUGARv2, a unified framework for learning diverse visual humanoid loco-manipulation skills through interaction reference synthesis and a shared training pipeline. To obtain diverse references at low cost, we adapt reusable task templates to different spatial configurations, synthesizing coordinated robot and object motions. To reduce task-specific reward design and training effort, we use reference-based imitation as a common learning objective across tasks. We first train privileged teachers to execute these references, then train visual policies through teacher distillation and reinforcement learning with the same reference-based mimic rewards, using diverse object instances and visual domain randomization to support visual generalization and sim-to-real transfer. Across five humanoid loco-manipulation tasks, SUGARv2 achieves a mean success rate of 91.0% in simulation. Deployment on a real humanoid robot further demonstrates successful sim-to-real transfer for whole-body manipulation. These results demonstrate the framework's potential for learning diverse visual humanoid loco-manipulation skills.
Figures & tables
Table 1: Comparison of humanoid loco-manipulation systems. No Teleop : no teleoperated demonstrations required. Whole-Body : joint upper- and lower-body control. Multi-task : pipeline reusable across tasks. Cost : cost of scaling up training data for each task. – : no data generation required.
Figure 2: Overview of SUGARv2. Left: each task uses an interaction template that is instantiated for different spatial configurations, yielding keyframes and timing to guide continuous interaction reference synthesis. Floor-to-Table is shown as an example, with four configurations c1 – c4 and their synthesized references τc1 – τc4 . Right: a privileged teacher is trained with a reference-based mimic reward to execute the interaction references through a frozen pretrained whole-body controller (WBC). The visual policy uses the same WBC and learns through teacher distillation and reinforcement learning with the same reward.
Figure 3: Synthesized interaction references across five tasks. Overlaid poses show references for different spatial configurations. Opacity indicates relative time within each displayed reference interval, with later poses shown more opaquely.
Figure 4: Visual diversity across (a) Floor-to-Table and (b) Basket Transport. Each paired example shows the same robot state and scene from a third-person view in the top row and an egocentric RGB view in the bottom row.
Data source
Learning method
Tabletop Pickup
Floor-to-Table
Box Carrying
Basket Transport
Same-Table
Mean
(a) Comparing learning methods
SUGARv2
VIRAL
63.2±0.9
13.4±4.6
20.4±9.1
41.4±1.2
59.7±4.8
39.6
SUGARv2
GR00T N1.7
38.9±2.3
13.8±2.4
15.4±1.4
20.6±3.4
28.2±2.2
23.4
(b) Comparing interaction reference sources
GRAIL
SUGARv2
43.3±0.7
0.6±0.2
83.1±1.3
11.7±2.1
2.1±0.1
28.2
(c) Full SUGARv2 framework
Table 2: Visual policy success rates (%) across five tasks. (a) VIRAL and GR00T using SUGARv2 references and teachers. (b) Our learning pipeline using GRAIL references. (c) The full SUGARv2 framework combining our reference synthesis and visual policy learning pipeline.
Task
Original
Unseen backgrounds
Unseen objects
Both unseen
Tabletop Pickup
92.8±1.5
90.6±0.3
86.9±1.1
83.7±1.0
Floor-to-Table
91.7±3.0
85.3±1.9
89.9±0.7
80.0±2.2
Box Carrying
87.6±1.9
83.5±1.4
85.2±1.4
81.6±1.7
Basket Transport
84.3±2.3
77.8±1.6
88.2±0.7
83.6±0.4
Same-Table
98.5±0.4
97.2±0.1
88.0±0.4
87.1±0.3
Mean
91.0
86.9
87.7
83.2
Table 3: SUGARv2 success rates (%) under unseen backgrounds and object instances, without additional training.
Setting
Tabletop Pickup
Floor-to-Table
Box Carrying
Basket Transport
Same-Table
Mean
Full SUGARv2
92.8±1.5
91.7±3.0
87.6±1.9
84.3±2.3
98.5±0.4
91.0
w/o teacher distillation
0.1±0.1
0.0±0.0
0.0±0.0
0.0±0.0
0.0±0.0
0.0
w/o reference-based RL
80.5±1.8
77.1±6.5
3.6±2.1
3.9±2.6
42.0±25.3
41.4
w/o continued teacher distillation
79.0±2.5
61.6±3.3
60.9±7.9
75.2±5.2
76.6±4.1
70.6
Table 4: Task success rates (%) for ablations of the learning strategy.
Metric
GRAIL
SUGARv2
Synthesis time (min/ref.) ↓
36.1
0.4
Teacher success (%) ↑
62.4
98.0
Object tracking error (cm) ↓
9.8
6.6
Robot tracking error (cm) ↓
13.8
10.0
Table 5: Reference synthesis cost and teacher performance, averaged across five tasks. Synthesis time includes failed attempts and postprocessing per retained reference. Tracking errors measure deviations of executed motions from references.
Figure 5: Visual policy success rates at different reference data scales. The dashed line denotes the mean across five tasks. Horizontal positions follow x0.6 ; tick labels show trajectory counts.
Building humanoid robots capable of generalizable whole-body loco-manipulation in the real world remains a fundamental challenge. Existing methods either rely on laborious task-specific reward engineering, rigidly replay reference motions that fail to generalize, or depend on costly teleoperation that limits scalability. While human videos capture diverse human behaviors, motion priors inferred from them are inherently imperfect, suffering from occlusion, contact artifacts, and retargeting errors that render them unsuitable for direct policy learning. To address this, we present SUGAR, a scalable data-driven framework that converts diverse human videos into deployable humanoid loco-manipulation skills, without any task-specific reward engineering or reference-motion conditioning at inference. SUGAR proceeds in three stages. First, a fully automated pipeline extracts kinematic interaction priors including human-object motion trajectories and contact labels from unstructured human videos. Second, a privileged physics-based refiner uses a unified mimic reward and progressive state pool to transform imperfect priors into physically feasible, high-fidelity skills. Third, refined skills are distilled into a hierarchical autonomous policy consisting of a command generator and a command tracker. We evaluate SUGAR on six representative loco-manipulation tasks in simulation and real-world humanoid hardware. Our method substantially outperforms reference-tracking baselines, and performance scales clearly with the amount of human video data. It also achieves zero-shot real-world transfer with reliable closed-loop execution, autonomous failure recovery, and stable long-horizon performance under external perturbations. Project Page: https://tianshuwu.github.io/sugar-humanoid/
Tianshu Wu, Xiangqi Kong, Yue Chen +5
1CFCS, School of Computer Science, Peking University · School of Computer Science and Engineering, Beihang University
Visual loco-manipulation policies that can generalize to novel scenes and objects have long been a goal of robotics research. However, today's data-hungry algorithms make collecting sufficient demonstrations a struggle for tabletop manipulation, and even more so for humanoids that must also walk and balance. Learning from simulated data and transferring that behavior to the real world, as is commonly done in locomotion, sidesteps this struggle, so we replicate that recipe for loco-manipulation. In doing so, we find that cloning synthetic demonstrations results in a low performance ceiling no matter the amount of training data. Reinforcement learning breaks through it, and refining the cloned policy with Flow-GRPO on a single sparse reward yields performance that synthetic behavior cloning cannot match. Together, these stages form our end-to-end sim-to-real pipeline spanning more than 150,000 scenes, which we use to train FetchMan. We evaluate it on FetchMan-Bench, a simulation benchmark we release, and deploy it zero-shot on a real Unitree G1, where our single-object reach-and-pick policy walks to and grasps a target across unseen scenes at 73.3% success. Finally, we extend this recipe to multi-object training, a first step toward loco-manipulation generalist policies at this data scale.
Omar Rayyan, Zhi Li, Max Argus +4
University of California, Los Angeles · Allen Institute for AI · University of Washington
Learning generalizable humanoid loco-manipulation remains bottlenecked by costly real-world demonstrations. Simulation provides a scalable alternative, but requires visually faithful, physics-ready scenes and executable whole-body motions. We present Zero-HLM, a real-to-sim-to-real framework that transforms smartphone captures and simulated source demonstrations into scalable training data for humanoid loco-manipulation. Gaussian-guided agentic real-to-sim reconstruction aligns visual and physical scenes; motion-environment co-adaptation uses rollout feedback to correct whole-body references and interaction geometry. We generate 4,800 physically validated demonstrations for zero-shot transfer across three Unitree G1 tasks. Simulation-trained GR00T N1.7 achieves 66.7% mean full-task success, versus 46.7% with real-data training. On generalized configurations, expanded simulation training raises pooled success from 23% for base-trained real-data policies to 69%, without additional real demonstrations. Project page: https://young-bit.github.io/zero-hlm/
Dianyi Yang, Mengya Liu, Zhumei Wang +9
Beijing Institute of Technology · State Key Laboratory of General Artificial Intelligence, BIGAI · Tsinghua University +1