Visual loco-manipulation policies that can generalize to novel scenes and objects have long been a goal of robotics research. However, today's data-hungry algorithms make collecting sufficient demonstrations a struggle for tabletop manipulation, and even more so for humanoids that must also walk and balance. Learning from simulated data and transferring that behavior to the real world, as is commonly done in locomotion, sidesteps this struggle, so we replicate that recipe for loco-manipulation. In doing so, we find that cloning synthetic demonstrations results in a low performance ceiling no matter the amount of training data. Reinforcement learning breaks through it, and refining the cloned policy with Flow-GRPO on a single sparse reward yields performance that synthetic behavior cloning cannot match. Together, these stages form our end-to-end sim-to-real pipeline spanning more than 150,000 scenes, which we use to train FetchMan. We evaluate it on FetchMan-Bench, a simulation benchmark we release, and deploy it zero-shot on a real Unitree G1, where our single-object reach-and-pick policy walks to and grasps a target across unseen scenes at 73.3% success. Finally, we extend this recipe to multi-object training, a first step toward loco-manipulation generalist policies at this data scale.
Figures & tables
Figure 1: FetchMan learns visual humanoid loco-manipulation entirely in simulation and deploys it zero-shot to a real Unitree G1. The resulting policy walks up to and grasps objects across diverse, unseen real-world scenes, with no real-world data or per-scene tuning.
Figure 2: Simulated observations. Example frames of the head-fisheye and wrist streams the policy receives, across scenes randomized in texture, lighting, and camera intrinsics and extrinsics.
Figure 3: Synthetic data generation pipeline. Each episode samples (1) a house from MolmoSpaces, (2) a pickable target object within it, (3) a contact-free start and grasp-standoff goal pose connected by a planned path the scripted controller follows, and (4) domain randomization applied independently per episode.
Figure 4: Policy architecture. The head and wrist RGB streams are encoded by a frozen DINOv3 backbone into per-patch image tokens, while the proprioceptive vector (ht,rt,pt,qtupper,gt) is mapped by an MLP to a single state token. A flow-matching DiT cross-attends to these tokens and denoises a 15-dimensional action chunk, split into upper-body targets (waist, arm, and gripper) tracked by a joint-level PD controller and a base command (vx,vy,ω,hˉt) executed by the pre-trained lower-body locomotion controller.
Figure 5: Two-stage training pipeline. Stage 1 (Imitation Pretraining). The task sampler instantiates diverse simulated environments in which the scripted G1 staged policy generates demonstration trajectories, and the resulting frozen dataset trains the loco-manipulation policy π by behavior cloning. Stage 2 (Online GRPO Refinement). The same sampler resets groups of environments to identical tasks (group size × number of groups), π collects on-policy trajectories that differ only in sampled action noise, and a group-relative GRPO update refines π in place.
Figure 6: Real-world rollouts and the effect of RL refinement. Left. Representative zero-shot rollouts of FetchMan on the Unitree G1 across diverse, unseen scenes, where the policy navigates to the target bowl and grasps it from head- and wrist-camera observations alone. Right. The locomotion-to-manipulation handoff before and after Flow-GRPO refinement, on a fixed initialization. The behavior-cloned policy grasps as soon as the target is nearby and misses from too far, while after RL it walks in until the object is within reach and only then grasps.
Figure 7: Behavior cloning saturates, RL breaks through. Left. Single-object loco-manipulation SR (%) on the 100-scene validation set. The first segment scales BC demonstrations (5k–150k) and plateaus after 50k, while the second continues the 150k checkpoint with GRPO, which pushes past the imitation ceiling. A second run applies the same refinement to the 5k checkpoint and does not improve comparably. Right. The text-conditioned multi-object policy, whose BC success rate peaks near 40% before GRPO lifts it to roughly 62%. Time in GPU hours.
Table 8
Figure 8: Multi-object rollouts. Head-camera views from the text-conditioned policy grasping different target objects, one episode per row, with the prompted object name shown at left.
Imitation learning is a promising approach for training humanoid robots to both walk and manipulate, but it requires a large number of demonstrations, which are time-intensive and difficult to collect via teleoperation. Existing data-generation algorithms can automatically synthesize demonstrations for manipulators, but they are ineffective on humanoids because their high-dimensional composite action spaces involve arms, legs, and torsos. We present HumanoidMimicGen, a method for generating humanoid legged loco-manipulation data. Our method adapts contact-rich whole-body skills from a handful of source demonstrations to new states, generalizing across changes in object pose. By interleaving these single- and dual-arm skills with whole-body locomotion and manipulation planning, the method generates stable, collision-free data across diverse scenes and layouts. To evaluate our approach, we introduce a new simulated loco-manipulation benchmark containing nine diverse tasks that test humanoid loco-manipulation capabilities. There, we demonstrate that HumanoidMimicGen automatically generates large datasets for imitation learning and enables a systematic study of how data generation and policy learning decisions impact model performance. We show that whole-body visuomotor policies co-trained with data generated by HumanoidMimicGen outperform those trained only on real-world data by 20%.
Recent progress in robot manipulation has been largely driven by learning from large-scale demonstrations. For humanoid robot loco-manipulation tasks, however, existing data sources force an unsatisfying tradeoff between trajectory quality and scalability. Real-world teleoperation provides the highest-quality trajectories but requires dedicated physical space and time-consuming scene resets. Simulation offers an alternative way out of this dilemma: it can produce clean, embodiment-aligned data at scale without any physical hardware. In this paper, we propose OASIS, a simulation-data-driven framework for humanoid loco-manipulation. OASIS automatically reconstructs realistic object assets from real-world images using a 3D generative model. Based on these assets, trajectories are first collected through teleoperation in simulation, and then augmented under diverse domain randomizations in a post-processing stage. With the resulting simulation data, we further design a hierarchical visuomotor policy for humanoid loco-manipulation. Extensive experiments on the real humanoid robot show that, under zero-shot deployment, the policy trained on our simulation data achieves higher success rates on most tasks than that trained on real-robot teleoperation data, owing largely to the broad lighting and environmental variations covered by our simulation rendering, which real-robot data fails to capture. The project page is available at https://oasis-humanoid.github.io/.
Zehao Yu, Jiakun Zheng, Weiji Xie +4
Institute of Artificial Intelligence (TeleAI), China Telecom · Fudan University · East China University of Science and Technology +1
Humanoid loco-manipulation requires adaptive whole-body coordination to seamlessly integrate locomotion and physical interaction. Despite recent advances, learning autonomous loco-manipulation remains challenging due to the scarcity of diverse, physically executable robot-object interaction data and the difficulty of learning unified whole-body control directly from onboard observations. We present ViLoMan, a scalable framework for autonomous humanoid loco-manipulation. ViLoMan first transforms partial kinematic demonstrations of human-object interactions into complete, physically executable robot trajectories. It then leverages these trajectories within a teacher-student distillation framework to learn a unified policy that maps egocentric depth observations and proprioceptive measurements directly to joint-level whole-body actions. During deployment, the policy requires neither reference motions nor intermediate commands. We evaluate ViLoMan on door-closing tasks across diverse door configurations and robot initial conditions in both simulation and the real world. Experimental results demonstrate that a single policy enables a Unitree G1 humanoid to complete the full task using only onboard depth sensing and proprioception, while generalizing robustly across task variations and transferring effectively from simulation to reality. Project page: viloman-anonymous.pages.dev.
Zejie Tian, Ruibing Hou, Bingpeng Ma +2
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, China.