Visual loco-manipulation policies that can generalize to novel scenes and objects have long been a goal of robotics research. However, today's data-hungry algorithms make collecting sufficient demonstrations a struggle for tabletop manipulation, and even more so for humanoids that must also walk and balance. Learning from simulated data and transferring that behavior to the real world, as is commonly done in locomotion, sidesteps this struggle, so we replicate that recipe for loco-manipulation. In doing so, we find that cloning synthetic demonstrations results in a low performance ceiling no matter the amount of training data. Reinforcement learning breaks through it, and refining the cloned policy with Flow-GRPO on a single sparse reward yields performance that synthetic behavior cloning cannot match. Together, these stages form our end-to-end sim-to-real pipeline spanning more than 150,000 scenes, which we use to train FetchMan. We evaluate it on FetchMan-Bench, a simulation benchmark we release, and deploy it zero-shot on a real Unitree G1, where our single-object reach-and-pick policy walks to and grasps a target across unseen scenes at 73.3% success. Finally, we extend this recipe to multi-object training, a first step toward loco-manipulation generalist policies at this data scale.
Figures & tables
Figure 1: FetchMan learns visual humanoid loco-manipulation entirely in simulation and deploys it zero-shot to a real Unitree G1. The resulting policy walks up to and grasps objects across diverse, unseen real-world scenes, with no real-world data or per-scene tuning.
Figure 2: Simulated observations. Example frames of the head-fisheye and wrist streams the policy receives, across scenes randomized in texture, lighting, and camera intrinsics and extrinsics.
Figure 3: Synthetic data generation pipeline. Each episode samples (1) a house from MolmoSpaces, (2) a pickable target object within it, (3) a contact-free start and grasp-standoff goal pose connected by a planned path the scripted controller follows, and (4) domain randomization applied independently per episode.
Figure 4: Policy architecture. The head and wrist RGB streams are encoded by a frozen DINOv3 backbone into per-patch image tokens, while the proprioceptive vector (ht,rt,pt,qtupper,gt) is mapped by an MLP to a single state token. A flow-matching DiT cross-attends to these tokens and denoises a 15-dimensional action chunk, split into upper-body targets (waist, arm, and gripper) tracked by a joint-level PD controller and a base command (vx,vy,ω,hˉt) executed by the pre-trained lower-body locomotion controller.
Figure 5: Two-stage training pipeline. Stage 1 (Imitation Pretraining). The task sampler instantiates diverse simulated environments in which the scripted G1 staged policy generates demonstration trajectories, and the resulting frozen dataset trains the loco-manipulation policy π by behavior cloning. Stage 2 (Online GRPO Refinement). The same sampler resets groups of environments to identical tasks (group size × number of groups), π collects on-policy trajectories that differ only in sampled action noise, and a group-relative GRPO update refines π in place.
Figure 6: Real-world rollouts and the effect of RL refinement. Left. Representative zero-shot rollouts of FetchMan on the Unitree G1 across diverse, unseen scenes, where the policy navigates to the target bowl and grasps it from head- and wrist-camera observations alone. Right. The locomotion-to-manipulation handoff before and after Flow-GRPO refinement, on a fixed initialization. The behavior-cloned policy grasps as soon as the target is nearby and misses from too far, while after RL it walks in until the object is within reach and only then grasps.
Figure 7: Behavior cloning saturates, RL breaks through. Left. Single-object loco-manipulation SR (%) on the 100-scene validation set. The first segment scales BC demonstrations (5k–150k) and plateaus after 50k, while the second continues the 150k checkpoint with GRPO, which pushes past the imitation ceiling. A second run applies the same refinement to the 5k checkpoint and does not improve comparably. Right. The text-conditioned multi-object policy, whose BC success rate peaks near 40% before GRPO lifts it to roughly 62%. Time in GPU hours.
Table 8
Figure 8: Multi-object rollouts. Head-camera views from the text-conditioned policy grasping different target objects, one episode per row, with the prompted object name shown at left.