Visual loco-manipulation of arbitrary in-the-wild objects requires accurate end-effector (EE) control and a generalizable understanding of the scene from visual inputs (eg, RGB-D images). Existing imitation and sim2real methods jointly learn both these aspects via monolithic end-to-end learning and are thus hard to scale. In this work, we bring to bear the best tools for each of these problems -- large vision models for generalizable scene understanding and simulated training for accurate EE control -- leading to an overall modular loco-manipulation system that exhibits strong generalization. Our core technical innovation is HERO, an accurate residual-aware EE tracking policy made possible by combining classical robotics with machine learning. It uses a) inverse kinematics to convert residual end-effector targets into reference trajectories, b) a learned neural forward model for accurate forward kinematics, and c) goal adjustment and replanning. Together, these innovations reduce the end-effector tracking error to 2.44cm, outperforming the strongest prior method by 5.5x. Our overall system operates in diverse real-world environments, from offices to coffee shops, where the robot reliably grasps various everyday objects (eg, mugs, apples, toys) on surfaces ranging from 43cm to 92cm in height. Systematic modular and end-to-end tests demonstrate the effectiveness of our proposed design. We believe our advances open up new ways of training humanoids to interact with daily objects. Project page: https://hero-humanoid.github.io/
Figures & tables
Figure 2: Overall architecture for our proposed modular system for open-vocabulary object grasping. Given a free-form natural language text query indicating which object needs to be picked, we use open-vocabulary large vision models (LVMs: Grounding DINO [ 47 ] and SAM [ 5 ] ) to segment out the object of interest and predict parallel jaw grasps (using the AnyGrasp model [ 11 ] ). We retarget the predicted grasp to a Dex-3 hand. We use our proposed whole-body end-effector tracker to convey the robot arm to the predicted grasp before picking up the object. By decomposing action planning ( i.e. identifying which object to pick and using what grasp) from action execution ( i.e. actual control of the robot), we inherit the strong visual generalization from pre-trained models as well as strong control capabilities for simulated training of the tracking policy.
Figure 3: HERO end-effector control. HERO outputs IK and planned references from an EE goal, then tracks them with a learned whole-body policy, which leverages residual EE error computed from learned FK and odometry. Replanning and goal adjustment compensate drift and sim2real bias.
Figure 4: Learned neural forward kinematics and odometry models. (a) A residual FK model η corrects analytical FK by predicting EE translation and rotation residuals. (b) A residual odometry model ξ estimates base motion from lower-body joints. (c) Whole-body reaching can move the object out of the egocentric view, making purely visual correction unreliable.
Figure 5: Visual perception for language-conditioned grasping. An example of target segmentation, grasp proposals, filtering, and retargeted EE poses from onboard RGB-D observations.
Figure 6: MOCAP and camera calibration. (a) Using measured marker-to-link offsets, Kabsch–Umeyama [ 35 , 76 ] recovers 6-DoF EE/base poses from MOCAP coordinates ( <1.5 mm RMSE). (b–c) The D435i camera is mounted on the G1 head with a passive neck pitch joint. (d–e) We manually move an ArUco board [ 18 ] with MOCAP markers through 60–70 poses, recording board poses in the camera frame and board/base poses in the MOCAP frame. Tsai–Lenz [ 74 ] then solves the camera-to-base transform. More details can be found in Sec. B.1 and Sec. B.2 .
Figure 7: Novel test setups and objects. (a-b) Standard ( 0.74m ) and short ( 0.56m ) table setups. (c) Diverse daily objects used for the test.
Figure 7Table 8Table 9
Method
a) End-effector Pose
b) Base Odometry
Translation
Rotation
Translation
Rotation
Error (cm)
Error (deg)
Error (cm)
Error (deg)
Analytical FK
1.30
6.03
1.00
0.49
Kinematic Calibration
0.58
2.83
0.51
0.51
Learned FK, no residual (ours)
4.44
7.06
0.53
0.75
Learned FK (ours)
0.50
3.18
0.52
0.46
Table 6: FK and odometry accuracy. EE/base translation and rotation errors from MOCAP; kinematic calibration and residuals reduce bias.
Figure 9: Failure mode examples . (a) Object slips from the hand. (b) Object knocked over.
Figure 10: Door opening with HERO . (a) The egocentric RGB-D visual inputs. (b) Given a language query for the door handle ( e.g. , ‘‘fridge handle’’ prompted here), our modular system obtains the grasping pose, same as the pipeline for picking up objects. (c) The robot executes the reaching trajectory and closes the hand when reaching the target poses closely, and then returns to the default pose with the door opened. Note that while the door is heavy, HERO successfully manages to open the door with a smooth and stable door motion.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Query type
Detection accuracy (%)
Canonical name
73.8
Canonical name + attribute
86.6
Canonical name + attribute + spatial relation
89.7
Overall (all five query types)
82.6
Appendix
Table 7: OCID-Ref query sensitivity. Detection accuracy on our 1,450-query evaluation (58 classes, five views, five query types). Selected types are shown; overall includes all five.
Figure 11: HERO is able to distinguish the target object via language queries. (a) Picking up a red apple ; (b) Picking up a green apple instead of red apple .
Figure 12: CDF analysis of end-effector tracking errors. Top row: Comparison of translation (left) and rotation (right) error distributions across HERO , FALCON [ 91 ] , and AMO [ 39 ] for all table heights. The steeper curves of HERO indicate consistently lower errors and tighter distributions. Bottom row: Ablation study showing translation (left) and rotation (right) error distributions with and without replanning ( Sec. 3.3 ). The steeper CDF curves with replanning demonstrate their significant contribution to tracking accuracy.
Figure 13: HERO enables a humanoid to grasp a moving object via visual closed-loop replanning. (a-b) Two examples of visual closed-loop replanning. The goal is to grasp black can .
Figure 14: Field of view (FoV) visualization using HERO . We let the robot stand at a random distance from the object ( e.g. , 1.28m), and the robot keeps walking forward until the target object ( stapler ) is detected. After successfully detecting the target object, the robot stops walking and grasps the object via whole-body coordination. The robot can only see the object within 0.6m, which makes it hard for the robot to search for the object in a random room.
Figure 15: Visualization of analytical forward kinematics error . We plot the 60 data points collected in the MOCAP room. The error is indicated in the color bar on the left side of the figure, and the size of the scatter also increases with the error.
Figure 16: Visual perception for language-conditioned grasping. Examples of target segmentation, grasp proposals, filtering, and retargeted EE poses from onboard RGB-D observations.
Figure 17: Impact of replanning on end-effector tracking error in the real world . We plot the end-effector translation error as a function of execution time steps. The plot shows 1 minute of execution at 50Hz. The transparent lines are individual 60 real-world rollouts, and the corresponding solid line indicates the average value. The gray vertical dashlines indicate the replanning every 6 seconds (0.15Hz). Cyan line shows HERO without replanning and while purple line shows HERO with re-planning. Re-planning leads to more accurate tracking. Orange line uses end-effector estimates from our neural model which leads to tracking performance very close to the oracle purple line that uses end-effector estimates from MOCAP.
Configuration
Single Arm ( m3 )
Both Arms ( m3 )
Arms-only (14 DoFs)
0.166
0.248
Arms+Waist (17 DoFs)
0.426
0.523
Appendix
Table 8: Reachable workspace volume across configurations.
Figure 18: HERO enables a humanoid picking up objects from a standard table ( 0.74 m) across a large workspace with open-vocabulary queries. (a-c) The robot can reach and pick up a red apple placed at different heights, poses, and locations.
Visual loco-manipulation policies that can generalize to novel scenes and objects have long been a goal of robotics research. However, today's data-hungry algorithms make collecting sufficient demonstrations a struggle for tabletop manipulation, and even more so for humanoids that must also walk and balance. Learning from simulated data and transferring that behavior to the real world, as is commonly done in locomotion, sidesteps this struggle, so we replicate that recipe for loco-manipulation. In doing so, we find that cloning synthetic demonstrations results in a low performance ceiling no matter the amount of training data. Reinforcement learning breaks through it, and refining the cloned policy with Flow-GRPO on a single sparse reward yields performance that synthetic behavior cloning cannot match. Together, these stages form our end-to-end sim-to-real pipeline spanning more than 150,000 scenes, which we use to train FetchMan. We evaluate it on FetchMan-Bench, a simulation benchmark we release, and deploy it zero-shot on a real Unitree G1, where our single-object reach-and-pick policy walks to and grasps a target across unseen scenes at 73.3% success. Finally, we extend this recipe to multi-object training, a first step toward loco-manipulation generalist policies at this data scale.
Omar Rayyan, Zhi Li, Max Argus +4
University of California, Los Angeles · Allen Institute for AI · University of Washington
Humanoid loco-manipulation requires adaptive whole-body coordination to seamlessly integrate locomotion and physical interaction. Despite recent advances, learning autonomous loco-manipulation remains challenging due to the scarcity of diverse, physically executable robot-object interaction data and the difficulty of learning unified whole-body control directly from onboard observations. We present ViLoMan, a scalable framework for autonomous humanoid loco-manipulation. ViLoMan first transforms partial kinematic demonstrations of human-object interactions into complete, physically executable robot trajectories. It then leverages these trajectories within a teacher-student distillation framework to learn a unified policy that maps egocentric depth observations and proprioceptive measurements directly to joint-level whole-body actions. During deployment, the policy requires neither reference motions nor intermediate commands. We evaluate ViLoMan on door-closing tasks across diverse door configurations and robot initial conditions in both simulation and the real world. Experimental results demonstrate that a single policy enables a Unitree G1 humanoid to complete the full task using only onboard depth sensing and proprioception, while generalizing robustly across task variations and transferring effectively from simulation to reality. Project page: viloman-anonymous.pages.dev.
Zejie Tian, Ruibing Hou, Bingpeng Ma +2
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, China.
Whole-body humanoid loco-manipulation requires coordinating the robot's entire kinematic chain. However, most existing systems typically decouple the upper and lower bodies into separate controllers, limiting such coordination and yielding behaviors similar to those of a wheeled dual-arm platform. In this paper, we ask what it takes to build a whole-body native vision-language-action (VLA) model that maps language and pixels directly to all of the humanoid's degrees of freedom. We conduct a systematic empirical study organized as a roadmap of one-variable-at-a-time experiments across three phases: whole-body teleoperation, VLA model design, and heterogeneous co-training. Our study yields several intriguing findings: a joint-based whole-body teleoperation interface outperforms alternatives that only partially expose the humanoid's degrees of freedom; a VLA pretrained on static and wheeled dual-arm platforms transfers surprisingly well to a humanoid's full action space; and co-training with HuMI, the humanoid analog of UMI, extends the policy to new objects and instructions without additional whole-body teleoperation on those targets. Following this roadmap yields OpenHLM, an open-source recipe for whole-body humanoid loco-manipulation. In a challenging long-horizon task that spans a wide vertical range of the humanoid, OpenHLM outperforms two state-of-the-art humanoid VLA baselines (GR00T N1.6 and Ψ0) using less than half the total demonstration time. Our code, training data, and model checkpoints are available at [https://openhlm-project.github.io/].
Yingdong Hu, Haodong Zhu, Boyuan Zheng +6
Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI