HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Grasping

Runpei Dong; Ziyan Li; Arjun Gupta; Xialin He; Saurabh Gupta

cs.ROFeb 18, 2026

HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Grasping

Authors: Runpei Dong, Ziyan Li, Arjun Gupta, Xialin He, Saurabh Gupta

Organizations: University of Illinois Urbana-Champaign

Abstract

Visual loco-manipulation of arbitrary in-the-wild objects requires accurate end-effector (EE) control and a generalizable understanding of the scene from visual inputs (eg, RGB-D images). Existing imitation and sim2real methods jointly learn both these aspects via monolithic end-to-end learning and are thus hard to scale. In this work, we bring to bear the best tools for each of these problems -- large vision models for generalizable scene understanding and simulated training for accurate EE control -- leading to an overall modular loco-manipulation system that exhibits strong generalization. Our core technical innovation is HERO, an accurate residual-aware EE tracking policy made possible by combining classical robotics with machine learning. It uses a) inverse kinematics to convert residual end-effector targets into reference trajectories, b) a learned neural forward model for accurate forward kinematics, and c) goal adjustment and replanning. Together, these innovations reduce the end-effector tracking error to 2.44cm, outperforming the strongest prior method by 5.5x. Our overall system operates in diverse real-world environments, from offices to coffee shops, where the robot reliably grasps various everyday objects (eg, mugs, apples, toys) on surfaces ranging from 43cm to 92cm in height. Systematic modular and end-to-end tests demonstrate the effectiveness of our proposed design. We believe our advances open up new ways of training humanoids to interact with daily objects. Project page: https://hero-humanoid.github.io/

Figures & tables

Figure 2: Overall architecture for our proposed modular system for open-vocabulary object grasping. Given a free-form natural language text query indicating which object needs to be picked, we use open-vocabulary large vision models (LVMs: Grounding DINO [ 47 ] and SAM [ 5 ] ) to segment out the object of interest and predict parallel jaw grasps (using the AnyGrasp model [ 11 ] ). We retarget the predicted grasp to a Dex-3 hand. We use our proposed whole-body end-effector tracker to convey the robot arm to the predicted grasp before picking up the object. By decomposing action planning ( i.e. identifying which object to pick and using what grasp) from action execution ( i.e. actual control of the robot), we inherit the strong visual generalization from pre-trained models as well as strong control capabilities for simulated training of the tracking policy.

Figure 3: HERO end-effector control. HERO outputs IK and planned references from an EE goal, then tracks them with a learned whole-body policy, which leverages residual EE error computed from learned FK and odometry. Replanning and goal adjustment compensate drift and sim2real bias.

Figure 4: Learned neural forward kinematics and odometry models. (a) A residual FK model $\eta$ corrects analytical FK by predicting EE translation and rotation residuals. (b) A residual odometry model $\xi$ estimates base motion from lower-body joints. (c) Whole-body reaching can move the object out of the egocentric view, making purely visual correction unreliable.

Figure 5: Visual perception for language-conditioned grasping. An example of target segmentation, grasp proposals, filtering, and retargeted EE poses from onboard RGB-D observations.

Figure 6: MOCAP and camera calibration. (a) Using measured marker-to-link offsets, Kabsch–Umeyama [ 35 , 76 ] recovers 6-DoF EE/base poses from MOCAP coordinates ( $<1.5$ mm RMSE). (b–c) The D435i camera is mounted on the G1 head with a passive neck pitch joint. (d–e) We manually move an ArUco board [ 18 ] with MOCAP markers through 60–70 poses, recording board poses in the camera frame and board/base poses in the MOCAP frame. Tsai–Lenz [ 74 ] then solves the camera-to-base transform. More details can be found in Sec. B.1 and Sec. B.2 .

Figure 7: Novel test setups and objects. (a-b) Standard ( $0.74\text{m}$ ) and short ( $0.56\text{m}$ ) table setups. (c) Diverse daily objects used for the test.

Table 8

Table 9

Method	a) End-effector Pose		b) Base Odometry
Method	Translation	Rotation	Translation	Rotation
	Error (cm)	Error (deg)	Error (cm)	Error (deg)
Analytical FK	1.30	6.03	1.00	0.49
Kinematic Calibration	0.58	2.83	0.51	0.51
Learned FK, no residual (ours)	4.44	7.06	0.53	0.75
Learned FK (ours)	0.50	3.18	0.52	0.46

Table 6: FK and odometry accuracy. EE/base translation and rotation errors from MOCAP; kinematic calibration and residuals reduce bias.

Panel of Figure 9 — Figure 9: Failure mode examples . (a) Object slips from the hand. (b) Object knocked over.

Figure 10: Door opening with HERO . (a) The egocentric RGB-D visual inputs. (b) Given a language query for the door handle ( e.g. , ‘‘fridge handle’’ prompted here), our modular system obtains the grasping pose, same as the pipeline for picking up objects. (c) The robot executes the reaching trajectory and closes the hand when reaching the target poses closely, and then returns to the default pose with the door opened. Note that while the door is heavy, HERO successfully manages to open the door with a smooth and stable door motion.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Query type	Detection accuracy (%)
Canonical name	73.8
Canonical name + attribute	86.6
Canonical name + attribute + spatial relation	89.7
Overall (all five query types)	82.6

Appendix

Table 7: OCID-Ref query sensitivity. Detection accuracy on our 1,450-query evaluation (58 classes, five views, five query types). Selected types are shown; overall includes all five.

Figure 11: HERO is able to distinguish the target object via language queries. (a) Picking up a red apple ; (b) Picking up a green apple instead of red apple .

Figure 12: CDF analysis of end-effector tracking errors. Top row: Comparison of translation (left) and rotation (right) error distributions across HERO , FALCON [ 91 ] , and AMO [ 39 ] for all table heights. The steeper curves of HERO indicate consistently lower errors and tighter distributions. Bottom row: Ablation study showing translation (left) and rotation (right) error distributions with and without replanning ( Sec. 3.3 ). The steeper CDF curves with replanning demonstrate their significant contribution to tracking accuracy.

Figure 13: HERO enables a humanoid to grasp a moving object via visual closed-loop replanning. (a-b) Two examples of visual closed-loop replanning. The goal is to grasp black can .

Figure 14: Field of view (FoV) visualization using HERO . We let the robot stand at a random distance from the object ( e.g. , 1.28m), and the robot keeps walking forward until the target object ( stapler ) is detected. After successfully detecting the target object, the robot stops walking and grasps the object via whole-body coordination. The robot can only see the object within 0.6m, which makes it hard for the robot to search for the object in a random room.

Figure 15: Visualization of analytical forward kinematics error . We plot the 60 data points collected in the MOCAP room. The error is indicated in the color bar on the left side of the figure, and the size of the scatter also increases with the error.

Figure 16: Visual perception for language-conditioned grasping. Examples of target segmentation, grasp proposals, filtering, and retargeted EE poses from onboard RGB-D observations.

Figure 17: Impact of replanning on end-effector tracking error in the real world . We plot the end-effector translation error as a function of execution time steps. The plot shows 1 minute of execution at 50Hz. The transparent lines are individual 60 real-world rollouts, and the corresponding solid line indicates the average value. The gray vertical dashlines indicate the replanning every 6 seconds (0.15Hz). Cyan line shows HERO without replanning and while purple line shows HERO with re-planning. Re-planning leads to more accurate tracking. Orange line uses end-effector estimates from our neural model which leads to tracking performance very close to the oracle purple line that uses end-effector estimates from MOCAP.

Configuration	Single Arm ( $\mathrm{m}^{3}$ )	Both Arms ( $\mathrm{m}^{3}$ )
Arms-only (14 DoFs)	0.166	0.248
Arms+Waist (17 DoFs)	0.426	0.523

Appendix

Table 8: Reachable workspace volume across configurations.

Figure 18: HERO enables a humanoid picking up objects from a standard table ( $\mathbf{0.74}$ m) across a large workspace with open-vocabulary queries. (a-c) The robot can reach and pick up a red apple placed at different heights, poses, and locations.

Robotic Control

Robotic control studies how machines use sensing and computation to produce reliable movement and interaction in changing environments.

See all 583 papers

FetchMan: Learning Visual Humanoid Loco-Manipulation Policies from Simulated Experiences
Aug 17, 2026Omar Rayyan, Zhi Li, Max Argus +4Robotic Loco-Manipulation Loco-Manipulation
University of California, Los Angeles · Allen Institute for AI · University of Washington
ViLoMan: Learning Visual-Proprioceptive Whole-Body Loco-Manipulation Skills for Humanoid Robots
Sep 16, 2026Zejie Tian, Ruibing Hou, Bingpeng Ma +2Visuomotor Policy Learning Whole-Body Robotic Manipulation
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, China.
OpenHLM: An Empirical Recipe for Whole-Body Humanoid Loco-Manipulation
Jun 20, 2026Yingdong Hu, Haodong Zhu, Boyuan Zheng +6Humanoid Loco-Manipulation Whole-Body Robotic Manipulation
Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI

Aug 17, 2026J/K move · Enter open · S save

FetchMan: Learning Visual Humanoid Loco-Manipulation Policies from Simulated Experiences

Omar Rayyan, Zhi Li, Max Argus, Yuxin Jiang+3

University of California, Los Angeles · Allen Institute for AI · University of Washington

Robotic Loco-Manipulation Loco-Manipulation Humanoid Loco-Manipulation Sim-to-Real Transfer RL for Robotics Zero-Shot Robotic Manipulation+1

Open paper AskPDF arXiv

Visual loco-manipulation policies that can generalize to novel scenes and objects have long been a goal of robotics research. However, today's data-hungry algorithms make collecting sufficient demonstrations a struggle for tabletop manipulation, and even more so for humanoids that must also walk and balance. Learning from simulated data and transferring that behavior to the real world, as is commonly done in locomotion, sidesteps this struggle, so we replicate that recipe for loco-manipulation. In doing so, we find that cloning synthetic demonstrations results in a low performance ceiling no matter the amount of training data. Reinforcement learning breaks through it, and refining the cloned policy with Flow-GRPO on a single sparse reward yields performance that synthetic behavior cloning cannot match. Together, these stages form our end-to-end sim-to-real pipeline spanning more than 150,000 scenes, which we use to train FetchMan. We evaluate it on FetchMan-Bench, a simulation benchmark we release, and deploy it zero-shot on a real Unitree G1, where our single-object reach-and-pick policy walks to and grasps a target across unseen scenes at 73.3% success. Finally, we extend this recipe to multi-object training, a first step toward loco-manipulation generalist policies at this data scale.

Abstract

Robotic Control

Explore similar work

FetchMan: Learning Visual Humanoid Loco-Manipulation Policies from Simulated Experiences

ViLoMan: Learning Visual-Proprioceptive Whole-Body Loco-Manipulation Skills for Humanoid Robots

OpenHLM: An Empirical Recipe for Whole-Body Humanoid Loco-Manipulation