Enabling vision-language models (VLMs) to control humanoid robots requires both reliable whole-body interfaces and knowledge of how to use the body that carries over across tasks. We present HITCH (Harness Interfaces To Command Humanoids), a harness in which perception returns the 3D position of any point the VLM selects in its image, and execution turns commands for the pelvis, torso, hands, walking, and gaze into whole-body motion with a learned tracker. We find that both measuring the scene and executing commands accurately raise task success. We then fine-tune a VLM on trajectories in which teacher policies solve five base tasks through HITCH, so that it learns how to operate the humanoid. Without task-specific trajectories, the fine-tuned model completes 51% of SIMPLE episodes and 42.5% of HumanoidArena episodes, and adding five trajectories per SIMPLE task raises its SIMPLE success to 95%. On a real G1, it carries a box and walks to a goal without real training data.
Figures & tables
Fig. 1: A VLM fine-tuned on teacher trajectories controls a G1 humanoid through a harness. Left: one evaluation episode of SIMPLE, with the robot drawn at several moments. The agent measures the box and the basket by choosing points in its head-camera images (insets) and receives their 3D positions (blue). It then commands the body: it lifts the box, turns, walks to the other table, and lowers its hand into the basket through waypoints (orange). Right: other tasks in SIMPLE, HumanoidArena, and on a real G1, and the success rate on SIMPLE of the pretrained model, of the model trained on the base tasks only, and of the model trained on the base tasks and 5 trajectories per SIMPLE task.
Fig. 2: Overview. Top left: in each turn, the agent πθ (Qwen3.8-27B) receives the instruction ℓ and the history ht of head-camera images and body states, and writes a response at that contains one call. Top right: the call goes to the harness. A perception call selects points in the image and returns their 3D positions in the observation frame O , measured from depth, together with the marked image. An execution call sets the body, hands, fingers, walking, and gaze; a planner turns it into a whole-body reference, which the HEFT tracker follows at 50Hz , and the call returns the next observation. Bottom: teacher code policies act through the same harness on five base tasks and six SIMPLE tasks. The agent is fine-tuned on their multi-turn trajectories and evaluated on new tasks without or with a few task trajectories, and on a real G1 in new scenes.
Fig. 3: Postures set by single commands in teacher episodes of the base tasks. A command gives positions in the observation frame O of the latest observation (axes), in metres. Orange dots mark the palm positions that the left and middle commands set; the dashed arrow on the right marks the walk target. Left: crouch, lean forward, and place a cube into a bowl. Middle: grip a box with both hands, each palm moving through two waypoints, while keeping the 40\SIUnitSymbolDegree lean of the previous command. Right: lift the box and step back. Below each frame are the fields of its command.
Fig. 4: Active perception. (a) The D455 on a 2-DoF mount on the G1’s head, on the real robot and in simulation, seen from the side and from above. The mount turns the view from level to 60∘ down and up to 45∘ either way. (b) A perception query of the agent in SIMPLE: it selects six points on the box and the table in the head-camera image and receives their positions in O .
Fig. 5: Episodes in time order, one row each, with the call that returned each frame below it. Left: teacher episodes of three base tasks. Right: the agent on SIMPLE after training on the base tasks and ten SIMPLE trajectories per task, and on HumanoidArena and the real G1 after training on the base tasks only. Third-person frames show the head-camera image (top right), the head camera’s view (blue), and the path walked so far (orange); the HumanoidArena and real G1 rows show head-camera images.
Tracking error
SIMPLE
HumanoidArena
Teacher
Tracker
Hand
Walk
Hand
Walk
success
HEFT
0.9
11
0.7
8
94.2
HEFT with harness data
0.6
5
0.7
3
97.9
SONIC
2.6
35
2.0
44
27.9
AMO
3.1
47
3.0
40
12.9
TABLE I: Execution. Hand error (cm), walk error (%), and teacher success rate (%).
SIMPLE
HumanoidArena
(subset)
N
0
5
10
30
0
10
Without base tasks
3.3 †
90.0
95.0
96.7
16.3 †
18.8
With base tasks
51.1
95.0
98.9
98.9
42.5
28.8
Without perception
With base tasks
2.8
–
80.6
–
27.5
21.3
TABLE II: Success rate (%) without reasoning, by the number N of SIMPLE trajectories per task.
SIMPLE
HumanoidArena
(subset)
Training data
w/o
w/
w/o
w/
Qwen3.8-27B
None (pretrained)
3.3
8.3
16.3
22.5
Base tasks
51.1
47.2
42.5
41.3
SIMPLE ( N=10 )
95.0
86.7
18.8
12.5
TABLE III: Success rate (%) without (w/o) and with (w/) reasoning.
Carrying
Navigation
s
%
s
%
Perception
12.1
20
5.4
20
Image capture
11.3
19
4.9
18
Perception queries
0.8
1
0.4
2
Decision (model requests)
11.5
19
5.0
19
Execution
37.0
61
16.4
61
TABLE IV: Time in successful real episodes (s) and its share (%), mean per episode, from the first image until the motion that completes the task ends.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Content
Body
Pelvis height, 0.30m to 0.79m ; torso pitch, up to 60\SIUnitSymbolDegree , and yaw, up to 45\SIUnitSymbolDegree either way, in steps of 5\SIUnitSymbolDegree , with limits that depend on the pelvis height; duration.
Hands
Per hand, 1 to 12 palm waypoints, each with a position, an optional orientation, and a duration of 0.4s to 8s ; or the hand stays fixed in space, moves with the pelvis or the torso, or returns to its default pose. An orientation is a named palm direction (inward, down, up, or forward) or a palm normal and a finger direction.
Fingers
Close or open, on dexterous hands.
Walking
A target position and turn in O , or 1 to 12 footsteps.
Gaze
Camera yaw, −45\SIUnitSymbolDegree to 45\SIUnitSymbolDegree , and pitch, 0\SIUnitSymbolDegree to 60\SIUnitSymbolDegree down, in steps of 15\SIUnitSymbolDegree .
Appendix
TABLE V: Fields of an execution command.
Task
Official
Ours
SIMPLE
XMovePick
move forward to pick up the cracker box
Walk to the table ahead and pick up the cracker box with one hand.
XMoveBendPick
xmove to the table and bend to pick up the cracker box
Walk to the low table ahead and pick up the cracker box with one hand.
BendPick
bend to pick up the cracker box
Pick up the cracker box from the low table in front of you with one hand.
Grasp
grasp the object on the tabletop.
Pick up the chips can from the table in front of you with one hand.
PickBetweenTables
pick up the cracker box from table1,locomotion to table2,and place on table2.
Pick up the cracker box from the table in front of you, carry it to the basket on the table behind you, and put it into the basket. Use one hand.
Appendix
TABLE VI: Official instructions and ours.
Official
Time (s)
Turns
Task
limit (s)
Mean
10th–90th
Mean (range)
SIMPLE
XMovePick
16
12.3
12.0–13.4
11.8 (9–13)
XMoveBendPick
16
14.0
13.9–14.0
11.5 (10–15)
BendPick
8
16.3
16.1–16.3
14.0 (11–17)
Grasp
27
14.2
14.1–14.4
11.0 (10–13)
Appendix
TABLE VII: Time limits and durations of successful episodes. Time: mean and the 10th to 90th percentile; turns: mean and range.
Whole-body humanoid loco-manipulation requires coordinating the robot's entire kinematic chain. However, most existing systems typically decouple the upper and lower bodies into separate controllers, limiting such coordination and yielding behaviors similar to those of a wheeled dual-arm platform. In this paper, we ask what it takes to build a whole-body native vision-language-action (VLA) model that maps language and pixels directly to all of the humanoid's degrees of freedom. We conduct a systematic empirical study organized as a roadmap of one-variable-at-a-time experiments across three phases: whole-body teleoperation, VLA model design, and heterogeneous co-training. Our study yields several intriguing findings: a joint-based whole-body teleoperation interface outperforms alternatives that only partially expose the humanoid's degrees of freedom; a VLA pretrained on static and wheeled dual-arm platforms transfers surprisingly well to a humanoid's full action space; and co-training with HuMI, the humanoid analog of UMI, extends the policy to new objects and instructions without additional whole-body teleoperation on those targets. Following this roadmap yields OpenHLM, an open-source recipe for whole-body humanoid loco-manipulation. In a challenging long-horizon task that spans a wide vertical range of the humanoid, OpenHLM outperforms two state-of-the-art humanoid VLA baselines (GR00T N1.6 and Ψ0) using less than half the total demonstration time. Our code, training data, and model checkpoints are available at [https://openhlm-project.github.io/].
Yingdong Hu, Haodong Zhu, Boyuan Zheng +6
Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI
A general-purpose vision-language model can understand a task goal without knowing how a particular robot's motion and functional parts produce the intended effect. We introduce KnowBody, a harness that makes these action-relevant body relations explicit, queryable, and revisable while keeping the model weights frozen. Initialized from one off-task trajectory, a partial body model guides action selection and the interpretation of past interactions. New evidence refines the model, and knowledge dependent on revised body estimates is rechecked before reuse. Across 32 fixed-budget trials on four real-robot tasks, initialized KnowBody achieves 75% completion versus 25% for the native harness and requires fewer planner rounds on successful trials in tasks completed by both. With persistent updates enabled, planner rounds decrease by 29-53% from the first to the fifth recorded success.
Zeyu Lou, Yanhong Zeng, Yong Wang +1
Nanjing University · Ant Group · Zhejiang University
Foundation vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging. We present Show-Harness, an Embodied Harness that enables VLMs to "play" robots through a compact semantic interface linking intent to action. Show-Harness exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions, keeping the VLM directly responsible for fine-grained physical decisions. Through the same interface, Show-Harness demonstrates the feasibility of (1) directly unlocking closed-source frontier VLMs for zero-shot robot control, and (2) adapting small-scale open-source VLMs for low-cost deployment with just a few GPU-hours of fine-tuning. We further develop GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to "play" robots across embodiments without specialized teleoperation hardware. Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms. These results suggest that the right interface can unlock substantial embodied capability from foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.