HITCH: Teaching a Vision-Language Model to Operate a Humanoid through a Whole-Body Harness

Qingzhou Lu; Chenxin Liu; Xinming Wang; Liu Cao; Yanjiang Guo; Mengdi Xu; Jianyu Chen

cs.RONov 5, 2026

HITCH: Teaching a Vision-Language Model to Operate a Humanoid through a Whole-Body Harness

Authors: Qingzhou Lu, Chenxin Liu, Xinming Wang, Liu Cao, Yanjiang Guo, Mengdi Xu, Jianyu Chen

Organizations: Tsinghua University. · Institute of Automation, Chinese Academy of Sciences. · Shanghai Qizhi Institute.

Abstract

Enabling vision-language models (VLMs) to control humanoid robots requires both reliable whole-body interfaces and knowledge of how to use the body that carries over across tasks. We present HITCH (Harness Interfaces To Command Humanoids), a harness in which perception returns the 3D position of any point the VLM selects in its image, and execution turns commands for the pelvis, torso, hands, walking, and gaze into whole-body motion with a learned tracker. We find that both measuring the scene and executing commands accurately raise task success. We then fine-tune a VLM on trajectories in which teacher policies solve five base tasks through HITCH, so that it learns how to operate the humanoid. Without task-specific trajectories, the fine-tuned model completes 51% of SIMPLE episodes and 42.5% of HumanoidArena episodes, and adding five trajectories per SIMPLE task raises its SIMPLE success to 95%. On a real G1, it carries a box and walks to a goal without real training data.

Figures & tables

Fig. 1: A VLM fine-tuned on teacher trajectories controls a G1 humanoid through a harness. Left: one evaluation episode of SIMPLE, with the robot drawn at several moments. The agent measures the box and the basket by choosing points in its head-camera images (insets) and receives their 3D positions (blue). It then commands the body: it lifts the box, turns, walks to the other table, and lowers its hand into the basket through waypoints (orange). Right: other tasks in SIMPLE, HumanoidArena, and on a real G1, and the success rate on SIMPLE of the pretrained model, of the model trained on the base tasks only, and of the model trained on the base tasks and 5 trajectories per SIMPLE task.

Fig. 2: Overview. Top left: in each turn, the agent $\pi_{\theta}$ (Qwen3.8-27B) receives the instruction $\ell$ and the history $h_{t}$ of head-camera images and body states, and writes a response $a_{t}$ that contains one call. Top right: the call goes to the harness. A perception call selects points in the image and returns their 3D positions in the observation frame $O$ , measured from depth, together with the marked image. An execution call sets the body, hands, fingers, walking, and gaze; a planner turns it into a whole-body reference, which the HEFT tracker follows at $50\text{\,}\mathrm{H}\mathrm{z}$ , and the call returns the next observation. Bottom: teacher code policies act through the same harness on five base tasks and six SIMPLE tasks. The agent is fine-tuned on their multi-turn trajectories and evaluated on new tasks without or with a few task trajectories, and on a real G1 in new scenes.

Fig. 3: Postures set by single commands in teacher episodes of the base tasks. A command gives positions in the observation frame $O$ of the latest observation (axes), in metres. Orange dots mark the palm positions that the left and middle commands set; the dashed arrow on the right marks the walk target. Left: crouch, lean forward, and place a cube into a bowl. Middle: grip a box with both hands, each palm moving through two waypoints, while keeping the $40\text{\,}\mathrm{\SIUnitSymbolDegree}$ lean of the previous command. Right: lift the box and step back. Below each frame are the fields of its command.

Fig. 4: Active perception. (a) The D455 on a 2-DoF mount on the G1’s head, on the real robot and in simulation, seen from the side and from above. The mount turns the view from level to $60^{\circ}$ down and up to $45^{\circ}$ either way. (b) A perception query of the agent in SIMPLE: it selects six points on the box and the table in the head-camera image and receives their positions in $O$ .

Fig. 5: Episodes in time order, one row each, with the call that returned each frame below it. Left: teacher episodes of three base tasks. Right: the agent on SIMPLE after training on the base tasks and ten SIMPLE trajectories per task, and on HumanoidArena and the real G1 after training on the base tasks only. Third-person frames show the head-camera image (top right), the head camera’s view (blue), and the path walked so far (orange); the HumanoidArena and real G1 rows show head-camera images.

	Tracking error
	SIMPLE		HumanoidArena		Teacher
Tracker	Hand	Walk	Hand	Walk	success
HEFT	0.9	11	0.7	8	94.2
HEFT with harness data	0.6	5	0.7	3	97.9
SONIC	2.6	35	2.0	44	27.9
AMO	3.1	47	3.0	40	12.9

TABLE I: Execution. Hand error (cm), walk error (%), and teacher success rate (%).

Without perception
	SIMPLE				HumanoidArena
					(subset)
$N$	0	5	10	30	0	10
Without base tasks	3.3 †	90.0	95.0	96.7	16.3 †	18.8
With base tasks	51.1	95.0	98.9	98.9	42.5	28.8
With base tasks	2.8	–	80.6	–	27.5	21.3

TABLE II: Success rate (%) without reasoning, by the number

N

of SIMPLE trajectories per task.

Qwen3.8-27B
	SIMPLE		HumanoidArena
			(subset)
Training data	w/o	w/	w/o	w/
None (pretrained)	3.3	8.3	16.3	22.5
Base tasks	51.1	47.2	42.5	41.3
SIMPLE ( $N{=}10$ )	95.0	86.7	18.8	12.5

TABLE III: Success rate (%) without (w/o) and with (w/) reasoning.

	Carrying		Navigation
	s	%	s	%
Perception	12.1	20	5.4	20
Image capture	11.3	19	4.9	18
Perception queries	0.8	1	0.4	2
Decision (model requests)	11.5	19	5.0	19
Execution	37.0	61	16.4	61

TABLE IV: Time in successful real episodes (s) and its share (%), mean per episode, from the first image until the motion that completes the task ends.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Content
Body	Pelvis height, $0.30\text{\,}\mathrm{m}$ to $0.79\text{\,}\mathrm{m}$ ; torso pitch, up to $60\text{\,}\mathrm{\SIUnitSymbolDegree}$ , and yaw, up to $45\text{\,}\mathrm{\SIUnitSymbolDegree}$ either way, in steps of $5\text{\,}\mathrm{\SIUnitSymbolDegree}$ , with limits that depend on the pelvis height; duration.
Hands	Per hand, 1 to 12 palm waypoints, each with a position, an optional orientation, and a duration of $0.4\text{\,}\mathrm{s}$ to $8\text{\,}\mathrm{s}$ ; or the hand stays fixed in space, moves with the pelvis or the torso, or returns to its default pose. An orientation is a named palm direction (inward, down, up, or forward) or a palm normal and a finger direction.
Fingers	Close or open, on dexterous hands.
Walking	A target position and turn in $O$ , or 1 to 12 footsteps.
Gaze	Camera yaw, $-45\text{\,}\mathrm{\SIUnitSymbolDegree}$ to $45\text{\,}\mathrm{\SIUnitSymbolDegree}$ , and pitch, $0\text{\,}\mathrm{\SIUnitSymbolDegree}$ to $60\text{\,}\mathrm{\SIUnitSymbolDegree}$ down, in steps of $15\text{\,}\mathrm{\SIUnitSymbolDegree}$ .

Appendix

TABLE V: Fields of an execution command.

Task	Official	Ours
SIMPLE
XMovePick	move forward to pick up the cracker box	Walk to the table ahead and pick up the cracker box with one hand.
XMoveBendPick	xmove to the table and bend to pick up the cracker box	Walk to the low table ahead and pick up the cracker box with one hand.
BendPick	bend to pick up the cracker box	Pick up the cracker box from the low table in front of you with one hand.
Grasp	grasp the object on the tabletop.	Pick up the chips can from the table in front of you with one hand.
PickBetweenTables	pick up the cracker box from table1,locomotion to table2,and place on table2.	Pick up the cracker box from the table in front of you, carry it to the basket on the table behind you, and put it into the basket. Use one hand.

Appendix

TABLE VI: Official instructions and ours.

SIMPLE
	Official	Time (s)		Turns
Task	limit (s)	Mean	10th–90th	Mean (range)
XMovePick	16	12.3	12.0–13.4	11.8 (9–13)
XMoveBendPick	16	14.0	13.9–14.0	11.5 (10–15)
BendPick	8	16.3	16.1–16.3	14.0 (11–17)
Grasp	27	14.2	14.1–14.4	11.0 (10–13)

Appendix

TABLE VII: Time limits and durations of successful episodes. Time: mean and the 10th to 90th percentile; turns: mean and range.

Humanoid Robot Control

Humanoid robot control develops methods for coordinating bipedal balance, locomotion, and whole-body interaction with people and objects.

See all 218 papers

OpenHLM: An Empirical Recipe for Whole-Body Humanoid Loco-Manipulation
Jun 20, 2026Yingdong Hu, Haodong Zhu, Boyuan Zheng +6Humanoid Loco-Manipulation Whole-Body Robotic Manipulation
Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI
Know Your Body: A Harness for Direct and Self-Improving Robot Control with VLMs
Sep 22, 2026Zeyu Lou, Yanhong Zeng, Yong Wang +1VLMs for Robotics Large Language Model-Based Robot Planning
Nanjing University · Ant Group · Zhejiang University
Show-Harness: Just a VLM Agent Can Play Robots
Sep 9, 2026Yanzhe Chen, Zechen Bai, Zhijun Cao +7Cross-Embodiment Robot Learning Language-Conditioned Robot Control
Show Lab, National University of Singapore

Jun 20, 2026J/K move · Enter open · S save

OpenHLM: An Empirical Recipe for Whole-Body Humanoid Loco-Manipulation

Yingdong Hu, Haodong Zhu, Boyuan Zheng, Yihang Hu+5

Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI

Humanoid Loco-Manipulation Whole-Body Robotic Manipulation Vision-Language-Action Models Long-Horizon Robotic Manipulation Robot Teleoperation Cross-Embodiment Robot Learning+2

Open paper AskPDF arXiv

Whole-body humanoid loco-manipulation requires coordinating the robot's entire kinematic chain. However, most existing systems typically decouple the upper and lower bodies into separate controllers, limiting such coordination and yielding behaviors similar to those of a wheeled dual-arm platform. In this paper, we ask what it takes to build a whole-body native vision-language-action (VLA) model that maps language and pixels directly to all of the humanoid's degrees of freedom. We conduct a systematic empirical study organized as a roadmap of one-variable-at-a-time experiments across three phases: whole-body teleoperation, VLA model design, and heterogeneous co-training. Our study yields several intriguing findings: a joint-based whole-body teleoperation interface outperforms alternatives that only partially expose the humanoid's degrees of freedom; a VLA pretrained on static and wheeled dual-arm platforms transfers surprisingly well to a humanoid's full action space; and co-training with HuMI, the humanoid analog of UMI, extends the policy to new objects and instructions without additional whole-body teleoperation on those targets. Following this roadmap yields OpenHLM, an open-source recipe for whole-body humanoid loco-manipulation. In a challenging long-horizon task that spans a wide vertical range of the humanoid, OpenHLM outperforms two state-of-the-art humanoid VLA baselines (GR00T N1.6 and

Ψ_0

) using less than half the total demonstration time. Our code, training data, and model checkpoints are available at [https://openhlm-project.github.io/].

Abstract

Humanoid Robot Control

Explore similar work

OpenHLM: An Empirical Recipe for Whole-Body Humanoid Loco-Manipulation

Know Your Body: A Harness for Direct and Self-Improving Robot Control with VLMs

Show-Harness: Just a VLM Agent Can Play Robots