Humans achieve complex manipulation through coordinated whole-body control, whereas most Vision-Language-Action (VLA) models treat robot body parts largely independently, making high-DoF humanoid control challenging and often unstable. We present HEX, a state-centric framework for coordinated manipulation on full-sized bipedal humanoid robots. HEX introduces a humanoid-aligned universal state representation for scalable learning across heterogeneous embodiments, and incorporates a Mixture-of-Experts Unified Proprioceptive Predictor to model whole-body coordination and temporal motion dynamics from large-scale multi-embodiment trajectory data. To efficiently capture temporal visual context, HEX uses lightweight history tokens to summarize past observations, avoiding repeated encoding of historical images during inference. It further employs a residual-gated fusion mechanism with a flow-matching action head to adaptively integrate visual-language cues with proprioceptive dynamics for action generation. Experiments on real-world humanoid manipulation tasks show that HEX achieves state-of-the-art performance in task success rate and generalization, particularly in fast-reaction and long-horizon scenarios.
Figures & tables
Figure 1: Overview of HEX. (a) HEX is, to the best of our knowledge, the first whole-body VLA framework for full-sized bipedal humanoid robots, pretrained on diverse cross-embodiment humanoid trajectory data. (b) HEX combines a high-level VLA module with a low-level whole-body controller for coordinated action generation and balance-preserving execution. (c) We evaluate HEX on Tienkung 2.0 and Tienkung 3.0 across whole-body, long-horizon, and fast-reaction tasks, demonstrating strong performance across diverse manipulation scenarios.
Figure 2: Overview of the proposed high-level VLA policy in HEX. Given a language instruction L , the current visual observation Vt , and a history query token Qt , the VLM encodes visual-language context together with lightweight temporal review cues summarized in a history cache. In parallel, humanoid-aligned proprioceptive states are organized into structured part-aware tokens and processed by a MoE-based Unified Proprioceptive Predictor, which captures whole-body interactions and forecasts future state dynamics. The resulting visual-language and predictive proprioceptive features are then integrated by the HEX Action Expert through adaptive fusion for action generation, producing task-relevant high-level actions over the prediction horizon.
Figure 3: Left and middle: Unified Proprioceptive Predictor (UPP). Morphology-based proprioceptive states are first mapped into canonical body-part tokens and augmented with learnable future query tokens. These spatio-temporal tokens are processed by a shared transformer backbone sandwiched by morphology-aware MoE adaptation modules, yielding future proprioceptive latents Hp . The middle panel details the morphology-aware MoE, where flattened part-time tokens are routed by a learned top- k gate to a set of routed experts, while a shared expert provides a common transformation across all tokens. This design enables token-wise specialization for embodiment- and part-dependent variations while preserving reusable dynamics across embodiments. Right: Action Expert. Noisy action tokens are encoded and conditioned on both visual-language features HVL and predicted proprioceptive features Hp through dual cross-attention. A learned gate adaptively injects the state branch on top of the visual-language branch, followed by self-attention and feed-forward refinement. The resulting denoised features are decoded into high-level actions for arm and hand control and for downstream whole-body execution.
Figure 4: Real-Robot teleoperation data collection Setup.
Method
Para.
Avg (%)
Tienkung 3.0 ( )
Tienkung 3.0 ( )
Tienkung 3.0 ( )
Kneel and manipulate the objects
Tidy Table
Bring box and pack all objects
ACT [ 60 ]
80M
57.1
83.3
8.3
8.3
SwitchVLA [ 24 ]
0.3B
40.5
0.0
8.3
0.0
GR00T N1.5 [ 5 ]
3B
70.2
100.0
41.7
33.3
π0.5 [ 20 ]
3.3B
71.8
100.0
35.7
25.0
HEX
2.4B
79.8
100.0
41.7
41.7
Table 1: Task success rates (%) of different methods on Tienkung 3.0 and Tienkung 2.0 tasks. The icons denote the main body parts involved in control for each task: arm ( ), hand ( ), waist ( ), and leg ( ).
Method
Tienkung 2.0 ( )
Grasp Box
Turn Around
Walk to Table
Place Box
ACT [ 60 ]
80.0
80.0
46.7
26.7
SwitchVLA [ 24 ]
73.3
60.0
33.3
13.3
GR00T N1.5 [ 5 ]
73.3
66.7
40.0
20.0
π0.5 [ 20 ]
100.0
100.0
73.3
40.0
HEX
100.0
100.0
73.3
53.3
Table 2: Task success rates (%) on the long-horizon box convey task on Tienkung 2.0.
Figure 5: Generalization tasks. Two distribution-shift variants for each of four seen tasks: Pose Mimic , Pouring , Box Carry , and Kneel Pick .
Figure 6: Generalization results across unseen task variants.
Figure 7: Ablation study. (a) Effect of pretraining. Left: state loss. Right: action loss together with success-rate comparisons at different training stages. Pretraining improves optimization. (b) Effect of key model components. Performance improves consistently as the history cache, UPP, and MoE design are progressively introduced, and the full HEX achieves the best success rates on both Pouring and Box Conveying .
Figure 8: Failure analysis across different methods and tasks. Each Sankey diagram shows how failed trials are distributed across task stages and fine-grained error types, with flow width proportional to the number of failures.
Figure 9: Comparison of MoE routing patterns before and after the transformer blocks during a long-horizon box conveyance task. Left: routing before the transformer blocks. Right: routing after the transformer blocks. The heatmaps show the selected expert ID for each body part over time, together with representative frames and subtask boundaries. Compared with the largely static routing before the transformer blocks, routing after the transformer blocks exhibits clearer phase-dependent switching, suggesting stronger state-dependent specialization.
Figure 10: Latency–accuracy comparison on a single NVIDIA RTX 4090 GPU, where bubble size indicates the number of model parameters.
Whole-body humanoid loco-manipulation requires coordinating the robot's entire kinematic chain. However, most existing systems typically decouple the upper and lower bodies into separate controllers, limiting such coordination and yielding behaviors similar to those of a wheeled dual-arm platform. In this paper, we ask what it takes to build a whole-body native vision-language-action (VLA) model that maps language and pixels directly to all of the humanoid's degrees of freedom. We conduct a systematic empirical study organized as a roadmap of one-variable-at-a-time experiments across three phases: whole-body teleoperation, VLA model design, and heterogeneous co-training. Our study yields several intriguing findings: a joint-based whole-body teleoperation interface outperforms alternatives that only partially expose the humanoid's degrees of freedom; a VLA pretrained on static and wheeled dual-arm platforms transfers surprisingly well to a humanoid's full action space; and co-training with HuMI, the humanoid analog of UMI, extends the policy to new objects and instructions without additional whole-body teleoperation on those targets. Following this roadmap yields OpenHLM, an open-source recipe for whole-body humanoid loco-manipulation. In a challenging long-horizon task that spans a wide vertical range of the humanoid, OpenHLM outperforms two state-of-the-art humanoid VLA baselines (GR00T N1.6 and Ψ0) using less than half the total demonstration time. Our code, training data, and model checkpoints are available at [https://openhlm-project.github.io/].
Yingdong Hu, Haodong Zhu, Boyuan Zheng +6
Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI
Humanoid robots hold great promise as general-purpose agents in human-centered environments, yet adapting generalist vision-language-action (VLA) policies to real-world humanoid loco-manipulation remains challenging. Unlike fixed-base or wheeled manipulators, humanoids must coordinate locomotion, posture, gaze, and dual-arm manipulation over a high-dimensional whole-body action space, where inaccurate base or waist motions can easily induce unstable and aggressive upper-body compensation. We propose HAF-VLA, a hierarchical action-flow framework for humanoid VLA control. Built upon a pretrained flow-matching VLA, HAF-VLA inherits the vision-language prefix encoder and action expert weights, and adapts them to humanoid whole-body loco-manipulation with learnable stage embeddings and cross-stage KV-cache conditioning. Instead of generating the full action chunk in a single pass, HAF-VLA progressively denoises a shared whole-body action space: first locomotion and head commands, then waist refinement, and finally full-body manipulation. To preserve structural dependencies across stages, we introduce a cross-stage KV cache conditioning mechanism, enabling subsequent action generation stages to refine and augment their outputs based on the action representations from preceding stages. Experimental results across six real-world tasks demonstrate that our framework significantly enhances the performance of existing models in humanoid loco-manipulation tasks, demonstrating an effective pathway for adapting strong mobile-manipulation VLAs to humanoid whole-body loco-manipulation.
Langzhe Gu, Chengkai Hou, Meng Li +12
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Innovation Center of Humanoid Robotics · Nankai University +2
Dexterous manipulation is essential for real-world robot autonomy, mirroring the central role of human hand coordination in daily activity. Humans rely on rich multimodal perception--vision, sound, and language-guided intent--to perform dexterous actions, motivating vision-based, language-conditioned manipulation systems for robots. However, training reliable vision-language-action (VLA) models for dexterous manipulation requires large-scale demonstrations across many robotic hands. In addition, as new dexterous embodiments appear rapidly, collecting data for each becomes costly and impractical, creating a need for scalable cross-embodiment learning. We introduce XL-VLA, a vision-language-action framework integrated with a unified latent action space shared across diverse dexterous hands. This embodiment-invariant latent space is directly pluggable into standard VLA architectures, enabling seamless cross-embodiment training and efficient reuse of both existing and newly collected data. Experimental results demonstrate that XL-VLA consistently outperforms baseline VLA models operating in raw joint spaces, establishing it as an effective solution for scalable cross-embodiment dexterous manipulation.