Humans achieve complex manipulation through coordinated whole-body control, whereas most Vision-Language-Action (VLA) models treat robot body parts largely independently, making high-DoF humanoid control challenging and often unstable. We present HEX, a state-centric framework for coordinated manipulation on full-sized bipedal humanoid robots. HEX introduces a humanoid-aligned universal state representation for scalable learning across heterogeneous embodiments, and incorporates a Mixture-of-Experts Unified Proprioceptive Predictor to model whole-body coordination and temporal motion dynamics from large-scale multi-embodiment trajectory data. To efficiently capture temporal visual context, HEX uses lightweight history tokens to summarize past observations, avoiding repeated encoding of historical images during inference. It further employs a residual-gated fusion mechanism with a flow-matching action head to adaptively integrate visual-language cues with proprioceptive dynamics for action generation. Experiments on real-world humanoid manipulation tasks show that HEX achieves state-of-the-art performance in task success rate and generalization, particularly in fast-reaction and long-horizon scenarios.
Figures & tables
Figure 1: Overview of HEX. (a) HEX is, to the best of our knowledge, the first whole-body VLA framework for full-sized bipedal humanoid robots, pretrained on diverse cross-embodiment humanoid trajectory data. (b) HEX combines a high-level VLA module with a low-level whole-body controller for coordinated action generation and balance-preserving execution. (c) We evaluate HEX on Tienkung 2.0 and Tienkung 3.0 across whole-body, long-horizon, and fast-reaction tasks, demonstrating strong performance across diverse manipulation scenarios.
Figure 2: Overview of the proposed high-level VLA policy in HEX. Given a language instruction L , the current visual observation Vt , and a history query token Qt , the VLM encodes visual-language context together with lightweight temporal review cues summarized in a history cache. In parallel, humanoid-aligned proprioceptive states are organized into structured part-aware tokens and processed by a MoE-based Unified Proprioceptive Predictor, which captures whole-body interactions and forecasts future state dynamics. The resulting visual-language and predictive proprioceptive features are then integrated by the HEX Action Expert through adaptive fusion for action generation, producing task-relevant high-level actions over the prediction horizon.
Figure 3: Left and middle: Unified Proprioceptive Predictor (UPP). Morphology-based proprioceptive states are first mapped into canonical body-part tokens and augmented with learnable future query tokens. These spatio-temporal tokens are processed by a shared transformer backbone sandwiched by morphology-aware MoE adaptation modules, yielding future proprioceptive latents Hp . The middle panel details the morphology-aware MoE, where flattened part-time tokens are routed by a learned top- k gate to a set of routed experts, while a shared expert provides a common transformation across all tokens. This design enables token-wise specialization for embodiment- and part-dependent variations while preserving reusable dynamics across embodiments. Right: Action Expert. Noisy action tokens are encoded and conditioned on both visual-language features HVL and predicted proprioceptive features Hp through dual cross-attention. A learned gate adaptively injects the state branch on top of the visual-language branch, followed by self-attention and feed-forward refinement. The resulting denoised features are decoded into high-level actions for arm and hand control and for downstream whole-body execution.
Figure 4: Real-Robot teleoperation data collection Setup.
Method
Para.
Avg (%)
Tienkung 3.0 ( )
Tienkung 3.0 ( )
Tienkung 3.0 ( )
Kneel and manipulate the objects
Tidy Table
Bring box and pack all objects
ACT [ 60 ]
80M
57.1
83.3
8.3
8.3
SwitchVLA [ 24 ]
0.3B
40.5
0.0
8.3
0.0
GR00T N1.5 [ 5 ]
3B
70.2
100.0
41.7
33.3
π0.5 [ 20 ]
3.3B
71.8
100.0
35.7
25.0
HEX
2.4B
79.8
100.0
41.7
41.7
Table 1: Task success rates (%) of different methods on Tienkung 3.0 and Tienkung 2.0 tasks. The icons denote the main body parts involved in control for each task: arm ( ), hand ( ), waist ( ), and leg ( ).
Method
Tienkung 2.0 ( )
Grasp Box
Turn Around
Walk to Table
Place Box
ACT [ 60 ]
80.0
80.0
46.7
26.7
SwitchVLA [ 24 ]
73.3
60.0
33.3
13.3
GR00T N1.5 [ 5 ]
73.3
66.7
40.0
20.0
π0.5 [ 20 ]
100.0
100.0
73.3
40.0
HEX
100.0
100.0
73.3
53.3
Table 2: Task success rates (%) on the long-horizon box convey task on Tienkung 2.0.
Figure 5: Generalization tasks. Two distribution-shift variants for each of four seen tasks: Pose Mimic , Pouring , Box Carry , and Kneel Pick .
Figure 6: Generalization results across unseen task variants.
Figure 7: Ablation study. (a) Effect of pretraining. Left: state loss. Right: action loss together with success-rate comparisons at different training stages. Pretraining improves optimization. (b) Effect of key model components. Performance improves consistently as the history cache, UPP, and MoE design are progressively introduced, and the full HEX achieves the best success rates on both Pouring and Box Conveying .
Figure 8: Failure analysis across different methods and tasks. Each Sankey diagram shows how failed trials are distributed across task stages and fine-grained error types, with flow width proportional to the number of failures.
Figure 9: Comparison of MoE routing patterns before and after the transformer blocks during a long-horizon box conveyance task. Left: routing before the transformer blocks. Right: routing after the transformer blocks. The heatmaps show the selected expert ID for each body part over time, together with representative frames and subtask boundaries. Compared with the largely static routing before the transformer blocks, routing after the transformer blocks exhibits clearer phase-dependent switching, suggesting stronger state-dependent specialization.
Figure 10: Latency–accuracy comparison on a single NVIDIA RTX 4090 GPU, where bubble size indicates the number of model parameters.
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Innovation Center of Humanoid Robotics · Nankai University +2