Humans achieve complex manipulation through coordinated whole-body control, whereas most Vision-Language-Action (VLA) models treat robot body parts largely independently, making high-DoF humanoid control challenging and often unstable. We present HEX, a state-centric framework for coordinated manipulation on full-sized bipedal humanoid robots. HEX introduces a humanoid-aligned universal state representation for scalable learning across heterogeneous embodiments, and incorporates a Mixture-of-Experts Unified Proprioceptive Predictor to model whole-body coordination and temporal motion dynamics from large-scale multi-embodiment trajectory data. To efficiently capture temporal visual context, HEX uses lightweight history tokens to summarize past observations, avoiding repeated encoding of historical images during inference. It further employs a residual-gated fusion mechanism with a flow-matching action head to adaptively integrate visual-language cues with proprioceptive dynamics for action generation. Experiments on real-world humanoid manipulation tasks show that HEX achieves state-of-the-art performance in task success rate and generalization, particularly in fast-reaction and long-horizon scenarios.
Figures & tables
Figure 1: Overview of HEX. (a) HEX is, to the best of our knowledge, the first whole-body VLA framework for full-sized bipedal humanoid robots, pretrained on diverse cross-embodiment humanoid trajectory data. (b) HEX combines a high-level VLA module with a low-level whole-body controller for coordinated action generation and balance-preserving execution. (c) We evaluate HEX on Tienkung 2.0 and Tienkung 3.0 across whole-body, long-horizon, and fast-reaction tasks, demonstrating strong performance across diverse manipulation scenarios.
Figure 2: Overview of the proposed high-level VLA policy in HEX. Given a language instruction L , the current visual observation Vt , and a history query token Qt , the VLM encodes visual-language context together with lightweight temporal review cues summarized in a history cache. In parallel, humanoid-aligned proprioceptive states are organized into structured part-aware tokens and processed by a MoE-based Unified Proprioceptive Predictor, which captures whole-body interactions and forecasts future state dynamics. The resulting visual-language and predictive proprioceptive features are then integrated by the HEX Action Expert through adaptive fusion for action generation, producing task-relevant high-level actions over the prediction horizon.
Figure 3: Left and middle: Unified Proprioceptive Predictor (UPP). Morphology-based proprioceptive states are first mapped into canonical body-part tokens and augmented with learnable future query tokens. These spatio-temporal tokens are processed by a shared transformer backbone sandwiched by morphology-aware MoE adaptation modules, yielding future proprioceptive latents Hp . The middle panel details the morphology-aware MoE, where flattened part-time tokens are routed by a learned top- k gate to a set of routed experts, while a shared expert provides a common transformation across all tokens. This design enables token-wise specialization for embodiment- and part-dependent variations while preserving reusable dynamics across embodiments. Right: Action Expert. Noisy action tokens are encoded and conditioned on both visual-language features HVL and predicted proprioceptive features Hp through dual cross-attention. A learned gate adaptively injects the state branch on top of the visual-language branch, followed by self-attention and feed-forward refinement. The resulting denoised features are decoded into high-level actions for arm and hand control and for downstream whole-body execution.
Figure 4: Real-Robot teleoperation data collection Setup.
Method
Para.
Avg (%)
Tienkung 3.0 ( )
Tienkung 3.0 ( )
Tienkung 3.0 ( )
Kneel and manipulate the objects
Tidy Table
Bring box and pack all objects
ACT [ 60 ]
80M
57.1
83.3
8.3
8.3
SwitchVLA [ 24 ]
0.3B
40.5
0.0
8.3
0.0
GR00T N1.5 [ 5 ]
3B
70.2
100.0
41.7
33.3
π0.5 [ 20 ]
3.3B
71.8
100.0
35.7
25.0
HEX
2.4B
79.8
100.0
41.7
41.7
Table 1: Task success rates (%) of different methods on Tienkung 3.0 and Tienkung 2.0 tasks. The icons denote the main body parts involved in control for each task: arm ( ), hand ( ), waist ( ), and leg ( ).
Method
Tienkung 2.0 ( )
Grasp Box
Turn Around
Walk to Table
Place Box
ACT [ 60 ]
80.0
80.0
46.7
26.7
SwitchVLA [ 24 ]
73.3
60.0
33.3
13.3
GR00T N1.5 [ 5 ]
73.3
66.7
40.0
20.0
π0.5 [ 20 ]
100.0
100.0
73.3
40.0
HEX
100.0
100.0
73.3
53.3
Table 2: Task success rates (%) on the long-horizon box convey task on Tienkung 2.0.
Figure 5: Generalization tasks. Two distribution-shift variants for each of four seen tasks: Pose Mimic , Pouring , Box Carry , and Kneel Pick .
Figure 6: Generalization results across unseen task variants.
Figure 7: Ablation study. (a) Effect of pretraining. Left: state loss. Right: action loss together with success-rate comparisons at different training stages. Pretraining improves optimization. (b) Effect of key model components. Performance improves consistently as the history cache, UPP, and MoE design are progressively introduced, and the full HEX achieves the best success rates on both Pouring and Box Conveying .
Figure 8: Failure analysis across different methods and tasks. Each Sankey diagram shows how failed trials are distributed across task stages and fine-grained error types, with flow width proportional to the number of failures.
Figure 9: Comparison of MoE routing patterns before and after the transformer blocks during a long-horizon box conveyance task. Left: routing before the transformer blocks. Right: routing after the transformer blocks. The heatmaps show the selected expert ID for each body part over time, together with representative frames and subtask boundaries. Compared with the largely static routing before the transformer blocks, routing after the transformer blocks exhibits clearer phase-dependent switching, suggesting stronger state-dependent specialization.
Figure 10: Latency–accuracy comparison on a single NVIDIA RTX 4090 GPU, where bubble size indicates the number of model parameters.
Whole-body humanoid loco-manipulation requires coordinating the robot's entire kinematic chain. However, most existing systems typically decouple the upper and lower bodies into separate controllers, limiting such coordination and yielding behaviors similar to those of a wheeled dual-arm platform. In this paper, we ask what it takes to build a whole-body native vision-language-action (VLA) model that maps language and pixels directly to all of the humanoid's degrees of freedom. We conduct a systematic empirical study organized as a roadmap of one-variable-at-a-time experiments across three phases: whole-body teleoperation, VLA model design, and heterogeneous co-training. Our study yields several intriguing findings: a joint-based whole-body teleoperation interface outperforms alternatives that only partially expose the humanoid's degrees of freedom; a VLA pretrained on static and wheeled dual-arm platforms transfers surprisingly well to a humanoid's full action space; and co-training with HuMI, the humanoid analog of UMI, extends the policy to new objects and instructions without additional whole-body teleoperation on those targets. Following this roadmap yields OpenHLM, an open-source recipe for whole-body humanoid loco-manipulation. In a challenging long-horizon task that spans a wide vertical range of the humanoid, OpenHLM outperforms two state-of-the-art humanoid VLA baselines (GR00T N1.6 and Ψ0) using less than half the total demonstration time. Our code, training data, and model checkpoints are available at [https://openhlm-project.github.io/].
Yingdong Hu, Haodong Zhu, Boyuan Zheng +6
Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI
Dexterous manipulation is essential for real-world robot autonomy, mirroring the central role of human hand coordination in daily activity. Humans rely on rich multimodal perception--vision, sound, and language-guided intent--to perform dexterous actions, motivating vision-based, language-conditioned manipulation systems for robots. However, training reliable vision-language-action (VLA) models for dexterous manipulation requires large-scale demonstrations across many robotic hands. In addition, as new dexterous embodiments appear rapidly, collecting data for each becomes costly and impractical, creating a need for scalable cross-embodiment learning. We introduce XL-VLA, a vision-language-action framework integrated with a unified latent action space shared across diverse dexterous hands. This embodiment-invariant latent space is directly pluggable into standard VLA architectures, enabling seamless cross-embodiment training and efficient reuse of both existing and newly collected data. Experimental results demonstrate that XL-VLA consistently outperforms baseline VLA models operating in raw joint spaces, establishing it as an effective solution for scalable cross-embodiment dexterous manipulation.
Vision-Language-Action (VLA) models have become a powerful paradigm for robot manipulation, but training a single generalist policy for heterogeneous robot embodiments remains an open problem. Existing methods have two main limitations. First, they underuse dynamics priors shared across diverse visual and interaction data, limiting cross-embodiment transfer. Second, they require extensive manual preprocessing to convert embodiment-specific actions into a common format. To overcome these limitations, we propose DyPES-VLA, a cross-embodiment VLA that learns shared Dynamics Priors and Embodiment-Specific control. First, we learn shared dynamics priors by training the vision-language model (VLM) with a future-prediction objective on cross-embodiment data, driving the shared query representation to capture object motion, contact, and interaction-induced scene changes. Second, an embodiment-specific Mixture-of-Experts (MoE) action head translates these shared dynamics priors into executable controls directly in each embodiment's native action space, without manually pre-aligning heterogeneous actions into a common format. This head shares attention layers to capture common temporal action structures, while its embodiment-specific feed-forward experts resolve the unique kinematic constraints and control semantics of distinct embodiments. As a generalist policy, our \ourmethod achieves state-of-the-art performance across simulation and real-world evaluations, reaching 98.0% success on LIBERO, 59.25% on RoboCasa-GR1, and 89.02% on RoboTwin~2.0.
Junfeng Li, Junjie He, Zhide Zhong +12
1The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · 2COCO Matrix, Shanghai, China