Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Innovation Center of Humanoid Robotics · Nankai University · Xi’an Jiaotong University
Humanoid robots hold great promise as general-purpose agents in human-centered environments, yet generalist vision-language-action (VLA) foundation models are not readily applicable to humanoid whole-body loco-manipulation. The high dimensionality and interdependence of humanoid motions make it challenging for conventional single-stage VLA architectures to coordinate locomotion, waist posture, and dual-arm manipulation effectively. Moreover, policies trained through offline behavior cloning can remain suboptimal during real-world deployment. Although online reinforcement learning can refine policies through real-world interaction, directly tuning large VLA backbones demands excessive computation and may introduce safety risks during real-robot exploration. To address these bottlenecks, we introduce HAF (Humanoid Adaptation Framework), a two-part framework consisting of HAF-VLA and HAF-Steer that transfers off-the-shelf generalist VLA foundation models to humanoid whole-body loco-manipulation. HAF-VLA is a hierarchical action-flow generator built on a pretrained flow-matching VLA. It splits full-body action denoising into three sequential stages with stage embeddings and cross-stage KV caches that retain kinematic dependencies, avoiding incoherent whole-body actions from one-shot generation. On top of the frozen HAF-VLA, HAF-Steer is a latent offline-to-online RL pipeline that leverages flow-matching invertibility and DCT-based dimensionality reduction to restrict RL optimization to a compact noise subspace and train a regularized SAC policy. This avoids updating the large VLA backbone and enables efficient real-world policy refinement. Evaluated on seven real-world humanoid loco-manipulation tasks, HAF surpasses vanilla single-stage VLA baselines and improves whole-body coordination and task performance. Project website: https://grange007.github.io/HAF .
Figures & tables
Figure 1: Overview of HAF. We demonstrate seven humanoid loco-manipulation tasks in real-world settings. HAF consists of two complementary components: HAF-VLA is a hierarchical action-flow vision-language-action model that generates whole-body motions stage by stage according to the kinematic dependencies among locomotion, torso adjustment, and dual-arm manipulation; HAF-Steer performs latent-space reinforcement learning on discrete cosine transform compressed temporal noise from frozen flow-based VLA policies. Together, they improve task performance in real-world settings.
Figure 2: Overview of HAF-VLA. A shared action expert progressively expands the active action space from locomotion and head control to waist posture and bimanual manipulation. Clean action caches from earlier stages condition subsequent generation, and only the final full-body action chunk is executed.
Figure 3: Overview of HAF-Steer. (a) A conventional flow-matching VLA samples full-dimensional Gaussian noise and decodes it into an action trajectory through a frozen flow generator. (b) HAF-Steer instead constructs a low-dimensional spectral action space from expert demonstrations: demonstrated trajectories are mapped back to their initial flow noise through numerical reverse integration, compressed by retaining the first 8 temporal DCT coefficients, and normalized to form expert spectral actions in the offline replay buffer. An RL policy is trained with mixed offline and online experience to predict spectral actions, which are de-normalized, reconstructed into full temporal noise by inverse DCT, and decoded by the frozen VLA to produce executable robot trajectories.
Figure 4: Robot teleoperation data collection setup. The teleoperation pipeline uses an IMU for head control, kinematically matched master arms for dual-arm manipulation, and a joystick to command leg locomotion and waist posture.
Figure 5: Seven real-world humanoid loco-manipulation tasks. Each row depicts a representative execution sequence requiring navigation, whole-body posture adjustment, and physical object interaction.
Task
HAF-VLA
π0.5
GR00T N1.7
Cosmos Policy
ACT
Laundry Loading
66.7
53.3
40.0
0.0
10.0
Clothes Retrieval
53.3
53.3
33.3
26.7
23.3
Table Tidy
80.0
70.0
40.0
16.7
23.3
Basket Transfer
63.3
50.0
43.3
33.3
16.7
Toy Storage
80.0
53.3
30.0
40.0
23.3
Ball Tossing
56.7
33.3
36.7
3.3
30.0
Table 1: Main HAF-VLA results on seven long-horizon humanoid loco–manipulation tasks. We report the average normalized task score (%) over 10 rollout trials. Higher values indicate better performance.
Method / Variant
Stage Design
Total Steps
Norm. Score (%)
HAF-VLA
Locomotion/Head → + Waist → + Manip.
30
66.7
All-Joint Denoising
All joints at every stage
30
53.3
Arm-First Hierarchy
Manip. → + Waist → + Locomotion/Head
30
50.0
π0.5 , 30 steps
Full-body action
30
20.0
Table 2: Ablation study of HAF-VLA’s hierarchical action-flow design. We report the average normalized task score (%) over 10 rollout trials on the Laundry Loading task. Higher scores are better.
Figure 6: Generalization experiments. Left: object-distraction test during Laundry Loading . Right: position-disturbance test during Clothes Retrieval .
Task
Disturbance
HAF-VLA
π0.5
Laundry Loading
Unseen chair near the path
40.0
26.7
Clothes Retrieval
20 cm backward start shift
43.3
36.7
Table 3: Generalization evaluation under controlled disturbances. We report the average normalized task score (%) over 10 rollout trials. Higher scores indicate stronger robustness.
Figure 7: In-distribution and out-of-distribution evaluation setups for HAF-Steer. Blue markers indicate destination-table locations covered during demonstration collection, while red markers indicate the OOD evaluation locations. For Toy Storage , the destination table is placed 30 cm beyond the demonstrated spatial range. For Basket Transfer , the destination table is shifted by 30 cm from its demonstrated position.
Figure 8: HAF-Steer performance under in-distribution and out-of-distribution settings. We report successful trials out of 10 rollouts on Toy Storage and Basket Transfer using both π0.5 and HAF-VLA backbones. Noise BC denotes the spectral actor after behavior-cloning initialization without SAC post-training. † DSRL failed to complete training because unsafe exploratory motions triggered early termination during real-robot rollouts.
Whole-body humanoid loco-manipulation requires coordinating the robot's entire kinematic chain. However, most existing systems typically decouple the upper and lower bodies into separate controllers, limiting such coordination and yielding behaviors similar to those of a wheeled dual-arm platform. In this paper, we ask what it takes to build a whole-body native vision-language-action (VLA) model that maps language and pixels directly to all of the humanoid's degrees of freedom. We conduct a systematic empirical study organized as a roadmap of one-variable-at-a-time experiments across three phases: whole-body teleoperation, VLA model design, and heterogeneous co-training. Our study yields several intriguing findings: a joint-based whole-body teleoperation interface outperforms alternatives that only partially expose the humanoid's degrees of freedom; a VLA pretrained on static and wheeled dual-arm platforms transfers surprisingly well to a humanoid's full action space; and co-training with HuMI, the humanoid analog of UMI, extends the policy to new objects and instructions without additional whole-body teleoperation on those targets. Following this roadmap yields OpenHLM, an open-source recipe for whole-body humanoid loco-manipulation. In a challenging long-horizon task that spans a wide vertical range of the humanoid, OpenHLM outperforms two state-of-the-art humanoid VLA baselines (GR00T N1.6 and Ψ0) using less than half the total demonstration time. Our code, training data, and model checkpoints are available at [https://openhlm-project.github.io/].
Yingdong Hu, Haodong Zhu, Boyuan Zheng +6
Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI
Vision-language-action (VLA) models using discrete action tokens have proven effective for controling robotic arms on manipulation tasks. For a humanoid, however, the whole-body action space -- legs, torso, arms, and hands -- is far higher-dimensional and heterogeneous, raising tokenization, training, and real-time inference challenges that the previous VLA models do not address. We present Holo-M, to our knowledge the first discrete VLA model for humanoid loco-manipulation that intrinsically exploits the language model by extending its vocabulary with action tokens. In this model, we devise a unified action tokenizer that decomposes the humanoid action space into four body-part-specific tokenizers -- end-effector, body, hand, and kinematics -- enabling training across drastically different embodiments and data sources, including humanoid teleoperation, ego-centric human video, and simulation. By extending the language model's vocabulary with these action tokens, we avoid the knowledge-insulation problem inherent to the models that use separate continuous action experts. To meet real-time control requirements, we decode each body part's action tokens through grouped discrete diffusion decoding, rather than using autoregression on the action tokens. We have conducted extensive experiments on the SIMPLE humanoid loco-manipulation benchmark, in which Holo-M achieves the highest success rates in both the generalist and specialist evaluations, leading the second best by significant margins. We will release all the code and model weights.
Vision-Language-Action (VLA) models have recently demonstrated strong generalization in robotic manipulation, yet their applicability to whole-body, contact-rich humanoid locomotion remains severely underexplored due to data scarcity, the absence of dynamically consistent demonstrations, and the difficulty of encoding optimality and safety in learning-based pipelines. This work introduces a unified framework WOLF-VLA that integrates whole-body optimal-control (OC) motion synthesis with large-scale multi-modal dataset to train VLAs capable of generating humanoid locomotion policies directly from natural-language instructions. We construct a comprehensive dataset of dynamically feasible humanoid trajectories across six locomotion-related task families, each parameterized by environmental variations, object colors, placements, and visual distractors. We train a VLA model using the collected joint trajectories, ego-centric visual observations and natural language instruction, yielding a policy that exhibits strong reasoning and robustness to initial-condition variability, and competitive performance across several tasks and environment settings. A systematic ablation study demonstrates the impact of each modality on the model performance. The full dataset, model checkpoints, and benchmarking simulation suite will be openly released, establishing a reproducible dynamically consistent benchmark for whole-body humanoid locomotion rich VLA control and enabling future research in scalable transfer of instruction-driven locomotion policies.
Melya Boukheddimi, Omar Adjali, Daniel Sontag +1
Robotics Innovation Center, DFKI, 28359 Bremen, Germany · Interactive Machine Learning, DFKI, 26129 Oldenburg, Germany · University of Oldenburg, 26129 Oldenburg, Germany +1