Complex whole-body loco-manipulation requires executing diverse maneuvers, such as crouching, grasping, and locomotion, within a single long-horizon task. While performing simple pick-and-place behaviors is well-established, scaling to complex, long-horizon tasks remains difficult. Currently, this challenge is addressed by relying on human motion data and correspondingly humanoid platforms, which limits platform independence and introduces substantial data processing overhead. Instead, we extend proximal policy optimization (PPO) to train a single policy for multi-phase tasks across diverse robotic hardware, leveraging prior task structure to shape behavior. Our method adapts PPO using a phase-conditioned critic, phase-gated reward functions, and a multi-phase reset curriculum to guide exploration. To support varied hardware, we employ a decoupled hierarchical control architecture that separates lower-level balancing dynamics from high-level phase execution. We demonstrate multi-stage loco-manipulation skills on hardware on both the Unitree G1 and a Boston Dynamics Spot equipped with a manipulator arm, and validate our design choices through ablations. Our approach enables the acquisition of complex loco-manipulation behaviors purely through environment interaction while directly incorporating high-level task structure into policy learning.
Figures & tables
Fig. 2 : Overview of the low-level whole-body policy. The policy commands leg and waist joints to execute stable locomotion, low crouching, and crouch-walking. Training stability is achieved via inverse kinematics (IK) keyframe initializations that spawn the robot in valid postures, coupled with a feasibility-constrained command space. For standing environments, this region is bounded by the convex hull of admissible IK keyframes, whereas for dynamic environments, the feasible command envelope dynamically contracts as target tracking velocity increases.
Fig. 3 : Overview of the phase-conditioned PPO framework. An environment state machine segments the task into phases ( ψk ) via local success conditions (e.g., distance d and phase-specific thresholds ϵ ). To tackle exploration bottlenecks, a multi-phase curriculum initially resets episodes into later stages before progressively shifting back to the initial phase as training advances. The actor outputs delta actions updating command targets ct , while an auxiliary head enforces phase internalization via cross-entropy loss ( LCE ) against the true environment phase. The critic conditions value estimation on state st and phase one-hot encoding ek , combining a base value Vbase with phase-specific residual heads. These are constrained by Lreg to keep the main learning burden on the base head. Finally, task rewards are dynamically gated based on the active phase ψk and combined with rewards for forward phase transitions and penalties for backward regression.
Group
Description
Dim.
Pelvis IMU
Angular velocity & projected gravity
6
Torso IMU
Angular velocity & projected gravity
6
Commands
Planar linear & angular velocity targets
3
Base height & pitch targets
2
Arm target joint configurations
14
Joints
Joint positions & velocities
58
TABLE I : Observation Space Specifications for humanoid low-level whole-body policy
TABLE II : Task-specific behavioral phases across the evaluated robotic platforms and environments.
Fig. 4 : Hardware deployment of the g1_lift_box task using onboard state estimation and ArUco marker detection. The right panel shows the Unitree G1 humanoid executing the lift, while the insets display the state estimation visualization (top-left) and the onboard first-person camera view tracking the ArUco marker (bottom-left).
Fig. 5 : Algorithmic Ablation Study on Spot and G1: Task success rate across training steps on (a) spot_tire_rolling and (b) g1_push_two_boxes , comparing our method ( Ours ) against baseline architectures (MLP, Multihead), an unguided variant ( No phase prediction ), and execution without multi-phase episode initialization ( Initial phase reset ). Solid curves and shaded regions denote the mean and variance across 3 random seeds, respectively. (c) Spot Phase Analysis: Evaluation of phase-prediction accuracy across 1,000 successful episodes of spot_tire_rolling task. Because individual phase durations vary across rollouts, evaluation episodes are temporally anchored at phase transitions with intermediate timesteps linearly warped to uniform phase durations. The top plot shows that the auxiliary head’s online phase predictions tightly track the ground-truth phase steps with minimal variance, while the bottom plot highlights the object-to-goal distance as a proxy for task progression.
Cluttered indoor environments, where large and heavy objects are scattered across diverse surfaces, require humanoid robots to sequentially navigate, grasp, transport, and accurately place each item at its target location within a single uninterrupted episode. This long-horizon, whole-body loco-manipulation task remains a significant challenge for current methods. Previous approaches often suffer from two main issues: easy-reward bias, where training overemphasizes early transport stages at the expense of later ones, and catastrophic forgetting, where focusing on later stages leads to a decline in earlier-stage performance. In this work, we introduce Humanoid Horizon, a unified policy framework designed to overcome these limitations through three interrelated mechanisms. The Parallel Training Strategy organizes N scenes into S concurrent stage streams governed by a shared policy, ensuring all transport stages receive continuous gradient updates and removing the bottleneck of sequential optimization. The Dynamic Starting Mechanism updates each environment's initial state with terminal states from upstream rollouts, gradually broadening transition coverage and enhancing robustness at stage boundaries. Reward Gating sets the reward to zero for the rest of the episode in later-stage streams when the immediately preceding object is displaced beyond a set threshold, so the shared policy learns not to disturb a just-placed object and earlier placements are preserved throughout the episode. Collectively, these strategies achieve per-stage success rates exceeding 80% on the two-object LHM-Humanoid benchmark (350 training scenes, 66 held-out scenes). As the number of sequentially transported objects grows beyond two, success declines with the horizon, but the degradation is graceful relative to the sharp drop seen in all baselines.
Haozhuo Zhang, Qiang Zhang, Jian Tang +4
The University of Manchester · University of Science and Technology of China · X-Humanoid +2
Long-horizon humanoid loco-manipulation requires composing versatile whole-body skills and reliable high-level decision making. Existing methods often coordinate pretrained skills with scripted planners, finite-state machines or task-specific model-free policies, restricting their ability to handle complex task sequences. To address this limitation, we propose LUCID, a hierarchical model-based reinforcement learning framework that plans over reusable skills through imagined rollouts of a learned dynamics model. LUCID first trains a structured latent-conditioned low-level policy via adversarial imitation and then freezes it while jointly learning a high-level policy and macro-dynamics world model. The world model predicts the temporally extended state transitions induced by latent decisions, enabling high-level policy optimization through imagined rollouts. We evaluate our framework across various simulated multi-object rearrangement scenarios. Experimental results show that LUCID improves the full-task success and partial-completion rates compared to prior baseline methods, demonstrating its effectiveness in complex sequential loco-manipulation tasks.
Cheng Guo, Mingzhe Ni, Angelo Cangelosi +1
Department of Computer Science, University of Manchester, Manchester, UK · Human-Robot Interfaces and Interaction Laboratory, Italian Institute of Technology, Genoa, Italy
Visual loco-manipulation policies that can generalize to novel scenes and objects have long been a goal of robotics research. However, today's data-hungry algorithms make collecting sufficient demonstrations a struggle for tabletop manipulation, and even more so for humanoids that must also walk and balance. Learning from simulated data and transferring that behavior to the real world, as is commonly done in locomotion, sidesteps this struggle, so we replicate that recipe for loco-manipulation. In doing so, we find that cloning synthetic demonstrations results in a low performance ceiling no matter the amount of training data. Reinforcement learning breaks through it, and refining the cloned policy with Flow-GRPO on a single sparse reward yields performance that synthetic behavior cloning cannot match. Together, these stages form our end-to-end sim-to-real pipeline spanning more than 150,000 scenes, which we use to train FetchMan. We evaluate it on FetchMan-Bench, a simulation benchmark we release, and deploy it zero-shot on a real Unitree G1, where our single-object reach-and-pick policy walks to and grasps a target across unseen scenes at 73.3% success. Finally, we extend this recipe to multi-object training, a first step toward loco-manipulation generalist policies at this data scale.
Omar Rayyan, Zhi Li, Max Argus +4
University of California, Los Angeles · Allen Institute for AI · University of Washington