Complex whole-body loco-manipulation requires executing diverse maneuvers, such as crouching, grasping, and locomotion, within a single long-horizon task. While performing simple pick-and-place behaviors is well-established, scaling to complex, long-horizon tasks remains difficult. Currently, this challenge is addressed by relying on human motion data and correspondingly humanoid platforms, which limits platform independence and introduces substantial data processing overhead. Instead, we extend proximal policy optimization (PPO) to train a single policy for multi-phase tasks across diverse robotic hardware, leveraging prior task structure to shape behavior. Our method adapts PPO using a phase-conditioned critic, phase-gated reward functions, and a multi-phase reset curriculum to guide exploration. To support varied hardware, we employ a decoupled hierarchical control architecture that separates lower-level balancing dynamics from high-level phase execution. We demonstrate multi-stage loco-manipulation skills on hardware on both the Unitree G1 and a Boston Dynamics Spot equipped with a manipulator arm, and validate our design choices through ablations. Our approach enables the acquisition of complex loco-manipulation behaviors purely through environment interaction while directly incorporating high-level task structure into policy learning.
Figures & tables
Fig. 2 : Overview of the low-level whole-body policy. The policy commands leg and waist joints to execute stable locomotion, low crouching, and crouch-walking. Training stability is achieved via inverse kinematics (IK) keyframe initializations that spawn the robot in valid postures, coupled with a feasibility-constrained command space. For standing environments, this region is bounded by the convex hull of admissible IK keyframes, whereas for dynamic environments, the feasible command envelope dynamically contracts as target tracking velocity increases.
Fig. 3 : Overview of the phase-conditioned PPO framework. An environment state machine segments the task into phases ( ψk ) via local success conditions (e.g., distance d and phase-specific thresholds ϵ ). To tackle exploration bottlenecks, a multi-phase curriculum initially resets episodes into later stages before progressively shifting back to the initial phase as training advances. The actor outputs delta actions updating command targets ct , while an auxiliary head enforces phase internalization via cross-entropy loss ( LCE ) against the true environment phase. The critic conditions value estimation on state st and phase one-hot encoding ek , combining a base value Vbase with phase-specific residual heads. These are constrained by Lreg to keep the main learning burden on the base head. Finally, task rewards are dynamically gated based on the active phase ψk and combined with rewards for forward phase transitions and penalties for backward regression.
Group
Description
Dim.
Pelvis IMU
Angular velocity & projected gravity
6
Torso IMU
Angular velocity & projected gravity
6
Commands
Planar linear & angular velocity targets
3
Base height & pitch targets
2
Arm target joint configurations
14
Joints
Joint positions & velocities
58
TABLE I : Observation Space Specifications for humanoid low-level whole-body policy
TABLE II : Task-specific behavioral phases across the evaluated robotic platforms and environments.
Fig. 4 : Hardware deployment of the g1_lift_box task using onboard state estimation and ArUco marker detection. The right panel shows the Unitree G1 humanoid executing the lift, while the insets display the state estimation visualization (top-left) and the onboard first-person camera view tracking the ArUco marker (bottom-left).
Fig. 5 : Algorithmic Ablation Study on Spot and G1: Task success rate across training steps on (a) spot_tire_rolling and (b) g1_push_two_boxes , comparing our method ( Ours ) against baseline architectures (MLP, Multihead), an unguided variant ( No phase prediction ), and execution without multi-phase episode initialization ( Initial phase reset ). Solid curves and shaded regions denote the mean and variance across 3 random seeds, respectively. (c) Spot Phase Analysis: Evaluation of phase-prediction accuracy across 1,000 successful episodes of spot_tire_rolling task. Because individual phase durations vary across rollouts, evaluation episodes are temporally anchored at phase transitions with intermediate timesteps linearly warped to uniform phase durations. The top plot shows that the auxiliary head’s online phase predictions tightly track the ground-truth phase steps with minimal variance, while the bottom plot highlights the object-to-goal distance as a proxy for task progression.
Department of Computer Science, University of Manchester, Manchester, UK · Human-Robot Interfaces and Interaction Laboratory, Italian Institute of Technology, Genoa, Italy