Humanoid loco-manipulation requires coordinated task-space motion planning with stable loco-manipulation command tracking under complex robot-environment dynamics and long-horizon tasks. While diffusion policies (DPs) show promise for learning from demonstrations, deploying them on humanoids poses critical challenges: the motion planner trained offline is decoupled from the loco-manipulation controller, leading to poor command tracking, compounding distribution shift, and task failures. The common approach of scaling demonstration data is prohibitively expensive for high-dimensional humanoid systems. To address this challenge, we present REFINE-DP (REinforcement learning FINE-tuning of Diffusion Policy), a hierarchical framework that jointly optimizes a DP motion planner and an RL-based loco-manipulation controller. The DP is fine-tuned via a PPO-based diffusion policy gradient to improve task success rate, while the controller is simultaneously updated to accurately track the planner's evolving command distribution, reducing the distributional mismatch that degrades motion quality. We validate REFINE-DP on a humanoid robot performing loco-manipulation tasks, including door traversal and long-horizon object transport. REFINE-DP achieves an over 90% success rate in simulation, even in out-of-distribution cases not seen in the pre-training data, and enables real-world execution without privileged state information. Our proposed method substantially outperforms pre-trained DP baselines and demonstrates that RL fine-tuning is key to reliable humanoid loco-manipulation. https://refine-dp.github.io/REFINE-DP/
Figures & tables
Fig. 1: Our pipeline consists of three stages: (a) Data Collection, where expert demonstrations are collected in a source environment using a frozen RL-based loco-manipulation controller; (b) Pre-training, where a diffusion policy (DP) motion planner is trained with the expert dataset containing human-demonstrated skills; and (c) Joint Optimization, where the pre-trained DP is jointly fine-tuned with the loco-manipulation controller in a target environment. During deployment, the DP generates Cartesian action chunks, and the loco-manipulation controller converts them into joint-position references for tracking.
Fig. 2: The sim-to-real RL of the loco-manipulation control policy, from motion prior generation to deployment.
Fig. 3: The robot autonomously executes loco-manipulation tasks: (a) door opening, (b) object transportation.
Fig. 4: Comparison of success rates across tasks in Sec. IV-A and methods in Sec. IV-B .
Fig. 5: Joint optimization achieves better end-effector tracking performance and velocities across tasks in Sec. IV-A .
Fig. 6: Comparison of success rate between scaling pre-training data and fine-tuning diffusion policy.
Fig. 7: The foot-placement policy enables the robot to walk on flat ground and stairs.
Fig. 8: Position and yaw tracking errors across 10 trials for velocity-based and foot-placement-based lower-body policies.
Observation
Actor
Critic
Manipulation
Projected gravity
✓
✓
Arm joint pos.
✓
✓
Arm joint vel.
✓
✓
Action
✓
✓
Hand pose cmd. ( pLcmd,pRcmd )
✓
✓
TABLE I: Observation terms for RL policy training
Term
Formulation
Weight
End-effector reference tracking (per foot)
Position ( σ=0.05 )
exp(−∥p−pref∥2/σ2)
+5.0
Yaw ( σ=0.1 )
exp(−(ψ−ψref)2/σ2)
+3.0
Base vel. ( σ=0.25 )
exp(−∥v−vref∥2/σ2)
+1.0
Joint reference tracking
Position ( σ=0.3 )
exp(−∥q−qref∥2/σ2)
+0.5
TABLE II: Reward terms for foot-placement tracking policy
Term
Formulation
Weight
End-effector position tracking (per hand)
Coarse ( σ=0.4 )
exp(−∥p−pcmd∥2/σ2)
+2.0
Fine ( σ=0.1 )
exp(−∥p−pcmd∥2/σ2)
+2.0
Precise ( σ=0.08 )
exp(−∥p−pcmd∥2/σ2)
+2.0
End-effector orientation tracking (per hand)
Coarse ( σ=0.8 )
exp(−∥equat∥2/σ2)
+1.0
TABLE III: Reward terms for hand-pose tracking policy
Fig. 9: Fine-tuning reduces task execution time in both simulation and hardware experiments.
Fig. 10: (a) Evaluation success rates on different robot initialization ranges. With curriculum checkpoint shows the best performance in high OOD regions. (b) Average sum of upper body joint jerk squared over 100 rollout environments.
Imitating human demonstrations has emerged as a dominant paradigm for learning humanoid loco-manipulation policies. However, scaling these approaches remains challenging due to the high cost of collecting diverse demonstrations and the need for continual human intervention to correct policy failures. In this paper, we present a self-supervised framework that bootstraps from sparse demonstrations and progressively expands its behavioral repertoire, enabling the learning of a goal-conditioned policy that automatically explores the goal space with minimal expert supervision. Our approach combines diffusion-based trajectory generation with reinforcement learning, where the latter is used to track goal-conditioned trajectories produced by the diffusion model for a range of loco-manipulation skills. Through extensive ablation studies and comparisons with state-of-the-art methods, we demonstrate the effectiveness of our framework on multiple humanoid loco-manipulation skills.
Pranav Debbad, Kanish Thiagarajan, Victor Dhédin +2
Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich (TUM), Germany.
Developing robust autonomous loco-manipulation skills for humanoids remains an open problem in robotics. While RL has been applied successfully to legged locomotion, applying it to complex, interaction-rich manipulation tasks is harder given long-horizon planning challenges for manipulation. A recent approach along these lines is DreamControl, which addresses these issues by leveraging off-the-shelf human motion diffusion models as a generative prior to guide RL policies during training. In this paper, we investigate the impact of DreamControl's motion prior and propose an improved framework that trains a guided diffusion model directly in the humanoid robot's motion space, aggregating diverse human and robot datasets into a unified embodiment space. We demonstrate that our approach captures a wider range of skills due to the larger training data mixture and establishes a more automated pipeline by removing the need for manual filtering interventions. Furthermore, we show that scaling the generation of reference trajectories is important for achieving robust downstream RL policies. We validate our approach through extensive experiments in simulation and on a real Unitree-G1.
Long-horizon humanoid loco-manipulation requires composing versatile whole-body skills and reliable high-level decision making. Existing methods often coordinate pretrained skills with scripted planners, finite-state machines or task-specific model-free policies, restricting their ability to handle complex task sequences. To address this limitation, we propose LUCID, a hierarchical model-based reinforcement learning framework that plans over reusable skills through imagined rollouts of a learned dynamics model. LUCID first trains a structured latent-conditioned low-level policy via adversarial imitation and then freezes it while jointly learning a high-level policy and macro-dynamics world model. The world model predicts the temporally extended state transitions induced by latent decisions, enabling high-level policy optimization through imagined rollouts. We evaluate our framework across various simulated multi-object rearrangement scenarios. Experimental results show that LUCID improves the full-task success and partial-completion rates compared to prior baseline methods, demonstrating its effectiveness in complex sequential loco-manipulation tasks.
Cheng Guo, Mingzhe Ni, Angelo Cangelosi +1
Department of Computer Science, University of Manchester, Manchester, UK · Human-Robot Interfaces and Interaction Laboratory, Italian Institute of Technology, Genoa, Italy