REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning

Zhaoyuan Gu; Yipu Chen; Zimeng Chai; Alfred Cueva; Thong Nguyen; Yifan Wu; Huishu Xue; Minji Kim

cs.ROMar 14, 2026

REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning

Authors: Zhaoyuan Gu, Yipu Chen, Zimeng Chai, Alfred Cueva, Thong Nguyen, Yifan Wu, Huishu Xue, Minji Kim, +6 more

Organizations: Institute for Robotics and Intelligent Machines, Georgia Institute of Technology, Atlanta, GA 30332 USA

Abstract

Humanoid loco-manipulation requires coordinated task-space motion planning with stable loco-manipulation command tracking under complex robot-environment dynamics and long-horizon tasks. While diffusion policies (DPs) show promise for learning from demonstrations, deploying them on humanoids poses critical challenges: the motion planner trained offline is decoupled from the loco-manipulation controller, leading to poor command tracking, compounding distribution shift, and task failures. The common approach of scaling demonstration data is prohibitively expensive for high-dimensional humanoid systems. To address this challenge, we present REFINE-DP (REinforcement learning FINE-tuning of Diffusion Policy), a hierarchical framework that jointly optimizes a DP motion planner and an RL-based loco-manipulation controller. The DP is fine-tuned via a PPO-based diffusion policy gradient to improve task success rate, while the controller is simultaneously updated to accurately track the planner's evolving command distribution, reducing the distributional mismatch that degrades motion quality. We validate REFINE-DP on a humanoid robot performing loco-manipulation tasks, including door traversal and long-horizon object transport. REFINE-DP achieves an over 90% success rate in simulation, even in out-of-distribution cases not seen in the pre-training data, and enables real-world execution without privileged state information. Our proposed method substantially outperforms pre-trained DP baselines and demonstrates that RL fine-tuning is key to reliable humanoid loco-manipulation. https://refine-dp.github.io/REFINE-DP/

Figures & tables

Fig. 1: Our pipeline consists of three stages: (a) Data Collection, where expert demonstrations are collected in a source environment using a frozen RL-based loco-manipulation controller; (b) Pre-training, where a diffusion policy (DP) motion planner is trained with the expert dataset containing human-demonstrated skills; and (c) Joint Optimization, where the pre-trained DP is jointly fine-tuned with the loco-manipulation controller in a target environment. During deployment, the DP generates Cartesian action chunks, and the loco-manipulation controller converts them into joint-position references for tracking.

Fig. 2: The sim-to-real RL of the loco-manipulation control policy, from motion prior generation to deployment.

Fig. 3: The robot autonomously executes loco-manipulation tasks: (a) door opening, (b) object transportation.

Fig. 4: Comparison of success rates across tasks in Sec. IV-A and methods in Sec. IV-B .

Fig. 5: Joint optimization achieves better end-effector tracking performance and velocities across tasks in Sec. IV-A .

Fig. 6: Comparison of success rate between scaling pre-training data and fine-tuning diffusion policy.

Fig. 7: The foot-placement policy enables the robot to walk on flat ground and stairs.

Fig. 8: Position and yaw tracking errors across 10 trials for velocity-based and foot-placement-based lower-body policies.

Observation	Actor	Critic
Manipulation
Projected gravity	✓	✓
Arm joint pos.	✓	✓
Arm joint vel.	✓	✓
Action	✓	✓
Hand pose cmd. ( $\mathbf{p}_{L}^{\text{cmd}},\mathbf{p}_{R}^{\text{cmd}}$ )	✓	✓

TABLE I: Observation terms for RL policy training

Term	Formulation	Weight
End-effector reference tracking (per foot)
Position ( $\sigma=0.05$ )	$\exp(-\lVert\mathbf{p}-\mathbf{p}^{\mathrm{ref}}\rVert^{2}/\sigma^{2})$	$+5.0$
Yaw ( $\sigma=0.1$ )	$\exp(-(\psi-\psi^{\mathrm{ref}})^{2}/\sigma^{2})$	$+3.0$
Base vel. ( $\sigma=0.25$ )	$\exp(-\lVert\mathbf{v}-\mathbf{v}^{\mathrm{ref}}\rVert^{2}/\sigma^{2})$	$+1.0$
Joint reference tracking
Position ( $\sigma=0.3$ )	$\exp(-\lVert\mathbf{q}-\mathbf{q}^{\mathrm{ref}}\rVert^{2}/\sigma^{2})$	$+0.5$

TABLE II: Reward terms for foot-placement tracking policy

Term	Formulation	Weight
End-effector position tracking (per hand)
Coarse ( $\sigma=0.4$ )	$\exp(-\lVert\mathbf{p}-\mathbf{p}^{\mathrm{cmd}}\rVert^{2}/\sigma^{2})$	$+2.0$
Fine ( $\sigma=0.1$ )	$\exp(-\lVert\mathbf{p}-\mathbf{p}^{\mathrm{cmd}}\rVert^{2}/\sigma^{2})$	$+2.0$
Precise ( $\sigma=0.08$ )	$\exp(-\lVert\mathbf{p}-\mathbf{p}^{\mathrm{cmd}}\rVert^{2}/\sigma^{2})$	$+2.0$
End-effector orientation tracking (per hand)
Coarse ( $\sigma=0.8$ )	$\exp(-\lVert\mathbf{e}_{\mathrm{quat}}\rVert^{2}/\sigma^{2})$	$+1.0$

TABLE III: Reward terms for hand-pose tracking policy

Fig. 9: Fine-tuning reduces task execution time in both simulation and hardware experiments.

Panel of Figure 13 — Fig. 10: (a) Evaluation success rates on different robot initialization ranges. With curriculum checkpoint shows the best performance in high OOD regions. (b) Average sum of upper body joint jerk squared over 100 rollout environments.

Explore similar work

Humanoid-DART: Humanoid Loco-Manipulation using Diffusion-guided Augmentation through Relabeling and Tracking
Jun 25, 2026Pranav Debbad, Kanish Thiagarajan, Victor Dhédin +2Humanoid Loco-Manipulation Robot Imitation Learning
Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich (TUM), Germany.
DreamControl-v2: Simpler and Scalable Autonomous Humanoid Skills via Trainable Guided Diffusion Priors
Mar 31, 2026Sudarshan Harithas, Sangkyung Kwak, Pushkal Katara +6Robot Skill Learning Humanoid Robot Control
General Robotics, USA · Brown University, USA · University of California, Berkeley, USA
LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation
Aug 7, 2026Cheng Guo, Mingzhe Ni, Angelo Cangelosi +1Humanoid Loco-Manipulation Latent Action Learning
Department of Computer Science, University of Manchester, Manchester, UK · Human-Robot Interfaces and Interaction Laboratory, Italian Institute of Technology, Genoa, Italy

Jun 25, 2026J/K move · Enter open · S save

Humanoid-DART: Humanoid Loco-Manipulation using Diffusion-guided Augmentation through Relabeling and Tracking

Pranav Debbad, Kanish Thiagarajan, Victor Dhédin, Shafeef Omar+1

Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich (TUM), Germany.

Humanoid Loco-Manipulation Robot Imitation Learning Goal-Conditioned RL

Open paper AskPDF arXiv

Imitating human demonstrations has emerged as a dominant paradigm for learning humanoid loco-manipulation policies. However, scaling these approaches remains challenging due to the high cost of collecting diverse demonstrations and the need for continual human intervention to correct policy failures. In this paper, we present a self-supervised framework that bootstraps from sparse demonstrations and progressively expands its behavioral repertoire, enabling the learning of a goal-conditioned policy that automatically explores the goal space with minimal expert supervision. Our approach combines diffusion-based trajectory generation with reinforcement learning, where the latter is used to track goal-conditioned trajectories produced by the diffusion model for a range of loco-manipulation skills. Through extensive ablation studies and comparisons with state-of-the-art methods, we demonstrate the effectiveness of our framework on multiple humanoid loco-manipulation skills.

Abstract

Explore similar work

Humanoid-DART: Humanoid Loco-Manipulation using Diffusion-guided Augmentation through Relabeling and Tracking

DreamControl-v2: Simpler and Scalable Autonomous Humanoid Skills via Trainable Guided Diffusion Priors

LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation