Humanoid Horizon: Extending Task Horizon in Whole-Body Loco-Manipulation via Parallel Training, Dynamic Starting, and Reward Gating

Haozhuo Zhang; Qiang Zhang; Jian Tang; Mingzhe Ni; Michele Caprio; Angelo Cangelosi; Wei Pan

cs.ROOct 6, 2026

Humanoid Horizon: Extending Task Horizon in Whole-Body Loco-Manipulation via Parallel Training, Dynamic Starting, and Reward Gating

Authors: Haozhuo Zhang, Qiang Zhang, Jian Tang, Mingzhe Ni, Michele Caprio, Angelo Cangelosi, Wei Pan

Organizations: The University of Manchester · University of Science and Technology of China · X-Humanoid · University of Warwick · Newcastle University

Abstract

Cluttered indoor environments, where large and heavy objects are scattered across diverse surfaces, require humanoid robots to sequentially navigate, grasp, transport, and accurately place each item at its target location within a single uninterrupted episode. This long-horizon, whole-body loco-manipulation task remains a significant challenge for current methods. Previous approaches often suffer from two main issues: easy-reward bias, where training overemphasizes early transport stages at the expense of later ones, and catastrophic forgetting, where focusing on later stages leads to a decline in earlier-stage performance. In this work, we introduce Humanoid Horizon, a unified policy framework designed to overcome these limitations through three interrelated mechanisms. The Parallel Training Strategy organizes $N$ scenes into $S$ concurrent stage streams governed by a shared policy, ensuring all transport stages receive continuous gradient updates and removing the bottleneck of sequential optimization. The Dynamic Starting Mechanism updates each environment's initial state with terminal states from upstream rollouts, gradually broadening transition coverage and enhancing robustness at stage boundaries. Reward Gating sets the reward to zero for the rest of the episode in later-stage streams when the immediately preceding object is displaced beyond a set threshold, so the shared policy learns not to disturb a just-placed object and earlier placements are preserved throughout the episode. Collectively, these strategies achieve per-stage success rates exceeding 80% on the two-object LHM-Humanoid benchmark (350 training scenes, 66 held-out scenes). As the number of sequentially transported objects grows beyond two, success declines with the horizon, but the degradation is graceful relative to the sharp drop seen in all baselines.

Figures & tables

Figure 1: Humanoid Horizon uses a single policy for long-horizon whole-body loco-manipulation across cluttered bedroom, kitchen, living room, and warehouse scenes without resets. Each row shows one uninterrupted multi-object transport episode with sequential navigation, grasping, carrying, and placement.

Figure 2: Overview of the Humanoid Horizon training pipeline. (A) Parallel Training : $S$ stage streams share one policy $\pi_{\theta}$ and are optimized jointly, each initialized from its state buffer ( $\mathcal{B}_{1}$ – $\mathcal{B}_{S}$ ). (B) Dynamic Starting : for each scene $n$ , the terminal state from stage $i$ overwrites the corresponding downstream buffer $\mathcal{B}_{i+1}^{n}$ (one state per environment) to initialize stage $i{+}1$ in the next epoch; repeated overwrites across epochs broaden transition-state coverage over time. (C) Reward Gating : for stage- $i$ streams ( $i>1$ ), reward is zeroed if object $o_{i-1}$ moves beyond tolerance $\epsilon$ , preserving previously completed placements. (D) VLA Distillation : the RL teacher is distilled via DAgger into a vision-language-action student conditioned on egocentric RGB/depth observations and language instructions.

Method	Succ1 $\uparrow$	Succ2 $\uparrow$	SuccAll $\uparrow$	Dist 1 (m) $\downarrow$	Dist 2 (m) $\downarrow$
End-to-End RL	2.3%	0.0%	0.0%	2.05	1.97
Curriculum RL	88.3%	48.5%	47.2%	0.26	1.01
Hierarchical RL	37.5%	25.0%	20.8%	0.78	1.13
HumanVLA Xu et al. (2024)	42.3%	36.7%	29.9%	0.65	0.97
InterMimic Xu et al. (2025)	20.6%	9.8%	7.5%	1.32	1.82
TokenHSI Pan et al. (2025)	40.5%	31.5%	27.6%	0.58	1.07

Table 1: Results on 350 training tasks.

Method	Succ1 $\uparrow$	Succ2 $\uparrow$	SuccAll $\uparrow$	Dist 1 (m) $\downarrow$	Dist 2 (m) $\downarrow$
End-to-End RL	1.4%	0.0%	0.0%	2.37	2.28
Curriculum RL	75.5%	40.4%	39.5%	0.36	1.11
Hierarchical RL	31.6%	21.8%	17.3%	0.97	1.39
HumanVLA Xu et al. (2024)	36.3%	31.3%	25.1%	0.72	1.16
InterMimic Xu et al. (2025)	17.2%	8.2%	5.9%	1.47	2.03
TokenHSI Pan et al. (2025)	33.9%	26.1%	22.9%	0.67	1.21

Table 2: Results on 66 unseen tasks.

Method	Succ1 $\uparrow$	Succ2 $\uparrow$	SuccAll $\uparrow$	Dist 1 (m) $\downarrow$	Dist 2 (m) $\downarrow$
End-to-End RL	1.3%	0.0%	0.0%	2.50	2.49
Curriculum RL	71.5%	41.1%	40.7%	0.37	1.18
Hierarchical RL	29.3%	20.7%	16.5%	1.04	1.45
HumanVLA Xu et al. (2024)	34.2%	29.8%	23.5%	0.70	1.26
InterMimic Xu et al. (2025)	16.3%	8.4%	6.2%	1.64	2.22
TokenHSI Pan et al. (2025)	32.0%	27.0%	23.7%	0.72	1.31

Table 3: VLA extension results (all methods distilled via the same DAgger Ross et al. (2011) pipeline).

Method	Succ1 $\uparrow$	Succ2 $\uparrow$	Succ3 $\uparrow$	Succ4 $\uparrow$	Succ5 $\uparrow$	SuccAll $\uparrow$
End-to-End RL	1.5%	0.0%	0.0%	0.0%	0.0%	0.0%
Curriculum RL	91.2%	54.2%	37.5%	25.6%	9.3%	3.2%
Hierarchical RL	38.6%	27.3%	16.6%	3.8%	0.0%	0.0%
HumanVLA Xu et al. (2024)	45.8%	40.5%	21.7%	5.6%	1.2%	0.1%
InterMimic Xu et al. (2025)	21.3%	10.2%	2.5%	0.4%	0.0%	0.0%
TokenHSI Pan et al. (2025)	41.7%	32.7%	18.9%	4.9%	1.1%	0.1%

Table 4: More than two objects sequential transport results. LHM-Humanoid † denotes the variant trained with five teacher policies strictly following the original LHM-Humanoid Zhang et al. (2026) pipeline.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
PPO epochs per iteration	4
Horizon length (rollout steps)	32
Number of mini-batches	8
Discount factor $\gamma$	0.98
GAE $\lambda$	0.95
PPO clip $\epsilon$	0.2

Appendix

Table 5: Teacher (RL) training hyperparameters.

Loss term	Weight
Actor ( $w_{a}$ )	1.0
Critic ( $w_{c}$ )	5.0
Bound mu ( $w_{b}$ )	10.0
Discriminator total ( $w_{d}$ )	5.0
Disc prediction	1.0
Disc logit regularization	0.01

Appendix

Table 6: Loss weights.

$w_{k}$	Reward term	Name	Formula
0.1	Approach velocity	$r_{\text{vel}}$	$\exp\!\bigl(-2(v^{*}-v_{\text{proj}})^{2}\bigr)$
0.1	Approach position	$r_{\text{pos}}$	$\exp\!\bigl(-0.5\cdot d_{\text{robot}\to\text{obj}}\bigr)$
0.1	Grasp proximity	$r_{\text{grasp}}$	$\frac{1}{2}\sum_{h}\exp\!\bigl(-5\cdot d_{h\to\text{obj}}\bigr)$
0.1	Lift height	$r_{\text{height}}$	$\frac{\min(z_{\text{obj}},z_{\text{target}})-z_{\text{init}}}{z_{\text{target}}-z_{\text{init}}}$
0.2	Object transport vel.	$r_{\text{obj\_vel}}$	$\exp\!\bigl(-2(v^{*}_{\text{obj}}-v_{\text{obj,proj}})^{2}\bigr)$
0.2	Object waypoint dist.	$r_{\text{obj\_wp}}$	$\exp\!\bigl(-1\cdot d_{\text{obj}\to\text{waypoint}}\bigr)$

Appendix

Table 7: Reward function components. All reward terms are bounded in

[0,1]

. The thresholds

\delta_{r}=0.5

m and

\delta_{o}=0.3

m gate the approach and grasp rewards once the robot/object is sufficiently close. When

d_{\text{obj}\to\text{goal}}<\delta_{o}

, the approach, grasp, and height rewards are clamped to 1.0 to avoid interfering with placement.

Hyperparameter	Value
Initial mixing coefficient $\beta_{0}$	0.998
$\beta$ decay schedule	Exponential
Number of step iterations per epoch	1
Number of train iterations per epoch	5
Replay buffer size	100,000
Training batch size	600

Appendix

Table 8: DAgger distillation hyperparameters.

Figure 3: Training success curves across three random seeds (seed 0, 1, 2) over 16,000 epochs. Solid lines show per-seed success rates. All three seeds converge to similar final performance (SuccAll $\approx$ 87–89%), demonstrating that Humanoid Horizon is robust to random initialization. The initial rapid increase (epochs 0–2,000) corresponds to the pretrained single-object checkpoint’s adaptation to the parallel multi-object regime, followed by steady improvement through the interplay of dynamic starting and reward gating.

Seed	success (SuccAll)	success_1 (Succ1)
Seed 0	87.7%	92.2%
Seed 1	87.3%	92.5%
Seed 2	87.8%	92.8%
Mean $\pm$ Std	87.6 $\pm$ 0.3%	92.5 $\pm$ 0.3%

Appendix

Table 9: Final training metrics across three random seeds (last 100 epochs). success refers to the SuccAll rate logged during training, and success_1 refers to the Succ1 rate.

Seen tasks (Table 1 )
	Succ1	Succ2	SuccAll
LHM-Humanoid	1.55	1.76	1.91
Humanoid Horizon	0.94	0.13	0.94
Unseen tasks (Table 2 )
LHM-Humanoid	3.27	3.78	3.78
Humanoid Horizon	0.71	1.43	1.24

Appendix

Table 10: Three-seed standard deviations of success rates, in percentage points.

Explore similar work

LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation
Aug 7, 2026Cheng Guo, Mingzhe Ni, Angelo Cangelosi +1Humanoid Loco-Manipulation Hierarchical Reinforcement Learning
Department of Computer Science, University of Manchester, Manchester, UK · Human-Robot Interfaces and Interaction Laboratory, Italian Institute of Technology, Genoa, Italy
LHM-Humanoid: Long-Horizon Human Motion Control for Continuous Object Transport in Cluttered Scenes
Aug 23, 2025Haozhuo Zhang, Jingkai Sun, Michele Caprio +5Full-Body Humanoid Control Goal-Conditioned Value Function
The University of Manchester, Manchester, UK · X-Humanoid, Beijing, China · University of Hong Kong, Hong Kong, China +1
Towards a General Humanoid Loco-Manipulation Model via Egocentric Whole-Body Human Data Pretraining
Sep 30, 2026Chongyang Xu, Zhao Wu, Jin Chen +9Humanoid Loco-Manipulation Egocentric Dataset
Alibaba Group · Sichuan University · Shanghai Innovation Institute +2

Aug 7, 2026, cs.LGJ/K move · Enter open · S save

LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation

Cheng Guo, Mingzhe Ni, Angelo Cangelosi, Arash Ajoudani

Department of Computer Science, University of Manchester, Manchester, UK · Human-Robot Interfaces and Interaction Laboratory, Italian Institute of Technology, Genoa, Italy

Humanoid Loco-Manipulation Hierarchical Reinforcement Learning Imagination Long-Horizon Task Planning

Open paper AskPDF arXiv

Long-horizon humanoid loco-manipulation requires composing versatile whole-body skills and reliable high-level decision making. Existing methods often coordinate pretrained skills with scripted planners, finite-state machines or task-specific model-free policies, restricting their ability to handle complex task sequences. To address this limitation, we propose \textbf{LUCID}, a hierarchical model-based reinforcement learning framework that plans over reusable skills through imagined rollouts of a learned dynamics model. LUCID first trains a structured latent-conditioned low-level policy via adversarial imitation and then freezes it while jointly learning a high-level policy and macro-dynamics world model. The world model predicts the temporally extended state transitions induced by latent decisions, enabling high-level policy optimization through imagined rollouts. We evaluate our framework across various simulated multi-object rearrangement scenarios. Experimental results show that LUCID improves the full-task success and partial-completion rates compared to prior baseline methods, demonstrating its effectiveness in complex sequential loco-manipulation tasks.

Abstract

Explore similar work

LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation

LHM-Humanoid: Long-Horizon Human Motion Control for Continuous Object Transport in Cluttered Scenes

Towards a General Humanoid Loco-Manipulation Model via Egocentric Whole-Body Human Data Pretraining