Generalizing from References using a Multi-Task Reference and Goal-Driven RL Framework

Jiashun Wang; M. Eva Mungai; He Li; Jean Pierre Sleiman; Jessica Hodgins; Farbod Farshidian

cs.ROFeb 23, 2026

Generalizing from References using a Multi-Task Reference and Goal-Driven RL Framework

Authors: Jiashun Wang, M. Eva Mungai, He Li, Jean Pierre Sleiman, Jessica Hodgins, Farbod Farshidian

Organizations: RAI Institute · Carnegie Mellon University

Abstract

Learning agile humanoid behaviors from human motion offers a powerful route to natural, coordinated control, but existing approaches face a persistent trade-off: reference-tracking policies are often brittle outside the demonstration dataset, while purely task-driven Reinforcement Learning (RL) can achieve adaptability at the cost of motion quality. We introduce a unified multi-task RL framework that bridges this gap by treating reference motion as a prior for behavioral shaping rather than a deployment-time constraint. A single goal-conditioned policy is trained jointly on two tasks that share the same observation and action spaces, but differ in their initialization schemes, command spaces, and reward structures: (i) a reference-guided imitation task in which reference trajectories define dense imitation rewards but are not provided as policy inputs, and (ii) a goal-conditioned generalization task in which goals are sampled independently of any reference and where rewards reflect only task success. By co-optimizing these objectives within a shared formulation, the policy acquires structured, human-like motor skills from dense reference supervision while learning to adapt these skills to novel goals and initial conditions. This is achieved without adversarial objectives, explicit trajectory tracking, phase variables, or reference-dependent inference. We evaluate the method on a challenging box-based parkour playground that demands diverse athletic behaviors (e.g., jumping and climbing), and show that the learned controller transfers beyond the reference distribution while preserving motion naturalness. Finally, we demonstrate long-horizon behavior generation by composing multiple learned skills, illustrating the flexibility of the learned polices in complex scenarios.

Figures & tables

Panel of Figure 1 — Fig. 2 : Success rate of our method under different initial conditions for walk-climb and walk-jump skills. When varying one initial condition, all other conditions are held at their nominal values. Orange markers show the nominal configuration of the initial state, while the purple markers show the randomness level of the initialization during training. The gray rectangle represents the box with its edge positioned at 2.3m.

Fig. 3 : Hardware experiments with varied initial conditions for the walk–climb, walk–jump, and climb–down skills. Each behavior is depicted from two different initial conditions. Despite changes in initial conditions, the robot adapts its strategy and successfully executes the skills.

Nominal Initializations
Methods	Walk-Jump			Walk-Climb			Climb-Down
	success rate $\uparrow$	root $R$ $\downarrow$	joint pos $q$ $\downarrow$	success rate $\uparrow$	root $R$ $\downarrow$	joint pos $q$ $\downarrow$	success rate $\uparrow$	root $R$ $\downarrow$	joint pos $q$ $\downarrow$
ZEST mocap	0.98	0.45	1.60	0.92	0.83	1.60	0.90	1.02	1.75
Tabula rasa RL	1.00	-	-	1.00	-	-	0.91	-	-
Ours	1.00	0.32	2.26	1.00	0.31	1.87	1.00	0.73	3.43
Beyond Nominal Initializations

TABLE I : Quantitative comparison with alternative training paradigms. Task success rate reflects robustness, while base orientation (root

R

) and joint position (joint pos

q

) errors measure similarity to the reference motion. The latter metrics are omitted for tabula rasa RL, which does not use reference information. Arrows in the table indicate the desired direction (increase or decrease) for each metric.

Fig. 4 : Multi-skill composition in a sim-to-sim evaluation in MuJoCo. Learned policies are composed to execute walk-climb, walk-jump, and climb-down behaviors over long horizons.

Nominal Initializations
Methods	Walk-Jump			Walk-Climb			Climb-Down
	success rate $\uparrow$	root $R$ $\downarrow$	joint pos $q$ $\downarrow$	success rate $\uparrow$	root $R$ $\downarrow$	joint pos $q$ $\downarrow$	success rate $\uparrow$	root $R$ $\downarrow$	joint pos $q$ $\downarrow$
Ours w/o task curriculum	0.00	0.38	2.71	0.00	0.51	2.70	0.96	0.58	2.83
Ours w/o imitation	0.00	4.52	11.3	0.00	2.88	8.00	0.98	1.88	16.12
Ours w/o generalization	0.99	0.29	2.25	0.97	0.31	1.99	0.91	0.56	2.56
Ours	1.00	0.32	2.26	1.00	0.31	1.87	1.00	0.73	3.43

TABLE II : Quantitative comparison with ablative baselines. Task success rate reflects robustness, while base orientation (root

R

) and joint position (joint pos

q

) errors measure similarity to the reference motion. Arrows in the table indicate the desired direction (increase or decrease) for each metric.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Term Name	Definition	Noise
Policy Observations
Torso angular velocity	${}^{T}\omega_{IT}$ (IMU on torso)	$\mathcal{N}(0,\;0.10^{2})$
Projected gravity	${}^{T}g_{I}$ (gravity direction expressed in torso/IMU frame)	$\mathcal{N}(0,\;0.015^{2})$
Joint positions (relative)	$q_{\text{rel}}$	$\mathcal{N}(0,\;0.005^{2})$
Joint velocities (relative)	$\dot{q}_{\text{rel}}$	$\mathcal{N}(0,\;0.25^{2})$
Previous action	$a_{t-1}$	–

Appendix

TABLE III: Observation terms summary. Noise is zero-mean Gaussian and additive. Privileged observations are used by the critic only.

Term Name	Definition (per-env scalar)	Weight	$\bm{\sigma_{i}}$
Tracking Rewards
Base position tracking	$\exp\!\left(-\kappa\left\\|r_{IB}-r^{\ast}_{IB}\right\\|^{2}/\sigma_{1}^{2}\right)$	$1$	$0.4$
Base orientation	$\exp\!\left(-\kappa\left\\|\Phi_{IB}\ominus\Phi^{\ast}_{IB}\right\\|^{2}/\sigma_{2}^{2}\right)$	$1$	$0.5$
Base angular velocity	$\exp\!\left(-\kappa\left\\|\omega_{IB}-\omega^{\ast}_{IB}\right\\|^{2}/\sigma_{3}^{2}\right)$	$1$	$1.5$
Base linear velocity	$\exp\!\left(-\kappa\left\\|v_{IB}-v^{\ast}_{IB}\right\\|^{2}/\sigma_{4}^{2}\right)$	$1$	$0.6$
Joint position	$\exp\!\left(-\kappa\left\\|q-q^{\ast}\right\\|^{2}/\sigma_{5}^{2}\right)$	$1$	$0.3\cdot\sqrt{n_{j}}$

Appendix

TABLE IV: Reward terms and hyperparameters used in training. For exponential tracking terms, we use the form

\exp(-\kappa\|e\|^{2}/\sigma_{i}^{2})

, where

\kappa

is a global sensitivity parameter shared across all tracking terms and

\sigma_{i}

is a per-term normalization scale. We use

r_{IB}

and

\Phi_{IB}

to denote the base position and orientation in the world frame,

v_{IB}

and

\omega_{IB}

for base linear and angular velocities, and

q

for joint positions; starred quantities

(\cdot)^{\ast}

denote reference values. The operator

\ominus

denotes the

\mathrm{SO}(3)

difference implemented via the rotation-log map. For the joint position term,

n_{j}

denotes the number of actuated joints. For contact-related terms, let

F^{w}_{t,h,b}\in\mathbbm{R}^{3}

denote the net contact force in the world frame acting on body

b

at history index

h

, and let

\mathcal{B}

denote the set of foot bodies. Let

v^{w}_{\text{foot}}(b)\in\mathbbm{R}^{2}

denote the planar velocity of foot

b

. In the inequality ankle position limit penalty,

A

and

b

define a convex polytope constraint on ankle joint configurations. In the flat ankle penalty,

\tilde{g}^{(i)}_{z}

denotes the

z

-component of the gravity vector expressed in the aligned local frame of ankle

i

. For the foot clearance reward,

h_{b}

denotes the vertical height of foot

b

h^{\ast}

is the target clearance height, and

\alpha

is a scalar controlling velocity gating inside the

\tanh(\cdot)

term. Finally,

\Delta t

denotes the simulation timestep used to compute foot jerk.

Component	Specification
Assistive wrench scaling	$\beta(\lambda)=(1-\lambda)\,\beta_{\max}$
Maximum assistive scale $\beta_{\max}$	$0.75$
Virtual force PD gains $(k_{p}^{v},k_{d}^{v})$	$(0,15)$
Virtual torque PD gains $(k_{p}^{\omega},k_{d}^{\omega})$	$(200,1)$
Imitation sampling probability	$p_{\text{imi}}(\lambda)=(1-\lambda)p_{0}+\lambda p_{\text{target}}$
Initial imitation probability $p_{0}$	1.0

Appendix

TABLE V: Curriculum hyperparameters.

MDP and Simulation Setup
Hyperparameter	Value
Episode length ( $\bar{L}_{\text{episode}}$ )	$10.0$ s
Simulation time-step ( $dt$ )	$0.004$ s
Control decimation	$5$
Control frequency	$50$ Hz
Policy and Value Networks

Appendix

TABLE VI: MDP configuration, network architecture, and PPO hyperparameters.

Task	$\Delta x$ (m)	$\Delta y$ (m)	Yaw (rad)	Roll / Pitch (rad)
Walk–jump	$\pm 0.4$	$\pm 0.4$	$\pm 0.8$	$\pm 0.15$
Walk–climb	$\pm 0.4$	$\pm 0.4$	$\pm 0.8$	$\pm 0.15$
Climb–down	$\pm 0.2$	$\pm 0.2$	$\pm 0.6$	$\pm 0.15$

Appendix

TABLE VII: Reference-based initialization perturbations applied to the base pose. All offsets are sampled uniformly within the specified ranges and applied relative to the reference state.

Term	Value
Static friction	$\mathcal{U}(0.8,\;2.5)$
Dynamic friction	$\mathcal{U}(0.7,\;2.5)$
Restitution	$\mathcal{U}(0.0,\;0.2)$
Torso mass	$\text{default}+\mathcal{U}(-2.5,\;4.0)\ \text{kg}$
Pelvis mass	$\text{default}+\mathcal{U}(-1.0,\;1.0)\ \text{kg}$
External disturbance (impulsive push at base)	Interval $=\mathcal{U}(0.0\,\text{s},\;4.0\,\text{s})$ , $v_{xy}=0.4\ \text{m\,s}^{-1}$

Appendix

TABLE VIII: Domain randomization terms used during training.

Explore similar work

Oct 7, 2026

HuMBLE: Human Motion-Driven Behavior Learning for Embodied Locomotion

Despite recent advances in humanoid locomotion, controllers optimized for command tracking and robustness tend to produce mechanical gaits, whereas controllers tied to human motion data often fail to generalize to commands outside the data distribution. This work introduces a learning framework that balances these competing objectives to synthesize real-time steerable, robust, and biomimetic locomotion policies from human data. Using an in-house curated locomotion dataset covering diverse speeds and directions, we first learn a natural locomotion prior policy through a teacher-student distillation process. Specifically, we train a full-body reference-conditioned policy with Reinforcement Learning (RL), then distill it into a lightweight prior policy conditioned solely on proprioception and a planar torso-velocity steering command. Next, we fine-tune the prior policy with multi-task RL to expand command coverage and robustness beyond the data distribution, pairing a goal-conditioned task that tracks arbitrary commands with a reference-guided task that tracks the human data as an explicit style regularizer. We validate our framework on three humanoid robots: the Boston Dynamics Atlas R1, Atlas D1, and Unitree G1. Experimental results demonstrate robust performance across real-world scenarios, including direct user-controlled locomotion in indoor and outdoor environments, and integration as the locomotion layer within hierarchical control stacks. Benchmarks against Tabula Rasa RL policies trained without human data and ablation studies confirm that our framework yields a lightweight, deployable policy that reconstructs coordinated whole-body behavior from a steering command, retaining the human gait characteristics while remaining robust and fully steerable.

Mike Zhang, Dongho Kang, Kevin Bergamin +19

RAI Institute, USA · Boston Dynamics, USA · Carnegie Mellon University, USA

Humanoid Robot Locomotion RL for Legged Locomotion Robot Imitation Learning

Ask PDF

Apr 19, 2026

Learning Whole-Body Humanoid Locomotion via Motion Generation and Motion Tracking

Whole-body humanoid locomotion is challenging due to high-dimensional control, morphological instability, and the need for real-time adaptation to various terrains using onboard perception. Directly applying reinforcement learning (RL) with reward shaping to humanoid locomotion often leads to lower-body-dominated behaviors, whereas imitation-based RL can learn more coordinated whole-body skills but is typically limited to replaying reference motions without a mechanism to adapt them online from perception for terrain-aware locomotion. To address this gap, we propose a whole-body humanoid locomotion framework that combines skills learned from reference motions with terrain-aware adaptation. We first train a diffusion model on retargeted human motions for real-time prediction of terrain-aware reference motions. Concurrently, we train a whole-body reference tracker with RL using this motion data. To improve robustness under imperfectly generated references, we further fine-tune the tracker with a frozen motion generator in a closed-loop setting. The resulting system supports directional goal-reaching control with terrain-aware whole-body adaptation, and can be deployed on a Unitree G1 humanoid robot with onboard perception and computation. The hardware experiments demonstrate successful traversal over boxes, hurdles, stairs, and mixed terrain combinations. Quantitative results further show the benefits of incorporating online motion generation and fine-tuning the motion tracker for improved generalization and robustness.

Zewei Zhang, Kehan Wen, Michael Xu +7

Robotic Systems Lab, ETH Zurich · EPFL · Simon Fraser University +2

Robot Trajectory Tracking Humanoid Robot Locomotion Terrain-Aware Robot Locomotion Robot Locomotion RL for Legged Locomotion Robot Imitation Learning+2

Ask PDF

May 15, 2026

Terrain Consistent Reference-Guided RL for Humanoid Navigation Autonomy

We present a method for training reference-guided, perceptive reinforcement learning locomotion policies for humanoid robots in which reference trajectories are modulated in training to be consistent with terrain geometry. Aiming to deploy our method with standard navigation autonomy infrastructure, we synthesize SE(2)-controllable reference trajectories inside the RL training loop, projecting desired footsteps onto valid footholds and adjusting swing-foot and center-of-mass trajectories to match the terrain. The resulting policy exposes a clean SE(2) velocity interface compatible with standard navigation planners. In simulation, environmentally-conditioned references significantly improve reference tracking performance compared to environment agnostic references. On hardware, we integrate the policy with an MPC + control barrier function planner and demonstrate long-horizon (>70m) closed-loop autonomous navigation on the Unitree G1 through outdoor environments containing rough terrain and consecutive flights of stairs, with all sensing and computation onboard.

William D. Compton, Zachary Olkin, Aaron D. Ames

Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA 91125

Humanoid Robot Locomotion Reinforcement Learning Robot Navigation Terrain-Aware Robot Locomotion Robot Locomotion RL for Legged Locomotion

Ask PDF