Continual Humanoid Motion Learning

Zhewen He; Hao Huang; Geeta Chandra Raju Bethala; Chong Yu; Tao Chen; Anthony Tzes; Yi Fang

cs.ROOct 3, 2026

Continual Humanoid Motion Learning

Authors: Zhewen He, Hao Huang, Geeta Chandra Raju Bethala, Chong Yu, Tao Chen, Anthony Tzes, Yi Fang

Organizations: New York University Abu Dhabi, Abu Dhabi, United Arab Emirates. · Fudan University, Shanghai, China.

Abstract

Humanoid whole-body controllers can now track a diverse set of dynamic motions, but they are typically trained offline and then frozen, so teaching such a controller a new skill tends to erode the skills it already mastered. We study continual learning for humanoid whole-body motion, where a single controller must acquire skills from a sequential task stream without revisiting past data. We introduce Similarity-guided LoRA-PNN, a progressive neural network (PNN) policy that prevents catastrophic forgetting by construction while reusing knowledge across skills through lightweight low-rank adaptation. A two-level motion-similarity measure, built from dynamic time warping aggregated by optimal transport, decides which prior skill to build on and how much new capacity to allocate, yielding strong forward transfer and large efficiency gains. Across six sequentially learned skill categories, our similarity-guided LoRA policy attains the best forward transfer (0.125 vs. 0.079) and the highest average accuracy among all methods, while saving up to 94.5% of trainable parameters and 40.8% of training time. The resulting controller reaches 96.13% sim-to-sim transfer and is deployed on a physical Unitree G1. Our code is available at https://anonymous.4open.science/r/continual-humanoid-learning-35D3.

Figures & tables

Figure 1: Representative whole-body motion skills (a subset of the six skill categories) learned sequentially in a continual-learning setting, where the policy must acquire new behaviors while retaining previously learned locomotion and athletic motion skills

Figure 2: Overview of the continual whole-body locomotion learning pipeline. A collection of retargeted motions Harvey et al. (2020) is organized into a sequential stream of tasks ( e.g . , jumping, dancing, and fighting) and learned using a teacher–student framework. The teacher policy, with access to privileged information, is expanded for each new task by adding a trainable column (solid blue) via similarity-guided LoRA, while previously learned columns (dashed orange) are kept frozen to preserve earlier skills. The teacher’s actions supervise the student policy through a distillation loss, and the student is deployed in the real world

Strategy	Rank Rule
Fixed	$r_{k}=r_{1}\qquad\forall\,1\leq k\leq K\kern 5.0pt.$
Binned	$\begin{aligned} &\bar{S}_{k}\coloneqq\mathrm{normalize}\big(S(p(k),k)\big)\in[0,1]\kern 5.0pt,\\ &r_{k}=\begin{cases}r_{\min},&\bar{S}_{k}\geq\gamma_{1}\kern 5.0pt,\\ r_{\text{mid}},&\gamma_{2}\leq\bar{S}_{k}<\gamma_{1}\kern 5.0pt,\\ r_{\max},&\bar{S}_{k}<\gamma_{2}\kern 5.0pt.\end{cases}\end{aligned}$
Linear	$\begin{aligned} &\bar{S}_{k}\coloneqq\mathrm{normalize}\big(S(p(k),k)\big)\in[0,1]\kern 5.0pt,\\ &r_{k}=\left\lfloor\mathrm{clip}\!\left(r_{\max}-\alpha\,\bar{S}_{k},\,r_{\min},\,r_{\max}\right)\right\rceil\kern 5.0pt.\end{aligned}$

Table 1: LoRA rank allocation strategies. The allowed ranks are

r_{\min},r_{\mathrm{mid}},r_{\max}\in\{4,8,16\}

;

\mathrm{normalize}

min–max rescales similarity to

[0,1]

over the candidate parents

\{l<k\}

;

\gamma_{1},\gamma_{2},\tau_{s},\alpha

are fixed (see text)

Figure 3: Motion-set distance matrix over the six categories (lighter = more similar)

Observation / signal	Teacher	Student	Deployable	Notes
Proprioception (common, deployable)
dof_pos	✓	✓	✓	Joint positions
dof_vel	✓	✓	✓	Joint velocities
actions	✓	✓	✓	Previous action
next_step_ref_motion	✓	✓	✓	Reference-motion conditioning (available in deployment when tracking a given motion)
history		✓	✓	Past $K{=}10$ frames of $[\texttt{dof\_\allowbreak pos},\texttt{dof\_\allowbreak vel},\texttt{actions}]$

Table 2: Observation design for teacher–student distillation. The teacher is trained with privileged observations; the student uses only deployable observations plus a history encoding

Networks	actor MLP $[512,512,256]$ , critic $[768,512,256]$ (LayerNorm), Conv1D encoder $128/64$ , SiLU
PPO	lr ${=}1\mathrm{e}{-4}$ , clip ${=}0.2$ , $\gamma{=}0.99$ , GAE ${=}0.95$ , entropy ${=}0.01$ , $24$ steps, $4$ mini-batches, $5$ epochs
Budget	$50$ K iterations, $1024$ envs, $100$ evaluation episodes per task
LoRA	$(r_{\min},r_{\mathrm{mid}},r_{\max}){=}(4,8,16)$ , $\alpha_{\text{LoRA}}{=}1.0$ , dropout ${=}0$
Rank rules	$\gamma_{1}{=}0.7$ , $\gamma_{2}{=}0.4$ , $\alpha{=}12.0$ , $\tau_{s}{=}1.0$ ( $1.5$ binned)
Sim. windows	$(L,S)\in\{(3,2),(5,3),(10,7),(15,10)\}$ s

Table 3: Key implementation hyperparameters. Full reward/termination weights and domain-randomization ranges follow standard whole-body tracking practice Cheng et al. (2024) ; Xie et al. (2025) ; Han et al. (2025)

Metric	Description	Definition
AA	Average accuracy over the tasks learned so far (overall continual performance after stage $t$ )	$\displaystyle\mathrm{AA}_{t}=\frac{1}{t}\sum_{i=1}^{t}A_{t,i}$
AIA	Average incremental accuracy: the historical average of AA up to stage $t$ (quality of the whole learning trajectory)	$\displaystyle\mathrm{AIA}_{t}=\frac{1}{t}\sum_{k=1}^{t}\mathrm{AA}_{k}$
FM	Forgetting measure: how much performance on previous tasks degrades after learning new tasks, relative to the historical best	$\displaystyle\mathrm{FM}_{t}=\frac{1}{t-1}\sum_{i=1}^{t-1}\Big(\max_{k\in\{i,\dots,t-1\}}A_{k,i}-A_{t,i}\Big)$
BWT	Backward transfer: net change on past tasks after learning subsequent tasks; negative values indicate forgetting	$\displaystyle\mathrm{BWT}_{t}=\frac{1}{t-1}\sum_{i=1}^{t-1}\big(A_{t,i}-A_{i,i}\big)$
IM	Intransigence: gap to the joint-training reference, i.e . , how far the continual learner is from the multi-task upper bound	$\displaystyle\mathrm{IM}_{t}=\frac{1}{t}\sum_{i=1}^{t}\big(A^{\mathrm{JT}}_{i}-A_{t,i}\big)$
FWT	Forward transfer: benefit (or harm) on a new task before learning it, relative to a single-task reference trained from scratch	$\displaystyle\mathrm{FWT}_{t}=\frac{1}{t-1}\sum_{i=2}^{t}\big(A_{i-1,i}-A^{\mathrm{ST}}_{i}\big)$

Table 4: Definitions of the continual-learning metrics.

A_{t,i}

denotes the score of the policy after learning tasks

1{:}t

, evaluated on task

i

;

A^{\mathrm{JT}}_{i}

and

A^{\mathrm{ST}}_{i}

are the scores of a joint-trained reference and of a single-task reference trained only on task

i

Metric	Unit	Meaning	Formula
Succ. $\uparrow$	%	Success rate over evaluation episodes (episode finishes without failure or early termination)	$\frac{1}{N}\sum_{n=1}^{N}\mathbb{I}[\text{episode$ n $succeeds}]$
$E_{\text{mpjpe}}\downarrow$	mm	Mean per-joint position error, root-relative (global translation removed by aligning the root)	$\frac{10^{3}}{TJ}\sum_{t,j}\big\lVert(\mathbf{p}_{t,j}-\mathbf{p}_{t,r})$ $\quad-(\hat{\mathbf{p}}_{t,j}-\hat{\mathbf{p}}_{t,r})\big\rVert_{2}$
$E_{\text{gmpjpe}}\downarrow$	mm	Global MPJPE: world-frame joint position error, no root alignment	$\frac{10^{3}}{TJ}\sum_{t,j}\big\lVert\mathbf{p}_{t,j}-\hat{\mathbf{p}}_{t,j}\big\rVert_{2}$
$E_{\text{mpbpe}}\downarrow$	mm	Mean per-body keypoint position error over a predefined set $\mathcal{B}$ (key bodies / end-effectors)	$\frac{10^{3}}{T\|\mathcal{B}\|}\sum_{t}\sum_{b\in\mathcal{B}}\big\lVert\mathbf{x}_{t,b}-\hat{\mathbf{x}}_{t,b}\big\rVert_{2}$
$E_{\text{mpjve}}\downarrow$	mm/s	Mean per-joint velocity error (finite difference)	$\frac{10^{3}}{(T-1)J}\sum_{t=1}^{T-1}\sum_{j}\big\lVert(\mathbf{p}_{t+1,j}-\mathbf{p}_{t,j})$ $\quad-(\hat{\mathbf{p}}_{t+1,j}-\hat{\mathbf{p}}_{t,j})\big\rVert_{2}$
$E_{\text{mpjae}}\downarrow$	mm/s 2	Mean per-joint acceleration error (second finite difference)	$\frac{10^{3}}{(T-2)J}\sum_{t=1}^{T-2}\sum_{j}\big\lVert(\mathbf{p}_{t+2,j}-2\mathbf{p}_{t+1,j}+\mathbf{p}_{t,j})$ $\quad-(\hat{\mathbf{p}}_{t+2,j}-2\hat{\mathbf{p}}_{t+1,j}+\hat{\mathbf{p}}_{t,j})\big\rVert_{2}$

Table 5: Definitions of the imitation metrics.

t\in\{1,\dots,T\}

indexes frames,

j\in\{1,\dots,J\}

joints,

r

the root (pelvis), and

b\in\mathcal{B}

body keypoints ( e.g . , torso, feet, hands). Predicted and reference positions are

\mathbf{p}_{t,j},\hat{\mathbf{p}}_{t,j}\in\mathbb{R}^{3}

(metres, converted to mm by the factor

10^{3}

)

Method	AA $\uparrow$	AIA $\uparrow$	FM $\downarrow$	BWT $\uparrow$	IM $\downarrow$	FWT $\uparrow$
Multi-motion humanoid baselines
KungfuBot2	0.920 $\pm$ 0.058	0.946 $\pm$ 0.053	0.088 $\pm$ 0.006	-0.089 $\pm$ 0.006	-0.028 $\pm$ 0.003	0.079 $\pm$ 0.005
ASAP	0.809 $\pm$ 0.044	0.917 $\pm$ 0.052	0.222 $\pm$ 0.122	-0.222 $\pm$ 0.122	-0.025 $\pm$ 0.003	0.059 $\pm$ 0.003
HumanoidVerse	0.827 $\pm$ 0.023	0.898 $\pm$ 0.026	0.204 $\pm$ 0.007	-0.203 $\pm$ 0.006	-0.028 $\pm$ 0.000	0.060 $\pm$ 0.002
Standard CL methods
EWC	0.488 $\pm$ 0.026	0.645 $\pm$ 0.032	0.220 $\pm$ 0.121	-0.220 $\pm$ 0.120	–	–

Table 6: Continual-learning metrics on success rate. We compare adapted multi-motion humanoid baselines and CL methods under a unified task-incremental protocol. Results are mean (upper) and standard deviation (lower,

\pm

) over 100 evaluation episodes. IM/FWT require joint-training and single-task references that are not applicable to EWC/PackNet (marked -- ). Best in bold

Figure 4: Radar plots of CL metrics. (Left) Success rate. (Right) Negative MPJPE score. Polygons show min–max normalized values with square-root transformation for visual separation; numbers denote original metric values

Method	Succ. $\uparrow$	$E_{\text{mpjpe}}\downarrow$	$E_{\text{gmpjpe}}\downarrow$	$E_{\text{mpbpe}}\downarrow$	$E_{\text{mpjve}}\downarrow$	$E_{\text{mpjae}}\downarrow$	$E_{\text{pbve}}\downarrow$	$E_{\text{pbae}}\downarrow$
KungfuBot2	100%	87.38	147.84	29.81	105.64	90.45	4.27	3.54
ASAP	100%	77.0	129.0	–	–	–	2.69	5.65
HumanoidVerse	97.8%	122.12	146.16	28.87	98.44	90.61	4.17	3.55
Ours	100%	80.7	135.36	30.67	91.79	84.92	4.72	4.23

Table 7: Imitation metrics (single-task training, not CL). Success rate (

\uparrow

) and tracking errors (

\downarrow

);

E_{\text{mpjpe}}/E_{\text{gmpjpe}}/E_{\text{mpbpe}}

\mathrm{m}\mathrm{m}

E_{\text{mpjve}}/E_{\text{mpjae}}

\mathrm{m}\mathrm{m}\mathrm{/}\mathrm{s}

and

\mathrm{m}\mathrm{m}\mathrm{/}\mathrm{s}^{2}

E_{\text{pbve}}/E_{\text{pbae}}

\mathrm{c}\mathrm{m}\mathrm{/}\mathrm{s}

and

\mathrm{c}\mathrm{m}\mathrm{/}\mathrm{s}^{2}

. Entries evaluated on our suite where available; -- not applicable. Best in bold

Variant	AA $\uparrow$	AIA $\uparrow$	FM $\downarrow$	Trainable Params $\downarrow$	Saved Params (%) $\uparrow$	Time (h/5k) $\downarrow$	Saved Time (%) $\uparrow$
w/o PNN (single backbone)	0.843 $\pm$ 0.024	0.892 $\pm$ 0.028	0.137 $\pm$ 0.004	2.4M	–	–	–
PNN, full FT (w/o LoRA)	0.938 $\pm$ 0.043	0.949 $\pm$ 0.044	0.000	2.4M	–	2.94	–
PNN + LoRA, w/o sim. init	0.896 $\pm$ 0.041	0.923 $\pm$ 0.044	0.000	132K	94.50	1.74	40.82
PNN + LoRA, fixed rank	0.895 $\pm$ 0.038	0.929 $\pm$ 0.041	0.000	132K	94.50	1.74	40.82
PNN + LoRA, binned rank	0.928 $\pm$ 0.025	0.946 $\pm$ 0.036	0.000	182K	92.42	2.28	22.55
PNN + LoRA, linear rank	0.945 $\pm$ 0.030	0.964 $\pm$ 0.032	0.000	218K	90.92	2.75	6.53

Table 8: Ablation and efficiency after learning all

K

tasks (CL metrics on success rate). We jointly ablate the architecture and report trainable parameters, training time per 5k iterations, and savings relative to PNN with full fine-tuning. FM/BWT are near-zero for all PNN-based variants; only the non-PNN ablation forgets. Best in bold

Figure 5: Similarity-guided LoRA rank allocation. Rank assignment under the linear rule, which smoothly increases rank as task similarity decreases, and the binned rule, which maps similarity into discrete rank levels

	Success rate		$-$ MPJPE
Rank rule	AA $\uparrow$	AIA $\uparrow$	AA $\uparrow$	AIA $\uparrow$
Fixed	0.895 $\pm$ 0.038	0.929 $\pm$ 0.041	-116.542 $\pm$ 3.793	-128.423 $\pm$ 4.841
Binned	0.928 $\pm$ 0.025	0.946 $\pm$ 0.036	-98.062 $\pm$ 2.783	-94.542 $\pm$ 3.767
Linear	0.945 $\pm$ 0.030	0.964 $\pm$ 0.032	-88.524 $\pm$ 4.173	-94.303 $\pm$ 2.510

Table 9: CL metrics under different LoRA rank-allocation rules, on success rate and on

-

MPJPE (higher is better for both). Mean (upper) and standard deviation (lower,

\pm

) over 100 evaluation episodes per task. FM and BWT are omitted because all PNN-based variants exhibit near-zero forgetting. Best in bold

Figure 6: Qualitative sim-to-sim comparison of the single distilled student on dance , fall_and_up , fight , and run : the same policy executed in Isaac Gym (left, with reference key-body markers) and MuJoCo (right)

Figure 7: Real-robot execution of the single continually trained student on the Unitree G1. Each panel shows one motion, with frames ordered left to right in time: (a) walk, (b) run, (c)–(d) two dance motions, and (e)–(f) two fight motions (a side kick and a horse-stance sequence). All six motions are executed by the same controller without any per-skill fine-tuning

Transfer Success Rate (%)	$\Delta E_{\text{mpjpe}}$ (%)	$\Delta E_{\text{mpjve}}$ (%)
$96.13$	$+6.52$	$+14.1$

Table 10: Sim-to-sim transfer (Isaac Gym

\rightarrow

MuJoCo). Transfer success rate of the deployable student, and relative change (

\Delta

, %) in MPJPE/MPJVE after transfer

Distillation strategy	Sim-to-sim SR (%)
Task-wise teachers $\rightarrow$ unified student	$94.07$
Continual teacher $\rightarrow$ continual student ( Ours )	$\mathbf{96.13}$

Table 11: Continual distillation ablation. Sim-to-sim success rate of the deployable student under two distillation strategies. Our continual teacher

\rightarrow

continual student yields a single deployable controller while retaining all skills

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value	Description
Learning rate	$1\mathrm{e}{-4}$	For both actor and critic
Clip parameter $\epsilon$	$0.2$	PPO clipping range
Discount factor $\gamma$	$0.99$	Reward discount
GAE $\lambda$	$0.95$	Generalized advantage estimation
Value-loss coefficient	$1.0$	Weight of the value-function loss
Entropy coefficient	$0.01$	Entropy regularization weight

Appendix

Table 12: PPO training hyperparameters

Component	Configuration
Motion encoder	Conv1D, hidden dimension $60$ , output dimension $128$ , SiLU
History encoder	Conv1D, hidden dimension $30$ , output dimension $64$ , SiLU
Privileged encoder	MLP $[64]$ , output dimension $64$ , SiLU
Task column (actor)	MLP $[512,\,512,\,256]$ , SiLU
Critic	MLP $[768,\,512,\,256]$ , SiLU, LayerNorm
Lateral connection	Linear adapter, zero-initialized

Appendix

Table 13: PNN network architecture

Parameter	Range
Link mass	Scaled by $[0.9,\,1.1]$
Link inertia	Scaled by $[0.9,\,1.1]$
Base COM offset	$x,y$ : $\pm$ 0.05\text{,}\mathrm{m} $; $z$ : $\pm$0.01\text{\,}\mathrm{m}$
PD gains ( $K_{p}$ , $K_{d}$ )	Scaled by $[0.9,\,1.1]$
Ground friction	Uniform in $[0.2,\,1.2]$
Torque RFI	Additive noise in $\pm 0.05$

Appendix

Table 14: Domain randomization settings

Motion-tracking rewards
Term	Weight $w$	$\boldsymbol{\sigma}$	Description
teleop_root_vel	1.0	1.0	Root velocity tracking
teleop_anchor_body_position	1.5	0.1	Anchor-body position tracking
teleop_anchor_body_rotation	1.5	0.15	Anchor-body rotation tracking
local_key_body_position	3.0	0.1	Local key-body position tracking
local_key_body_rotation	2.0	0.15	Local key-body rotation tracking

Appendix

Table 15: Reward and termination design. Tracking rewards use

r=w\exp(-\lVert e\rVert^{2}/\sigma^{2})

, where

e

is the tracking error

Explore similar work

Any2Any: Efficient Cross-Embodiment Transfer for Humanoid Whole-Body Tracking
May 22, 2026Ming Yang, Tao Yu, Feng Li +1Humanoid Robot Control Cross-Embodiment Robot Learning
LimX Dynamics
TRACE: Efficient Transfer Learning for Neural Whole-Body Control Across Humanoid Embodiments
Oct 13, 2026Qingyu Lan, Zhihao Tian, Zifan Wang +1Humanoid Robot Control Cross-Embodiment Robot Learning
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology
LHM-Humanoid: Long-Horizon Human Motion Control for Continuous Object Transport in Cluttered Scenes
Aug 23, 2025Haozhuo Zhang, Jingkai Sun, Michele Caprio +5Human Motion Generation Goal-Conditioned RL
The University of Manchester, Manchester, UK · X-Humanoid, Beijing, China · University of Hong Kong, Hong Kong, China +1

May 22, 2026J/K move · Enter open · S save

Any2Any: Efficient Cross-Embodiment Transfer for Humanoid Whole-Body Tracking

Ming Yang, Tao Yu, Feng Li, Hua Chen

LimX Dynamics

Humanoid Robot Control Cross-Embodiment Robot Learning Humanoid Motion Tracking Cross-Embodiment Transfer

Open paper AskPDF arXiv

Whole-body tracking (WBT) models have become a key foundation for humanoid robots, enabling them to imitate diverse motions with high fidelity. Training such models from scratch requires large-scale data and computation, making rapid deployment on new humanoid platforms costly. This raises a natural question: Can pretrained WBT models transfer across embodiments with minimal adaptation? To answer this question, we propose Any2Any, a paradigm that efficiently transfers an existing WBT specialist to a new humanoid embodiment with only a small amount of data and compute. Any2Any first performs kinematic alignment between source and target humanoids, aligning their input and output spaces so that the pretrained source policy can be meaningfully reused on the target embodiment.Any2Any then performs dynamics adaptation by applying lightweight parameter-efficient fine-tuning (PEFT) components to selected dynamics-sensitive modules, preserving useful behavioral priors while enabling targeted adaptation to the target robot. Extensive experiments on multiple humanoid platforms and pretrained backbones show that Any2Any substantially accelerates convergence and reduces training cost compared with training from scratch, while achieving competitive or superior tracking performance. Notably, using only 1% of the compute and data required for full training, Any2Any successfully transfers Sonic models pre-trained on Unitree G1 to LimX Oli and LimX Luna. These results suggest that pretrained WBT specialists can be efficiently reused across embodiments, providing a scalable path toward deploying humanoid whole-body control on new robots. More results and videos are available on our project page: https://any2any.top/.

Abstract

Explore similar work

Any2Any: Efficient Cross-Embodiment Transfer for Humanoid Whole-Body Tracking

TRACE: Efficient Transfer Learning for Neural Whole-Body Control Across Humanoid Embodiments

LHM-Humanoid: Long-Horizon Human Motion Control for Continuous Object Transport in Cluttered Scenes