Progressively Learning Heterogeneous Skills in a Unified Latent Space

Yue-Yi Zhang; Ming Gong; Linpu He; Wei-Shi Zheng; Zhilin Zhao

cs.CVAug 24, 2026

Progressively Learning Heterogeneous Skills in a Unified Latent Space

Authors: Yue-Yi Zhang, Ming Gong, Linpu He, Wei-Shi Zheng, Zhilin Zhao

Organizations: Sun Yat-sen University, China

Abstract

We propose HetSkills, a novel framework designed to progressively learn heterogeneous skills within a unified latent space for physics-based character control. The core idea is to treat this latent space as a shared executable interface, enabling seamless integration of skills learned from diverse data sources, supervision forms, and tasks. HetSkills begins by learning a tracking skill that establishes a strong foundation in motion control and creates a shared motion decoder, which can be reused across tasks without the need for retraining or separate controllers. To prevent the text-to-motion skill from exploiting shortcut pathways instead of learning language semantics, we introduce motion intuition distillation to ground text-to-motion generation in language semantics and a task-guidance module that dynamically adjusts actions based on high-level language instructions. This enables HetSkills to preserve natural motion while continuously expanding its skill repertoire, making it highly adaptable for long-horizon tasks. Experimental results demonstrate the effectiveness in motion tracking, text-to-motion generation, motion completion, and downstream task adaptation, achieving impressive success rates even under challenging conditions.

Figures & tables

Figure 1. Overview of HetSkills. From left to right, the results illustrate four representative skill categories: motion tracking, text-to-motion generation, motion completion, and downstream task adaptation. Although these skills are learned from heterogeneous data sources, supervision signals, and task objectives, they are all represented in a unified executable latent space and decoded by a shared physics-based motion controller, enabling progressive skill accumulation, reuse, and composition.

Figure 2. HetSkills progressively integrates four representative heterogeneous skills, including motion tracking $\mathscr{F}^{trc}$ , text-to-motion $\mathscr{F}^{t2m}$ , motion completion $\mathscr{F}^{moc}$ , and downstream task adaptation $\mathscr{F}^{task^{*}}$ , within a unified latent space. Each skill maps its heterogeneous goal signals to part-wise arm and body latents via its own task encoder. After motion tracking learns the shared state encoder and motion decoder, these modules are frozen and reused by all subsequent skills. The downstream task adaptation adopts a language-guided compositional module to blend frozen motion priors with task-conditioned residual corrections, enabling lightweight adaptation without task-specific demonstrations or reference motions.

Figure 3. Architecture of the tracking skill

\mathscr{F}^{trc}

. The tracking skill takes proprioception, the previous action, and future target poses as input, then predicts separate latent commands for the arm and body branches. A shared state encoder extracts the current physical context, while part-wise controllers decode the arm and body latents into joint-level target actions. After tracking training, the shared state encoder and part-wise controllers are frozen as the common motion decoder for all later skills, while

\mathscr{F}^{trc}

itself is retained as one heterogeneous skill.

Figure 4. Visualization models of different HetSkills stages. The leftmost character is the physically simulated humanoid used for control, while the other four styled characters visualize the learned skill families, including a cowboy for motion tracking $\mathscr{F}^{trc}$ , a Santa for text-to-motion $\mathscr{F}^{t2m}$ , a gamer for motion completion $\mathscr{F}^{moc}$ , and a courier for downstream task adaptation $\mathscr{F}^{task^{*}}$ . These visual appearances distinguish skills in demonstrations. All skills share the same underlying physical character and latent controller.

Figure 5. Learned body and arm latent spaces. t-SNE projections show that motions from different categories, including locomotion, gesture, acrobatics, and other behaviors, form distinguishable distributions, even though no explicit category labels are used during latent learning. This indicates that the deterministic and compact latent representation captures semantically meaningful motion structure, which supports later skill reuse and composition.

Figure 6. Architecture of the text-to-motion skill

\mathscr{F}^{t2m}

. The frozen TMR text encoder provides a semantic language embedding, while the current proprioception-action input and recent history states are projected into tokens. Two part-wise Transformer encoders predict arm and body latents in the shared latent space, which are then decoded by the frozen state encoder and part-wise controllers into physical actions. This design isolates language grounding from low-level control and allows text-conditioned motion generation to reuse the tracking-learned controller.

Figure 7. Examples of human-scene interaction generated by HetSkills. The characters interact with diverse everyday objects such as chairs, sofas, and tables, producing sitting, reclining, and other object-conditioned motions within the same shared control framework.

Panel of Figure 8 — Figure 8. Downstream task examples solved by HetSkills. The same frozen latent controller is adapted to diverse downstream tasks: (a) path follow, (b) strike with a kick, (c) strike with a push, and (d) pick-and-place. No task-specific motion demonstrations are used. Each task is learned through reward-driven lightweight adaptation that composes language-conditioned latent priors with task-conditioned residual corrections.

Panel of Figure 9 — Figure 9. Qualitative results of motion tracking. Each row shows temporally ordered rollout frames generated by the tracking skill $\mathscr{F}^{trc}$ . The examples demonstrate that the shared decoder can reproduce diverse reference motions with stable balance, plausible contacts, and coordinated full-body control.

	Train		Test
Method	Success	MPJPE	Success	MPJPE
MaskedMimic	99.4%	32.9	99.2%	35.1
PULSE	99.8%	39.2	97.1%	54.1
HetSkills-1step	99.9%	29.2	99.3%	43.6
HetSkills-2step	99.9%	28.2	100%	39.4
HetSkills-1part	99.9%	31.8	100%	42.3

Table 1. Motion-tracking performance on AMASS. Success rate reports the percentage of rollouts that remain below the 0.5 m MPJPE failure threshold on flat terrain. The upper block compares HetSkills with MaskedMimic and PULSE, including one-frame and two-frame inference variants of HetSkills. The lower block ablates the number of latent body partitions, showing the trade-off between tracking accuracy, part-wise interpretability, and computational efficiency.

Panel of Figure 10 — Figure 10. HetSkills supports diverse motion completion tasks including (a) human-scene interaction, (b) motion in-betweening, and (c) VR tracking, recovering coherent full-body motion from spatially or temporally sparse observations within a single shared controller.

	Train		Test
Method	Success	MPJPE	Success	MPJPE
HetSkills	99.9%	39.9	97.8%	82.3
PULSE	99.5%	57.8	93.4%	88.6
MaskedMimic	98.6%	50.0	98.1%	58.1
ASE	79.8%	103.0	37.6%	120.5

Table 2. VR-driven body tracking from sparse end-effector observations. Only the head and both hands are provided as goal constraints, while the remaining full-body motion must be inferred by the controller. Success rate measures whether the visible end-effectors remain within the 0.5 m tracking-error threshold, and MPJPE reports full-body reconstruction error in millimeters on the AMASS train and test splits.

Panel of Figure 11 — Figure 11. Qualitative results of text-to-motion generation. Given only natural-language descriptions, $\mathscr{F}^{\mathrm{t2m}}$ generates diverse full-body motions from a neutral-pose initialization, including acrobatic movements, locomotion, dance, crouching, lying down, running, and hand waving. These results show that the text-to-motion skill can produce semantically aligned and physically plausible motions even when the initial state does not match the target motion.

	threshold: 0.3 m		threshold: 0.5 m
Method	First frame	Neutral	First frame	Neutral
MaskedMimic	76.1%	19.8%	83.5%	34.0%
CLoSD	30.9%	27.1%	44.2%	38.5%
HetSkills	92.2%	60.2%	96.9%	81.7%
w/o RMI	92.6%	48.5%	97.7%	70.0%
w/ residual	0%	0%	0%	0%

Table 3. Text-to-motion success rates on HumanML3D under two initialization protocols and two MPJPE failure thresholds. The upper block reports the main results, where rollouts start from either the ground-truth first pose or a neutral pose. The lower block ablates MID components, highlighting the importance of Randomized Memory Initialization and showing that the residual shortcut weakens language grounding.

Method	R@1 $\uparrow$	R@2 $\uparrow$	R@3 $\uparrow$	R@5 $\uparrow$	MedR $\downarrow$
Ground Truth	71%	86%	91%	96%	1.00
HetSkills	65%	82%	88%	93%	1.03
CLoSD	41%	58%	67%	77%	2.08
MaskedMimic	38%	54%	62%	72%	2.41

Table 4. Text-to-motion retrieval performance under the HumanML3D evaluation protocol ( 7 ) with neutral-pose initialization. Retrieval metrics are computed using the pretrained TMR model. R@N measures the fraction of correct text-motion matches ranked within the top N retrieved results, while MedR denotes the median rank of the correct match. Higher R@N and lower MedR indicate better text-motion alignment.

Figure 12. MoE gating dynamics during the pick-and-place task. The rendered sequence at the top shows rollout frames sampled every $20$ frames across the pick-up, carry-to, and put-down stages. The heatmap below shows how the Arm and Body branches assign time-varying weights to different language-conditioned priors. The changing weights indicate that the task-guidance module dynamically selects and blends different semantic priors as the task stage evolves; the full language prompts are listed in Table 5 .

Figure 13. Downstream training curves on path follow, strike push, and pick-up. Episode rewards are plotted against environment samples and averaged over three random seeds. Compared with Scratch, ASE, and PULSE, HetSkills converges stably to high task rewards across all tasks, while preserving natural motion through the frozen language-conditioned prior.

Figure 14. Long-horizon composition of heterogeneous skills. A high-level task is decomposed into a sequence of task units, including pick-up, carry-to, put-down, path follow, human-scene interaction, and text-to-motion, with each unit executed by its corresponding skill module within the same unified latent space. This demonstrates that skills learned from different data sources, supervision forms, and task objectives can be composed seamlessly without modifying the shared controller or manually designing low-level transition rules.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Branch	Instructions
Pick-and-Place	Arm	“a man lifts something on his left and places it down on his right” (Lift)
(all stages)		“the person is moving or carrying something” (Carry)
	Body	“a person walks to his right” (Right)
		“the person was walking forward then turn left” (Left)
		“a man stands still” (Stand)
Strike (kick)	Arm & Body	“a person does a high kick with their right leg”

Appendix

Table 5. Language-prior instruction sets for downstream adaptation. The table lists the natural-language prompts used by

\mathscr{F}^{t2m}

to construct candidate motion priors for each downstream task and body branch. The arm and body prompt sets provide part-specific semantic priors for tasks such as pick-and place, strike, and path follow, while the MoE gating network dynamically blends these priors according to the current task observation.

Figure 16. Long-horizon skill composition results for the two-box sequential placement task. The humanoid first picks up the first box from the source table and places it on the target desk, then walks to the second source table, picks up the second box, carries it back to the desk, and places it at the target position.

Figure 17. Long-horizon skill composition results for the sit-strike-celebrate task. The humanoid walks to the chair, sits down and stands up, approaches the target object, strikes it down, and then performs a jumping celebration motion.

Figure 18. Long-horizon rollout of sequential text-conditioned motion generation. The humanoid switches across three language conditions and performs a left punch, a forward kick, and a cartwheel in sequence.

Explore similar work

Sep 1, 2026

ADAPT: Agile Diffusion Action Priors for Robust and Steerable Online Text-Driven Humanoid Control

We present ADAPT, an end-to-end framework for interactive, text-conditioned humanoid whole-body control. Unlike dominant text-to-motion pipelines that generate kinematic motions for a separate tracker, ADAPT solves language control with an end-to-end closed-loop control framework, where the robot must continuously respond to changing commands while maintaining balance, natural motion, and smooth transitions. ADAPT learns a diffusion-based action prior from text-labeled humanoid state-action trajectories, enabling diverse motion skills to be directly executed from language commands. To improve long-horizon robustness and smooth prompt switching, we train a lightweight residual reinforcement learning policy on top of the frozen diffusion controller. We further show that the same diffusion policy can be reused as a steerable text-conditioned motion prior for downstream task adaptation. Experiments demonstrate robust language-grounded skill execution, smooth interactive transitions, and style-preserving downstream control.

Yan Wu, Chenhao Li, Kaifeng Zhao +3

ETH Zurich, Switzerland

Humanoid Robot Control Diffusion Policy Language-Conditioned Robot Control Robot Learning

Ask PDF

Sep 29, 2026

Learning Expressive and Compositional Motion Representation via Spectral Skills

Robotic foundation models offer a promising path toward general-purpose humanoid robot control, often through hierarchical architectures. However, their effectiveness depends on the command interface between the planner and the controller, which must support accurate execution while remaining easy to predict, and ideally allow new behaviors to be composed from prior ones. In this work, we introduce spectral skills, a latent representation of this interface that meets these requirements through predictive representation learning. By design, spectral skills compactly encode short motion segments and are learned by predicting subsequent motion rather than reconstructing the encoder input. On a 29-DoF humanoid, a controller conditioned on spectral skills reduces global tracking error by 62% relative to the state of the art. The same frozen controller chains independently encoded skills without a separate transition policy. It also composes new behaviors by adding orthogonal directions to any compatible base skill, producing combinations unseen in the training data. We demonstrate tracking, chaining, and composition, as well as control through a language-conditioned planner, on Unitree G1 hardware. Project page: https://spectral-skill.github.io

Feiyang Wu, Chenxiao Gao, Chen Yang +3

Georgia Institute of Technology

Robot Skill Learning Humanoid Robot Control Predictive Representation Learning Motion Representation Learning Compositional Representation Learning

Ask PDF

Sep 11, 2026

CHOREO: Every Humanoid Skill as a Trajectory

Recent advances in humanoid robotics have produced diverse skills through reinforcement learning, motion imitation, and generative modeling. Yet these capabilities remain siloed because they are built around incompatible representations, interfaces, and controllers. We present CHOREO, a framework for training-free composition of heterogeneous humanoid skills. Our key observation is that, regardless of how a skill is learned, it can ultimately be expressed as an executable motion trajectory. Based on this observation, CHOREO converts each capability into SkillMotion, a unified representation that combines motion states, contacts, semantics, and boundary conditions. Skills are composed through direct continuation, cubic Hermite blending, or validated bridge motions, without retraining source models or updating models at test time. On Unitree G1 in MuJoCo, CHOREO organizes 2,950 admitted SkillMotion assets derived from heterogeneous sources and achieves 95.4% sequence success across 130 multi-action tasks, including 93.8% success on eight-action sequences. These results demonstrate that executable trajectories provide a scalable interface for accumulating and composing pretrained humanoid capabilities.

Ziyi Sun, Jingwen Chen, Yuxi Wang +4

College of Engineering, Ocean University of China · Faculty of Information Science and Engineering, Ocean University of China · School of Artificial Intelligence, University of Chinese Academy of Sciences +1

Humanoid Robotics Humanoid Robot Control Robot Motion Generation Robot Skill Learning

Ask PDF