cs.CVAug 24, 2026
SaveProgressively Learning Heterogeneous Skills in a Unified Latent Space
Organizations: Sun Yat-sen University, China
Abstract
We propose HetSkills, a novel framework designed to progressively learn heterogeneous skills within a unified latent space for physics-based character control. The core idea is to treat this latent space as a shared executable interface, enabling seamless integration of skills learned from diverse data sources, supervision forms, and tasks. HetSkills begins by learning a tracking skill that establishes a strong foundation in motion control and creates a shared motion decoder, which can be reused across tasks without the need for retraining or separate controllers. To prevent the text-to-motion skill from exploiting shortcut pathways instead of learning language semantics, we introduce motion intuition distillation to ground text-to-motion generation in language semantics and a task-guidance module that dynamically adjusts actions based on high-level language instructions. This enables HetSkills to preserve natural motion while continuously expanding its skill repertoire, making it highly adaptable for long-horizon tasks. Experimental results demonstrate the effectiveness in motion tracking, text-to-motion generation, motion completion, and downstream task adaptation, achieving impressive success rates even under challenging conditions.
Figures & tables
Figure 1. Overview of HetSkills. From left to right, the results illustrate four representative skill categories: motion tracking, text-to-motion generation, motion completion, and downstream task adaptation. Although these skills are learned from heterogeneous data sources, supervision signals, and task objectives, they are all represented in a unified executable latent space and decoded by a shared physics-based motion controller, enabling progressive skill accumulation, reuse, and composition.
Figure 2. HetSkills progressively integrates four representative heterogeneous skills, including motion tracking , text-to-motion , motion completion , and downstream task adaptation , within a unified latent space. Each skill maps its heterogeneous goal signals to part-wise arm and body latents via its own task encoder. After motion tracking learns the shared state encoder and motion decoder, these modules are frozen and reused by all subsequent skills. The downstream task adaptation adopts a language-guided compositional module to blend frozen motion priors with task-conditioned residual corrections, enabling lightweight adaptation without task-specific demonstrations or reference motions.
Figure 3. Architecture of the tracking skill . The tracking skill takes proprioception, the previous action, and future target poses as input, then predicts separate latent commands for the arm and body branches. A shared state encoder extracts the current physical context, while part-wise controllers decode the arm and body latents into joint-level target actions. After tracking training, the shared state encoder and part-wise controllers are frozen as the common motion decoder for all later skills, while itself is retained as one heterogeneous skill.
Figure 4. Visualization models of different HetSkills stages. The leftmost character is the physically simulated humanoid used for control, while the other four styled characters visualize the learned skill families, including a cowboy for motion tracking , a Santa for text-to-motion , a gamer for motion completion , and a courier for downstream task adaptation . These visual appearances distinguish skills in demonstrations. All skills share the same underlying physical character and latent controller.
Figure 5. Learned body and arm latent spaces. t-SNE projections show that motions from different categories, including locomotion, gesture, acrobatics, and other behaviors, form distinguishable distributions, even though no explicit category labels are used during latent learning. This indicates that the deterministic and compact latent representation captures semantically meaningful motion structure, which supports later skill reuse and composition.
Figure 6. Architecture of the text-to-motion skill . The frozen TMR text encoder provides a semantic language embedding, while the current proprioception-action input and recent history states are projected into tokens. Two part-wise Transformer encoders predict arm and body latents in the shared latent space, which are then decoded by the frozen state encoder and part-wise controllers into physical actions. This design isolates language grounding from low-level control and allows text-conditioned motion generation to reuse the tracking-learned controller.
Figure 7. Examples of human-scene interaction generated by HetSkills. The characters interact with diverse everyday objects such as chairs, sofas, and tables, producing sitting, reclining, and other object-conditioned motions within the same shared control framework.
Figure 8. Downstream task examples solved by HetSkills. The same frozen latent controller is adapted to diverse downstream tasks: (a) path follow, (b) strike with a kick, (c) strike with a push, and (d) pick-and-place. No task-specific motion demonstrations are used. Each task is learned through reward-driven lightweight adaptation that composes language-conditioned latent priors with task-conditioned residual corrections.
Figure 9. Qualitative results of motion tracking. Each row shows temporally ordered rollout frames generated by the tracking skill . The examples demonstrate that the shared decoder can reproduce diverse reference motions with stable balance, plausible contacts, and coordinated full-body control.
| Train | Test | |||
| Method | Success | MPJPE | Success | MPJPE |
| MaskedMimic | 99.4% | 32.9 | 99.2% | 35.1 |
| PULSE | 99.8% | 39.2 | 97.1% | 54.1 |
| HetSkills-1step | 99.9% | 29.2 | 99.3% | 43.6 |
| HetSkills-2step | 99.9% | 28.2 | 100% | 39.4 |
| HetSkills-1part | 99.9% | 31.8 | 100% | 42.3 |
Table 1. Motion-tracking performance on AMASS. Success rate reports the percentage of rollouts that remain below the 0.5 m MPJPE failure threshold on flat terrain. The upper block compares HetSkills with MaskedMimic and PULSE, including one-frame and two-frame inference variants of HetSkills. The lower block ablates the number of latent body partitions, showing the trade-off between tracking accuracy, part-wise interpretability, and computational efficiency.
Figure 10. HetSkills supports diverse motion completion tasks including (a) human-scene interaction, (b) motion in-betweening, and (c) VR tracking, recovering coherent full-body motion from spatially or temporally sparse observations within a single shared controller.
| Train | Test | |||
|---|---|---|---|---|
| Method | Success | MPJPE | Success | MPJPE |
| HetSkills | 99.9% | 39.9 | 97.8% | 82.3 |
| PULSE | 99.5% | 57.8 | 93.4% | 88.6 |
| MaskedMimic | 98.6% | 50.0 | 98.1% | 58.1 |
| ASE | 79.8% | 103.0 | 37.6% | 120.5 |
Table 2. VR-driven body tracking from sparse end-effector observations. Only the head and both hands are provided as goal constraints, while the remaining full-body motion must be inferred by the controller. Success rate measures whether the visible end-effectors remain within the 0.5 m tracking-error threshold, and MPJPE reports full-body reconstruction error in millimeters on the AMASS train and test splits.
Figure 11. Qualitative results of text-to-motion generation. Given only natural-language descriptions, generates diverse full-body motions from a neutral-pose initialization, including acrobatic movements, locomotion, dance, crouching, lying down, running, and hand waving. These results show that the text-to-motion skill can produce semantically aligned and physically plausible motions even when the initial state does not match the target motion.
| threshold: 0.3 m | threshold: 0.5 m | |||
| Method | First frame | Neutral | First frame | Neutral |
| MaskedMimic | 76.1% | 19.8% | 83.5% | 34.0% |
| CLoSD | 30.9% | 27.1% | 44.2% | 38.5% |
| HetSkills | 92.2% | 60.2% | 96.9% | 81.7% |
| w/o RMI | 92.6% | 48.5% | 97.7% | 70.0% |
| w/ residual | 0% | 0% | 0% | 0% |
Table 3. Text-to-motion success rates on HumanML3D under two initialization protocols and two MPJPE failure thresholds. The upper block reports the main results, where rollouts start from either the ground-truth first pose or a neutral pose. The lower block ablates MID components, highlighting the importance of Randomized Memory Initialization and showing that the residual shortcut weakens language grounding.
| Method | R@1 | R@2 | R@3 | R@5 | MedR |
|---|---|---|---|---|---|
| Ground Truth | 71% | 86% | 91% | 96% | 1.00 |
| HetSkills | 65% | 82% | 88% | 93% | 1.03 |
| CLoSD | 41% | 58% | 67% | 77% | 2.08 |
| MaskedMimic | 38% | 54% | 62% | 72% | 2.41 |
Table 4. Text-to-motion retrieval performance under the HumanML3D evaluation protocol ( 7 ) with neutral-pose initialization. Retrieval metrics are computed using the pretrained TMR model. R@N measures the fraction of correct text-motion matches ranked within the top N retrieved results, while MedR denotes the median rank of the correct match. Higher R@N and lower MedR indicate better text-motion alignment.
Figure 12. MoE gating dynamics during the pick-and-place task. The rendered sequence at the top shows rollout frames sampled every frames across the pick-up, carry-to, and put-down stages. The heatmap below shows how the Arm and Body branches assign time-varying weights to different language-conditioned priors. The changing weights indicate that the task-guidance module dynamically selects and blends different semantic priors as the task stage evolves; the full language prompts are listed in Table 5 .
Figure 13. Downstream training curves on path follow, strike push, and pick-up. Episode rewards are plotted against environment samples and averaged over three random seeds. Compared with Scratch, ASE, and PULSE, HetSkills converges stably to high task rewards across all tasks, while preserving natural motion through the frozen language-conditioned prior.
Figure 14. Long-horizon composition of heterogeneous skills. A high-level task is decomposed into a sequence of task units, including pick-up, carry-to, put-down, path follow, human-scene interaction, and text-to-motion, with each unit executed by its corresponding skill module within the same unified latent space. This demonstrates that skills learned from different data sources, supervision forms, and task objectives can be composed seamlessly without modifying the shared controller or manually designing low-level transition rules.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Branch | Instructions |
| Pick-and-Place | Arm | “a man lifts something on his left and places it down on his right” (Lift) |
| (all stages) | “the person is moving or carrying something” (Carry) | |
| Body | “a person walks to his right” (Right) | |
| “the person was walking forward then turn left” (Left) | ||
| “a man stands still” (Stand) | ||
| Strike (kick) | Arm & Body | “a person does a high kick with their right leg” |
Table 5. Language-prior instruction sets for downstream adaptation. The table lists the natural-language prompts used by to construct candidate motion priors for each downstream task and body branch. The arm and body prompt sets provide part-specific semantic priors for tasks such as pick-and place, strike, and path follow, while the MoE gating network dynamically blends these priors according to the current task observation.
Figure 16. Long-horizon skill composition results for the two-box sequential placement task. The humanoid first picks up the first box from the source table and places it on the target desk, then walks to the second source table, picks up the second box, carries it back to the desk, and places it at the target position.
Figure 17. Long-horizon skill composition results for the sit-strike-celebrate task. The humanoid walks to the chair, sits down and stands up, approaches the target object, strikes it down, and then performs a jumping celebration motion.
Figure 18. Long-horizon rollout of sequential text-conditioned motion generation. The humanoid switches across three language conditions and performs a left punch, a forward kick, and a cartwheel in sequence.