Movens-Zero: Scaling Humanoid Motion Foundation Models with Large-Scale Human Videos
Organizations: Zhejiang University · Joint Laboratory of Embodied Intelligence, ZJU & Unitree Robotics · Bit Oasis · Unitree Robotics ·
Abstract
Scaling humanoid motion foundation models is constrained by the scarcity of diverse, robot-executable data and by the difficulty of jointly learning from heterogeneous multimodal inputs while supporting real-time closed-loop control. We present Movens-5M, constructed from raw Internet videos through clip generation, quality filtering, semantic annotation, motion recovery, and physical refinement with humanoid retargeting and execution verification. It contains 15 million human-motion sequences and 5 million physically executable humanoid-motion sequences spanning over 10,000 hours, aligned with text, RGB video, speech, and music. Building on this data, we introduce Movens-Zero, a scalable understanding-generation-execution architecture that supports joint learning across modalities and real-time closed-loop execution. Task-mixed flow-matching training jointly optimizes a single action expert across seven generation tasks, enabling direct motion generation from text, speech, music, and video. Multimodal Real-Time Chunking connects overlapping chunks under continuously updated instructions and robot states for continuous closed-loop execution. We further introduce Movens-Intent, a four-track benchmark with 8,192 expert-verified condition-motion pairs across text, speech, music, and video. Experiments show consistent gains across all four modalities when scaling execution-verified data from 1% to 100% and action-expert capacity from 64M to 911M parameters, while demonstrating that data quality and cross-modal supervision improve generation and execution.
Figures & tables
| Dataset | Open | # Sequences | Hours | Frames | Diversity | Quality checks | Aligned modalities | Source | ||||||
| Human | Humanoid | Types | Scenes | Constraints | Policies | Text | Music | Speech | RGB | |||||
| Human motion datasets | ||||||||||||||
| AMASS ( Mahmood et al., 2019 ) | ✓ | 11.3K | – | 40.0h | 4.32M | – | 1 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | MoCap agg. |
| LAFAN1 ( Harvey et al., 2020 ) | ✓ | 77 | – | 4.6h | 496.7K | 15 | 1 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | MoCap |
| KIT-ML ( Plappert et al., 2016 ) | ✓ | 3.9K | – | 11.2h | 4.03M | – | 1 | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | MoCap agg. |
| HumanML3D ( Guo et al., 2022 ) | ✓ | 14.6K | – | 28.6h | 2.06M | – | 1 | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | MoCap agg. |
| Dataset | Dataset Scale | Reference Motion Quality | Closed-loop Rollout Quality | |||||||||||||||
| Hours | Frames | # Humanoid Sequences | Temporal Quality | Foot–Support Consistency | Kinematic | Tracking Fidelity | Execution Validity | |||||||||||
| Jerk | MSNR | C-Slide | PFC | Non- Skating | Non- Floating | Non- Pen. | Joint Feas. | MPJPE | g-MPJPE | Tracking Score | Fall | J-Limit | ||||||
| MoCap-based datasets | ||||||||||||||||||
| AMASS ( Mahmood et al., 2019 ) | 40.0 | 4.32M | — | 121.55 | 14.01 | 0.345 | 0.916 | 71.17 | 72.97 | 75.82 | 75.61 | 41.59 | 695.83 | 0.326 | 4.260 | 0.292 | 7.10 | 19.40 |
| LAFAN1 ( Harvey et al., 2020 ) | 4.6 | 496.7K | — | 194.32 | 11.65 | 0.356 | 0.798 | 46.63 | 23.88 | 97.55 | 41.27 | 153.28 | 3028.86 | 0.592 | 7.713 | 0.084 | 63.64 | 98.70 |
| OmniRetarget ( Yang et al., 2025 ) | 4.0 | 432K | 2.1K | 204.61 | 13.11 | 0.221 | 0.250 | 80.69 | 66.27 | 100.00 | 100.00 | 59.23 | 443.92 | 0.322 | 5.866 | 0.197 | 13.70 | 32.30 |
| Method | Motion Generation Quality | Tracking Fidelity and Execution Failures | ||||||||||||
| FID | Matching | R@1 | R@2 | R@3 | Diversity | C-Slide | Jerk | MPJPE | g-MPJPE | Fall | J-Limit | |||
| OMG benchmark | ||||||||||||||
| GENMO ( Li et al., 2025a ) | 43.78 | 1.33 | 12.99 | 23.83 | 30.86 | 1.28 | 2.66 | 173.54 | 50.35 | 150.71 | 6.91 | 2.51 | 1.46 | 29.10 |
| HY-Motion ( Wen et al., 2025 ) | 24.68 | 1.30 | 31.84 | 48.73 | 57.71 | 1.29 | 12.20 | 112.56 | 69.05 | 287.71 | 11.94 | 3.24 | 4.98 | 55.57 |
| TextOp ( Xie et al., 2026 ) | 59.91 | 1.30 | 11.52 | 19.43 | 25.00 | 1.15 | 3.80 | 219.67 | 38.00 | 99.08 | 6.56 | 2.62 | 2.15 | 20.70 |
| Kimodo-SMPLX-RP ( Rempe et al., 2026 ) | 38.44 | 1.31 | 26.27 | 39.36 | 46.48 | 1.25 | 4.52 | 189.39 | 43.51 | 198.96 | 7.80 | 2.64 | 2.64 | 23.63 |
| Method | World-space | Camera-space | |||||||
| WA-MPJPE 100 | W-MPJPE 100 | RTE | Jitter | Foot-Sliding | PA-MPJPE | MPJPE | PVE | Accel | |
| RICH | |||||||||
| GENMO ( Li et al., 2025a ) | 75.3 | 118.6 | 1.9 | 15.0 | 6.7 | 39.1 | 66.8 | 75.4 | 4.1 |
| WHAM ( Shin et al., 2024 ) | 113.0 | 191.7 | 4.3 | 18.6 | 3.7 | 44.3 | 80.0 | 91.2 | 5.3 |
| TRAM ( Wang et al., 2024 ) | 103.5 | 168.0 | 2.5 | 18.5 | 12.5 | 44.1 | 70.5 | 78.7 | 5.4 |
| GVHMR ( Shen et al., 2024 ) | 78.8 | 126.3 | 2.4 | 12.8 | 3.0 | 39.5 | 66.0 | 74.4 | 4.1 |
| Method | Music Alignment and Target Motion Quality | Target Physical Quality and Tracker Fidelity | ||||||||||
| BeatAlign | FID-k | FID-g | PFC | C-Slide | Jerk | MPJPE | g-MPJPE | Fall | J-Limit | |||
| OMG benchmark | ||||||||||||
| GENMO ( Li et al., 2025a ) | 0.54 | 207.39 | 17.70 | 0.08 | 0.13 | 185.99 | 39.79 | 58.51 | 3.54 | 1.80 | 0.00 | 2.05 |
| LODGE ( Li et al., 2024 ) | 0.58 | 80.46 | 2.66 | 0.21 | 0.30 | 213.86 | 48.60 | 101.19 | 7.55 | 3.06 | 1.59 | 17.95 |
| Bailando ( Siyao et al., 2022 ) | 0.59 | 328.15 | 16.85 | 0.19 | 0.17 | 103.60 | 40.02 | 80.01 | 4.55 | 1.48 | 0.23 | 7.95 |
| Movens-Zero (Ours) | 0.60 | 53.44 | 2.31 | 0.10 | 0.11 | 87.25 | 37.79 | 50.39 | 3.40 | 1.75 | 0.00 | 15.32 |
| Method | Motion Generation Quality | Tracking Fidelity and Execution Failures | ||||||||||||
| FID | Matching | R@1 | R@2 | R@3 | Diversity | C-Slide | Jerk | MPJPE | g-MPJPE | Fall | J-Limit | |||
| GENMO ( Li et al., 2025a ) | 77.30 | 1.35 | 14.70 | 23.83 | 31.45 | 1.21 | 0.86 | 237.09 | 67.68 | 134.19 | 2.94 | 1.95 | 0.10 | 11.18 |
| HY-Motion ( Wen et al., 2025 ) | 30.66 | 1.25 | 36.67 | 52.49 | 62.40 | 1.33 | 1.44 | 49.35 | 45.63 | 117.10 | 3.30 | 0.94 | 1.03 | 13.62 |
| TextOp ( Xie et al., 2026 ) | 105.46 | 1.48 | 5.91 | 10.35 | 14.89 | 0.90 | 1.43 | 89.54 | 38.05 | 129.41 | 3.04 | 1.10 | 0.20 | 5.42 |
| Kimodo-SMPLX-RP ( Rempe et al., 2026 ) | 45.18 | 1.29 | 25.00 | 37.26 | 47.17 | 1.25 | 1.60 | 242.13 | 36.89 | 102.85 | 3.93 | 2.17 | 0.24 | 9.28 |
| Kimodo-G1-RP ( Rempe et al., 2026 ) | 38.84 | 1.29 | 25.00 | 37.21 | 46.24 | 1.23 | 1.76 | 215.91 | 35.50 | 89.83 | 3.28 | 1.87 | 0.39 | 8.25 |
| Movens-5M fraction | Unique hours | Text R@3 | Video MPJPE | Audio BeatAlign | Speech R@3 | Tracking Score |
| 122 | 12.72 | 132.82 | 0.23 | 13.98 | 0.21 | |
| 367 | 15.81 | 129.31 | 0.29 | 17.88 | 0.28 | |
| 1,223 | 24.92 | 110.55 | 0.38 | 21.76 | 0.36 | |
| 3,669 | 41.80 | 87.90 | 0.45 | 54.34 | 0.50 | |
| 12,231 | 62.57 | 40.90 | 0.61 | 74.30 | 0.72 |
| Scale | Action expert | Val. FM Loss | Text R@3 | Video MPJPE | Audio BeatAlign | Speech R@3 | Tracking Score |
| S | 64M | 0.487 | 10.30 | 108.50 | 0.20 | 25.37 | 0.18 |
| M | 188M | 0.392 | 22.96 | 92.80 | 0.26 | 41.25 | 0.24 |
| L | 500M | 0.186 | 47.80 | 73.84 | 0.53 | 63.98 | 0.40 |
| XL | 911M | 0.159 | 62.57 | 40.90 | 0.61 | 74.30 | 0.72 |
| Variant | Text R@3 | Video MPJPE | Audio BeatAlign | Speech R@3 | Tracking Score |
| Data curation | |||||
| Raw recovered motion | 48.72 | 126.82 | 0.48 | 35.31 | 0.41 |
| + visual and motion filtering | 54.36 | 84.57 | 0.52 | 49.43 | 0.48 |
| + closed-loop physical refinement | 57.84 | 64.91 | 0.55 | 64.35 | 0.58 |
| + tracking verification | 61.92 | 57.62 | 0.58 | 70.23 | 0.67 |
| + distribution balancing (full) | 62.57 | 40.90 | 0.61 | 74.30 | 0.72 |
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
| Gate | Signal | Retention rule |
| Human relevance | Qwen3-VL yes/no likelihood | |
| Temporal rate | RGB-window class posterior | Slow or normal playback/encoding rate |
| Articulated motion | Body-normalized limb amplitude | |
| Temporal integrity | Human/background flow statistics | No robust discontinuity test fires |
| Executability | Selected rollout tracking score | |
| Record completeness | Metadata and derived artifacts | All mandatory annotations, motions, rollout states, logs, and paths are present and readable |
| Metric | Threshold |
| Foot contact score | |
| Root jerk | |
| Position high-frequency ratio | |
| Minimum pelvis height | |
| Maximum pelvis height | |
| Pelvis distance to base of support |
| Scale | Hidden width | Depth | Heads | Head dimension | DiT parameters | Action-expert parameters |
| S | 768 | 3 | 12 | 64 | 56.1M | 64.2M |
| M | 1,152 | 4 | 18 | 64 | 174.0M | 187.9M |
| L | 1,536 | 6 | 24 | 64 | 479.2M | 500.1M |
| XL | 1,920 | 7 | 30 | 64 | 881.4M | 910.5M |
| Task | Streams | Exposure | Input |
| Text-to-Motion | 1 | 19.48% | Generate a motion sequence that matches the following description. Description: [text description]. |
| Music-to-Motion | 2 | 2.83% | [Music] |
| (Music, Text)-to-Motion | 2 | 2.83% | [Music] Generate a dance motion that follows the rhythm and energy of this music. |
| Speech-to-Motion | 1 | 9.74% | [Spoken command] |
| (Speech, Text)-to-Motion | 1 | 9.74% | [Spoken command] Parse the utterance in the audio and move according to that instruction. |
| Video-to-Motion | 3 | 27.69% | [Video] |
| Model and motion | Optimization | ||||
| Context features | Qwen2.5-Omni-3B, frozen | 2,048-D | Optimizer | AdamW | |
| Action expert | 6 blocks, 24 heads | width 1,536; head dim. 64 | Learning rate | minimum | |
| Action-expert params | 479.2M DiT | 500.1M total | Schedule | cosine | 1,000-step warmup |
| Motion/state token | 40-D | q01–q99 norm. + clip | Weight decay / | 0 | |
| Progress / time embed. | 5-D / 256-D | — | Maximum steps | seed 292285 | |
| Temporal and modality sampling | Distributed execution | ||||
| Dataset | Sampled | Sources | Ref. | Flat | Slide | Contact | Exec. |
| BONES-SEED | 1000 | 1000 | 1000 | 1000 | 988 | 988 | 1000 |
| OmniRetarget | 1000 | 1000 | 1000 | 1000 | 928 | 938 | 1000 |
| PHUMA | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 |
| OMG-Data | 1000 | 1000 | 1000 | 1000 | 297 | 325 | 1000 |
| AMASS | 1000 | 1000 | 1000 | 1000 | 997 | 999 | 1000 |
| LAFAN1 | 77 | 77 | 77 | 77 | 61 | 63 | 77 |
Human Motion Generation
Human motion generation creates articulated movement from language, audio, video, spatial constraints, or interaction goals.
See all 307 papers