World2Motion: Turning Video World Models into 3D Human Motion Generators

Fangyuan Tu; Xiangyue Zhang; Yiyi Cai; Yichen Peng; Kunhang Li; Bo Zheng; Zhixiang Wang; Kaipeng Zhang

cs.CVSep 29, 2026

World2Motion: Turning Video World Models into 3D Human Motion Generators

Authors: Fangyuan Tu, Xiangyue Zhang, Yiyi Cai, Yichen Peng, Kunhang Li, Bo Zheng, Zhixiang Wang, Kaipeng Zhang, +3 more

Organizations: Japan Advanced Institute of Science and Technology · The University of Tokyo · Institute of Science Tokyo · Alaya Lab

Abstract

We present World2Motion, a framework that generates scene-aware 3D human motion and corresponding video from a single image and a text prompt. While existing 3D motion generators learn from motion datasets, their generalization is constrained by limited coverage of environments. In contrast, video world models such as Cosmos 3 offer broader environmental priors but are not designed for full-body motion generation; recovering motion from their generated videos requires costly two-stage inference. To address these, we turn Cosmos 3 into a single-stage 3D motion generator. This adaptation has two challenges: the scarcity of paired video--motion data and temporal instability in the generated motion. First, we construct a training dataset combining synthetic video--motion pairs with real videos paired with estimated 3D motion. Second, we propose a shift-decoupled noise schedule that assigns different noise levels to video and motion through shared denoising progress. This design accommodates the different denoising requirements of the two modalities, reducing motion jitter. Experiments on a multi-source interaction benchmark show that World2Motion has better motion--text alignment and scene interaction compared with the evaluated 3D motion generators. It also matches the interaction success rate of the two-stage baseline while achieving approximately 3.3 $\times$ faster inference. Our project page is available at https://fyantu.github.io/World2Motion/.

Figures & tables

Figure 1: World2Motion adapts a video world model to jointly generate scene-aware 3D human motion and corresponding video from a single image and a text prompt. The example shows a person approaching a stool, turning, and sitting down. The top row shows generated video frames, while the bottom row shows the corresponding 3D motion rendered in a scene for visualization.

Approach	Direct 3D motion output	No 3D object input	Scene conditioning
T2M	✓	✓
I2V + HMR		✓	✓
HOI	✓		✓
World2Motion	✓	✓	✓

Table 1: Motion generation methods comparison.

Figure 2: Overview of World2Motion. From an image and a text prompt, the adapted Cosmos 3 backbone jointly generates 3D human motion and video. CameraHMR estimates the initial pose from the image. The shift-decoupled schedule uses modality-specific noise shifts with shared denoising progress.

Figure 3: Shift-decoupled video–motion denoising. (a) Video and motion noise levels during inference with $s_{v}=10$ and $s_{m}=1$ . (b) The corresponding training noise pairs follow a shared curve, with lower motion noise than video noise at intermediate steps. (c) Intermediate video frames and motion states. Rows show denoising steps, and column groups show video times. The chair mesh is reference geometry for visualization.

Figure 4: Qualitative comparison of scene interactions: (a) lifting a plastic box, (b) walking to a chair and sitting, and (c) walking to a sofa and sitting. Each method shows successive motion states; video-generating methods also show corresponding video frames. Human–object interaction baselines include their predicted object trajectories.

Translation	Raw Jerk $\downarrow$ ( $10^{3}\,\mathrm{m/s^{3}}$ )	Contact FSR $\downarrow$ (0–1)
Absolute	4.355	0.766
Relative	1.251	0.457

Table 3: Root translation ablation on 35 cases without post-processing. Raw Jerk uses 25 fps (

10^{3}\,\mathrm{m/s^{3}}

). Contact FSR is the sliding fraction among contacting foot-joint/frame pairs on a normalized skeleton (0–1); its definition differs from Table 2 .

	Raw Jerk ( $10^{3}\,\mathrm{m/s^{3}}$ ) $\downarrow$			Contact FSR (0–1) $\downarrow$
Training $s_{m}$	1	10	15	1	10	15
1	1.251	8.003	9.703	0.457	0.986	1.000
10	7.021	1.635	2.557	0.985	0.635	0.830
15	8.413	2.127	1.809	0.981	0.700	0.649

Table 4: Noise-schedule ablation on the same 35 cases, using Raw Jerk and contact-conditioned FSR as defined in Table 3 . These unfiltered ablation scores are not directly comparable to Table 2 . Columns give the inference motion shift

s_{m}

;

s_{v}=10

is fixed. Bold marks the lowest value per metric within each row.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Qualitative results of World2Motion for box lifting and carrying, sitting on a stool or armchair, and two yoga movements. Each example shows five successive video frames (top) and corresponding 3D motion renderings (bottom), ordered from left to right. The input text prompt appears below each example.

Figure 6: Additional qualitative results of World2Motion for sofa sitting, picking up and carrying a vacuum cleaner, standing up from a stool, and two dance sequences. Each example shows five successive video frames (top) and corresponding 3D motion renderings (bottom), ordered from left to right. The input text prompt appears below each example.

Explore similar work

MoVT: Video-Augmented Motion Tokenizer for Text-to-Motion Generation
Sep 14, 2026Beibei Jing, Tianle Guo, Youjia Zhang +5Human Motion Generation Text-To-Motion Generation
Huazhong University of Science and Technology School of Computer Science and Technology Wuhan, Hubei, China · Zhejiang University School of Software Technology Ningbo, China
ScaleMoGen: Autoregressive Next-Scale Prediction for Human Motion Generation
May 12, 2026Inwoo Hwang, Hojun Jang, Bing Zhou +3Human Motion Generation Motion Generation
Seoul National University · Snap Inc. · Meta Reality Labs
ABot-3DWorld 0: A Universal World Model to Explore Any 3D Space
Jul 13, 2026Mingchao Sun, Luyang Tang, Yu Liu +343D World World
AMAP CV Lab Alibaba Group

Sep 14, 2026, cs.CVJ/K move · Enter open · S save

MoVT: Video-Augmented Motion Tokenizer for Text-to-Motion Generation

Beibei Jing, Tianle Guo, Youjia Zhang, Zikai Song+4

Huazhong University of Science and Technology School of Computer Science and Technology Wuhan, Hubei, China · Zhejiang University School of Software Technology Ningbo, China

Human Motion Generation Text-To-Motion Generation Motion

Open paper AskPDF arXiv

Text-driven 3D human motion generation models face significant challenges in responding to diverse and unconstrained textual prompts, primarily due to the limited availability of 3D motion training data. To address this, we introduce MoVT, a novel framework that effectively leverages the extensive range of human action videos to enhance text-to-motion generation. At the core of our approach is the cross-modal augmented motion tokenizer, which projects discrete 3D motion tokens into the 2D domain. This projection allows us to enrich the motion codebook with complex, real-world motion patterns derived from videos. The enriched discrete tokens are then mapped back to the 3D domain, resulting in aligned 3D and 2D codebooks with an enhanced capacity to represent intricate motions. These enhanced codebooks are integrated into a generative masked transformer, which predicts masked motion token indices in a modality-agnostic manner. This enables the use of text-index pairs, generated from the 2D codebook and annotated motion videos, to further enhance the generator. Extensive empirical evaluations show that MoVT performs favorably against prior state-of-the-art methods across multiple key metrics.

Abstract

Explore similar work

MoVT: Video-Augmented Motion Tokenizer for Text-to-Motion Generation

ScaleMoGen: Autoregressive Next-Scale Prediction for Human Motion Generation

ABot-3DWorld 0: A Universal World Model to Explore Any 3D Space