Synthesizing text-driven 3D human motion within realistic scenes requires learning both semantic intent ("walk to the couch") and physical feasibility (e.g., avoiding collisions). Current methods use generative frameworks that simultaneously learn high-level planning and low-level contact reasoning, and rely on computationally expensive 3D scene data such as point clouds or voxel occupancy grids. We propose SceMoS, a scene-aware motion synthesis framework that shows that structured 2D scene representations can serve as a powerful alternative to full 3D supervision in physically grounded motion synthesis. SceMoS disentangles global planning from local execution using lightweight 2D cues and relying on (1) a text-conditioned autoregressive global motion planner that operates on a bird's-eye-view (BEV) image rendered from an elevated corner of the scene, encoded with DINOv2 features, as the scene representation, and (2) a geometry-grounded motion tokenizer trained via a conditional VQ-VAE, that uses 2D local scene heightmap, thus embedding surface physics directly into a discrete vocabulary. This 2D factorization reaches an efficiency-fidelity trade-off: BEV semantics capture spatial layout and affordance for global reasoning, while local heightmaps enforce fine-grained physical adherence without full 3D volumetric reasoning. SceMoS achieves state-of-the-art motion realism and contact accuracy on the TRUMANS benchmark, reducing the number of trainable parameters for scene encoding by over 50%, showing that 2D scene cues can effectively ground 3D human-scene interaction.
Figures & tables
Figure 1: The introduced scene-aware 3D human motion synthesis framework, SceMoS , uses 2D scene cues and text instructions to generate physically consistent and realistic 3D motions. We use a bird’s-eye-view (BEV) image rendered from an elevated corner of the input scene, and extract DINOv2 features for high-level semantic planning. For fine-grained contact reasoning, we use the local 2D heightmap of the scene around the root of the person’s initial pose.
Figure 2: Overview of the SceMoS framework. SceMoS disentangles text-conditioned scene-aware human motion synthesis into two stages. (a) The global motion planner predicts discrete motion tokens from text input and DINOv2 scene features extracted from a BEV image. (b) Our geometry-grounded motion tokenizer learns a scene-aware motion vocabulary for mapping these discrete tokens to a continuous 3D human motion. We use 2D local heightmaps around poses to condition our interaction decoder (top right) for fine-grained interaction generation. The red dotted line implies used only during training. Blue arrows follow through the inference pipeline.
Figure 3: Visualization of long-range motion synthesis in a cluttered indoor environment. SceMoS performs geometry-grounded planning by recalculating heightmaps every t frames, enabling globally coherent yet locally feasible motion planning that respects scene geometry. The BEV image input is shown in the inset.
Method
Scene Representation
Scene Encoder
# Trainable Scene Params
FID ↓
Div →
PFC ↓
Cont. ↑
Penetration ↓
mean
max
Ground truth
-
-
-
-
2.75
0.24
-
-
-
TRUMANS [ 23 ]
Voxelized 3D occupancy grid
ViT-B Encoder [ 7 ]
∼86M
0.34
2.71
0.58
0.98
1.83
11.75
TeSMo [ 61 ]
2D floormap + 3D point clouds
ResNet + transformer
∼35M
0.48
2.02
0.65
0.92
2.11
11.98
Humanise [ 59 ]
3D point clouds
Point Transformer [ 66 ]
∼55M
0.82
1.95
1.21
0.96
1.95
14.37
Scene Diffuser [ 20 ]
3D point clouds
Point Transformer [ 66 ]
∼55M
0.75
2.03
1.09
0.91
1.71
17.49
Table 1: Quantitative Evaluation of Motion Generation. Comparison in terms of scene representation, number of parameters needed for scene encoding, final motion generation quality between baselines, ablated versions, and our full method on the TRUMANS dataset. Bold indicates best. Underline indicates second best.
Variant
MPJPE ↓ (mm)
MPJVE ↓ (mm)
Cont. ↑
Penetration ↓
mean
max
A1: Scene-agnostic VQVAE
25.89
15.98
0.86
4.43
18.91
A2a: Heightmap grid (16×16)
21.48
11.02
0.98
2.12
15.01
A2b: Heightmap grid (64×64)
22.56
10.99
0.97
1.88
14.21
A3: Voxel occupancy grid
21.32
12.39
0.92
3.88
16.95
A4a: FiLM-style modulation
27.89
15.66
0.91
2.12
14.89
Table 2: Quantitative Evaluation of VQ Reconstruction . Comparison of motion reconstruction quality after tokenization across ablated versions on the TRUMANS dataset. Bold indicates the best performance.
Figure 4: Qualitative Comparison of SceMoS with recent HSI models. SceMoS generates motions that are semantically aligned with the input text instructions while maintaining stable contact and smooth transitions. In contrast, we observe some penetrations and misalignment (red circle) in some frames of the baselines.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
GT
SceMoS
Trumans
TeSMo
Humanise
Realism
4.36±0.1
3.41±0.2
3.38±0.1
2.19±0.4
1.83±0.3
Semantics
4.63±0.2
4.20±0.6
3.65±0.7
3.79±1.3
3.41±1.4
Appendix
Table A.1 : User study evaluation across all methods. We report the mean ± standard deviation of participant ratings for Realism and Semantics in a 5-point Likert scale.
Figure A.1 : Interface of our user study where we ask participants to rank the motion clips based on ‘realism’ and ‘semantics’, in a 5-point Likert scale.
Figure A.2 : User rating distribution across score categories. SceMoS receives the fewest “Poor / Bad” ratings and over 60% of its ratings fall into the “Good / Excellent” categories.
Method
TRUMANS
HUMANISE
Goal Acc. ↑
R-Prec. ↑
FID ↓
Div. →
Cont. ↑
SceMoS
0.62
0.58
0.95
1.33
0.74
Trumans
0.59
0.57
1.02
1.65
0.74
TeSMo
0.49
0.34
1.17
1.21
0.72
Humanise
0.28
0.31
0.99
1.22
0.72
Appendix
Table A.2 : Extended quantitative evaluation. Left: Task-completion metrics on TRUMANS dataset (Goal Accuracy and R-Precision). Right: Generalization performance on the HUMANISE dataset.
We present World2Motion, a framework that generates scene-aware 3D human motion and corresponding video from a single image and a text prompt. While existing 3D motion generators learn from motion datasets, their generalization is constrained by limited coverage of environments. In contrast, video world models such as Cosmos 3 offer broader environmental priors but are not designed for full-body motion generation; recovering motion from their generated videos requires costly two-stage inference. To address these, we turn Cosmos 3 into a single-stage 3D motion generator. This adaptation has two challenges: the scarcity of paired video--motion data and temporal instability in the generated motion. First, we construct a training dataset combining synthetic video--motion pairs with real videos paired with estimated 3D motion. Second, we propose a shift-decoupled noise schedule that assigns different noise levels to video and motion through shared denoising progress. This design accommodates the different denoising requirements of the two modalities, reducing motion jitter. Experiments on a multi-source interaction benchmark show that World2Motion has better motion--text alignment and scene interaction compared with the evaluated 3D motion generators. It also matches the interaction success rate of the two-stage baseline while achieving approximately 3.3× faster inference. Our project page is available at https://fyantu.github.io/World2Motion/.
Fangyuan Tu, Xiangyue Zhang, Yiyi Cai +8
Japan Advanced Institute of Science and Technology · The University of Tokyo · Institute of Science Tokyo +1
This paper tackles the problem of physics-aware human motion synthesis in a dynamic scene. Unlike existing works which mainly tend to generate physically unrealistic motions due to limited contact modeling, typically restricted to hands, in this paper, we introduce a physics-aware human motion generation framework that explicitly models the full spectrum of human-related forces, including human-object, human-scene, and internal body dynamics.~Our method imposes soft physical constraints to maintain force and torque balance, ensuring physically grounded motion synthesis. We further propose a novel continuous distance-based force model that generalizes contact modeling to arbitrary surfaces, capturing interactions not only with static environments but also with dynamic, moving objects. Extensive experiments show that our approach significantly improves physical plausibility and generalizes well to complex scenes, setting a new benchmark for physically consistent human motion generation.
Chaoyue Xing, Wei Mao, Miaomiao Liu
Australian National University, Canberra, Australia
We present ScaleMoGen, a scale-wise autoregressive framework for text-driven human motion generation. Unlike conventional autoregressive approaches that rely on standard next-token prediction, ScaleMoGen frames motion generation as a coarse-to-fine process. We quantize 3D motions into compositional discrete tokens across multiple skeletal-emporal scales of increasing granularity, learning to generate motion by autoregressively predicting next-scale token maps. To maintain structural integrity, our motion tokenizers and quantizers are explicitly designed so that discrete tokens at every scale strictly preserve the skeletal hierarchy. Additionally, we employ bitwise quantization and prediction, which efficiently scale up the tokenizer vocabulary to preserve motion details and stabilize optimization. Extensive experiments demonstrate that ScaleMoGen achieves state-of-the-art performance, establishing an FID of 0.030 (vs. 0.045 for MoMask) on HumanML3D and a CLIP Score of 0.693 (vs. 0.685 for MoMask++) on the SnapMoGen dataset. Furthermore, we demonstrate that our skeletal-temporal multi-scale representation naturally facilitates training-free, text-guided motion editing.
Inwoo Hwang, Hojun Jang, Bing Zhou +3
Seoul National University · Snap Inc. · Meta Reality Labs