Image Diffusion Models Are Good 3D Human Motion Generators

Siyuan Yang; Chong Wang; Hao Cheng; Jun Liu; Bihan Wen; Alex Kot; Hedvig Kjellstrom

cs.CVOct 9, 2026

Image Diffusion Models Are Good 3D Human Motion Generators

Authors: Siyuan Yang, Chong Wang, Hao Cheng, Jun Liu, Bihan Wen, Alex Kot, Hedvig Kjellstrom

Organizations: Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Artificial Intelligence, Hebei University of Technology, China · School of Computing and Communications, Lancaster University, United Kingdom · Faculty of Engineering, Shenzhen MSU-BIT University, China · College of Engineering and Computer Science, VinUniversity, Vietnam

Abstract

Text-to-motion generation typically relies on dataset-specific skeletal representations, with models trained separately for benchmarks with different joint layouts. We investigate whether image diffusion architectures can model 3D human motion through a common image-compatible representation that accommodates heterogeneous skeletal layouts while preserving motion-specific kinematic structure. To this end, we organize root-centered joint trajectories into topology-informed joint--time images and model global root dynamics through a complementary latent stream. A motion autoencoder conditions local-motion decoding on the root latent, while a text-conditioned Scalable Interpolant Transformer (SiT) jointly generates both latent streams, with supervision on decoded 3D joint positions further constraining generation in motion space. In addition to dataset-specific models, we train a single motion autoencoder on HumanML3D and KIT-ML, followed by a single text-conditioned SiT shared across both datasets, preserving their respective skeletal definitions without skeleton retargeting to a common joint layout. Experiments on HumanML3D and KIT-ML show that our dataset-specific models achieve state-of-the-art performance, while shared training yields further gains on both benchmarks, demonstrating the benefits of shared modeling across heterogeneous skeletal layouts.

Figures & tables

Figure 1: Dataset-specific versus shared motion modeling. (a) Separate models for HumanML3D and KIT-ML. (b) Our method uses a single model across both datasets despite their different skeletal definitions. The motion-to-image mapping is detailed in Figure 2 .

Figure 2: Topology-informed skeleton image construction. Local joint coordinates are reordered according to body-part grouping and kinematic-chain topology, arranged into a joint–time array with $x$ , $y$ , and $z$ coordinate channels, and resized to $256\times 256$ .

Figure 3: Overview of the proposed framework. (a) Separate encoders map skeleton images and root motion into two latent spaces. Root-conditioned decoding and motion recovery reconstruct the input motion. (b) With the autoencoder frozen, noisy image and root latents are tokenized and jointly processed by a text-conditioned SiT to predict their flow-matching velocities.

Motion Discrete / Token-based
Method	R-Precision $\uparrow$			FID $\downarrow$	Matching $\downarrow$	CLIP-score $\uparrow$
Method	Top-1	Top-2	Top-3	FID $\downarrow$	Matching $\downarrow$	CLIP-score $\uparrow$
T2M-GPT ( Zhang et al., 2023a )	$0.470^{\pm.003}$	$0.659^{\pm.002}$	$0.758^{\pm.002}$	$0.335^{\pm.003}$	$3.505^{\pm.017}$	$0.607^{\pm.005}$
MMM ( Pinyoanuntapong et al., 2024b )	$0.487^{\pm.003}$	$0.683^{\pm.002}$	$0.782^{\pm.001}$	$0.132^{\pm.004}$	$3.359^{\pm.009}$	$0.635^{\pm.003}$
MoMask ( Guo et al., 2024 )	$0.490^{\pm.004}$	$0.687^{\pm.003}$	$0.786^{\pm.003}$	$0.116^{\pm.006}$	$3.353^{\pm.010}$	$0.637^{\pm.003}$
BAMM ( Pinyoanuntapong et al., 2024a )	$0.480^{\pm.003}$	$0.671^{\pm.003}$	$0.772^{\pm.003}$	$0.147^{\pm.005}$	–	$0.630^{\pm.001}$

Table 1: Quantitative comparison on HumanML3D using the MARDM evaluation protocol. Bold and underlined values indicate the best and second-best results, respectively. Shared models are jointly trained on HumanML3D and KIT-ML. Confidence intervals are included where reported.

Motion Discrete / Token-based
Method	R-Precision $\uparrow$			FID $\downarrow$	Matching $\downarrow$	CLIP-score $\uparrow$
Method	Top-1	Top-2	Top-3	FID $\downarrow$	Matching $\downarrow$	CLIP-score $\uparrow$
T2M-GPT ( Zhang et al., 2023a )	$0.359^{\pm.007}$	$0.553^{\pm.007}$	$0.690^{\pm.013}$	$0.593^{\pm.053}$	$3.765^{\pm.046}$	$0.651^{\pm.005}$
MMM ( Pinyoanuntapong et al., 2024b )	$0.363^{\pm.005}$	$0.569^{\pm.006}$	$0.724^{\pm.006}$	$0.478^{\pm.034}$	$3.629^{\pm.028}$	$0.660^{\pm.003}$
MoMask ( Guo et al., 2024 )	$0.369^{\pm.005}$	$0.588^{\pm.005}$	$0.731^{\pm.005}$	$0.411^{\pm.026}$	$3.577^{\pm.021}$	$0.669^{\pm.002}$
Motion Diffusion / Flow-based

Table 2: Quantitative comparison on KIT-ML using the MARDM evaluation protocol.

Figure 4: Qualitative comparisons on HumanML3D. We compare our method with MoMask and MARDM-SiT. (a) Motion snapshots with root trajectories, where lighter-to-darker meshes indicate temporal progression. (b)–(c) Motion sequences shown chronologically from left to right.

Model scaling
Setting	FID $\downarrow$	R-Precision $\uparrow$
Setting	FID $\downarrow$	Top-1	Top-2	Top-3
SiT-S	$0.139^{\pm.004}$	$0.527^{\pm.004}$	$0.713^{\pm.002}$	$0.804^{\pm.002}$
SiT-B	$0.106^{\pm.006}$	$0.533^{\pm.003}$	$0.717^{\pm.002}$	$0.810^{\pm.002}$
SiT-L	$\mathbf{0.089^{\pm.005}}$	$\mathbf{0.540^{\pm.004}}$	$\mathbf{0.727^{\pm.002}}$	$\mathbf{0.818^{\pm.002}}$
ImageNet initialization

Table 3: Model ablations on HumanML3D.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Body part	HumanML3D	KIT-ML
Left leg	Foot (10), ankle (7), knee (4), hip (1)	Toe (20), mid-foot (19), ankle (18), knee (17), hip (16)
Left arm	Wrist (20), elbow (18), shoulder (16), collar (13)	Wrist (10), elbow (9), shoulder (8)
Torso and head	Spine1 (3), spine2 (6), spine3 (9), neck (12), head (15)	Pelvis spine (1), thorax (2), lower neck (3), upper neck (4)
Right arm	Collar (14), shoulder (17), elbow (19), wrist (21)	Shoulder (5), elbow (6), wrist (7)
Right leg	Hip (2), knee (5), ankle (8), foot (11)	Hip (11), knee (12), ankle (13), mid-foot (14), toe (15)

Appendix

Table 4: Body-part grouping and joint ordering for skeleton image construction. Rows follow the group order along the joint axis; joints within each cell are listed in traversal order. Parentheses indicate zero-based native joint indices, before root removal.

Model	Depth	Hidden dimension	Heads	Parameters
SiT-S	12	384	6	33.4M
SiT-B	12	768	12	132.3M
SiT-L	24	1024	16	461.3M

Appendix

Table 5: SiT configurations and trainable parameter counts.

Setting	HumanML3D	KIT-ML	Shared
Training window (frames)	128	64	128 (HumanML3D)
Training window (frames)	128	64	64 (KIT-ML)
Effective batch size	256	512	256
Epochs	100	50	100
Optimizer	AdamW
Learning rate	$10^{-3}$

Appendix

Table 6: Motion autoencoder training settings.

Setting	HumanML3D	KIT-ML	Shared
Maximum motion length (frames)	196
Effective batch size	256	16	256
Epochs	2000
Optimizer	AdamW
Adam betas $(\beta_{1},\beta_{2})$	$(0.9,0.999)$
Learning rate	$10^{-4}$

Appendix

Table 7: SiT training settings.

Table 12

Explore similar work

ScaleMoGen: Autoregressive Next-Scale Prediction for Human Motion Generation
May 12, 2026Inwoo Hwang, Hojun Jang, Bing Zhou +3Text-to-Motion Generation Human Motion Generation
Seoul National University · Snap Inc. · Meta Reality Labs
DC-Motion: Decoupling Structure and Details via Discrete-Continuous Tokens for Human Motion Generation
May 28, 2026Hequan Wang, Xuean Chen, Jiaxu Zhang +2Text-to-Motion Generation Human Motion Generation
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430072, China · Geophysical Exploration Brigade of Hubei Provincial Geological Bureau, Wuhan 430058, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, 487372, Singapore
Kimodo: Scaling Controllable Human Motion Generation
Mar 16, 2026Davis Rempe, Mathis Petrovich, Ye Yuan +21Human Motion Generation Diffusion Models for Motion Generation
NVIDIA

May 12, 2026J/K move · Enter open · S save

ScaleMoGen: Autoregressive Next-Scale Prediction for Human Motion Generation

Inwoo Hwang, Hojun Jang, Bing Zhou, Jian Wang+2

Seoul National University · Snap Inc. · Meta Reality Labs

Text-to-Motion Generation Human Motion Generation Motion Representation Learning

Open paper AskPDF arXiv

We present ScaleMoGen, a scale-wise autoregressive framework for text-driven human motion generation. Unlike conventional autoregressive approaches that rely on standard next-token prediction, ScaleMoGen frames motion generation as a coarse-to-fine process. We quantize 3D motions into compositional discrete tokens across multiple skeletal-emporal scales of increasing granularity, learning to generate motion by autoregressively predicting next-scale token maps. To maintain structural integrity, our motion tokenizers and quantizers are explicitly designed so that discrete tokens at every scale strictly preserve the skeletal hierarchy. Additionally, we employ bitwise quantization and prediction, which efficiently scale up the tokenizer vocabulary to preserve motion details and stabilize optimization. Extensive experiments demonstrate that ScaleMoGen achieves state-of-the-art performance, establishing an FID of 0.030 (vs. 0.045 for MoMask) on HumanML3D and a CLIP Score of 0.693 (vs. 0.685 for MoMask++) on the SnapMoGen dataset. Furthermore, we demonstrate that our skeletal-temporal multi-scale representation naturally facilitates training-free, text-guided motion editing.

Abstract

Explore similar work

ScaleMoGen: Autoregressive Next-Scale Prediction for Human Motion Generation

DC-Motion: Decoupling Structure and Details via Discrete-Continuous Tokens for Human Motion Generation

Kimodo: Scaling Controllable Human Motion Generation