Spatiotemporally Decoupled Autoregressive Diffusion Model for Human Motion Generation

Chengqun Yang; Liang Xu; Yanping Li; Fulong Liu; Jingnan Gao; Weili Zeng; Yichao Yan

cs.CVAug 24, 2026

Spatiotemporally Decoupled Autoregressive Diffusion Model for Human Motion Generation

Authors: Chengqun Yang, Liang Xu, Yanping Li, Fulong Liu, Jingnan Gao, Weili Zeng, Yichao Yan

Organizations: MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University

Abstract

Text-driven human motion synthesis has made substantial development with two core modules of motion representation and generative architecture. For representation, Vector Quantization (VQ)-based methods compress motion data into discrete tokens while latent-based models operate directly in continuous space. However, both of these representations exhibit significant limitations. VQ-based methods suffer from inherent information loss, which compromises the quality, diversity, and generalization of generated motions, while continuous representation on holistic whole-body motion hinders part-level flexibility. For architecture, diffusion and autoregressive diffusion models have demonstrated their superiority, yet the fine-grained controllability over individual body parts is also limited. Thus, we propose a unified spatiotemporally decoupled framework named DeMoDiff, which jointly redesigns representation and architecture. To enhance representation extraction capabilities and offer greater part-level controllability, we present a spatial-temporal VAE that encodes each body joint rather than compressing the whole-body motion into a single latent space. Then, we incorporate spatial-temporal masking and attention mechanisms into an autoregressive diffusion generator, achieving both generative capability and controllable editability. Extensive experiments on the HumanML3D and KIT-ML datasets demonstrate that our model achieves state-of-the-art reconstruction performance and compelling motion generation results. Moreover, our framework demonstrates strong temporal and spatial editing capabilities, further validating its effectiveness. Our project page: https://rex0191.github.io/DeMoDiff/

Figures & tables

Fig. 1: Comparison of motion representation (A1-3) and generation model (B1-3). Our proposed DeMoDiff is composed of spatial-temporal VAE and spatial-temporal autoregressive diffusion model for enhanced motion representation, generation, and spatiotemporal editing capabilities.

Fig. 2: Overview of DeMoDiff. (A) Spatial-Temporal VAE encodes motion into 2D latent and decodes it for reconstruction. (B) Spatial-Temporal Autoregressive Diffusion includes Spatial-Temporal Attention, Spatial Attention and Temporal Attention and is trained via mask modeling, using a diffusion head to predict motion latent. (C) During inference, Spatial-Temporal Autoregressive Diffusion iteratively predicts the latent representations, which are decoded into the motion sequence.

Datasets	Methods	FID $\downarrow$	Top1 $\uparrow$	Top2 $\uparrow$	Top3 $\uparrow$	MM-Dist $\downarrow$	Diversity $\rightarrow$
HumanML3D	Ground Truth	$0.002^{\pm.000}$	$0.511^{\pm.003}$	$0.703^{\pm.003}$	$0.797^{\pm.002}$	$2.974^{\pm.008}$	$9.503^{\pm.065}$
	MDM [ 20 ]	$0.544^{\pm.044}$	$0.320^{\pm.005}$	$0.498^{\pm.004}$	$0.611^{\pm.007}$	$5.566^{\pm.027}$	$\mathbf{9.559}^{\pm.086}$
	MLD [ 2 ]	$0.473^{\pm.013}$	$0.481^{\pm.003}$	$0.673^{\pm.003}$	$0.772^{\pm.002}$	$3.196^{\pm.010}$	$9.724^{\pm.082}$
	MotionDiffuse [ 25 ]	$0.630^{\pm.001}$	$0.491^{\pm.001}$	$0.681^{\pm.001}$	$0.782^{\pm.001}$	$3.113^{\pm.001}$	$\underline{9.410}^{\pm.049}$
	ReMoDiffuse [ 26 ]	$\mathbf{0.103}^{\pm.004}$	$\underline{0.510}^{\pm.005}$	$\underline{0.698}^{\pm.006}$	$\underline{0.795}^{\pm.004}$	$\underline{2.974}^{\pm.016}$	$9.018^{\pm.075}$
	DeMoDiff (Ours)	$\underline{0.164}^{\pm.012}$	$\mathbf{0.517}^{\pm.003}$	$\mathbf{0.712}^{\pm.004}$	$\textbf{0.808}^{\pm.002}$	$\mathbf{2.907}^{\pm.013}$	$9.399^{\pm.096}$

TABLE I: Evaluation on the HumanML3D dataset (upper half) and KIT-ML dataset (lower half). We repeat the evaluation for 20 times and report the average with 95% confidence interval.

\downarrow

means the lower, the better, while

\uparrow

represents the opposite.

\rightarrow

indicates the closer to the real motion the better. We use bold to indicate the best result and underscore for the second best.

Fig. 3: Visual comparison of generated results. The color from light blue to dark blue indicates the motion sequential order.

Methods	Generation		Reconstruction
	FID $\downarrow$	Diversity $\rightarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	ACCL $\downarrow$
TM2T [ 7 ]	0.320	9.705	57.5	42.3	7.8
T2M-GPT [ 24 ]	0.119	9.705	58.0	41.1	7.0
MoMask [ 5 ]	0.019	9.662	29.5	20.3	5.3
MoGenTS [ 23 ]	0.005	9.547	16.4	8.0	3.0
VPoser-t [ 13 ]	1.430	8.336	75.6	48.6	9.3

TABLE II: Evaluation of motion compression methods on HumanML3D dataset. MPJPE and PAMPJPE are measured in millimeters. ACCL indicates acceleration error.

Different VAE	FID $\downarrow$	Top1 $\uparrow$	Top2 $\uparrow$	Top3 $\uparrow$	MM-Dist $\downarrow$	Diversity $\rightarrow$	MPJPE $\downarrow$
1D Temporal (16 latent dim)	$0.0144^{\pm.0002}$	$0.5071^{\pm.0024}$	$0.6984^{\pm.0025}$	$0.7936^{\pm.0020}$	$3.0092^{\pm.0096}$	$9.4839^{\pm.1060}$	$28.0^{\pm.1}$
Ours (16 latent dim)	$0.0054^{\pm.0001}$	$\mathbf{0.5142}^{\pm.0026}$	$0.7015^{\pm.00025}$	$\underline{0.7983}^{\pm.0024}$	$\mathbf{2.9740}^{\pm.0085}$	$9.5407^{\pm.0736}$	$\underline{9.4}^{\pm.1}$
Ours (32 latent dim)	$\underline{0.0048}^{\pm.0000}$	$\underline{0.5126}^{\pm.0023}$	$\mathbf{0.7042}^{\pm.0018}$	$\mathbf{0.7988}^{\pm.0019}$	$\underline{2.9757}^{\pm.0073}$	$\mathbf{9.5081}^{\pm.0710}$	$9.8^{\pm.1}$
Ours (64 latent dim)	$\mathbf{0.0041^{\pm.0001}}$	$0.5111^{\pm.0020}$	$\underline{0.7019}^{\pm.0018}$	$0.7951^{\pm.0016}$	$2.9806^{\pm.0067}$	$\underline{9.4896}^{\pm.0716}$	$\mathbf{7.7^{\pm.1}}$

TABLE III: Ablation study of Spatial-Temporal VAE on the HumanML3D dataset.

Methods	FID $\downarrow$	Top1 $\uparrow$	Top2 $\uparrow$	Top3 $\uparrow$	MM-Dist $\downarrow$	Diversity $\rightarrow$
1D Temporal VAE + temporal attention	$0.665^{\pm.017}$	$0.452^{\pm.003}$	$0.633^{\pm.004}$	$0.735^{\pm.003}$	$3.361^{\pm.011}$	$9.114^{\pm.070}$
w/o all attention	$37.769^{\pm.122}$	$0.086^{\pm.003}$	$0.156^{\pm.004}$	$0.212^{\pm.003}$	$7.511^{\pm.011}$	$3.374^{\pm.083}$
w/o temporal&spatial attention	$1.315^{\pm.034}$	$0.443^{\pm.007}$	$0.639^{\pm.007}$	$0.748^{\pm.006}$	$3.269^{\pm.013}$	$8.905^{\pm.147}$
w/o spatial attention	$0.783^{\pm.015}$	$0.463^{\pm.003}$	$\underline{0.661}^{\pm.004}$	$\underline{0.765}^{\pm.007}$	$3.212^{\pm.022}$	$\underline{9.232}^{\pm.127}$
w/o temporal attention	$\underline{0.752}^{\pm.026}$	$\underline{0.466}^{\pm.005}$	$0.657^{\pm.004}$	$0.761^{\pm.004}$	$\underline{3.168}^{\pm.018}$	$9.193^{\pm.095}$
DeMoDiff (Full Model)	$\mathbf{0.164}^{\pm.012}$	$\mathbf{0.517}^{\pm.003}$	$\mathbf{0.712}^{\pm.004}$	$\mathbf{0.808}^{\pm.002}$	$\mathbf{2.907}^{\pm.013}$	$\mathbf{9.399}^{\pm.096}$

TABLE IV: Ablation study of the attention mechanisms choices on the HumanML3D dataset.

Fig. 4: Comparison of different motion compression methods. The dark blue represents ground-truth, the light blue represents predicted results ( Note that the global position of SMPL is the global position of motion. We have not performed any additional alignment processing ).

Fig. 5: Visualization of editing results. The color from light blue to dark blue indicates the motion sequence order. The green color indicates the edited regions.

Explore similar work

Feb 26, 2026

Causal Motion Diffusion Models for Autoregressive Motion Generation

Recent advances in motion diffusion models have substantially improved the realism of human motion synthesis. However, existing approaches either rely on full-sequence diffusion models with bidirectional generation, which limits temporal causality and real-time applicability, or autoregressive models that suffer from instability and cumulative errors. In this work, we present Causal Motion Diffusion Models (CMDM), a unified framework for autoregressive motion generation based on a causal diffusion transformer that operates in a semantically aligned latent space. CMDM builds upon a Motion-Language-Aligned Causal VAE (MAC-VAE), which encodes motion sequences into temporally causal latent representations. On top of this latent representation, an autoregressive diffusion transformer is trained using causal diffusion forcing to perform temporally ordered denoising across motion frames. To achieve fast inference, we introduce a frame-wise sampling schedule with causal uncertainty, where each subsequent frame is predicted from partially denoised previous frames. The resulting framework supports high-quality text-to-motion generation, streaming synthesis, and long-horizon motion generation at interactive rates. Experiments on HumanML3D and SnapMoGen demonstrate that CMDM outperforms existing diffusion and autoregressive models in both semantic fidelity and temporal smoothness, while substantially reducing inference latency.

Qing Yu, Akihisa Watanabe, Kent Fujiwara

LY Corporation · Waseda University

Human Motion Generation Diffusion Models for Motion Generation Autoregressive Diffusion Text-to-Motion Generation Diffusion Models Diffusion Model Inference Acceleration

Ask PDF

Oct 9, 2026

Image Diffusion Models Are Good 3D Human Motion Generators

Text-to-motion generation typically relies on dataset-specific skeletal representations, with models trained separately for benchmarks with different joint layouts. We investigate whether image diffusion architectures can model 3D human motion through a common image-compatible representation that accommodates heterogeneous skeletal layouts while preserving motion-specific kinematic structure. To this end, we organize root-centered joint trajectories into topology-informed joint--time images and model global root dynamics through a complementary latent stream. A motion autoencoder conditions local-motion decoding on the root latent, while a text-conditioned Scalable Interpolant Transformer (SiT) jointly generates both latent streams, with supervision on decoded 3D joint positions further constraining generation in motion space. In addition to dataset-specific models, we train a single motion autoencoder on HumanML3D and KIT-ML, followed by a single text-conditioned SiT shared across both datasets, preserving their respective skeletal definitions without skeleton retargeting to a common joint layout. Experiments on HumanML3D and KIT-ML show that our dataset-specific models achieve state-of-the-art performance, while shared training yields further gains on both benchmarks, demonstrating the benefits of shared modeling across heterogeneous skeletal layouts.

Siyuan Yang, Chong Wang, Hao Cheng +4

Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Artificial Intelligence, Hebei University of Technology, China +3

Text-to-Motion Generation Diffusion Models for Motion Generation Motion Representation Learning Diffusion Transformer

Ask PDF

Jun 11, 2024

RDM: Recurrent Diffusion Model for Human Motion Generation

Human motion generation is a challenging task due to its high dimensionality and the difficulty of generating fine-grained motions. Diffusion methods have been proposed due to their high sample quality and expressiveness. Early approaches treat the entire sequence as a whole, which is computationally expensive and restricts sequence length. In contrast, autoregressive diffusion models generate longer sequences. However, their reliance on fully denoising previous frames complicates training and inference. Consequently, we propose RDM, a new recurrent diffusion formulation inspired by Recurrent Neural Networks (RNNs). RDM explicitly conditions the diffusion process on preceding noisy frames, avoiding the cost of full denoising. To overcome the challenge of maintaining its probabilistic nature, we employ Normalizing Flows to model recurrent connections. Our evaluations demonstrate RDM's effectiveness: it achieves comparable performance to autoregressive baselines and generates long sequences that remain aligned with the text. RDM also skips diffusion steps during inference, reducing computational cost by up to 18x over autoregressive baselines. Project page: https://www.mirgahney.com/rfdm.io/.

Mirgahney Mohamed, Harry Jake Cunningham, Riccardo Barbano +2

Department of Computer Science, University College London

Diffusion Models for Motion Generation Diffusion Models Text-to-Motion Generation Diffusion Model Inference Acceleration

Ask PDF