Spatiotemporally Decoupled Autoregressive Diffusion Model for Human Motion Generation

Chengqun Yang; Liang Xu; Yanping Li; Fulong Liu; Jingnan Gao; Weili Zeng; Yichao Yan

cs.CVAug 24, 2026

Spatiotemporally Decoupled Autoregressive Diffusion Model for Human Motion Generation

Authors: Chengqun Yang, Liang Xu, Yanping Li, Fulong Liu, Jingnan Gao, Weili Zeng, Yichao Yan

Organizations: MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University

Abstract

Text-driven human motion synthesis has made substantial development with two core modules of motion representation and generative architecture. For representation, Vector Quantization (VQ)-based methods compress motion data into discrete tokens while latent-based models operate directly in continuous space. However, both of these representations exhibit significant limitations. VQ-based methods suffer from inherent information loss, which compromises the quality, diversity, and generalization of generated motions, while continuous representation on holistic whole-body motion hinders part-level flexibility. For architecture, diffusion and autoregressive diffusion models have demonstrated their superiority, yet the fine-grained controllability over individual body parts is also limited. Thus, we propose a unified spatiotemporally decoupled framework named DeMoDiff, which jointly redesigns representation and architecture. To enhance representation extraction capabilities and offer greater part-level controllability, we present a spatial-temporal VAE that encodes each body joint rather than compressing the whole-body motion into a single latent space. Then, we incorporate spatial-temporal masking and attention mechanisms into an autoregressive diffusion generator, achieving both generative capability and controllable editability. Extensive experiments on the HumanML3D and KIT-ML datasets demonstrate that our model achieves state-of-the-art reconstruction performance and compelling motion generation results. Moreover, our framework demonstrates strong temporal and spatial editing capabilities, further validating its effectiveness. Our project page: https://rex0191.github.io/DeMoDiff/

Figures & tables

Fig. 1: Comparison of motion representation (A1-3) and generation model (B1-3). Our proposed DeMoDiff is composed of spatial-temporal VAE and spatial-temporal autoregressive diffusion model for enhanced motion representation, generation, and spatiotemporal editing capabilities.

Fig. 2: Overview of DeMoDiff. (A) Spatial-Temporal VAE encodes motion into 2D latent and decodes it for reconstruction. (B) Spatial-Temporal Autoregressive Diffusion includes Spatial-Temporal Attention, Spatial Attention and Temporal Attention and is trained via mask modeling, using a diffusion head to predict motion latent. (C) During inference, Spatial-Temporal Autoregressive Diffusion iteratively predicts the latent representations, which are decoded into the motion sequence.

Datasets	Methods	FID $\downarrow$	Top1 $\uparrow$	Top2 $\uparrow$	Top3 $\uparrow$	MM-Dist $\downarrow$	Diversity $\rightarrow$
HumanML3D	Ground Truth	$0.002^{\pm.000}$	$0.511^{\pm.003}$	$0.703^{\pm.003}$	$0.797^{\pm.002}$	$2.974^{\pm.008}$	$9.503^{\pm.065}$
	MDM [ 20 ]	$0.544^{\pm.044}$	$0.320^{\pm.005}$	$0.498^{\pm.004}$	$0.611^{\pm.007}$	$5.566^{\pm.027}$	$\mathbf{9.559}^{\pm.086}$
	MLD [ 2 ]	$0.473^{\pm.013}$	$0.481^{\pm.003}$	$0.673^{\pm.003}$	$0.772^{\pm.002}$	$3.196^{\pm.010}$	$9.724^{\pm.082}$
	MotionDiffuse [ 25 ]	$0.630^{\pm.001}$	$0.491^{\pm.001}$	$0.681^{\pm.001}$	$0.782^{\pm.001}$	$3.113^{\pm.001}$	$\underline{9.410}^{\pm.049}$
	ReMoDiffuse [ 26 ]	$\mathbf{0.103}^{\pm.004}$	$\underline{0.510}^{\pm.005}$	$\underline{0.698}^{\pm.006}$	$\underline{0.795}^{\pm.004}$	$\underline{2.974}^{\pm.016}$	$9.018^{\pm.075}$
	DeMoDiff (Ours)	$\underline{0.164}^{\pm.012}$	$\mathbf{0.517}^{\pm.003}$	$\mathbf{0.712}^{\pm.004}$	$\textbf{0.808}^{\pm.002}$	$\mathbf{2.907}^{\pm.013}$	$9.399^{\pm.096}$

TABLE I: Evaluation on the HumanML3D dataset (upper half) and KIT-ML dataset (lower half). We repeat the evaluation for 20 times and report the average with 95% confidence interval.

\downarrow

means the lower, the better, while

\uparrow

represents the opposite.

\rightarrow

indicates the closer to the real motion the better. We use bold to indicate the best result and underscore for the second best.

Fig. 3: Visual comparison of generated results. The color from light blue to dark blue indicates the motion sequential order.

Methods	Generation		Reconstruction
	FID $\downarrow$	Diversity $\rightarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	ACCL $\downarrow$
TM2T [ 7 ]	0.320	9.705	57.5	42.3	7.8
T2M-GPT [ 24 ]	0.119	9.705	58.0	41.1	7.0
MoMask [ 5 ]	0.019	9.662	29.5	20.3	5.3
MoGenTS [ 23 ]	0.005	9.547	16.4	8.0	3.0
VPoser-t [ 13 ]	1.430	8.336	75.6	48.6	9.3

TABLE II: Evaluation of motion compression methods on HumanML3D dataset. MPJPE and PAMPJPE are measured in millimeters. ACCL indicates acceleration error.

Different VAE	FID $\downarrow$	Top1 $\uparrow$	Top2 $\uparrow$	Top3 $\uparrow$	MM-Dist $\downarrow$	Diversity $\rightarrow$	MPJPE $\downarrow$
1D Temporal (16 latent dim)	$0.0144^{\pm.0002}$	$0.5071^{\pm.0024}$	$0.6984^{\pm.0025}$	$0.7936^{\pm.0020}$	$3.0092^{\pm.0096}$	$9.4839^{\pm.1060}$	$28.0^{\pm.1}$
Ours (16 latent dim)	$0.0054^{\pm.0001}$	$\mathbf{0.5142}^{\pm.0026}$	$0.7015^{\pm.00025}$	$\underline{0.7983}^{\pm.0024}$	$\mathbf{2.9740}^{\pm.0085}$	$9.5407^{\pm.0736}$	$\underline{9.4}^{\pm.1}$
Ours (32 latent dim)	$\underline{0.0048}^{\pm.0000}$	$\underline{0.5126}^{\pm.0023}$	$\mathbf{0.7042}^{\pm.0018}$	$\mathbf{0.7988}^{\pm.0019}$	$\underline{2.9757}^{\pm.0073}$	$\mathbf{9.5081}^{\pm.0710}$	$9.8^{\pm.1}$
Ours (64 latent dim)	$\mathbf{0.0041^{\pm.0001}}$	$0.5111^{\pm.0020}$	$\underline{0.7019}^{\pm.0018}$	$0.7951^{\pm.0016}$	$2.9806^{\pm.0067}$	$\underline{9.4896}^{\pm.0716}$	$\mathbf{7.7^{\pm.1}}$

TABLE III: Ablation study of Spatial-Temporal VAE on the HumanML3D dataset.

Methods	FID $\downarrow$	Top1 $\uparrow$	Top2 $\uparrow$	Top3 $\uparrow$	MM-Dist $\downarrow$	Diversity $\rightarrow$
1D Temporal VAE + temporal attention	$0.665^{\pm.017}$	$0.452^{\pm.003}$	$0.633^{\pm.004}$	$0.735^{\pm.003}$	$3.361^{\pm.011}$	$9.114^{\pm.070}$
w/o all attention	$37.769^{\pm.122}$	$0.086^{\pm.003}$	$0.156^{\pm.004}$	$0.212^{\pm.003}$	$7.511^{\pm.011}$	$3.374^{\pm.083}$
w/o temporal&spatial attention	$1.315^{\pm.034}$	$0.443^{\pm.007}$	$0.639^{\pm.007}$	$0.748^{\pm.006}$	$3.269^{\pm.013}$	$8.905^{\pm.147}$
w/o spatial attention	$0.783^{\pm.015}$	$0.463^{\pm.003}$	$\underline{0.661}^{\pm.004}$	$\underline{0.765}^{\pm.007}$	$3.212^{\pm.022}$	$\underline{9.232}^{\pm.127}$
w/o temporal attention	$\underline{0.752}^{\pm.026}$	$\underline{0.466}^{\pm.005}$	$0.657^{\pm.004}$	$0.761^{\pm.004}$	$\underline{3.168}^{\pm.018}$	$9.193^{\pm.095}$
DeMoDiff (Full Model)	$\mathbf{0.164}^{\pm.012}$	$\mathbf{0.517}^{\pm.003}$	$\mathbf{0.712}^{\pm.004}$	$\mathbf{0.808}^{\pm.002}$	$\mathbf{2.907}^{\pm.013}$	$\mathbf{9.399}^{\pm.096}$

TABLE IV: Ablation study of the attention mechanisms choices on the HumanML3D dataset.

Fig. 4: Comparison of different motion compression methods. The dark blue represents ground-truth, the light blue represents predicted results ( Note that the global position of SMPL is the global position of motion. We have not performed any additional alignment processing ).

Fig. 5: Visualization of editing results. The color from light blue to dark blue indicates the motion sequence order. The green color indicates the edited regions.

Explore similar work

Causal Motion Diffusion Models for Autoregressive Motion Generation
Feb 26, 2026Qing Yu, Akihisa Watanabe, Kent FujiwaraHuman Motion Generation Diffusion Models for Motion Generation
LY Corporation · Waseda University
Image Diffusion Models Are Good 3D Human Motion Generators
Oct 9, 2026Siyuan Yang, Chong Wang, Hao Cheng +4Text-to-Motion Generation Diffusion Models for Motion Generation
Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Artificial Intelligence, Hebei University of Technology, China +3
RDM: Recurrent Diffusion Model for Human Motion Generation
Jun 11, 2024Mirgahney Mohamed, Harry Jake Cunningham, Riccardo Barbano +2Diffusion Models for Motion Generation Diffusion Models
Department of Computer Science, University College London

Feb 26, 2026J/K move · Enter open · S save

Causal Motion Diffusion Models for Autoregressive Motion Generation

Qing Yu, Akihisa Watanabe, Kent Fujiwara

LY Corporation · Waseda University

Human Motion Generation Diffusion Models for Motion Generation Autoregressive Diffusion Text-to-Motion Generation Diffusion Models Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Recent advances in motion diffusion models have substantially improved the realism of human motion synthesis. However, existing approaches either rely on full-sequence diffusion models with bidirectional generation, which limits temporal causality and real-time applicability, or autoregressive models that suffer from instability and cumulative errors. In this work, we present Causal Motion Diffusion Models (CMDM), a unified framework for autoregressive motion generation based on a causal diffusion transformer that operates in a semantically aligned latent space. CMDM builds upon a Motion-Language-Aligned Causal VAE (MAC-VAE), which encodes motion sequences into temporally causal latent representations. On top of this latent representation, an autoregressive diffusion transformer is trained using causal diffusion forcing to perform temporally ordered denoising across motion frames. To achieve fast inference, we introduce a frame-wise sampling schedule with causal uncertainty, where each subsequent frame is predicted from partially denoised previous frames. The resulting framework supports high-quality text-to-motion generation, streaming synthesis, and long-horizon motion generation at interactive rates. Experiments on HumanML3D and SnapMoGen demonstrate that CMDM outperforms existing diffusion and autoregressive models in both semantic fidelity and temporal smoothness, while substantially reducing inference latency.

Abstract

Explore similar work

Causal Motion Diffusion Models for Autoregressive Motion Generation

Image Diffusion Models Are Good 3D Human Motion Generators

RDM: Recurrent Diffusion Model for Human Motion Generation