Coordinate-Based Dual-Constrained Autoregressive Motion Generation

Kang Ding; Hongsong Wang; Jie Gui; Liang Wang

cs.CVApr 9, 2026

Coordinate-Based Dual-Constrained Autoregressive Motion Generation

Authors: Kang Ding, Hongsong Wang, Jie Gui, Liang Wang

Organizations: School of Cyber Science and Engineering, Southeast University, Nanjing 210096, China · School of Computer Science and Engineering, Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, Southeast University, Nanjing 210096, China · School of Cyber Science and Engineering, Southeast University, also with Purple Mountain Laboratories, Nanjing 211111, China, and also with Engineering Research Center of Blockchain Application, Supervision And Management (Southeast University), Ministry of Education, Southeast University, Nanjing 210096, China · New Laboratory of Pattern Recognition (NLPR), also with State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA), and also with School of Artificial Intelligence, University of Chinese Academy of Sciences

Abstract

Text-to-motion generation has attracted increasing attention in the research community recently, with potential applications in animation, virtual reality, robotics, and human-computer interaction. Diffusion and autoregressive models are two popular and parallel research directions for text-to-motion generation. However, diffusion models often suffer from error amplification during noise prediction, while autoregressive models exhibit mode collapse due to motion discretization. To address these limitations, we propose a flexible, high-fidelity, and semantically faithful text-to-motion framework, named Coordinate-based Dual-constrained Autoregressive Motion Generation (CDAMD). With motion coordinates as input, CDAMD follows the autoregressive paradigm and leverages diffusion-inspired multi-layer perceptrons to enhance the fidelity of predicted motions. Furthermore, a Dual-Constrained Causal Mask is introduced to guide autoregressive generation, where motion tokens act as priors and are concatenated with textual encodings. Since there is limited work on coordinate-based motion synthesis, we establish new benchmarks for both text-to-motion generation and motion editing. Experimental results demonstrate that our approach achieves state-of-the-art performance in terms of both fidelity and semantic consistency on these benchmarks.

Figures & tables

Fig. 1: Comparison of our approach with existing methods: (a) The existing autoregressive model (BAMM [ 11 ] ) fails to generate text-aligned motion for this example text prompt; (b) Top-1 R-Precision and FID results on the HumanML3D (bubble areas represent the 95% confidence interval of the FID). Our approach achieves excellent performance in both FID and R-Precision.

Fig. 2: Architecture illustration of CDAMD. (a) Hybrid Motion Encoders encodes the raw motion sequence into a compact fine-grained latent space. (b) CDAMD model learns to autoregressively predict next tokens conditioned on text embedding from CLIP and compressed motion tokens from RVQ-VAE. (c) Dual-Constrained Causal Attention (DCCA) enforces both temporal and conditional causality, ensuring that motion generation proceeds autoregressively and preserving semantic conditioning.

Methods	FID $\bm{\downarrow}$	R-Precision $\bm{\uparrow}$			Matching $\bm{\downarrow}$	MModality $\bm{\uparrow}$	Diversity $\bm{\uparrow}$
Methods	FID $\bm{\downarrow}$	Top 1	Top 2	Top 3	Matching $\bm{\downarrow}$	MModality $\bm{\uparrow}$	Diversity $\bm{\uparrow}$
MDM-50Step ∗ [ 20 ]	$0.395^{\pm.065}$	$0.435^{\pm.013}$	$0.627^{\pm.0014}$	$0.737^{\pm.011}$	$3.443^{\pm.060}$	$2.182^{\pm.055}$	$9.812^{\pm.148}$
MotionDiffuse ∗ [ 48 ]	$1.334^{\pm.035}$	$0.435^{\pm.008}$	$0.618^{\pm.006}$	$0.725^{\pm.005}$	$3.542^{\pm.015}$	$1.833^{\pm.078}$	$9.634^{\pm.048}$
ReMoDiffuse ∗ [ 23 ]	$0.207^{\pm.006}$	$0.467^{\pm.001}$	$0.654^{\pm.004}$	$0.748^{\pm.002}$	$3.289^{\pm.022}$	$\textbf{2.560}^{\pm.289}$	$9.483^{\pm.245}$
MotionLCM V2 ∗ [ 35 ]	$0.152^{\pm.007}$	$0.511^{\pm.007}$	$\underline{0.707}^{\pm.003}$	$\textbf{0.802}^{\pm.002}$	$3.005^{\pm.009}$	$1.993^{\pm.085}$	$9.703^{\pm.097}$
MMM ∗ [ 49 ]	$0.093^{\pm.004}$	$0.495^{\pm.008}$	$0.687^{\pm.000}$	$0.782^{\pm.003}$	$3.165^{\pm.037}$	$1.303^{\pm.085}$	$9.786^{\pm.352}$

TABLE I: Evaluation of text-to-motion generation on the HumanML3D datasets. More than 10 baselines are re-implemented using joint coordinates to reproduce their results on this dataset. The evaluation is repeated 20 times, and the average is reported with a 95% confidence interval. Bold indicates the best result, and underscore denotes the second best. ∗ refers the result of our re-implementation. VQ denotes the motion quantizers in the Residual Vector Quantized Variational Autoencoder (RVQ-VAE).

Methods	FID $\bm{\downarrow}$	R-Precision $\bm{\uparrow}$			Matching $\bm{\downarrow}$	MModality $\bm{\uparrow}$	Diversity $\bm{\uparrow}$
Methods	FID $\bm{\downarrow}$	Top 1	Top 2	Top 3	Matching $\bm{\downarrow}$	MModality $\bm{\uparrow}$	Diversity $\bm{\uparrow}$
MDM [ 20 ]	$0.782^{\pm.066}$	$0.390^{\pm.007}$	$0.609^{\pm.010}$	$0.733^{\pm.012}$	$3.956^{\pm.110}$	$\textbf{2.402}^{\pm.257}$	$11.019^{\pm.139}$
MotionDiffuse [ 48 ]	$2.793^{\pm.192}$	$0.357^{\pm.010}$	$0.553^{\pm.009}$	$0.667^{\pm.012}$	$3.957^{\pm.062}$	$1.538^{\pm.062}$	$11.316^{\pm.153}$
ReMoDiffuse [ 23 ]	$\textbf{0.257}^{\pm.021}$	$\underline{0.412}^{\pm.006}$	$\underline{0.630}^{\pm.012}$	$\underline{0.754}^{\pm.009}$	$\textbf{3.097}^{\pm.043}$	$1.814^{\pm.103}$	$\underline{11.451}^{\pm.183}$
MoMask [ 17 ]	$0.523^{\pm.022}$	$0.392^{\pm.006}$	$0.604^{\pm.008}$	$0.732^{\pm.006}$	$3.383^{\pm.030}$	$\underline{1.892}^{\pm.085}$	$11.143^{\pm.072}$
CDAMD (ours)	$\underline{0.270}^{\pm.020}$	$\textbf{0.416}^{\pm.006}$	$\textbf{0.632}^{\pm.005}$	$\textbf{0.759}^{\pm.004}$	$\underline{3.160}^{\pm.027}$	$1.211^{\pm.075}$	$\textbf{11.902}^{\pm.154}$

TABLE II: Evaluation of coordinate-based text-to-motion generation on the KIT dataset. Baselines are re-implemented using joint coordinates to reproduce their results on this dataset. Bold indicates the best result, and underscore denotes the second best.

Tasks	Methods	R-Precision $\bm{\uparrow}$			FID $\bm{\uparrow}$	Matching $\bm{\downarrow}$	CLIP-score $\bm{\uparrow}$
Tasks	Methods	Top 1	Top 2	Top 3	FID $\bm{\uparrow}$	Matching $\bm{\downarrow}$	CLIP-score $\bm{\uparrow}$
Temporal Inpainting	BAMM [ 11 ]	0.387	0.554	0.649	0.385	4.046	0.574
	MARDM [ 10 ]	0.503	0.702	0.795	0.120	3.051	0.671
	CDAMD (ours)	0.500	0.691	0.790	0.103	3.049	0.665
Temporal Outpainting	BAMM [ 11 ]	0.433	0.605	0.707	0.206	3.615	0.613
	MARDM [ 10 ]	0.512	0.705	0.797	0.114	3.065	0.671

TABLE III: Results of temporal motion editing on the HumanML3D. We use joint coordinates as motion representation, and re-implement results of BAMM [ 11 ] and MARDM [ 10 ] .

Methods	FID $\bm{\downarrow}$	R-Precision $\bm{\uparrow}$			Matching $\bm{\downarrow}$	Diversity $\bm{\uparrow}$	CLIP-score $\bm{\uparrow}$
Methods	FID $\bm{\downarrow}$	Top 1	Top 2	Top 3	Matching $\bm{\downarrow}$	Diversity $\bm{\uparrow}$	CLIP-score $\bm{\uparrow}$
DCCM	0.082	0.522	0.708	0.800	2.966	9.680	0.679
BCM	0.154	0.520	0.710	0.805	2.925	9.770	0.683
CM	0.231	0.515	0.696	0.792	3.055	9.132	0.671

TABLE IV: Comparison between different masking strategies in the transformer backbone. We compare the proposed Dual-Constrained Causal Mask (DCCM) with the Bidirectional Causal Mask (BCM) [ 11 ] and the Causal Mask (CM) on the HumanML3D dataset.

Methods	FID $\bm{\downarrow}$	R-Precision $\bm{\uparrow}$			Matching $\bm{\downarrow}$	Diversity $\bm{\uparrow}$	CLIP-score $\bm{\uparrow}$
Methods	FID $\bm{\downarrow}$	Top 1	Top 2	Top 3	Matching $\bm{\downarrow}$	Diversity $\bm{\uparrow}$	CLIP-score $\bm{\uparrow}$
CDAMD (ours)	0.082	0.522	0.708	0.800	2.966	9.680	0.679
w/o CMP	0.136	0.520	0.704	0.796	3.015	9.495	0.676
w/o DCCA	0.231	0.515	0.696	0.792	3.055	9.132	0.671
CDAMD (ours) †	0.090	0.523	0.712	0.808	2.942	9.728	0.684
w/o CMP †	0.122	0.516	0.699	0.791	3.109	9.686	0.681

TABLE V: Ablation study of the design choices of CDAMD on the HumanML3D dataset. CMP and DCCA denote Conditioned Motion Prior and Dual-Constrained Causal Attention, respectively. † denotes architectural components under the standard 263-dimensional mixed representation.

Fig. 3: Visualization of temporal editing tasks, inpainting, outpainting, prefix, and suffix where orange indicates conditioned motion and blue refers to generated parts.

Fig. 4: Qualitative results of text-to-motion generation on HumanML3D. Our approach is compared with BAMM [ 11 ] and MoMask [ 17 ] , which are representative autoregressive motion models.

Fig. 5: The failure cases of text-to-motion on HumanML3D test set.

Fig. 6: Visualization comparison if textual to motion to state-of-the-art methods. Both the BAMM [ 11 ] and MoMask [ 17 ] generate motion of insufficient quality, while CDAMD generates higher quality and is more correlated with textual descriptions.

Fig. 7: Validation loss curves with and without conditional motion priors. Loss curves show faster convergence and lower error when motion priors are incorporated.

CDAMD (ours)	MoMask	BAMM	Ground-Truth
39.8	21.5	10.4	28.3

TABLE VI: Comparison of motion preferences among CDAMD, MoMask [ 17 ] , BAMM [ 11 ] , and ground truth in the user study.

Explore similar work

Jan 21, 2026

Reconstruction-Anchored Diffusion Model for Text-to-Motion Generation

Diffusion models have seen widespread adoption for text-driven human motion generation and related tasks due to their impressive generative capabilities and flexibility. However, current motion diffusion models face two major limitations: a representational gap caused by pre-trained text encoders that lack motion-specific information, and error propagation during the iterative denoising process. This paper introduces Reconstruction-Anchored Diffusion Model (RAM) to address these challenges. First, RAM leverages a motion latent space as intermediate supervision for text-to-motion generation. To this end, RAM co-trains a motion reconstruction branch with two key objective functions: self-regularization to enhance the discrimination of the motion space and motion-centric latent alignment to enable accurate mapping from text to the motion latent space. Second, we propose Reconstructive Error Guidance (REG), a testing-stage guidance mechanism that exploits the motion diffusion model's inherent self-correction ability to mitigate error propagation. At each denoising step, REG uses the motion reconstruction branch to reconstruct the previous estimate, reproducing the prior error patterns. By amplifying the residual between the current prediction and the reconstructed estimate, REG highlights the improvements in the current prediction. Extensive experiments demonstrate that RAM achieves significant improvements and state-of-the-art performance. Our code will be released.

Yifei Liu, Changxing Ding, Ling Guo +2

South China University of Technology · Joy Future Academy

Diffusion Model Guidance Text-to-Motion Generation Human Motion Generation Diffusion Models for Motion Generation

Ask PDF

Apr 18, 2026

Marrying Text-to-Motion Generation with Skeleton-Based Action Recognition

Human action recognition and motion generation are two active research problems in human-centric computer vision, both aiming to align motion with textual semantics. However, most existing works study these two problems separately, without uncovering the links between them, namely that motion generation requires semantic comprehension. This work investigates unified action recognition and motion generation by leveraging skeleton coordinates for both motion understanding and generation. We propose Coordinates-based Autoregressive Motion Diffusion (CoAMD), which synthesizes motion in a coarse-to-fine manner. As a core component of CoAMD, we design a Multi-modal Action Recognizer (MAR) that provides gradient-based semantic guidance for motion generation. Furthermore, we establish a rigorous benchmark by evaluating baselines on absolute coordinates. Our model can be applied to four important tasks, including skeleton-based action recognition, text-to-motion generation, text-motion retrieval, and motion editing. Extensive experiments on 13 benchmarks across these tasks demonstrate that our approach achieves state-of-the-art performance, highlighting its effectiveness and versatility for human motion modeling. Code is available at https://github.com/jidongkuang/CoAMD.

Jidong Kuang, Hongsong Wang, Jie Gui

School of Cyber Science and Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China

Skeleton-Based Action Recognition Human Activity Recognition Text-to-Motion Generation Motion Diffusion Models Human Motion Generation Diffusion Models

Ask PDF

Feb 26, 2026

Causal Motion Diffusion Models for Autoregressive Motion Generation

Recent advances in motion diffusion models have substantially improved the realism of human motion synthesis. However, existing approaches either rely on full-sequence diffusion models with bidirectional generation, which limits temporal causality and real-time applicability, or autoregressive models that suffer from instability and cumulative errors. In this work, we present Causal Motion Diffusion Models (CMDM), a unified framework for autoregressive motion generation based on a causal diffusion transformer that operates in a semantically aligned latent space. CMDM builds upon a Motion-Language-Aligned Causal VAE (MAC-VAE), which encodes motion sequences into temporally causal latent representations. On top of this latent representation, an autoregressive diffusion transformer is trained using causal diffusion forcing to perform temporally ordered denoising across motion frames. To achieve fast inference, we introduce a frame-wise sampling schedule with causal uncertainty, where each subsequent frame is predicted from partially denoised previous frames. The resulting framework supports high-quality text-to-motion generation, streaming synthesis, and long-horizon motion generation at interactive rates. Experiments on HumanML3D and SnapMoGen demonstrate that CMDM outperforms existing diffusion and autoregressive models in both semantic fidelity and temporal smoothness, while substantially reducing inference latency.

Qing Yu, Akihisa Watanabe, Kent Fujiwara

LY Corporation · Waseda University

Human Motion Generation Diffusion Models for Motion Generation Autoregressive Diffusion Text-to-Motion Generation Diffusion Models Diffusion Model Inference Acceleration

Ask PDF