Exploring Motion-Language Alignment for Text-driven Motion Generation

Ruxi Gu; Zilei Wang; Wei Wang

cs.CVApr 3, 2026

Exploring Motion-Language Alignment for Text-driven Motion Generation

Authors: Ruxi Gu, Zilei Wang, Wei Wang

Organizations: Department of Automation, University of Science and Technology of China, Hefei, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China

Abstract

Text-driven human motion generation aims to synthesize realistic motion sequences that follow textual descriptions. Despite recent advances, accurately aligning motion dynamics with textual semantics remains a fundamental challenge. In this paper, we revisit text-to-motion generation from the perspective of motion-language alignment and propose MLA-Gen, a framework that integrates global motion priors with fine-grained local conditioning. This design enables the model to capture common motion patterns, while establishing detailed alignment between texts and motions. Furthermore, we identify a previously overlooked attention sink phenomenon in human motion generation, where attention disproportionately concentrates on the start text token, limiting the utilization of informative textual cues and leading to degraded semantic grounding. To analyze this issue, we introduce SinkRatio, a metric for measuring attention concentration, and develop alignment-aware masking and control strategies to regulate attention during generation. Extensive experiments demonstrate that our approach consistently improves both motion quality and motion-language alignment over strong baselines.

Figures & tables

Figure 1: Failure cases from previous text-to-motion generation framework [ 26 ] . The color gradient from dark to light represents the temporal flow.

Figure 2: Overview of our MLA-Gen framework. It comprises three complementary components: Memory Slots for capturing global motion priors, Motion-Language Alignment for providing fine-grained textual semantics, and a SinkRatio-based mechanism that models and mitigates the attention sink during both attention computation ( sink-mask ) and sampling ( sink-ctrl ).

Figure 3: Heatmap of the memory slots activation. Regions rendered in brighter yellow indicate higher attention weights between the corresponding motion frames and memory slots.

Figure 4: Heatmap of motion-language alignment. Regions in brighter yellow indicate higher attention weights between the corresponding motion frames and text tokens.

Figure 5: Heatmaps comparison of alignment on the masked model (left) and the unmasked model (right).

Figure 6: SinkRatio curves for masked and unmasked models. Each curve depicts the mean SinkRatio across all batch samples over timesteps, with the shaded region indicating the standard deviation.

Methods	FID $\downarrow$	R-Precision $\uparrow$			Matching $\downarrow$	MModality $\uparrow$	CLIP-score $\uparrow$
Methods	FID $\downarrow$	Top 1	Top 2	Top 3	Matching $\downarrow$	MModality $\uparrow$	CLIP-score $\uparrow$
Real	0.000 ±.000	0.503 ±.002	0.696 ±.001	0.795 ±.002	3.244 ±.005	-	0.639 ±.001
MotionDiffuse	0.778 ±.005	0.450 ±.006	0.641 ±.005	0.753 ±.005	3.490 ±.023	3.179 ±.046	0.606 ±.004
ReMoDiffuse	0.883 ±.021	0.468 ±.003	0.653 ±.003	0.754 ±.005	3.414 ±.020	2.703 ±.154	0.621 ±.003
MLD++	2.027 ±.021	0.500 ±.003	0.691 ±.002	0.789 ±.001	3.220 ±.008	1.924 ±.065	0.639 ±.002
MotionLCM V2	2.267 ±.023	0.501 ±.002	0.693 ±.002	0.790 ±.002	3.192 ±.009	1.780 ±.062	0.640 ±.003

Table 1: Quantitative text-to-motion evaluation on HumanML3D dataset. We repeat the evaluation 20 times and report the average with 95% confidence interval. We use bold face/underline to indicate the best/2nd results, and asterisk (*) to indicate the better results between our method and ACMDM [ 26 ] .

Method	FID $\downarrow$	R-Top1 $\uparrow$	R-Top2 $\uparrow$	R-Top3 $\uparrow$
MotionDiffuse	3.845	0.344	0.536	0.658
ReMoDiffuse	1.725	0.356	0.572	0.706
MARDM- $\epsilon$	0.340	0.375	0.597	0.739
MARDM-v	0.242	0.387	0.610	0.749
ACMDM	0.237	0.391	0.615	0.752
MLA-Gen-S	0.199	0.392	0.615	0.744

Table 2: Evaluation results on KIT-ML dataset.

Figure 7: Visualization comparison between ACMDM-S [ 26 ] and our MLA-Gen-S.

Methods	FID $\downarrow$	R-Top1 $\uparrow$	R-Top2 $\uparrow$	R-Top3 $\uparrow$
+ $M$ + $A$ + $S$	0.056	0.507	0.698	0.795
+ $M$ + $A$	0.076	0.508	0.703	0.799
+ $M$	0.110	0.506	0.699	0.797
+ $A$	0.101	0.508	0.702	0.799
w/o any modules	0.120	0.507	0.702	0.798

Table 3: Ablation study of MLA-Gen. We use bold face to denote the original configurations.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Name	Meaning	Value
ep	Number of training epochs	500
opt	Training optimizer	AdamW
lr	Learning rate	2e-4
lr-decay	Weight decay	0.1
$S$	Number of slots	16
$\lambda$	Local cond scale	0.2

Appendix

Table 4: Hyperparameter settings. Hyperparameters listed above the dividing line are those required for training, while those below correspond specifically to the MLA-Gen model.

Methods	FID $\downarrow$	R-Precision $\uparrow$
Methods	FID $\downarrow$	Top 1	Top 2	Top 3
$\lambda=0.1$	0.089	0.508	0.703	0.799
$\lambda=0.2$	0.076	0.508	0.703	0.798
$\lambda=0.4$	0.089	0.501	0.695	0.793
$\lambda=0.6$	0.095	0.499	0.692	0.788
learnable $\lambda$	0.118	0.510	0.703	0.799

Appendix

Table 5: Ablation study of the local-to-global fusion factor

\lambda

. We use gray shade and bold face to indicate the original configurations in our MLA-Gen.

$S$	FID $\downarrow$	R-Top1 $\uparrow$	R-Top2 $\uparrow$	R-Top3 $\uparrow$
4	0.081	0.501	0.696	0.793
8	0.081	0.503	0.696	0.791
12	0.056	0.507	0.697	0.796
16	0.056	0.507	0.698	0.795

Appendix

Table 6: Ablation study on MLA-Gen-S of slot number

S

Methods	FID $\downarrow$	R-Precision $\uparrow$
Methods	FID $\downarrow$	Top 1	Top 2	Top 3
top-1	0.051	0.506	0.698	0.793
top-2	0.056	0.507	0.698	0.795
top-3	0.059	0.507	0.699	0.794

Appendix

Table 7: Ablation study of the top-

K

algorithm in SinkRatio.

Methods	FID $\downarrow$	Top-1 $\uparrow$	Top-2 $\uparrow$	Top-3 $\uparrow$
strong mask	0.056	0.507	0.698	0.795
weak mask	0.069	0.505	0.697	0.795
no mask	0.099	0.508	0.700	0.797

Appendix

Table 8: Ablation of the sink-mask mechanism. Strong and weak masks correspond to

t_{\text{thresh}}=0.2

and

0.6

Methods	FID $\downarrow$	Top-1 $\uparrow$	Top-2 $\uparrow$	Top-3 $\uparrow$
$C_{l}$	0.056	0.507	0.698	0.795
$0.5C_{l}$	0.070	0.506	0.699	0.795
$0$	0.101	0.505	0.698	0.793

Appendix

Table 9: Ablation of local conditioning in the CFG unconditional branch.

C_{l}

0.5C_{l}

, and

0

indicate full, limited, and no conditional information in

X_{\text{uncond}}

generation, respectively.

Video Files	Text Description
./videos/ACMDM_sample_1.mp4 ./videos/Ours_sample_1.mp4	A person stands in a defensive stance with the left arm and leg forward, and then uses the right forearm to perform a blocking motion across the body.
./videos/ACMDM_sample_2.mp4 ./videos/Ours_sample_2.mp4	A person is dancing and clapping their hands above the head.
./videos/ACMDM_sample_3.mp4 ./videos/Ours_sample_3.mp4	A person kicks two balls with the left foot, first to the right and then to the left.
./videos/ACMDM_sample_4.mp4 ./videos/Ours_sample_4.mp4	A person walks and steps onto an elevated surface before stepping down.

Appendix

Table 10: Mapping between video pair paths and their corresponding textual descriptions. Each pair contains results from ACMDM and our method (MLA-Gen), respectively.

Methods	FID $\downarrow$	Top-1 $\uparrow$	Top-2 $\uparrow$	Top-3 $\uparrow$
sink-ctrl ( $w=3.5$ )	0.044	0.501	0.693	0.790
sink-ctrl ( $w=4$ )	0.056	0.507	0.698	0.795
sink-ctrl ( $w=4.5$ )	0.084	0.509	0.700	0.796
cfg-ctrl ( $w=3.5$ )	0.048	0.502	0.694	0.791
cfg-ctrl ( $w=4$ )	0.068	0.506	0.699	0.794
cfg-ctrl ( $w=4.5$ )	0.110	0.507	0.701	0.796

Appendix

Table 11: Ablation of CFG strategies under different guidance scales.

Figure 8: A failure case of MLA-Gen with a very long textual description.

Figure 9: Heatmap sequences of the masked model across increasing timesteps. Heatmaps are arranged in a left-to-right, top-to-bottom order. Brighter yellow regions indicate higher attention weights between motion frames and text tokens.

Figure 10: Heatmap sequences of the unmasked model across increasing timesteps.

Explore similar work

MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation
Dec 15, 2025Yannan He, Garvita Tiwari, Xiaohan Zhang +4Latent Diffusion Models Text-to-Motion Generation
University of Tübingen, Germany · Tübingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany +2
FlowCoMotion: Text-to-Motion Generation via Token-Latent Flow Modeling
Apr 13, 2026Dawei Guan, Di Yang, Chengjie Jin +1Text-to-Motion Generation Human Motion Generation
School of Artificial Intelligence and Data Science, University of Science and Technology of China · Suzhou Institute for Advanced Research, University of Science and Technology of China
FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervision
Aug 2, 2026Tongyan Wang, Zhengyuan Li, Muhan Lin +5Cross-Modal Alignment
1Purdue University · University of Illinois Urbana-Champaign

Dec 15, 2025J/K move · Enter open · S save

MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation

Yannan He, Garvita Tiwari, Xiaohan Zhang, Pankaj Bora+3

University of Tübingen, Germany · Tübingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany+2

Latent Diffusion Models Text-to-Motion Generation Cross-Modal Alignment Motion Diffusion Models Human Motion Generation Diffusion Models for Motion Generation

Open paper AskPDF arXiv

We introduce MoLingo, a text-to-motion (T2M) model that generates realistic, lifelike human motion by denoising in a continuous latent space. Recent works perform latent space diffusion, either on the whole latent at once or auto-regressively over multiple latents. In this paper, we study how to make diffusion on continuous motion latents work best. We focus on two questions: (1) how to build a semantically aligned latent space so diffusion becomes more effective, and (2) how to best inject text conditioning so the motion follows the description closely. We propose a semantic-aligned motion encoder trained with frame-level text labels so that latents with similar text meaning stay close, which makes the latent space more diffusion-friendly. We also compare single-token conditioning with a multi-token cross-attention scheme and find that cross-attention gives better motion realism and text-motion alignment. With semantically aligned latents, auto-regressive generation, and cross-attention text conditioning, our model sets a new state of the art in human motion generation on standard metrics and in a user study. We will release our code and models for further research and downstream usage.

Abstract

Explore similar work

MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation

FlowCoMotion: Text-to-Motion Generation via Token-Latent Flow Modeling

FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervision