Bilingual Text-to-Motion Generation: A New Benchmark and Baselines
Authors: Wanjiang Weng, Xiaofeng Tan, Xiangbo Shu, Guo-Sen Xie, Pan Zhou, Hongsong Wang
Organizations: School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Nanjing University of Science and Technology · Singapore Management University
Text-to-motion generation holds significant potential for cross-linguistic applications, yet it is hindered by the lack of bilingual datasets and the poor cross-lingual semantic understanding of existing language models. To address these gaps, we introduce BiHumanML3D, the first bilingual text-to-motion benchmark, constructed via LLM-assisted annotation and rigorous manual correction. Furthermore, we propose a simple yet effective baseline, Bilingual Motion Diffusion (BiMD), featuring Cross-Lingual Alignment (CLA). CLA explicitly aligns semantic representations across languages, creating a robust conditional space that enables high-quality motion generation from bilingual inputs, including zero-shot code-switching scenarios. Extensive experiments demonstrate that BiMD with CLA achieves an FID of 0.045 vs. 0.169 and R@3 of 82.8% vs. 80.8%, significantly outperforms monolingual diffusion models and translation baselines on BiHumanML3D, underscoring the critical necessity and reliability of our dataset and the effectiveness of our alignment strategy for cross-lingual motion synthesis. The dataset and code are released at https://wengwanjiang.github.io/BilingualT2M-page
Figures & tables
Figure 1: Code-switching in bilingual text-to-motion generation. Existing text-to-motion methods (e.g., MLD [ 2 ] and MotionLCM [ 6 ] ) struggle to interpret mixed-language inputs, leading to incorrect motion semantics, while our BiMD generates semantically consistent motions.
Figure 2: Visual comparison of bilingual text-to-motion generation. Existing methods such as MDM [ 31 ] and MLD [ 2 ] exhibit limitations in bilingual processing. Directly using pretrained multilingual encoders results in imbalanced performance across languages. Our BiMD with Cross-Lingual Alignment (CLA) generates accurate motions from both English and Chinese descriptions.
Figure 3: Pipeline for constructing our bilingual HumanML3D dataset. The data collection and filtering process removes unsuitable motions, ensuring high-quality motion-text pairs for annotation. The annotation pipeline begins with an initial translation stage, followed by a refinement stage to address translation issues. Finally, human annotators manually verify and correct the translation with LLM, ensuring linguistic and contextual accuracy.
Figure 4: Framework for training the bilingual motion diffusion model. We align English and Chinese text embeddings in a shared latent space by freezing the teacher model EΦt and fine-tuning the student model Eϕs with the cross-lingual alignment loss LCLA in Eq. ( 1 ). The aligned student model Eϕs then provides text conditions for training the diffusion model ϵθ , enabling bilingual motion generation while minimizing LBiMD in Eq. ( 6 ).
Figure 5: Visual comparison of zero‑shot cross‑lingual motion generation from Chinese‑only training to unseen English texts. We find that BiMD with CLA generates accurate, semantically aligned motions despite never having seen English during training.
Method
CLA
Lang.
R Precision ↑
FID ↓
MM Dist ↓
Div →
Top 1
Top 2
Top 3
Real
-
CN
0.543
0.732
0.821
0.002
3.338
10.750
EN
0.511
0.703
0.797
0.002
2.974
9.503
Mo.Diffuse [ 43 ]
✗
CN
0.468 ±0.002
0.652 ±0.003
0.764 ±0.004
1.024 ±0.011
3.512 ±0.007
11.586 ±0.078
EN
0.476 ±0.001
0.666 ±0.004
0.775 ±0.003
0.991 ±0.028
3.356 ±0.006
9.972 ±0.076
✓
CN
0.510 ±0.003
0.677 ±0.002
0.797 ±0.001
0.493 ±0.009
3.138 ±0.014
10.610 ±0.066
Table 2: Comparison of text-to-motion generation performance on the BiHumanML3D dataset. “Lang.” indicates the evaluated language. English and Chinese results are evaluated using the T2MT evaluator [ 9 ] and our proposed evaluator, respectively. Bold highlights the best results. The symbol “✓” at “CLA” denotes methods that employ our cross-lingual alignment representation, indicating the use of a unified model for generation, whereas the symbol “✗” means methods retrained with XLM as text encoder.
Text Source
R-Precision ↑
FID ↓
MMDist ↓
Div →
R1
R2
R3
DeepL-API
0.377
0.585
0.687
0.776
4.763
10.766
GPT-Translated
0.387
0.593
0.700
0.721
4.401
10.396
DeepSeek-Translated
0.393
0.601
0.713
0.645
4.253
10.523
HumanML3D
0.481
0.673
0.772
0.473
3.196
9.724
Table 3: Performance comparison of the MLD model conditioned by translated English texts versus HumanML3D (native English texts).
Table 4: User study. (a) Motion preference: BiMD (Zero-shot) vs. MLD vs. ground truth on English prompts. (b) Semantic consistency for paired Chinese–English prompts with/without CLA. Percentages are averaged over prompts.
Train
Test
CLA
R Precision ↑
FID ↓
MM Dist ↓
Div →
Top 1
Top 2
Top 3
Real
CN
–
0.543
0.732
0.821
0.002
3.338
10.750
EN
0.511
0.703
0.797
0.002
2.974
9.503
EN
CN
✗
0.173
0.288
0.364
3.533
6.134
9.231
EN
CN
✓
0.425
0.608
0.701
0.072
3.584
9.720
CN
EN
✗
0.184
0.298
0.388
3.397
6.258
9.228
Table 5: Cross-lingual motion generation performance under different training and test language settings. Train and Test denote the training and evaluation languages, respectively. “CLA” indicates whether Cross-lingual Alignment is applied. The arrows ↑ , ↓ , and → indicate higher, lower, and closer-to-real-motion values are better, respectively. Bold highlights the best results.
Figure 6: A translation failure case. Direct translation may omit motion-critical details (e.g., direction), resulting in under-specified descriptions. Our annotation preserves such information, leading to more precise motion semantics.
Figure 7: t-SNE visualizations of text representations, showing the distribution of 1000 randomly selected text descriptions from the testing set of HumanML3D. Dots of the same color represent text of the same language, with potential alignment effects observed.
We introduce MoLingo, a text-to-motion (T2M) model that generates realistic, lifelike human motion by denoising in a continuous latent space. Recent works perform latent space diffusion, either on the whole latent at once or auto-regressively over multiple latents. In this paper, we study how to make diffusion on continuous motion latents work best. We focus on two questions: (1) how to build a semantically aligned latent space so diffusion becomes more effective, and (2) how to best inject text conditioning so the motion follows the description closely. We propose a semantic-aligned motion encoder trained with frame-level text labels so that latents with similar text meaning stay close, which makes the latent space more diffusion-friendly. We also compare single-token conditioning with a multi-token cross-attention scheme and find that cross-attention gives better motion realism and text-motion alignment. With semantically aligned latents, auto-regressive generation, and cross-attention text conditioning, our model sets a new state of the art in human motion generation on standard metrics and in a user study. We will release our code and models for further research and downstream usage.
Yannan He, Garvita Tiwari, Xiaohan Zhang +4
University of Tübingen, Germany · Tübingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany +2
Text-to-motion generation and motion-to-text captioning are two fundamental tasks in human motion modeling, both grounded in the same underlying motion-text correspondence. Existing unified approaches mostly rely on autoregressive modeling, which imposes a fixed generation order and is therefore poorly suited to the bidirectional dependencies between language and motion, allowing early prediction errors to persist as fixed context and degrade both temporal coherence and cross-modal consistency. Masked discrete diffusion, which models sequences through iterative bidirectional prediction, offers a natural remedy. We therefore propose BiMoGen (Bidirectional Motion-text Generation), a unified masked discrete diffusion framework for bidirectional motion-text modeling. To stabilize training, we design Decoupled Uni- and Cross-Modal Training, in which masked pretraining first establishes cross-modal correspondence on paired motion-text sequences, after which supervised fine-tuning specializes the model for bidirectional generation. Masked diffusion nonetheless introduces its own source of error, as the model is trained on clean ground-truth context yet encounters self-generated and potentially erroneous context at inference, with errors committed under heavily masked states propagating through subsequent steps. We further introduce Generation-Aware Self-Correction that exposes the model to its own predictions during training and applies correction passes at early sampling steps to revise unreliably committed tokens. Extensive experiments on HumanML3D and KIT-ML demonstrate competitive performance on both tasks, validating the effectiveness of the proposed two-stage training and self-correction designs. The project page is available at https://wengwanjiang.github.io/BiMoGen-Page.
Wanjiang Weng, Yongliang Wu, Xiaofeng Tan +3
Department of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, Nanjing, China · National University of Singapore +1
Text-driven human motion generation aims to synthesize realistic motion sequences that follow textual descriptions. Despite recent advances, accurately aligning motion dynamics with textual semantics remains a fundamental challenge. In this paper, we revisit text-to-motion generation from the perspective of motion-language alignment and propose MLA-Gen, a framework that integrates global motion priors with fine-grained local conditioning. This design enables the model to capture common motion patterns, while establishing detailed alignment between texts and motions. Furthermore, we identify a previously overlooked attention sink phenomenon in human motion generation, where attention disproportionately concentrates on the start text token, limiting the utilization of informative textual cues and leading to degraded semantic grounding. To analyze this issue, we introduce SinkRatio, a metric for measuring attention concentration, and develop alignment-aware masking and control strategies to regulate attention during generation. Extensive experiments demonstrate that our approach consistently improves both motion quality and motion-language alignment over strong baselines.
Ruxi Gu, Zilei Wang, Wei Wang
Department of Automation, University of Science and Technology of China, Hefei, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China