Bilingual Text-to-Motion Generation: A New Benchmark and Baselines

Wanjiang Weng; Xiaofeng Tan; Xiangbo Shu; Guo-Sen Xie; Pan Zhou; Hongsong Wang

cs.CVMar 26, 2026

Bilingual Text-to-Motion Generation: A New Benchmark and Baselines

Authors: Wanjiang Weng, Xiaofeng Tan, Xiangbo Shu, Guo-Sen Xie, Pan Zhou, Hongsong Wang

Organizations: School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Nanjing University of Science and Technology · Singapore Management University

Abstract

Text-to-motion generation holds significant potential for cross-linguistic applications, yet it is hindered by the lack of bilingual datasets and the poor cross-lingual semantic understanding of existing language models. To address these gaps, we introduce BiHumanML3D, the first bilingual text-to-motion benchmark, constructed via LLM-assisted annotation and rigorous manual correction. Furthermore, we propose a simple yet effective baseline, Bilingual Motion Diffusion (BiMD), featuring Cross-Lingual Alignment (CLA). CLA explicitly aligns semantic representations across languages, creating a robust conditional space that enables high-quality motion generation from bilingual inputs, including zero-shot code-switching scenarios. Extensive experiments demonstrate that BiMD with CLA achieves an FID of 0.045 vs. 0.169 and R@3 of 82.8% vs. 80.8%, significantly outperforms monolingual diffusion models and translation baselines on BiHumanML3D, underscoring the critical necessity and reliability of our dataset and the effectiveness of our alignment strategy for cross-lingual motion synthesis. The dataset and code are released at https://wengwanjiang.github.io/BilingualT2M-page

Figures & tables

Figure 1: Code-switching in bilingual text-to-motion generation. Existing text-to-motion methods (e.g., MLD [ 2 ] and MotionLCM [ 6 ] ) struggle to interpret mixed-language inputs, leading to incorrect motion semantics, while our BiMD generates semantically consistent motions.

Figure 2: Visual comparison of bilingual text-to-motion generation. Existing methods such as MDM [ 31 ] and MLD [ 2 ] exhibit limitations in bilingual processing. Directly using pretrained multilingual encoders results in imbalanced performance across languages. Our BiMD with Cross-Lingual Alignment (CLA) generates accurate motions from both English and Chinese descriptions.

Figure 3: Pipeline for constructing our bilingual HumanML3D dataset. The data collection and filtering process removes unsuitable motions, ensuring high-quality motion-text pairs for annotation. The annotation pipeline begins with an initial translation stage, followed by a refinement stage to address translation issues. Finally, human annotators manually verify and correct the translation with LLM, ensuring linguistic and contextual accuracy.

Figure 4: Framework for training the bilingual motion diffusion model. We align English and Chinese text embeddings in a shared latent space by freezing the teacher model $E^{t}_{\Phi}$ and fine-tuning the student model $E^{s}_{\phi}$ with the cross-lingual alignment loss $\mathcal{L}_{CLA}$ in Eq. ( 1 ). The aligned student model $E^{s}_{\phi}$ then provides text conditions for training the diffusion model $\epsilon_{\theta}$ , enabling bilingual motion generation while minimizing $\mathcal{L}_{\text{BiMD}}$ in Eq. ( 6 ).

Figure 5: Visual comparison of zero‑shot cross‑lingual motion generation from Chinese‑only training to unseen English texts. We find that BiMD with CLA generates accurate, semantically aligned motions despite never having seen English during training.

Method	CLA	Lang.	R Precision $\uparrow$			FID $\downarrow$	MM Dist $\downarrow$	Div $\rightarrow$
Method	CLA	Lang.	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist $\downarrow$	Div $\rightarrow$
Real	-	CN	0.543	0.732	0.821	0.002	3.338	10.750
Real	-	EN	0.511	0.703	0.797	0.002	2.974	9.503
Mo.Diffuse [ 43 ]	✗	CN	0.468 ±0.002	0.652 ±0.003	0.764 ±0.004	1.024 ±0.011	3.512 ±0.007	11.586 ±0.078
	✗	EN	0.476 ±0.001	0.666 ±0.004	0.775 ±0.003	0.991 ±0.028	3.356 ±0.006	9.972 ±0.076
	✓	CN	0.510 ±0.003	0.677 ±0.002	0.797 ±0.001	0.493 ±0.009	3.138 ±0.014	10.610 ±0.066

Table 2: Comparison of text-to-motion generation performance on the BiHumanML3D dataset. “Lang.” indicates the evaluated language. English and Chinese results are evaluated using the T2MT evaluator [ 9 ] and our proposed evaluator, respectively. Bold highlights the best results. The symbol “✓” at “CLA” denotes methods that employ our cross-lingual alignment representation, indicating the use of a unified model for generation, whereas the symbol “✗” means methods retrained with XLM as text encoder.

Text Source	R-Precision $\uparrow$			FID $\downarrow$	MMDist $\downarrow$	Div $\rightarrow$
Text Source	R1	R2	R3	FID $\downarrow$	MMDist $\downarrow$	Div $\rightarrow$
DeepL-API	0.377	0.585	0.687	0.776	4.763	10.766
GPT-Translated	0.387	0.593	0.700	0.721	4.401	10.396
DeepSeek-Translated	0.393	0.601	0.713	0.645	4.253	10.523
HumanML3D	0.481	0.673	0.772	0.473	3.196	9.724

Table 3: Performance comparison of the MLD model conditioned by translated English texts versus HumanML3D (native English texts).

Zero-shot MLD GT Win Rate 35.4 17.6 47.0 (a) Motion Preference (%)

w/ CLA w/o CLA Neither Win Rate 68.1 24.8 7.1 (b) Semantic Consistency (%)

Table 4: User study. (a) Motion preference: BiMD (Zero-shot) vs. MLD vs. ground truth on English prompts. (b) Semantic consistency for paired Chinese–English prompts with/without CLA. Percentages are averaged over prompts.

Train	Test	CLA	R Precision $\uparrow$			FID $\downarrow$	MM Dist $\downarrow$	Div $\rightarrow$
Train	Test	CLA	Top 1	Top 2	Top 3	FID $\downarrow$	MM Dist $\downarrow$	Div $\rightarrow$
Real	CN	–	0.543	0.732	0.821	0.002	3.338	10.750
Real	EN	–	0.511	0.703	0.797	0.002	2.974	9.503
EN	CN	✗	0.173	0.288	0.364	3.533	6.134	9.231
EN	CN	✓	0.425	0.608	0.701	0.072	3.584	9.720
CN	EN	✗	0.184	0.298	0.388	3.397	6.258	9.228

Table 5: Cross-lingual motion generation performance under different training and test language settings. Train and Test denote the training and evaluation languages, respectively. “CLA” indicates whether Cross-lingual Alignment is applied. The arrows

\uparrow

\downarrow

, and

\rightarrow

indicate higher, lower, and closer-to-real-motion values are better, respectively. Bold highlights the best results.

Figure 6: A translation failure case. Direct translation may omit motion-critical details (e.g., direction), resulting in under-specified descriptions. Our annotation preserves such information, leading to more precise motion semantics.

Panel of Figure 7 — Figure 7: t-SNE visualizations of text representations, showing the distribution of 1000 randomly selected text descriptions from the testing set of HumanML3D. Dots of the same color represent text of the same language, with potential alignment effects observed.

Explore similar work

MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation
Dec 15, 2025Yannan He, Garvita Tiwari, Xiaohan Zhang +4Latent Diffusion Models Text-to-Motion Generation
University of Tübingen, Germany · Tübingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany +2
BiMoGen: Bidirectional Motion-Text Generation via Unified Masked Discrete Diffusion
Sep 28, 2026Wanjiang Weng, Yongliang Wu, Xiaofeng Tan +3Masked Diffusion Models Text-to-Motion Generation
Department of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, Nanjing, China · National University of Singapore +1
Exploring Motion-Language Alignment for Text-driven Motion Generation
Apr 3, 2026Ruxi Gu, Zilei Wang, Wei WangText-to-Motion Generation Human Motion Generation
Department of Automation, University of Science and Technology of China, Hefei, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China

Dec 15, 2025J/K move · Enter open · S save

MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation

Yannan He, Garvita Tiwari, Xiaohan Zhang, Pankaj Bora+3

University of Tübingen, Germany · Tübingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany+2

Latent Diffusion Models Text-to-Motion Generation Cross-Modal Alignment Motion Diffusion Models Human Motion Generation Diffusion Models for Motion Generation

Open paper AskPDF arXiv

We introduce MoLingo, a text-to-motion (T2M) model that generates realistic, lifelike human motion by denoising in a continuous latent space. Recent works perform latent space diffusion, either on the whole latent at once or auto-regressively over multiple latents. In this paper, we study how to make diffusion on continuous motion latents work best. We focus on two questions: (1) how to build a semantically aligned latent space so diffusion becomes more effective, and (2) how to best inject text conditioning so the motion follows the description closely. We propose a semantic-aligned motion encoder trained with frame-level text labels so that latents with similar text meaning stay close, which makes the latent space more diffusion-friendly. We also compare single-token conditioning with a multi-token cross-attention scheme and find that cross-attention gives better motion realism and text-motion alignment. With semantically aligned latents, auto-regressive generation, and cross-attention text conditioning, our model sets a new state of the art in human motion generation on standard metrics and in a user study. We will release our code and models for further research and downstream usage.

Abstract

Explore similar work

MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation

BiMoGen: Bidirectional Motion-Text Generation via Unified Masked Discrete Diffusion

Exploring Motion-Language Alignment for Text-driven Motion Generation