Open-UniMo: Towards Unified Motion-Language Understanding and Generation in the Open World

Guocun Wang; Kenkun Liu; Guorui Song; Jing Lin; Zhe Huang; Luyuan Zhang; Dake Zhong; Choo Sin Wai

cs.CVSep 13, 2026

Open-UniMo: Towards Unified Motion-Language Understanding and Generation in the Open World

Authors: Guocun Wang, Kenkun Liu, Guorui Song, Jing Lin, Zhe Huang, Luyuan Zhang, Dake Zhong, Choo Sin Wai, +2 more

Organizations: Tsinghua University, Beijing 100190, China · SSE, The Chinese University of Hong Kong, Shenzhen 518172, China · MMLab, Nanyang Technological University, Singapore 639798

Abstract

Unified motion generation and understanding is crucial for embodied AI systems that can both synthesize and interpret human actions in open-world environments. Existing motion-language models often treat motion as an auxiliary modality of a language model, leading to text-dominated representations and limited cross-modal interaction. Moreover, the next-token prediction paradigm is not naturally suited to long motion sequences, where autoregressive generation may accumulate prediction errors. To address these challenges, we propose Open-UniMo, a unified Large Motion-Language Model (LMLM) trained on million-scale open-world motion-language data. Open-UniMo promotes modality parity by extending Qwen's vocabulary of about 150K text tokens with 64K motion tokens, enabling motion and language to share a unified token space. We further introduce motion-consistent Chain-of-Thought reasoning as an intermediate representation to bridge language semantics and motion dynamics. Open-UniMo is trained with a two-stage pipeline, where supervised fine-tuning establishes CoT-guided bidirectional motion-language mapping and Group Relative Policy Optimization (GRPO) improves semantic alignment while mitigating cumulative errors in autoregressive motion-token generation. To support comprehensive evaluation, we propose Open-MoBench, a VLM-guided benchmark for assessing text-to-motion (T2M) generation, motion-to-text (M2T) understanding, and bidirectional consistency. Extensive experiments show that Open-UniMo achieves state-of-the-art performance on both conventional metrics and Open-MoBench. Furthermore, ablation studies reveal that M2T understanding is not primarily limited by motion-token vocabulary size; instead, coupling M2T with the learnable T2M generation path yields stronger cross-modal representations, demonstrating that generation can facilitate understanding in AR-based motion-language modeling.

Figures & tables

Fig. 2: Overview of the Open-UniMo framework. Open-UniMo is trained in two stages with CoT reasoning. In the SFT stage, the model learns CoT-guided bidirectional motion-language mappings for both T2M and M2T tasks. In the GRPO stage, reinforcement learning further refines format compliance and motion-language alignment through multiple reward signals.

Fig. 3: Illustration of the CoT annotation process. Human joint sequences are rendered in Blender and paired with captions, which are further processed by Qwen2.5-VL-72B to generate reasoning traces.

Fig. 4: Qualitative comparison on Open-MoBench for the T2M task. Compared with existing representative approaches [ 11 , 8 , 18 ] , Open-UniMo generates motions that better capture semantic details and natural dynamics.

Fig. 5: Qualitative comparison on Open-MoBench for the M2T task. Compared with open-source models [ 15 , 11 , 38 ] , Open-UniMo generates descriptions that are more coherent, contextually relevant, and semantically faithful to the input motions.

Methods	R-Precision ↑			FID ↓	MM-Dist ↓	Diversity ↑
	Top1	Top2	Top3
ScaMo [ 19 ]	0.670	0.810	0.870	89.000	-	-
GoToZero [ 18 ]	0.790	0.905	0.944	10.568	24.574	46.272
Open-UniMo SFT (w/o CoT)	0.786	0.894	0.933	10.909	24.281	46.385
Open-UniMo SFT+GRPO (w/o CoT) ▽	0.812	0.915	0.950	10.444	23.628	46.457
Open-UniMo SFT+GRPO (w/o CoT)	0.814	0.917	0.951	10.526	23.555	46.495

TABLE I: Quantitative results of the T2M task on the MotionMillion dataset. The best scores are highlighted in bold, and the second-best scores are underlined. Results marked with

\triangledown

denote variants trained with an additional symmetric M2T reward that mirrors the T2M reward design.

Methods	BLEU-1	BLEU-4	ROUGE-L	CIDEr	BertScore
TM2T [ 38 ]	28.60	2.46	27.34	3.59	27.20
MotionGPT [ 15 ]	34.69	4.33	25.01	5.53	31.81
MotionAgent [ 11 ]	36.77	5.75	27.86	6.73	35.96
Open-UniMo SFT (w/o CoT)	52.80	11.78	31.96	10.51	40.29
Open-UniMo SFT+GRPO (w/o CoT) ▽	51.22	10.80	31.60	9.55	39.68
Open-UniMo SFT+GRPO (w/o CoT)	52.63	11.97	32.35	11.48	40.72

TABLE II: Quantitative results of the M2T task on the M2T subset of Open-MoBench. The best scores are highlighted in bold, and the second-best scores are underlined. Results marked with

\triangledown

denote variants trained with an additional symmetric M2T reward that mirrors the T2M reward design. Results marked with

\lozenge

indicate evaluation on the full validation set of the MotionMillion dataset.

Methods	R-Precision ↑			FID ↓	MM-Dist ↓	Diversity ↑	MModality ↑
	Top1	Top2	Top3
MDM [ 44 ]	$0.320^{\pm 0.005}$	$0.498^{\pm 0.004}$	$0.611^{\pm 0.007}$	$0.544^{\pm 0.044}$	$5.566^{\pm 0.027}$	$9.559^{\pm 0.086}$	2.799 ±0.074
MLD [ 6 ]	$0.481^{\pm 0.003}$	$0.673^{\pm 0.003}$	$0.772^{\pm 0.002}$	$0.473^{\pm 0.013}$	$3.196^{\pm 0.010}$	$9.724^{\pm 0.082}$	$2.413^{\pm 0.072}$
MotionDiffuse [ 7 ]	$0.491^{\pm 0.001}$	$0.681^{\pm 0.001}$	$0.782^{\pm 0.001}$	$0.630^{\pm 0.001}$	$3.113^{\pm 0.001}$	$9.410^{\pm 0.049}$	$1.553^{\pm 0.064}$
T2M [ 9 ]	$0.457^{\pm 0.002}$	$0.559^{\pm 0.007}$	$0.740^{\pm 0.003}$	$1.067^{\pm 0.002}$	$3.340^{\pm 0.008}$	$9.188^{\pm 0.002}$	$2.090^{\pm 0.088}$
TM2T [ 38 ]	$0.424^{\pm 0.003}$	$0.618^{\pm 0.003}$	$0.729^{\pm 0.002}$	$1.501^{\pm 0.017}$	$3.467^{\pm 0.011}$	$8.589^{\pm 0.076}$	2.424 ±0.079

TABLE III: Quantitative results of the T2M task on the HumanML3D dataset. Each evaluation is repeated 20 times with average metrics and 95% confidence intervals. The best scores are highlighted in bold, and the second-best scores are underlined.

Methods	B@1↑	B@4↑	R-L↑	CIDEr↑	Bert↑
TM2T [ 38 ]	48.9	8.27	38.1	15.8	32.2
LaMPM2T [ 79 ]	47.8	13.04	37.1	28.9	32.7
MoTe [ 80 ]	46.7	11.15	37.4	31.5	30.3
MotionGPT [ 15 ]	48.2	12.47	37.4	29.2	32.4
MotionGPT-2 [ 16 ]	48.7	13.80	37.6	29.8	32.6
MotionChain [ 48 ]	48.1	12.56	33.9	33.7	36.9

TABLE IV: Quantitative results of the M2T task on the HumanML3D dataset. The best scores are highlighted in bold, and the second-best scores are underlined.

CoT	$r_{\text{motion}}$	$r_{\text{semantic}}$	$r_{\text{format}}$	R-Precision ↑			FID $\downarrow$	MM-Dist $\downarrow$	Diversity $\uparrow$	B@1 $\uparrow$	B@4 $\uparrow$	R-L $\uparrow$	CIDEr $\uparrow$	Bert $\uparrow$
CoT	$r_{\text{motion}}$	$r_{\text{semantic}}$	$r_{\text{format}}$	Top1	Top2	Top3	FID $\downarrow$	MM-Dist $\downarrow$	Diversity $\uparrow$	B@1 $\uparrow$	B@4 $\uparrow$	R-L $\uparrow$	CIDEr $\uparrow$	Bert $\uparrow$
				0.786	0.894	0.933	10.909	24.281	46.385	53.85	12.27	32.67	11.73	44.04
	$\checkmark$	$\checkmark$		0.809	0.914	0.949	10.440	23.673	46.789	54.83	12.82	33.19	12.93	44.67
	$\checkmark$		$\checkmark$	0.807	0.915	0.949	10.623	23.775	46.195	54.83	12.64	33.01	12.69	44.53
		$\checkmark$	$\checkmark$	0.797	0.906	0.944	10.672	23.940	46.504	54.68	12.67	33.04	12.56	44.50
	$\checkmark$	$\checkmark$	$\checkmark$	0.814	0.917	0.951	10.526	23.555	46.495	55.02	12.85	33.17	12.87	44.61

TABLE V: Ablation study on the effectiveness of CoT and different GRPO reward components on the full MotionMillion validation set.

Train Task	Stage	R-Precision $\uparrow$			FID $\downarrow$	MM-Dist $\downarrow$	Div $\uparrow$
		Top1	Top2	Top3
T2M	SFT	0.438	0.613	0.702	0.201	3.588	9.748
T2M	SFT+RL	0.529	0.739	0.832	0.203	2.743	9.780
T2M+M2T	SFT	0.460	0.642	0.735	0.292	3.360	9.726
T2M+M2T	SFT+RL	0.539	0.738	0.831	0.177	2.768	10.042

TABLE VI: Ablation study of the synergy effect of unified modeling on the HumanML3D and MotionMillion Datasets. All experiments on HumanML3D are conducted with CoT. Results on MotionMillion additionally compare settings with and without CoT.

Vocabulary Size	B@1 $\uparrow$	B@4 $\uparrow$	R-L $\uparrow$	CIDEr $\uparrow$	Bert $\uparrow$
65536	34.32	2.61	22.87	1.35	32.17
16384	36.28	2.84	24.04	1.31	33.39
4096	36.16	3.22	23.21	1.01	31.48
2048	33.42	2.59	21.68	0.93	27.85
512	37.95	2.95	20.92	1.20	29.67

TABLE VII: Ablation study on the impact of motion token vocabulary size for the M2T task on the full validation set of the MotionMillion dataset.

Methods	Phy. Plaus.	Sem. Align.	Temp. Coh.	Avg.	Bi. Consis.
GoToZero [ 18 ]	3.141	1.903	2.197	2.414	-
Open-UniMo SFT (w/o CoT)	3.218	1.871	2.229	2.439	1.587
Open-UniMo SFT+GRPO (w/o CoT)	3.189	1.912	2.272	2.458	1.686
Open-UniMo SFT+GRPO	3.281	1.990	2.312	2.528	1.742

TABLE VIII: Quantitative results of the T2M task on the Open-MoBench. The best scores are highlighted in bold. Phy. Plaus. : Physical Plausibility (0-5); Sem. Align. : Semantic Alignment (0-5); Temp. Coh. : Temporal Coherence (0-5); Avg. : Average Score (0-5); Bi. Consis. : Bidirectional Consistency (0-5).

Methods	Detail Consis.	Sem. Align.	Temp. Coh.	Avg.	Bi. Consis.
TM2T [ 38 ]	0.995	1.145	1.301	1.147	1.714
MotionGPT [ 15 ]	1.082	1.182	1.383	1.216	1.862
MotionAgent [ 11 ]	1.133	1.302	1.491	1.309	1.813
Open-UniMo SFT (w/o CoT)	2.346	2.976	3.096	2.806	2.369
Open-UniMo SFT+GRPO (w/o CoT)	2.433	3.052	3.233	2.906	2.362
Open-UniMo SFT+GRPO	2.439	3.077	3.308	2.941	2.288

TABLE IX: Quantitative results of the M2T task on the Open-MoBench. The best scores are highlighted in bold. Detail Consis. : Detail Consistency (0-5); Sem. Align. : Semantic Alignment (0-5); Temp. Coh. : Temporal Coherence (0-5); Avg. : Average Score (0-5); Bi. Consis. : Bidirectional Consistency (0-5).

Fig. 6: Spearman rank correlation between human preference and VLM-as-Judge evaluation on Open-MoBench. Each point represents one model, where the x-axis denotes the human preference win ratio and the y-axis denotes the VLM evaluation win ratio under the same task and evaluation dimension. The fitted line visualizes the ranking trend, and $\rho$ denotes the Spearman rank correlation between model-level human and VLM rankings.

Fig. 7: Left–right limb annotation for VLM-based motion evaluation. Blender-rendered motion frames are processed by YOLOv11-Pose [ 74 ] , with green indicating left body limbs and red indicating right body limbs.

Fig. 8: t-SNE visualization of the M2T $\rightarrow$ T2M bidirectional consistency evaluation in the motion feature space. Open-UniMo ( green ) shows stronger overlap with ground-truth clusters ( blue ), suggesting better preservation of motion semantics.

Panel of Figure 17 — Fig. 9: A comparative word cloud analysis of the T2M evaluation in Open-MoBench (top) and HumanML3D [ 9 ] (bottom).

Fig. 10: Qualitative comparison of our method with prior models [ 15 , 11 ] on bidirectional consistency. Our approach better maintains semantic alignment between text and motion across both T2M and M2T directions.

Fig. 11: Human preference testing interface. Annotators compare anonymized model outputs under the same criterion and select the preferred result.

Fig. 12: Prompt templates for CoT annotation and model training. The prompts define structured reasoning and task-specific output formats for unified T2M generation and M2T understanding.

Fig. 13: Prompt templates for Open-MoBench VLM-as-Judge evaluation, covering T2M generation, M2T understanding, and bidirectional consistency scoring.

Explore similar work

UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation
Mar 23, 2026Ziyi Wang, Xinshun Wang, Shuang Chen +2Cross-Modal Learning Unified Multimodal Models
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China · Donghua University, China · School of Automation Science and Engineering, South China University of Technology
Re $^2$ MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement
Apr 20, 2026Jiakun Zheng, Ting Xiao, Shiqin Cao +3Text-to-Motion Generation Human Motion Generation
1East China University of Science and Technology · 2Hong Kong University of Science and Technology · Institute of Artificial Intelligence (TeleAI), China Telecom
IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation
Dec 11, 2025Yuan-Ming Li, Qize Yang, Nan Lei +5Text-to-Motion Generation
Sun Yat-sen University · Tongyi Lab, Alibaba Group · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China +1

Mar 23, 2026J/K move · Enter open · S save

UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

Ziyi Wang, Xinshun Wang, Shuang Chen, Yang Cong+1

State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China · Donghua University, China · School of Automation Science and Engineering, South China University of Technology

Cross-Modal Learning Unified Multimodal Models Cross-Modal Alignment Human Motion Generation Multimodal Foundation Models Multimodal Understanding

Open paper AskPDF arXiv

We present UniMotion, to our knowledge the first unified framework for simultaneous understanding and generation of human motion, natural language, and RGB images within a single architecture. Existing unified models handle only restricted modality subsets (e.g., Motion-Text or static Pose-Image) and predominantly rely on discrete tokenization, which introduces quantization errors and disrupts temporal continuity. UniMotion overcomes both limitations through a core principle: treating motion as a first-class continuous modality on equal footing with RGB. A novel Cross-Modal Aligned Motion VAE (CMA-VAE) and symmetric dual-path embedders construct parallel continuous pathways for Motion and RGB within a shared LLM backbone. To inject visual-semantic priors into motion representations without requiring images at inference, we propose Dual-Posterior KL Alignment (DPA), which distills a vision-fused encoder's richer posterior into the motion-only encoder. To address the cold-start problem -- where text supervision alone is too sparse to calibrate the newly introduced motion pathway -- we further propose Latent Reconstruction Alignment (LRA), a self-supervised pre-training strategy that uses dense motion latents as unambiguous conditions to co-calibrate the embedder, backbone, and flow head, establishing a stable motion-aware foundation for all downstream tasks. UniMotion achieves state-of-the-art performance across seven tasks spanning any-to-any understanding, generation, and editing among the three modalities, with especially strong advantages on cross-modal compositional tasks.

Abstract

Explore similar work

UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

Re2^22MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement

IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation

Re $^2$ MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement