Open-UniMo: Towards Unified Motion-Language Understanding and Generation in the Open World
Authors: Guocun Wang, Kenkun Liu, Guorui Song, Jing Lin, Zhe Huang, Luyuan Zhang, Dake Zhong, Choo Sin Wai, +2 more
Organizations: Tsinghua University, Beijing 100190, China · SSE, The Chinese University of Hong Kong, Shenzhen 518172, China · MMLab, Nanyang Technological University, Singapore 639798
Unified motion generation and understanding is crucial for embodied AI systems that can both synthesize and interpret human actions in open-world environments. Existing motion-language models often treat motion as an auxiliary modality of a language model, leading to text-dominated representations and limited cross-modal interaction. Moreover, the next-token prediction paradigm is not naturally suited to long motion sequences, where autoregressive generation may accumulate prediction errors. To address these challenges, we propose Open-UniMo, a unified Large Motion-Language Model (LMLM) trained on million-scale open-world motion-language data. Open-UniMo promotes modality parity by extending Qwen's vocabulary of about 150K text tokens with 64K motion tokens, enabling motion and language to share a unified token space. We further introduce motion-consistent Chain-of-Thought reasoning as an intermediate representation to bridge language semantics and motion dynamics. Open-UniMo is trained with a two-stage pipeline, where supervised fine-tuning establishes CoT-guided bidirectional motion-language mapping and Group Relative Policy Optimization (GRPO) improves semantic alignment while mitigating cumulative errors in autoregressive motion-token generation. To support comprehensive evaluation, we propose Open-MoBench, a VLM-guided benchmark for assessing text-to-motion (T2M) generation, motion-to-text (M2T) understanding, and bidirectional consistency. Extensive experiments show that Open-UniMo achieves state-of-the-art performance on both conventional metrics and Open-MoBench. Furthermore, ablation studies reveal that M2T understanding is not primarily limited by motion-token vocabulary size; instead, coupling M2T with the learnable T2M generation path yields stronger cross-modal representations, demonstrating that generation can facilitate understanding in AR-based motion-language modeling.
Figures & tables
Fig. 2: Overview of the Open-UniMo framework. Open-UniMo is trained in two stages with CoT reasoning. In the SFT stage, the model learns CoT-guided bidirectional motion-language mappings for both T2M and M2T tasks. In the GRPO stage, reinforcement learning further refines format compliance and motion-language alignment through multiple reward signals.
Fig. 3: Illustration of the CoT annotation process. Human joint sequences are rendered in Blender and paired with captions, which are further processed by Qwen2.5-VL-72B to generate reasoning traces.
Fig. 4: Qualitative comparison on Open-MoBench for the T2M task. Compared with existing representative approaches [ 11 , 8 , 18 ] , Open-UniMo generates motions that better capture semantic details and natural dynamics.
Fig. 5: Qualitative comparison on Open-MoBench for the M2T task. Compared with open-source models [ 15 , 11 , 38 ] , Open-UniMo generates descriptions that are more coherent, contextually relevant, and semantically faithful to the input motions.
Methods
R-Precision ↑
FID ↓
MM-Dist ↓
Diversity ↑
Top1
Top2
Top3
ScaMo [ 19 ]
0.670
0.810
0.870
89.000
-
-
GoToZero [ 18 ]
0.790
0.905
0.944
10.568
24.574
46.272
Open-UniMo SFT (w/o CoT)
0.786
0.894
0.933
10.909
24.281
46.385
Open-UniMo SFT+GRPO (w/o CoT) ▽
0.812
0.915
0.950
10.444
23.628
46.457
Open-UniMo SFT+GRPO (w/o CoT)
0.814
0.917
0.951
10.526
23.555
46.495
TABLE I: Quantitative results of the T2M task on the MotionMillion dataset. The best scores are highlighted in bold, and the second-best scores are underlined. Results marked with ▽ denote variants trained with an additional symmetric M2T reward that mirrors the T2M reward design.
Methods
BLEU-1
BLEU-4
ROUGE-L
CIDEr
BertScore
TM2T [ 38 ]
28.60
2.46
27.34
3.59
27.20
MotionGPT [ 15 ]
34.69
4.33
25.01
5.53
31.81
MotionAgent [ 11 ]
36.77
5.75
27.86
6.73
35.96
Open-UniMo SFT (w/o CoT)
52.80
11.78
31.96
10.51
40.29
Open-UniMo SFT+GRPO (w/o CoT) ▽
51.22
10.80
31.60
9.55
39.68
Open-UniMo SFT+GRPO (w/o CoT)
52.63
11.97
32.35
11.48
40.72
TABLE II: Quantitative results of the M2T task on the M2T subset of Open-MoBench. The best scores are highlighted in bold, and the second-best scores are underlined. Results marked with ▽ denote variants trained with an additional symmetric M2T reward that mirrors the T2M reward design. Results marked with ◊ indicate evaluation on the full validation set of the MotionMillion dataset.
Methods
R-Precision ↑
FID ↓
MM-Dist ↓
Diversity ↑
MModality ↑
Top1
Top2
Top3
MDM [ 44 ]
0.320±0.005
0.498±0.004
0.611±0.007
0.544±0.044
5.566±0.027
9.559±0.086
2.799 ±0.074
MLD [ 6 ]
0.481±0.003
0.673±0.003
0.772±0.002
0.473±0.013
3.196±0.010
9.724±0.082
2.413±0.072
MotionDiffuse [ 7 ]
0.491±0.001
0.681±0.001
0.782±0.001
0.630±0.001
3.113±0.001
9.410±0.049
1.553±0.064
T2M [ 9 ]
0.457±0.002
0.559±0.007
0.740±0.003
1.067±0.002
3.340±0.008
9.188±0.002
2.090±0.088
TM2T [ 38 ]
0.424±0.003
0.618±0.003
0.729±0.002
1.501±0.017
3.467±0.011
8.589±0.076
2.424 ±0.079
TABLE III: Quantitative results of the T2M task on the HumanML3D dataset. Each evaluation is repeated 20 times with average metrics and 95% confidence intervals. The best scores are highlighted in bold, and the second-best scores are underlined.
Methods
B@1↑
B@4↑
R-L↑
CIDEr↑
Bert↑
TM2T [ 38 ]
48.9
8.27
38.1
15.8
32.2
LaMPM2T [ 79 ]
47.8
13.04
37.1
28.9
32.7
MoTe [ 80 ]
46.7
11.15
37.4
31.5
30.3
MotionGPT [ 15 ]
48.2
12.47
37.4
29.2
32.4
MotionGPT-2 [ 16 ]
48.7
13.80
37.6
29.8
32.6
MotionChain [ 48 ]
48.1
12.56
33.9
33.7
36.9
TABLE IV: Quantitative results of the M2T task on the HumanML3D dataset. The best scores are highlighted in bold, and the second-best scores are underlined.
CoT
rmotion
rsemantic
rformat
R-Precision ↑
FID ↓
MM-Dist ↓
Diversity ↑
B@1 ↑
B@4 ↑
R-L ↑
CIDEr ↑
Bert ↑
Top1
Top2
Top3
0.786
0.894
0.933
10.909
24.281
46.385
53.85
12.27
32.67
11.73
44.04
✓
✓
0.809
0.914
0.949
10.440
23.673
46.789
54.83
12.82
33.19
12.93
44.67
✓
✓
0.807
0.915
0.949
10.623
23.775
46.195
54.83
12.64
33.01
12.69
44.53
✓
✓
0.797
0.906
0.944
10.672
23.940
46.504
54.68
12.67
33.04
12.56
44.50
✓
✓
✓
0.814
0.917
0.951
10.526
23.555
46.495
55.02
12.85
33.17
12.87
44.61
TABLE V: Ablation study on the effectiveness of CoT and different GRPO reward components on the full MotionMillion validation set.
Train Task
Stage
R-Precision ↑
FID ↓
MM-Dist ↓
Div ↑
Top1
Top2
Top3
T2M
SFT
0.438
0.613
0.702
0.201
3.588
9.748
T2M
SFT+RL
0.529
0.739
0.832
0.203
2.743
9.780
T2M+M2T
SFT
0.460
0.642
0.735
0.292
3.360
9.726
T2M+M2T
SFT+RL
0.539
0.738
0.831
0.177
2.768
10.042
TABLE VI: Ablation study of the synergy effect of unified modeling on the HumanML3D and MotionMillion Datasets. All experiments on HumanML3D are conducted with CoT. Results on MotionMillion additionally compare settings with and without CoT.
Vocabulary Size
B@1 ↑
B@4 ↑
R-L ↑
CIDEr ↑
Bert ↑
65536
34.32
2.61
22.87
1.35
32.17
16384
36.28
2.84
24.04
1.31
33.39
4096
36.16
3.22
23.21
1.01
31.48
2048
33.42
2.59
21.68
0.93
27.85
512
37.95
2.95
20.92
1.20
29.67
TABLE VII: Ablation study on the impact of motion token vocabulary size for the M2T task on the full validation set of the MotionMillion dataset.
Methods
Phy. Plaus.
Sem. Align.
Temp. Coh.
Avg.
Bi. Consis.
GoToZero [ 18 ]
3.141
1.903
2.197
2.414
-
Open-UniMo SFT (w/o CoT)
3.218
1.871
2.229
2.439
1.587
Open-UniMo SFT+GRPO (w/o CoT)
3.189
1.912
2.272
2.458
1.686
Open-UniMo SFT+GRPO
3.281
1.990
2.312
2.528
1.742
TABLE VIII: Quantitative results of the T2M task on the Open-MoBench. The best scores are highlighted in bold. Phy. Plaus. : Physical Plausibility (0-5); Sem. Align. : Semantic Alignment (0-5); Temp. Coh. : Temporal Coherence (0-5); Avg. : Average Score (0-5); Bi. Consis. : Bidirectional Consistency (0-5).
Methods
Detail Consis.
Sem. Align.
Temp. Coh.
Avg.
Bi. Consis.
TM2T [ 38 ]
0.995
1.145
1.301
1.147
1.714
MotionGPT [ 15 ]
1.082
1.182
1.383
1.216
1.862
MotionAgent [ 11 ]
1.133
1.302
1.491
1.309
1.813
Open-UniMo SFT (w/o CoT)
2.346
2.976
3.096
2.806
2.369
Open-UniMo SFT+GRPO (w/o CoT)
2.433
3.052
3.233
2.906
2.362
Open-UniMo SFT+GRPO
2.439
3.077
3.308
2.941
2.288
TABLE IX: Quantitative results of the M2T task on the Open-MoBench. The best scores are highlighted in bold. Detail Consis. : Detail Consistency (0-5); Sem. Align. : Semantic Alignment (0-5); Temp. Coh. : Temporal Coherence (0-5); Avg. : Average Score (0-5); Bi. Consis. : Bidirectional Consistency (0-5).
Fig. 6: Spearman rank correlation between human preference and VLM-as-Judge evaluation on Open-MoBench. Each point represents one model, where the x-axis denotes the human preference win ratio and the y-axis denotes the VLM evaluation win ratio under the same task and evaluation dimension. The fitted line visualizes the ranking trend, and ρ denotes the Spearman rank correlation between model-level human and VLM rankings.
Fig. 7: Left–right limb annotation for VLM-based motion evaluation. Blender-rendered motion frames are processed by YOLOv11-Pose [ 74 ] , with green indicating left body limbs and red indicating right body limbs.
Fig. 8: t-SNE visualization of the M2T → T2M bidirectional consistency evaluation in the motion feature space. Open-UniMo ( green ) shows stronger overlap with ground-truth clusters ( blue ), suggesting better preservation of motion semantics.
Fig. 9: A comparative word cloud analysis of the T2M evaluation in Open-MoBench (top) and HumanML3D [ 9 ] (bottom).
Fig. 10: Qualitative comparison of our method with prior models [ 15 , 11 ] on bidirectional consistency. Our approach better maintains semantic alignment between text and motion across both T2M and M2T directions.
Fig. 11: Human preference testing interface. Annotators compare anonymized model outputs under the same criterion and select the preferred result.
Fig. 12: Prompt templates for CoT annotation and model training. The prompts define structured reasoning and task-specific output formats for unified T2M generation and M2T understanding.
Fig. 13: Prompt templates for Open-MoBench VLM-as-Judge evaluation, covering T2M generation, M2T understanding, and bidirectional consistency scoring.
We present UniMotion, to our knowledge the first unified framework for simultaneous understanding and generation of human motion, natural language, and RGB images within a single architecture. Existing unified models handle only restricted modality subsets (e.g., Motion-Text or static Pose-Image) and predominantly rely on discrete tokenization, which introduces quantization errors and disrupts temporal continuity. UniMotion overcomes both limitations through a core principle: treating motion as a first-class continuous modality on equal footing with RGB. A novel Cross-Modal Aligned Motion VAE (CMA-VAE) and symmetric dual-path embedders construct parallel continuous pathways for Motion and RGB within a shared LLM backbone. To inject visual-semantic priors into motion representations without requiring images at inference, we propose Dual-Posterior KL Alignment (DPA), which distills a vision-fused encoder's richer posterior into the motion-only encoder. To address the cold-start problem -- where text supervision alone is too sparse to calibrate the newly introduced motion pathway -- we further propose Latent Reconstruction Alignment (LRA), a self-supervised pre-training strategy that uses dense motion latents as unambiguous conditions to co-calibrate the embedder, backbone, and flow head, establishing a stable motion-aware foundation for all downstream tasks. UniMotion achieves state-of-the-art performance across seven tasks spanning any-to-any understanding, generation, and editing among the three modalities, with especially strong advantages on cross-modal compositional tasks.
Ziyi Wang, Xinshun Wang, Shuang Chen +2
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China · Donghua University, China · School of Automation Science and Engineering, South China University of Technology
Text-to-motion (T2M) generation aims to control the behavior of a target character via textual descriptions. Leveraging text-motion paired datasets, existing T2M models have achieved impressive performance in generating high-quality motions within the distribution of their training data. However, their performance deteriorates notably when the motion descriptions differ significantly from the training texts. To address this issue, we propose Re2MoGen, a Reasoning and Refinement open-vocabulary Motion Generation framework that leverages enhanced Large Language Model (LLM) reasoning to generate an initial motion planning and then refine its physical plausibility via reinforcement learning (RL) post-training. Specifically, Re2MoGen consists of three stages: We first employ Monte Carlo tree search to enhance the LLM's reasoning ability in generating reasonable keyframes of the motion based on text prompts, specifying only the root and several key joints' positions to ease the reasoning process. Then, we apply a human pose model as a prior to optimize the full-body poses based on the planned keyframes and use the resulting incomplete motion to supervise fine-tuning a pre-trained motion generator via a dynamic temporal matching objective, enabling spatiotemporal completion. Finally, we use post-training with physics-aware reward to refine motion quality to eliminate physical implausibility in LLM-planned motions. Extensive experiments demonstrate that our framework can generate semantically consistent and physically plausible motions and achieve state-of-the-art performance in open-vocabulary motion generation.
Jiakun Zheng, Ting Xiao, Shiqin Cao +3
1East China University of Science and Technology · 2Hong Kong University of Science and Technology · Institute of Artificial Intelligence (TeleAI), China Telecom
Recent advances in motion-aware large language models have shown remarkable promise for jointly learning motion understanding and generation knowledge. However, these models typically treat understanding and generation separately, limiting the mutual benefits that could arise from interactive feedback between tasks. In this work, we reveal that motion assessment and refinement tasks can act as crucial bridges to enable knowledge flow from motion understanding to generation. Specifically, we propose Interleaved Reasoning for Motion Generation (IRMoGen), a novel paradigm that tightly couples motion generation with assessment and refinement through iterative text-motion dialogue. To realize this, we introduce IRG-MotionLLM, the first model that seamlessly interleaves motion generation, assessment, and refinement to improve the alignment between generated motion and goal text. IRG-MotionLLM is developed progressively with a novel three-stage training scheme, initializing and subsequently enhancing native IRMoGen capabilities. To facilitate this development, we construct an automated data engine to synthesize interleaved reasoning annotations from existing text-motion datasets. Extensive experiments demonstrate the properties brought by IRMoGen training, and the advanced cross-benchmark and cross-evaluator performance of IRG-MotionLLM. Code and models are available at https://github.com/HumanMLLM/IRG-MotionLLM.
Yuan-Ming Li, Qize Yang, Nan Lei +5
Sun Yat-sen University · Tongyi Lab, Alibaba Group · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China +1