Learn to Modify: Self-evolving Multi-agent Refinement Framework for Conversational Text-to-Motion Generation
Authors: Jianan Li, Jiawei Zhao, Yangtao Zhou, Zhifu Zhao, Shaoxin Wu, Yutao Wang, Dezhen Wang, Hua Chu, Fu Li
Organizations: School of Computer Science and Technology, Xidian University, Xi’an, China · School of Artificial Intelligence, Xidian University, Xi’an, China · School of Computer Science and Technology, Tongji University, Shanghai, China
Despite rapid progress in text-to-motion generation, existing methods are commonly designed for single-shot generation. When users are dissatisfied with specific motion details, they are often forced to regenerate the entire sequence from scratch, which is computationally inefficient and potentially destroys previously established kinematic context. A practical generation system should instead support iterative user feedback and refine only the affected spatio-temporal regions while preserving the remaining motion. Therefore, we introduce a conversational motion generation paradigm that requires the system to coordinate instruction understanding, motion generation, localized editing, and quality assessment. Recent advances in LLM-driven multi-agent systems provide a promising foundation for such role-specialized coordination. To tackle these challenges, a Self-evolving Multi-Agent RefinemenT Framework (SMART) is proposed for conversational motion generation. Within SMART, to preserve semantic consistency, we design a semantic-guided masked motion editing mechanism, which precisely grounds natural language feedback into spatiotemporal motion regions to enable localized refinements without altering unedited regions. To reduce redundant exploration, we propose a memory-based self-evolving strategy that dynamically accumulates motion editing knowledge from interaction history and reuses past successful experience. Extensive experiments demonstrate that SMART effectively supports multi-turn conversational motion generation and enables efficient motion refinement.
Figures & tables
Figure 1: Comparison of single-shot generation and iterative refinement. Conventional methods regenerate the entire motion, hindering fine-grained correction. Our multi-agent framework uses multi-turn feedback to progressively refine motion details while preserving established context.
Figure 2: Overview of SMART. (a) Iterative motion refinement through conversational motion generation. (b) Multi-agent collaboration among the Chatter, Generator, Editor, and Checker for instruction alignment, motion synthesis, localized editing, and semantic verification. (c) Self-evolving strategy that accumulates generation experiences for motion refinement. (d) Semantic-guided motion editing that localizes user feedback with a Vision-Language Model (VLM) and refines selected motion regions while preserving unmasked content.
Method
Multi-turn Dialogue
Motion Editing
Automatic Checking
Cross-turn Context
No Extra Training
Motion-Agent
✓
✓
✗
✓
✗
MotionChain
✓
✓
✗
✓
✗
MotionGPT-2
✗
✓
✗
✗
✗
MotionFix
✗
✓
✗
✗
✗
SMART
✓
✓
✓
✓
✓
Table 1: Capability comparison with representative motion generation and editing methods. ✓ and ✗ denote supported and unsupported capabilities, respectively.
Datasets
Methods
Top-1 ↑
Top-2 ↑
Top-3 ↑
FID ↓
MM Dist ↓
Diversity →
HumanML3D
Real Motion
0.511 ± .003
0.703 ± .003
0.797 ± .002
0.002 ± .000
2.974 ± .008
9.503 ± .065
MDM ( Tevet et al. 2022 )
0.320 ± .005
0.498 ± .004
0.611 ± .007
0.544 ± .044
5.566 ± .027
9.559 ± .086
MLD ( Chen et al. 2023 )
0.481 ± .003
0.673 ± .003
0.772 ± .002
0.473 ± .013
3.196 ± .010
9.724 ± .082
MotionDiffuse ( Zhang et al. 2024 )
0.491 ± .001
0.681 ± .001
0.782 ± .001
0.630 ± .001
3.113 ± .001
9.410 ± .049
T2M-GPT ( Zhang et al. 2023a )
0.491 ± .003
0.680 ± .003
0.775 ± .002
0.116 ± .004
3.118 ± .011
9.761 ± .081
MoMask ( Guo et al. 2024 )
0.521 ± .002
0.713 ± .002
0.807 ± .002
0.045 ± .002
2.958 ± .008
–
Table 2: Comparison results on the HumanML3D and KIT-ML test sets. The best results among generative methods are highlighted in bold. Higher is better for R-Precision, while lower is better for FID and MM Dist. Diversity values closer to Real Motion are better.
Cha.
Gen.
Che.
Edi.
R-Prec@1 ↑
FID ↓
Diversity →
✗
✓
✗
✗
0.485 ± .002
0.554 ± .015
9.557 ± .228
✗
✓
✓
✓
0.482 ± .004
0.578 ± .009
9.571 ± .234
✓
✓
✗
✓
0.483 ± .005
0.577 ± .010
9.571 ± .110
✓
✓
✓
✗
0.514 ± .005
0.531 ± .034
9.500 ± .522
✓
✓
✓
✓
0.535 ± .003
0.503 ± .009
9.662 ± .065
Table 3: Ablation study of multi-agent components.
Figure 3: Ablation results of memory-based refinement and editing strategies.
Figure 4: Illustrative example of the iterative motion refinement process, with semantic misalignment in red, semantic alignment in green, and personality instruction in purple.
Figure 5: Human evaluation results of MDM and SMART.
Figure 6: Evaluation of memory-based refinement and VLM-based motion assessment.
Figure 7: Qualitative comparison of different text-to-motion methods on compositional prompts. Key action phrases are highlighted in red.
Recent advances in motion-aware large language models have shown remarkable promise for jointly learning motion understanding and generation knowledge. However, these models typically treat understanding and generation separately, limiting the mutual benefits that could arise from interactive feedback between tasks. In this work, we reveal that motion assessment and refinement tasks can act as crucial bridges to enable knowledge flow from motion understanding to generation. Specifically, we propose Interleaved Reasoning for Motion Generation (IRMoGen), a novel paradigm that tightly couples motion generation with assessment and refinement through iterative text-motion dialogue. To realize this, we introduce IRG-MotionLLM, the first model that seamlessly interleaves motion generation, assessment, and refinement to improve the alignment between generated motion and goal text. IRG-MotionLLM is developed progressively with a novel three-stage training scheme, initializing and subsequently enhancing native IRMoGen capabilities. To facilitate this development, we construct an automated data engine to synthesize interleaved reasoning annotations from existing text-motion datasets. Extensive experiments demonstrate the properties brought by IRMoGen training, and the advanced cross-benchmark and cross-evaluator performance of IRG-MotionLLM. Code and models are available at https://github.com/HumanMLLM/IRG-MotionLLM.
Yuan-Ming Li, Qize Yang, Nan Lei +5
Sun Yat-sen University · Tongyi Lab, Alibaba Group · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China +1
Text-to-motion (T2M) generation aims to control the behavior of a target character via textual descriptions. Leveraging text-motion paired datasets, existing T2M models have achieved impressive performance in generating high-quality motions within the distribution of their training data. However, their performance deteriorates notably when the motion descriptions differ significantly from the training texts. To address this issue, we propose Re2MoGen, a Reasoning and Refinement open-vocabulary Motion Generation framework that leverages enhanced Large Language Model (LLM) reasoning to generate an initial motion planning and then refine its physical plausibility via reinforcement learning (RL) post-training. Specifically, Re2MoGen consists of three stages: We first employ Monte Carlo tree search to enhance the LLM's reasoning ability in generating reasonable keyframes of the motion based on text prompts, specifying only the root and several key joints' positions to ease the reasoning process. Then, we apply a human pose model as a prior to optimize the full-body poses based on the planned keyframes and use the resulting incomplete motion to supervise fine-tuning a pre-trained motion generator via a dynamic temporal matching objective, enabling spatiotemporal completion. Finally, we use post-training with physics-aware reward to refine motion quality to eliminate physical implausibility in LLM-planned motions. Extensive experiments demonstrate that our framework can generate semantically consistent and physically plausible motions and achieve state-of-the-art performance in open-vocabulary motion generation.
Jiakun Zheng, Ting Xiao, Shiqin Cao +3
1East China University of Science and Technology · 2Hong Kong University of Science and Technology · Institute of Artificial Intelligence (TeleAI), China Telecom
Generative models excel at motion synthesis for a fixed number of agents but struggle to generalize with variable agents. Based on limited, domain-specific data, existing methods employ autoregressive models to generate motion recursively, which suffer from inefficiency and error accumulation. We propose Unified Motion Flow (UMF), which consists of Pyramid Motion Flow (P-Flow) and Semi-Noise Motion Flow (S-Flow). UMF decomposes the number-free motion generation into a single-pass motion prior generation stage and multi-pass reaction generation stages. Specifically, UMF utilizes a unified latent space to bridge the distribution gap between heterogeneous motion datasets, enabling effective unified training. For motion prior generation, P-Flow operates on hierarchical resolutions conditioned on different noise levels, thereby mitigating computational overheads. For reaction generation, S-Flow learns a joint probabilistic path that adaptively performs reaction transformation and context reconstruction, alleviating error accumulation. Extensive results and user studies demonstrate UMF' s effectiveness as a generalist model for multi-person motion generation from text. Project page: https://githubhgh.github.io/umf/.