Learn to Modify: Self-evolving Multi-agent Refinement Framework for Conversational Text-to-Motion Generation

Jianan Li; Jiawei Zhao; Yangtao Zhou; Zhifu Zhao; Shaoxin Wu; Yutao Wang; Dezhen Wang; Hua Chu

cs.CVOct 20, 2026

Learn to Modify: Self-evolving Multi-agent Refinement Framework for Conversational Text-to-Motion Generation

Authors: Jianan Li, Jiawei Zhao, Yangtao Zhou, Zhifu Zhao, Shaoxin Wu, Yutao Wang, Dezhen Wang, Hua Chu,

Organizations: School of Computer Science and Technology, Xidian University, Xi’an, China · School of Artificial Intelligence, Xidian University, Xi’an, China · School of Computer Science and Technology, Tongji University, Shanghai, China

Abstract

Despite rapid progress in text-to-motion generation, existing methods are commonly designed for single-shot generation. When users are dissatisfied with specific motion details, they are often forced to regenerate the entire sequence from scratch, which is computationally inefficient and potentially destroys previously established kinematic context. A practical generation system should instead support iterative user feedback and refine only the affected spatio-temporal regions while preserving the remaining motion. Therefore, we introduce a conversational motion generation paradigm that requires the system to coordinate instruction understanding, motion generation, localized editing, and quality assessment. Recent advances in LLM-driven multi-agent systems provide a promising foundation for such role-specialized coordination. To tackle these challenges, a Self-evolving Multi-Agent RefinemenT Framework (SMART) is proposed for conversational motion generation. Within SMART, to preserve semantic consistency, we design a semantic-guided masked motion editing mechanism, which precisely grounds natural language feedback into spatiotemporal motion regions to enable localized refinements without altering unedited regions. To reduce redundant exploration, we propose a memory-based self-evolving strategy that dynamically accumulates motion editing knowledge from interaction history and reuses past successful experience. Extensive experiments demonstrate that SMART effectively supports multi-turn conversational motion generation and enables efficient motion refinement.

Figures & tables

Figure 1: Comparison of single-shot generation and iterative refinement. Conventional methods regenerate the entire motion, hindering fine-grained correction. Our multi-agent framework uses multi-turn feedback to progressively refine motion details while preserving established context.

Figure 2: Overview of SMART. (a) Iterative motion refinement through conversational motion generation. (b) Multi-agent collaboration among the Chatter, Generator, Editor, and Checker for instruction alignment, motion synthesis, localized editing, and semantic verification. (c) Self-evolving strategy that accumulates generation experiences for motion refinement. (d) Semantic-guided motion editing that localizes user feedback with a Vision-Language Model (VLM) and refines selected motion regions while preserving unmasked content.

Method	Multi-turn Dialogue	Motion Editing	Automatic Checking	Cross-turn Context	No Extra Training
Motion-Agent	✓	✓	✗	✓	✗
MotionChain	✓	✓	✗	✓	✗
MotionGPT-2	✗	✓	✗	✗	✗
MotionFix	✗	✓	✗	✗	✗
SMART	✓	✓	✓	✓	✓

Table 1: Capability comparison with representative motion generation and editing methods. ✓ and ✗ denote supported and unsupported capabilities, respectively.

Datasets	Methods	Top-1 $\uparrow$	Top-2 $\uparrow$	Top-3 $\uparrow$	FID $\downarrow$	MM Dist $\downarrow$	Diversity $\rightarrow$
HumanML3D	Real Motion	0.511 $\pm$ .003	0.703 $\pm$ .003	0.797 $\pm$ .002	0.002 $\pm$ .000	2.974 $\pm$ .008	9.503 $\pm$ .065
	MDM ( Tevet et al. 2022 )	0.320 $\pm$ .005	0.498 $\pm$ .004	0.611 $\pm$ .007	0.544 $\pm$ .044	5.566 $\pm$ .027	9.559 $\pm$ .086
	MLD ( Chen et al. 2023 )	0.481 $\pm$ .003	0.673 $\pm$ .003	0.772 $\pm$ .002	0.473 $\pm$ .013	3.196 $\pm$ .010	9.724 $\pm$ .082
	MotionDiffuse ( Zhang et al. 2024 )	0.491 $\pm$ .001	0.681 $\pm$ .001	0.782 $\pm$ .001	0.630 $\pm$ .001	3.113 $\pm$ .001	9.410 $\pm$ .049
	T2M-GPT ( Zhang et al. 2023a )	0.491 $\pm$ .003	0.680 $\pm$ .003	0.775 $\pm$ .002	0.116 $\pm$ .004	3.118 $\pm$ .011	9.761 $\pm$ .081
	MoMask ( Guo et al. 2024 )	0.521 $\pm$ .002	0.713 $\pm$ .002	0.807 $\pm$ .002	0.045 $\pm$ .002	2.958 $\pm$ .008	–

Table 2: Comparison results on the HumanML3D and KIT-ML test sets. The best results among generative methods are highlighted in bold. Higher is better for R-Precision, while lower is better for FID and MM Dist. Diversity values closer to Real Motion are better.

Cha.	Gen.	Che.	Edi.	R-Prec@1 $\uparrow$	FID $\downarrow$	Diversity $\rightarrow$
✗	✓	✗	✗	0.485 $\pm$ .002	0.554 $\pm$ .015	9.557 $\pm$ .228
✗	✓	✓	✓	0.482 $\pm$ .004	0.578 $\pm$ .009	9.571 $\pm$ .234
✓	✓	✗	✓	0.483 $\pm$ .005	0.577 $\pm$ .010	9.571 $\pm$ .110
✓	✓	✓	✗	0.514 $\pm$ .005	0.531 $\pm$ .034	9.500 $\pm$ .522
✓	✓	✓	✓	0.535 $\pm$ .003	0.503 $\pm$ .009	9.662 $\pm$ .065

Table 3: Ablation study of multi-agent components.

Figure 3: Ablation results of memory-based refinement and editing strategies.

Figure 4: Illustrative example of the iterative motion refinement process, with semantic misalignment in red, semantic alignment in green, and personality instruction in purple.

Panel of Figure 5 — Figure 5: Human evaluation results of MDM and SMART.

Panel of Figure 6 — Figure 6: Evaluation of memory-based refinement and VLM-based motion assessment.

Figure 7: Qualitative comparison of different text-to-motion methods on compositional prompts. Key action phrases are highlighted in red.

Text-to-Motion Generation

Text-to-motion generation converts natural-language descriptions into semantically aligned, temporally coherent 3D human movement.

See all 96 papers

IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation
Dec 11, 2025Yuan-Ming Li, Qize Yang, Nan Lei +5Text-to-Motion Generation
Sun Yat-sen University · Tongyi Lab, Alibaba Group · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China +1
Re $^2$ MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement
Apr 20, 2026Jiakun Zheng, Ting Xiao, Shiqin Cao +3Text-to-Motion Generation Human Motion Generation
1East China University of Science and Technology · 2Hong Kong University of Science and Technology · Institute of Artificial Intelligence (TeleAI), China Telecom
Unified Number-Free Text-to-Motion Generation Via Flow Matching
Mar 27, 2026Guanhe Huang, Oya CeliktutanText-to-Motion Generation Human Motion Generation
King’s College London

Dec 11, 2025J/K move · Enter open · S save

IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation

Yuan-Ming Li, Qize Yang, Nan Lei, Shenghao Fu+4

Sun Yat-sen University · Tongyi Lab, Alibaba Group · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China+1

Text-to-Motion Generation

Open paper AskPDF arXiv

Recent advances in motion-aware large language models have shown remarkable promise for jointly learning motion understanding and generation knowledge. However, these models typically treat understanding and generation separately, limiting the mutual benefits that could arise from interactive feedback between tasks. In this work, we reveal that motion assessment and refinement tasks can act as crucial bridges to enable knowledge flow from motion understanding to generation. Specifically, we propose Interleaved Reasoning for Motion Generation (IRMoGen), a novel paradigm that tightly couples motion generation with assessment and refinement through iterative text-motion dialogue. To realize this, we introduce IRG-MotionLLM, the first model that seamlessly interleaves motion generation, assessment, and refinement to improve the alignment between generated motion and goal text. IRG-MotionLLM is developed progressively with a novel three-stage training scheme, initializing and subsequently enhancing native IRMoGen capabilities. To facilitate this development, we construct an automated data engine to synthesize interleaved reasoning annotations from existing text-motion datasets. Extensive experiments demonstrate the properties brought by IRMoGen training, and the advanced cross-benchmark and cross-evaluator performance of IRG-MotionLLM. Code and models are available at https://github.com/HumanMLLM/IRG-MotionLLM.

Abstract

Text-to-Motion Generation

Explore similar work

IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation

Re2^22MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement

Unified Number-Free Text-to-Motion Generation Via Flow Matching

Re $^2$ MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement