LaMoGen: Language to Motion Generation Through LLM-Guided Symbolic Inference

Junkun Jiang; Ho Yin Au; Jingyu Xiang; Jie Chen

cs.CVMar 12, 2026

LaMoGen: Language to Motion Generation Through LLM-Guided Symbolic Inference

Authors: Junkun Jiang, Ho Yin Au, Jingyu Xiang, Jie Chen

Organizations: Department of Computer Science, Hong Kong Baptist University, HKSAR

Abstract

Human motion is highly expressive and naturally aligned with language, yet prevailing methods relying heavily on joint text-motion embeddings struggle to synthesize temporally accurate, detailed motions and often lack explainability. To address these limitations, we introduce LabanLite, a motion representation developed by adapting and extending the Labanotation system. Unlike black-box text-motion embeddings, LabanLite encodes each atomic body-part action (e.g., a single left-foot step) as a discrete Laban symbol paired with a textual template. This abstraction decomposes complex motions into interpretable symbol sequences and body-part instructions, establishing a symbolic link between high-level language and low-level motion trajectories. Building on LabanLite, we present LaMoGen, a Text-to-LabanLite-to-Motion Generation framework that enables large language models (LLMs) to compose motion sequences through symbolic reasoning. The LLM interprets motion patterns, relates them to textual descriptions, and recombines symbols into executable plans, producing motions that are both interpretable and linguistically grounded. To support rigorous evaluation, we introduce a Labanotation-based benchmark with structured description-motion pairs and three metrics that jointly measure text-motion alignment across symbolic, temporal, and harmony dimensions. Experiments demonstrate that LaMoGen establishes a new baseline for both interpretability and controllability, outperforming prior methods on our benchmark and two public datasets. These results highlight the advantages of symbolic reasoning and agent-based design for language-driven motion synthesis.

Figures & tables

Figure 1: Given a structured text description, methods based on text-motion joint embeddings often fail to generate semantically consistent motion. In contrast, our approach leverages symbolic motion representations, allowing for accurate motion generation. As each symbol is associated with one conceptual description, this design enables LLMs to compose symbolic motion via retrieval augmentation prompting.

Figure 2: Overview of LaMoGen: (a) The Laban-motion Encoder-Decoder enables bidirectional conversion between motion and Laban instances. These instances are human-readable and LLM-editable, as each instance has a symbolic appearance and a conceptual description, stored in the Conceptual Description Database. (b) LLMs perform high-level symbolic planning via retrieval-augmented prompting, generating sequences of conceptual symbols. The Kinematic Detail Augmentor then enriches these sequences with details through autoregressive generation. Enriched symbol sequences are converted to instances, encoded as codes, and decoded into fine-grained motions.

Method	SMT $\uparrow$				TMP $\uparrow$				HMN $\uparrow$			R@3 $\uparrow$	FID $\downarrow$
Method	supL	supR	armL	armR	supL	supR	armL	armR	arm-arm	arm-sup	sup-sup	R@3 $\uparrow$	FID $\downarrow$
Real data	-	-	-	-	-	-	-	-	-	-	-	0.216	0.001
MDM [ 28 ]	0.380	0.380	0.335	0.257	0.316	0.316	0.329	0.231	0.119	0.226	0.258	0.180	22.81
ReMoDiff [ 37 ]	0.470	0.470	0.427	0.395	0.377	0.377	0.385	0.322	0.179	0.264	0.351	0.192	7.121
MoDiff [ 36 ]	0.491	0.491	0.470	0.411	0.362	0.362	0.412	0.328	0.180	0.281	0.361	0.196	5.701
CoMo [ 10 ]	0.358	0.358	0.474	0.382	0.211	0.211	0.284	0.250	0.203	0.298	0.252	0.176	21.94

Table 1: Quantitative comparisons on our Laban Benchmark, using the proposed Labanotation-based metrics, R-precision Top-3 (R@3) and FID. Bold and underlined values indicate the best and the second-best performance, respectively.

Figure 3: Qualitative comparisons on HumanML3D and HumanML3D-Laban test sets, with generated motions progressing from left to right. The LLM-composed conceptual symbols are shown below our results. Misalignments between texts and motions are highlighted. Note how our method preserves correct sequencing, repetition, and timing of actions, while precisely controlling body parts and motion attributes, and demonstrating compositional generation enabled by LLM-driven symbolic inference—where existing methods fail.

Method		R-precision $\uparrow$			FID $\downarrow$	MM-Dist $\downarrow$	Diversity $\rightarrow$	Multi-Mod. $\uparrow$
Method		Top-1	Top-2	Top-3	FID $\downarrow$	MM-Dist $\downarrow$	Diversity $\rightarrow$	Multi-Mod. $\uparrow$
HumanML3D	Real data	$0.511^{\pm.003}$	$0.703^{\pm.003}$	$0.797^{\pm.002}$	$0.002^{\pm.000}$	$2.974^{\pm.008}$	$9.503^{\pm.085}$	-
	Guo et al. [9]	$0.457^{\pm.002}$	$0.639^{\pm.003}$	$0.740^{\pm.003}$	$1.067^{\pm.002}$	$3.340^{\pm.008}$	$9.188^{\pm.002}$	$\underline{2.090^{\pm.083}}$
	MDM [ 28 ]	$0.320^{\pm.005}$	$0.498^{\pm.004}$	$0.611^{\pm.007}$	$0.544^{\pm.044}$	$5.566^{\pm.027}$	$\mathbf{9.559^{\pm.086}}$	$\mathbf{2.799^{\pm.072}}$
	ReMoDiff [ 37 ]	$\underline{0.510^{\pm.005}}$	$\underline{0.698^{\pm.006}}$	$\mathit{0.795^{\pm.004}}$	$\mathbf{0.103^{\pm.004}}$	$\mathbf{2.974^{\pm.016}}$	$9.018^{\pm.075}$	$1.795^{\pm.043}$
	MoDiff [ 36 ]	$0.491^{\pm.001}$	$0.681^{\pm.001}$	$0.782^{\pm.001}$	$0.630^{\pm.001}$	$3.113^{\pm.001}$	$\mathit{9.410^{\pm.049}}$	$1.533^{\pm.042}$

Table 2: Quantitative comparisons with state-of-the-art methods on the HumanML3D test and KIT-ML test datasets, under standard protocols. Bold , underlined , and italicised values denote the best, second-best, and third-best performance, respectively.

Method	avg. SMT $\uparrow$	avg. TMP $\uparrow$	avg. HMN $\uparrow$
Ikeuchi et al. [11]	0.751	0.632	0.611
Ours	0.871	0.852	0.786

Table 3: Performance of Laban symbol detection on the Laban-annotated motion dataset, using average Laban metrics. Bold values denote the best performance.

Figure 1: Preliminary of Labanotation . (a) Illustration of a Laban staff. (b) A segment of a Laban score [ 29 ] . We further extend Labanotation into LabanLite . LabanLite categorizes Laban symbols into (c) Conceptual symbols, where “L”, “R”, “F”, “M”, and “B” denote left, right, forward, middle, and backward directions. “Hi.”, “Mi.”, and “Lo.” denote high, middle, and low levels. (d) Detailed symbols, where “Orientation”, “Bend” and “Moving-effort” describe the kinematic details.

Figure 2: Illustration of a partial Laban score. This figure provides a visual explanation of the annotation process for a forward walk movement.

Figure 3: Illustration of the relationships among Body-Part Groups, attribute fields, Laban staff columns, and Laban instances.

Figure 4: Visual comparison between a human-interpretable symbol sequence and a computational-ready instance sequence. The illustrated Laban staff columns correspond to Left support and Right support.

Figure 5: Illustration of extracting lower-body conceptual symbols: (a) Given a motion sequence; (b) we segment the sequence into dynamic and hold intervals by computing foot velocity; (c) in parallel, frame-wise Laban symbols are identified; (d) for each interval, aggregated frame-wise symbols yield the most representative symbol for the interval.

Figure 6: Illustration of extracting Direction and Level symbols for the right hand, in a canonical space.

Figure 7: Distribution of action classes in the HumanML3D-Laban dataset.

Figure 8: Distribution of the number of frames per motion sequence in the HumanML3D-Laban dataset.

Figure 9: WordCloud of the most frequent words in the HumanML3D-Laban descriptions, generated using the Python package wordcloud [ 23 ] .

Encoder	Num.
In Dim.	512
Feat. Dim.	512
Depth	8
Head	8
Head Dim.	64
FFN Dim.	1024

Table 1: The detailed structure of the Decoder in the Laban-Motion Encoder-Decoder module.

Figure 10: Illustration of the user study form.

Figure 11: Illustration of the user study result.

Method	LLM-related Procedure	Objective	$\sim$ Token Count (k)
CoMo [ 10 ]	Decompose Texts	Generate	6.5
	Identify frames	Edit	51
	Identify body parts and edit	Edit	53
	Total		110.5
LaMoGen	Compose CDs	Generate	5
	Modify CDs	Edit	5

Table 2: Quantitative comparison for LLM usage cost. The approximate token count per motion sequence for generation and modification is reported.

Method	SMT $\uparrow$				TMP $\uparrow$				HMN $\uparrow$			R@3 $\uparrow$	FID $\downarrow$
Method	supL	supR	armL	armR	supL	supR	armL	armR	arm-arm	arm-sup	sup-sup	R@3 $\uparrow$	FID $\downarrow$
Real data	-	-	-	-	-	-	-	-	-	-	-	0.797	0.002
LaMoGen (Dec)	0.843	0.843	0.745	0.731	0.848	0.848	0.779	0.765	0.537	0.631	0.842	0.793	0.095
MDM [ 28 ]	0.413	0.413	0.276	0.315	0.372	0.372	0.369	0.308	0.177	0.284	0.502	0.611	0.544
ReMoDiff [ 37 ]	0.455	0.455	0.348	0.405	0.335	0.405	0.367	0.320	0.179	0.275	0.501	0.795	0.103
MoDiff [ 36 ]	0.482	0.482	0.386	0.360	0.401	0.401	0.388	0.334	0.190	0.299	0.513	0.782	0.630

Table 3: Quantitative comparisons on the HumanML3D test set, using the proposed Labanotation-based metrics: Semantic Alignment (SMT), Temporal Alignment (TMP), and Harmonious Alignment (HMN), along with Text-to-Motion metrics: R-precision Top-3 (R@3) and FID. Bold and underlined values indicate the best and the second-best performance, respectively.

Figure 12: Qualitative comparison on fine-grained temporal structure modification. Left: the baseline motion, condition on “walk forward”. Middle: the motion conditioned on “walk forward slow”. Right: the motion conditioned on “walk forward in 5 seconds”. We highlight the step number and the corresponding duration. We calculate the average step duration to assess the degree of the speed change.

Figure 13: Visual examples for motion editing, with LLM-composed description. Left: the initial generated motion conditioned by the user input. Right: an ambiguous edit instruction is given to modify the motion. The corresponding LLM-composed description follows the LabanLite-defined format “ <body-part group>, <moving semantic>, <time> ”. We highlight the refined part with red to show LLMs’ dialogue context understanding capability.

Figure 14: Qualitative comparisons on HumanML3D test sets, with motions progressing from left to right. Misalignments between text and generated motions are highlighted.

Top ref.	R@3 $\uparrow$	FID $\downarrow$	MM-Dist $\downarrow$	Multi-Mod. $\uparrow$
1	0.755	0.595	3.840	0.954
3	0.796	0.252	3.087	1.131
5	0.782	0.371	3.191	1.030
7	0.774	0.324	3.225	0.971
Mask rat.	R@3 $\uparrow$	FID $\downarrow$	MM-Dist $\downarrow$	Multi-Mod. $\uparrow$
0.1	0.789	0.258	3.008	1.092

Table 4: Ablation study of different numbers of top-matched references and masking ratios on the HumanML3D test set.

Code Dim	R@3 $\uparrow$	FID $\downarrow$	AITS $\downarrow$
CoMo [ 10 ]	0.790	0.262	0.58
256	0.759	0.310	0.28
384	0.783	0.295	0.32
512	0.796	0.252	0.45
768	0.802	0.241	0.57
1024	0.804	0.227	0.65

Table 5: Ablation study of different code dimensions on the HumanML3D test set. AITS measures the average inference time cost.

Method	avg. SMT $\uparrow$	avg. TMP $\uparrow$	avg. HMN $\uparrow$	R@3 $\uparrow$	FID $\downarrow$
Real data	-	-	-	0.216	0.001
MDM	0.338	0.298	0.201	0.180	22.81
MDM lbn	0.475	0.438	0.271	0.204	1.932
GPT4.1 $\dagger$	0.533	0.501	0.392	0.208	1.861
GPT4.1 ip	0.496	0.467	0.328	0.200	3.408

Table 6: Evaluation on Laban Benchmark.

\dagger

: main paper setting.

Method		R@3 $\uparrow$	FID $\downarrow$	MM-Dist $\downarrow$	Div. $\rightarrow$	Multi-Mod. $\uparrow$	Token Cost (k) $\downarrow$
Real data		0.797	0.002	2.974	9.503	-	-
MDM		0.611	0.544	5.566	9.559	2.799	-
CoMo		0.790	0.262	3.032	9.936	1.013	6.5
Interval	3 $\dagger$	0.796	0.252	3.087	9.124	1.131	5.0
	5	0.796	0.198	3.026	9.859	1.360	7.7
	7	0.800	0.184	2.992	9.840	1.413	10.3

Table 7: Evaluation on HumanML3D. We examine the impact of varying the number of evenly spaced intervals for the Direction and Level symbols.

\dagger

: main paper setting.

Prompt #1: Conceptual cues composer
There are five digit collections describing movements, where each line consists of: [number] [Caption] - a general description of the motion sequence. [Support] - detailed descriptions of the movements of the supporting body parts, specifically the left and right feet, using a series of triplets. [Left hand] - detailed descriptions of the movements of the left hand, using a series of tuples. [Right hand] - detailed descriptions of the movements of the right hand, using a series of tuples. In the detailed descriptions, we specify the movement details for each body part and their duration in seconds. For the support movements, the details must be selected from these 54 categories: [1: steps to rest position, …, 54: holds in knee-flexed backward diagonally to left position]. For the hand movements, the details must be selected from these 81 categories: [1: moves close to shoulder, …, 81: moves relatively low backward diagonally to left]. For example, for the [Support] line, the triplet list would be like: (left, 1, 0.25), (right, 2, 0.25), (left, 1, 0.25) while (right, 2, 0.25). This means that the first movement is “left foot steps to rest position in 0.25 seconds”. The second movement is “right foot steps forward in 0.25 seconds”. The third movement is “left foot steps to rest position in 0.25 seconds while right foot steps forward in 0.25 seconds”. For the [Left hand] line, the tuple list would be like: (1, 0.5), (2, 0.2). This means that the first movement is “left hand moves close to shoulder in 0.5 seconds” and the second movement is “left hand moves forward in 0.2 seconds”. For the [Right hand] line, the structure and definition are similar to [Left hand] lines. Below is the main body of the digit collection describing the movements. You should strictly imitate the following content and create only one digit collection of YOUR_INPUT . Reply without explanation.
Prompt #2: Rephraser for generating motion descriptions
Rephrase the sentence creatively YOUR_INPUT . the step number is YOUR_INPUT , with the step order: YOUR_INPUT .

Table 8: Prompts used in our LaMoGen to process natural language via Large Language Models.

Table 9: Illustration of the Direction symbols and their corresponding partial semantics.

Table 10: Illustration of the Level symbols and their corresponding partial semantics.

Table 11: Illustration of the Hold symbols and their corresponding partial semantics. Note that if the attribute field for a symbol is left empty, it indicates that this body part is dynamic.

Table 12: Illustration of the Orientation symbols and their corresponding partial semantics. Note that according to Labanotation, each symbol does not have a specific name. In LabanLite, we simply assign them sequential names for convenience.

Table 13: Illustration of the Bend symbols and their corresponding partial semantics. Note that according to Labanotation, each symbol does not have a specific name. In LabanLite, we simply assign them sequential names for convenience.

Table 14: Illustration of the Moving-effort symbols and their corresponding partial semantics. Note that according to Labanotation, each symbol does not have a specific name. In LabanLite, we simply assign them sequential names for convenience. If the attribute field for a symbol is left empty, it indicates “Moving-effort 1”.

Code Idx.	Body-Part Group	SMPL Joint	Attribute	is Concpt.	Staff Col.
$1\sim 3$	Support-L	Left foot	Direction (L/M/R)	✓	Left support
$4\sim 6$		Left foot	Direction (B/M/F)	✓	Left support
$7\sim 9$		Left foot	Level (Lo./Mi./Hi.)	✓	Left support
$10\sim 15$		Left knee	Bend	✗	Left leg gesture
$16\sim 21$		Left hip	Bend	✗	Left leg gesture
$22\sim 23$		Left foot & knee & hip	Hold	✓	Left support

Table 15: Definition of the Laban codebook. Each codebook entry is described by its index (Code Idx.), associated Body-Part Group, corresponding SMPL key joint name (SMPL Joint), relevant attribute (Attribute), a marker indicating whether the attribute is conceptual (is Concpt.), and the Laban staff column name (Staff Col.).

Index	Semantics	Index	Semantics
1	steps to rest position	28	holds in rest position
2	steps forward	29	holds in forward position
3	steps backward	30	holds in backward position
4	steps to right	31	holds in right position
5	steps to left	32	holds in left position
6	steps forward diagonally to right	33	holds in forward diagonally to right position

Table 16: Support body-part’s moving semantic lookup table.

Index	Semantics	Index	Semantics	Index	Semantics
1	moves close to shoulder	28	holds close to shoulder position	55	moves relatively to previous position
2	moves forward	29	holds forward position	56	moves relatively forward
3	moves backward	30	holds backward position	57	moves relatively backward
4	moves to right	31	holds right position	58	moves to relatively right
5	moves to left	32	holds left position	59	moves to relatively left
6	moves forward diagonally to right	33	holds forward diagonally to right position	60	moves relatively forward diagonally to right

Table 17: Arm body-part’s moving semantic lookup table.

Explore similar work

Mar 13, 2026

LingoMotion: An Interpretable and Unambiguous Symbolic Representation for Human Motion

Existing representations for human motion, such as MotionGPT, often operate as black-box latent vectors with limited interpretability and build on joint positions which can cause ambiguity. Inspired by the hierarchical structure of natural languages - from letters to words, phrases, and sentences - we propose LingoMotion, a motion language that facilitates interpretable and unambiguous symbolic representation for both simple and complex human motion. In this paper, we introduce the concept design of LingoMotion, including the definitions of motion alphabet based on joint angles, the morphology for forming words and phrases to describe simple actions like walking and their attributes like speed and scale, as well as the syntax for describing more complex human activities with sequences of words and phrases. The preliminary results, including the implementation and evaluation of motion alphabet using a large-scale motion dataset Motion-X, demonstrate the high fidelity of motion representation.

Yao Zhang, Zhuchenyang Liu, Yu Xiao

Aalto University, Espoo, Finland

Motion Representation Learning Interpretable ML Human Motion Generation

Ask PDF

Apr 3, 2026

Exploring Motion-Language Alignment for Text-driven Motion Generation

Text-driven human motion generation aims to synthesize realistic motion sequences that follow textual descriptions. Despite recent advances, accurately aligning motion dynamics with textual semantics remains a fundamental challenge. In this paper, we revisit text-to-motion generation from the perspective of motion-language alignment and propose MLA-Gen, a framework that integrates global motion priors with fine-grained local conditioning. This design enables the model to capture common motion patterns, while establishing detailed alignment between texts and motions. Furthermore, we identify a previously overlooked attention sink phenomenon in human motion generation, where attention disproportionately concentrates on the start text token, limiting the utilization of informative textual cues and leading to degraded semantic grounding. To analyze this issue, we introduce SinkRatio, a metric for measuring attention concentration, and develop alignment-aware masking and control strategies to regulate attention during generation. Extensive experiments demonstrate that our approach consistently improves both motion quality and motion-language alignment over strong baselines.

Ruxi Gu, Zilei Wang, Wei Wang

Department of Automation, University of Science and Technology of China, Hefei, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China

Text-to-Motion Generation Human Motion Generation Cross-Modal Alignment Attention Sinks

Ask PDF

Dec 11, 2025

IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation

Recent advances in motion-aware large language models have shown remarkable promise for jointly learning motion understanding and generation knowledge. However, these models typically treat understanding and generation separately, limiting the mutual benefits that could arise from interactive feedback between tasks. In this work, we reveal that motion assessment and refinement tasks can act as crucial bridges to enable knowledge flow from motion understanding to generation. Specifically, we propose Interleaved Reasoning for Motion Generation (IRMoGen), a novel paradigm that tightly couples motion generation with assessment and refinement through iterative text-motion dialogue. To realize this, we introduce IRG-MotionLLM, the first model that seamlessly interleaves motion generation, assessment, and refinement to improve the alignment between generated motion and goal text. IRG-MotionLLM is developed progressively with a novel three-stage training scheme, initializing and subsequently enhancing native IRMoGen capabilities. To facilitate this development, we construct an automated data engine to synthesize interleaved reasoning annotations from existing text-motion datasets. Extensive experiments demonstrate the properties brought by IRMoGen training, and the advanced cross-benchmark and cross-evaluator performance of IRG-MotionLLM. Code and models are available at https://github.com/HumanMLLM/IRG-MotionLLM.

Yuan-Ming Li, Qize Yang, Nan Lei +5

Sun Yat-sen University · Tongyi Lab, Alibaba Group · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China +1

Text-to-Motion Generation

Ask PDF