LaMoGen: Language to Motion Generation Through LLM-Guided Symbolic Inference

Junkun Jiang; Ho Yin Au; Jingyu Xiang; Jie Chen

Figure 1: Given a structured text description, methods based on text-motion joint embeddings often fail to generate semantically consistent motion. In contrast, our approach leverages symbolic motion representations, allowing for accurate motion generation. As each symbol is associated with one conceptual description, this design enables LLMs to compose symbolic motion via retrieval augmentation prompting.

Figure 2: Overview of LaMoGen: (a) The Laban-motion Encoder-Decoder enables bidirectional conversion between motion and Laban instances. These instances are human-readable and LLM-editable, as each instance has a symbolic appearance and a conceptual description, stored in the Conceptual Description Database. (b) LLMs perform high-level symbolic planning via retrieval-augmented prompting, generating sequences of conceptual symbols. The Kinematic Detail Augmentor then enriches these sequences with details through autoregressive generation. Enriched symbol sequences are converted to instances, encoded as codes, and decoded into fine-grained motions.

Method	SMT $\uparrow$				TMP $\uparrow$				HMN $\uparrow$			R@3 $\uparrow$	FID $\downarrow$
Method	supL	supR	armL	armR	supL	supR	armL	armR	arm-arm	arm-sup	sup-sup	R@3 $\uparrow$	FID $\downarrow$
Real data	-	-	-	-	-	-	-	-	-	-	-	0.216	0.001
MDM [ 28 ]	0.380	0.380	0.335	0.257	0.316	0.316	0.329	0.231	0.119	0.226	0.258	0.180	22.81
ReMoDiff [ 37 ]	0.470	0.470	0.427	0.395	0.377	0.377	0.385	0.322	0.179	0.264	0.351	0.192	7.121
MoDiff [ 36 ]	0.491	0.491	0.470	0.411	0.362	0.362	0.412	0.328	0.180	0.281	0.361	0.196	5.701
CoMo [ 10 ]	0.358	0.358	0.474	0.382	0.211	0.211	0.284	0.250	0.203	0.298	0.252	0.176	21.94

Table 1: Quantitative comparisons on our Laban Benchmark, using the proposed Labanotation-based metrics, R-precision Top-3 (R@3) and FID. Bold and underlined values indicate the best and the second-best performance, respectively.

Figure 3: Qualitative comparisons on HumanML3D and HumanML3D-Laban test sets, with generated motions progressing from left to right. The LLM-composed conceptual symbols are shown below our results. Misalignments between texts and motions are highlighted. Note how our method preserves correct sequencing, repetition, and timing of actions, while precisely controlling body parts and motion attributes, and demonstrating compositional generation enabled by LLM-driven symbolic inference—where existing methods fail.

Method		R-precision $\uparrow$			FID $\downarrow$	MM-Dist $\downarrow$	Diversity $\rightarrow$	Multi-Mod. $\uparrow$
Method		Top-1	Top-2	Top-3	FID $\downarrow$	MM-Dist $\downarrow$	Diversity $\rightarrow$	Multi-Mod. $\uparrow$
HumanML3D	Real data	$0.511^{\pm.003}$	$0.703^{\pm.003}$	$0.797^{\pm.002}$	$0.002^{\pm.000}$	$2.974^{\pm.008}$	$9.503^{\pm.085}$	-
	Guo et al. [9]	$0.457^{\pm.002}$	$0.639^{\pm.003}$	$0.740^{\pm.003}$	$1.067^{\pm.002}$	$3.340^{\pm.008}$	$9.188^{\pm.002}$	$\underline{2.090^{\pm.083}}$
	MDM [ 28 ]	$0.320^{\pm.005}$	$0.498^{\pm.004}$	$0.611^{\pm.007}$	$0.544^{\pm.044}$	$5.566^{\pm.027}$	$\mathbf{9.559^{\pm.086}}$	$\mathbf{2.799^{\pm.072}}$
	ReMoDiff [ 37 ]	$\underline{0.510^{\pm.005}}$	$\underline{0.698^{\pm.006}}$	$\mathit{0.795^{\pm.004}}$	$\mathbf{0.103^{\pm.004}}$	$\mathbf{2.974^{\pm.016}}$	$9.018^{\pm.075}$	$1.795^{\pm.043}$
	MoDiff [ 36 ]	$0.491^{\pm.001}$	$0.681^{\pm.001}$	$0.782^{\pm.001}$	$0.630^{\pm.001}$	$3.113^{\pm.001}$	$\mathit{9.410^{\pm.049}}$	$1.533^{\pm.042}$

Table 2: Quantitative comparisons with state-of-the-art methods on the HumanML3D test and KIT-ML test datasets, under standard protocols. Bold , underlined , and italicised values denote the best, second-best, and third-best performance, respectively.

Method	avg. SMT $\uparrow$	avg. TMP $\uparrow$	avg. HMN $\uparrow$
Ikeuchi et al. [11]	0.751	0.632	0.611
Ours	0.871	0.852	0.786

Table 3: Performance of Laban symbol detection on the Laban-annotated motion dataset, using average Laban metrics. Bold values denote the best performance.

Figure 1: Preliminary of Labanotation . (a) Illustration of a Laban staff. (b) A segment of a Laban score [ 29 ] . We further extend Labanotation into LabanLite . LabanLite categorizes Laban symbols into (c) Conceptual symbols, where “L”, “R”, “F”, “M”, and “B” denote left, right, forward, middle, and backward directions. “Hi.”, “Mi.”, and “Lo.” denote high, middle, and low levels. (d) Detailed symbols, where “Orientation”, “Bend” and “Moving-effort” describe the kinematic details.

Figure 2: Illustration of a partial Laban score. This figure provides a visual explanation of the annotation process for a forward walk movement.

Figure 3: Illustration of the relationships among Body-Part Groups, attribute fields, Laban staff columns, and Laban instances.

Figure 4: Visual comparison between a human-interpretable symbol sequence and a computational-ready instance sequence. The illustrated Laban staff columns correspond to Left support and Right support.

Figure 5: Illustration of extracting lower-body conceptual symbols: (a) Given a motion sequence; (b) we segment the sequence into dynamic and hold intervals by computing foot velocity; (c) in parallel, frame-wise Laban symbols are identified; (d) for each interval, aggregated frame-wise symbols yield the most representative symbol for the interval.

Figure 6: Illustration of extracting Direction and Level symbols for the right hand, in a canonical space.

Figure 7: Distribution of action classes in the HumanML3D-Laban dataset.

Figure 8: Distribution of the number of frames per motion sequence in the HumanML3D-Laban dataset.

Figure 9: WordCloud of the most frequent words in the HumanML3D-Laban descriptions, generated using the Python package wordcloud [ 23 ] .

Encoder	Num.
In Dim.	512
Feat. Dim.	512
Depth	8
Head	8
Head Dim.	64
FFN Dim.	1024

Table 1: The detailed structure of the Decoder in the Laban-Motion Encoder-Decoder module.

Figure 10: Illustration of the user study form.

Figure 11: Illustration of the user study result.

Method	LLM-related Procedure	Objective	$\sim$ Token Count (k)
CoMo [ 10 ]	Decompose Texts	Generate	6.5
	Identify frames	Edit	51
	Identify body parts and edit	Edit	53
	Total		110.5
LaMoGen	Compose CDs	Generate	5
	Modify CDs	Edit	5

Table 2: Quantitative comparison for LLM usage cost. The approximate token count per motion sequence for generation and modification is reported.

Method	SMT $\uparrow$				TMP $\uparrow$				HMN $\uparrow$			R@3 $\uparrow$	FID $\downarrow$
Method	supL	supR	armL	armR	supL	supR	armL	armR	arm-arm	arm-sup	sup-sup	R@3 $\uparrow$	FID $\downarrow$
Real data	-	-	-	-	-	-	-	-	-	-	-	0.797	0.002
LaMoGen (Dec)	0.843	0.843	0.745	0.731	0.848	0.848	0.779	0.765	0.537	0.631	0.842	0.793	0.095
MDM [ 28 ]	0.413	0.413	0.276	0.315	0.372	0.372	0.369	0.308	0.177	0.284	0.502	0.611	0.544
ReMoDiff [ 37 ]	0.455	0.455	0.348	0.405	0.335	0.405	0.367	0.320	0.179	0.275	0.501	0.795	0.103
MoDiff [ 36 ]	0.482	0.482	0.386	0.360	0.401	0.401	0.388	0.334	0.190	0.299	0.513	0.782	0.630

Table 3: Quantitative comparisons on the HumanML3D test set, using the proposed Labanotation-based metrics: Semantic Alignment (SMT), Temporal Alignment (TMP), and Harmonious Alignment (HMN), along with Text-to-Motion metrics: R-precision Top-3 (R@3) and FID. Bold and underlined values indicate the best and the second-best performance, respectively.

Figure 12: Qualitative comparison on fine-grained temporal structure modification. Left: the baseline motion, condition on “walk forward”. Middle: the motion conditioned on “walk forward slow”. Right: the motion conditioned on “walk forward in 5 seconds”. We highlight the step number and the corresponding duration. We calculate the average step duration to assess the degree of the speed change.

Figure 13: Visual examples for motion editing, with LLM-composed description. Left: the initial generated motion conditioned by the user input. Right: an ambiguous edit instruction is given to modify the motion. The corresponding LLM-composed description follows the LabanLite-defined format “ <body-part group>, <moving semantic>, <time> ”. We highlight the refined part with red to show LLMs’ dialogue context understanding capability.

Figure 14: Qualitative comparisons on HumanML3D test sets, with motions progressing from left to right. Misalignments between text and generated motions are highlighted.

Top ref.	R@3 $\uparrow$	FID $\downarrow$	MM-Dist $\downarrow$	Multi-Mod. $\uparrow$
1	0.755	0.595	3.840	0.954
3	0.796	0.252	3.087	1.131
5	0.782	0.371	3.191	1.030
7	0.774	0.324	3.225	0.971
Mask rat.	R@3 $\uparrow$	FID $\downarrow$	MM-Dist $\downarrow$	Multi-Mod. $\uparrow$
0.1	0.789	0.258	3.008	1.092

Table 4: Ablation study of different numbers of top-matched references and masking ratios on the HumanML3D test set.

Code Dim	R@3 $\uparrow$	FID $\downarrow$	AITS $\downarrow$
CoMo [ 10 ]	0.790	0.262	0.58
256	0.759	0.310	0.28
384	0.783	0.295	0.32
512	0.796	0.252	0.45
768	0.802	0.241	0.57
1024	0.804	0.227	0.65

Table 5: Ablation study of different code dimensions on the HumanML3D test set. AITS measures the average inference time cost.

Method	avg. SMT $\uparrow$	avg. TMP $\uparrow$	avg. HMN $\uparrow$	R@3 $\uparrow$	FID $\downarrow$
Real data	-	-	-	0.216	0.001
MDM	0.338	0.298	0.201	0.180	22.81
MDM lbn	0.475	0.438	0.271	0.204	1.932
GPT4.1 $\dagger$	0.533	0.501	0.392	0.208	1.861
GPT4.1 ip	0.496	0.467	0.328	0.200	3.408

Table 6: Evaluation on Laban Benchmark.

\dagger

: main paper setting.

Method		R@3 $\uparrow$	FID $\downarrow$	MM-Dist $\downarrow$	Div. $\rightarrow$	Multi-Mod. $\uparrow$	Token Cost (k) $\downarrow$
Real data		0.797	0.002	2.974	9.503	-	-
MDM		0.611	0.544	5.566	9.559	2.799	-
CoMo		0.790	0.262	3.032	9.936	1.013	6.5
Interval	3 $\dagger$	0.796	0.252	3.087	9.124	1.131	5.0
	5	0.796	0.198	3.026	9.859	1.360	7.7
	7	0.800	0.184	2.992	9.840	1.413	10.3

Table 7: Evaluation on HumanML3D. We examine the impact of varying the number of evenly spaced intervals for the Direction and Level symbols.

\dagger

: main paper setting.

Prompt #1: Conceptual cues composer
There are five digit collections describing movements, where each line consists of: [number] [Caption] - a general description of the motion sequence. [Support] - detailed descriptions of the movements of the supporting body parts, specifically the left and right feet, using a series of triplets. [Left hand] - detailed descriptions of the movements of the left hand, using a series of tuples. [Right hand] - detailed descriptions of the movements of the right hand, using a series of tuples. In the detailed descriptions, we specify the movement details for each body part and their duration in seconds. For the support movements, the details must be selected from these 54 categories: [1: steps to rest position, …, 54: holds in knee-flexed backward diagonally to left position]. For the hand movements, the details must be selected from these 81 categories: [1: moves close to shoulder, …, 81: moves relatively low backward diagonally to left]. For example, for the [Support] line, the triplet list would be like: (left, 1, 0.25), (right, 2, 0.25), (left, 1, 0.25) while (right, 2, 0.25). This means that the first movement is “left foot steps to rest position in 0.25 seconds”. The second movement is “right foot steps forward in 0.25 seconds”. The third movement is “left foot steps to rest position in 0.25 seconds while right foot steps forward in 0.25 seconds”. For the [Left hand] line, the tuple list would be like: (1, 0.5), (2, 0.2). This means that the first movement is “left hand moves close to shoulder in 0.5 seconds” and the second movement is “left hand moves forward in 0.2 seconds”. For the [Right hand] line, the structure and definition are similar to [Left hand] lines. Below is the main body of the digit collection describing the movements. You should strictly imitate the following content and create only one digit collection of YOUR_INPUT . Reply without explanation.
Prompt #2: Rephraser for generating motion descriptions
Rephrase the sentence creatively YOUR_INPUT . the step number is YOUR_INPUT , with the step order: YOUR_INPUT .

Table 8: Prompts used in our LaMoGen to process natural language via Large Language Models.

Table 9: Illustration of the Direction symbols and their corresponding partial semantics.

Table 10: Illustration of the Level symbols and their corresponding partial semantics.

Table 11: Illustration of the Hold symbols and their corresponding partial semantics. Note that if the attribute field for a symbol is left empty, it indicates that this body part is dynamic.

Table 12: Illustration of the Orientation symbols and their corresponding partial semantics. Note that according to Labanotation, each symbol does not have a specific name. In LabanLite, we simply assign them sequential names for convenience.

Table 13: Illustration of the Bend symbols and their corresponding partial semantics. Note that according to Labanotation, each symbol does not have a specific name. In LabanLite, we simply assign them sequential names for convenience.

Table 14: Illustration of the Moving-effort symbols and their corresponding partial semantics. Note that according to Labanotation, each symbol does not have a specific name. In LabanLite, we simply assign them sequential names for convenience. If the attribute field for a symbol is left empty, it indicates “Moving-effort 1”.

Code Idx.	Body-Part Group	SMPL Joint	Attribute	is Concpt.	Staff Col.
$1\sim 3$	Support-L	Left foot	Direction (L/M/R)	✓	Left support
$4\sim 6$		Left foot	Direction (B/M/F)	✓	Left support
$7\sim 9$		Left foot	Level (Lo./Mi./Hi.)	✓	Left support
$10\sim 15$		Left knee	Bend	✗	Left leg gesture
$16\sim 21$		Left hip	Bend	✗	Left leg gesture
$22\sim 23$		Left foot & knee & hip	Hold	✓	Left support

Table 15: Definition of the Laban codebook. Each codebook entry is described by its index (Code Idx.), associated Body-Part Group, corresponding SMPL key joint name (SMPL Joint), relevant attribute (Attribute), a marker indicating whether the attribute is conceptual (is Concpt.), and the Laban staff column name (Staff Col.).

Index	Semantics	Index	Semantics
1	steps to rest position	28	holds in rest position
2	steps forward	29	holds in forward position
3	steps backward	30	holds in backward position
4	steps to right	31	holds in right position
5	steps to left	32	holds in left position
6	steps forward diagonally to right	33	holds in forward diagonally to right position

Table 16: Support body-part’s moving semantic lookup table.

Index	Semantics	Index	Semantics	Index	Semantics
1	moves close to shoulder	28	holds close to shoulder position	55	moves relatively to previous position
2	moves forward	29	holds forward position	56	moves relatively forward
3	moves backward	30	holds backward position	57	moves relatively backward
4	moves to right	31	holds right position	58	moves to relatively right
5	moves to left	32	holds left position	59	moves to relatively left
6	moves forward diagonally to right	33	holds forward diagonally to right position	60	moves relatively forward diagonally to right

Table 17: Arm body-part’s moving semantic lookup table.

LaMoGen: Language to Motion Generation Through LLM-Guided Symbolic Inference

Abstract

Figures & tables

Explore similar work

LingoMotion: An Interpretable and Unambiguous Symbolic Representation for Human Motion

Exploring Motion-Language Alignment for Text-driven Motion Generation

IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation