Self-Consistency for LLM-Based Motion Trajectory Generation and Verification

Jiaju Ma; R. Kenny Jones; Jiajun Wu; Maneesh Agrawala

cs.CVMar 31, 2026

Self-Consistency for LLM-Based Motion Trajectory Generation and Verification

Authors: Jiaju Ma, R. Kenny Jones, Jiajun Wu, Maneesh Agrawala

Organizations: Stanford University

Abstract

Self-consistency has proven to be an effective technique for improving LLM performance on natural language reasoning tasks in a lightweight, unsupervised manner. In this work, we study how to adapt self-consistency to visual domains. Specifically, we consider the generation and verification of LLM-produced motion graphics trajectories. Given a prompt (e.g., "Move the circle in a spiral path"), we first sample diverse motion trajectories from an LLM, and then identify groups of consistent trajectories via clustering. Our key insight is to model the family of shapes associated with a prompt as a prototype trajectory paired with a group of geometric transformations (e.g., rigid, similarity, and affine). Two trajectories can then be considered consistent if one can be transformed into the other under the warps allowable by the transformation group. We propose an algorithm that automatically recovers a shape family, using hierarchical relationships between a set of candidate transformation groups. Our approach improves the accuracy of LLM-based trajectory generation by 4-6%. We further extend our method to support verification, observing 11% precision gains over VLM baselines. Our code and dataset are available at https://majiaju.io/trajectory-self-consistency .

Figures & tables

Figure 2: We define a shape family

\mathcal{F}(o,W)

with a prototype trajectory

o

and a Lie transformation group

W

. The shape family is then the set of all trajectories that can be warped into one another

w(o)

by all warps

w\in W

(left). In this example, any of the figure 8’s in the family can serve as the prototype since they can all be warped into one another. We can check whether a trajectory

t_{i}

is a member of the family by checking if there is a warp

w\in W

that can transform

t_{i}

into

o

(right).

Figure 3: Hierarchy of Lie transformation groups and shape families they induce. Each node represents a transformation group and depicts a prototype square-shaped trajectory

o

as well as other trajectories

w(o)

within the corresponding shape family.

Figure 4: We present clustering results on LLM-generated samples from three example prompts in our dataset. Each trajectory has a ground truth label on the upper right. Clusters colored green are the chosen largest clusters, and we note that their sizes vary, sometimes less than half of the total number of trajectories (middle). In the pentagon case, failed samples (Cluster 2–4) appear visually distinct from the true ones, while some of the failed deltoids (Cluster 4–8) look closer to the correct one with their triangular forms. The rightmost parabola is a skewed one that would have been grouped into the largest cluster under the affine warp.

Method	Decision Criteria	GPT-4.1	GPT-5
LLM-Direct	—	62.1	79.1
Ours	Majority-Consensus	68.0	83.3
Ours	Hierarchical-Consistency	66.7	82.6
Ours	Oracle	68.5	83.5

Table 1: Motion trajectory generation experiment (see Sec. 5.2 ). We evaluate the accuracy of LLM-generated motion trajectories (GPT-4.1 and GPT-5) under different decoding strategies. Under different decision criteria for selecting a transformation group

W

, our self-consistency approach improves upon the baseline alternative of directly generating a single sample (LLM-Direct).

Method	Decision Criteria	Precision	Recall	F1 Score
GPT-4.1	—	62.0	96.9	75.6
GPT-5	—	74.0	84.7	79.0
Ours	Majority-Consensus	85.8	66.1	74.6
Ours	Hierarchical-Consistency	80.5	89.0	84.6
Ours	Oracle	87.9	83.3	85.6

Table 2: Motion trajectory verification experiment (see Sec. 5.3 ). We compare our self-consistency-based approach for verifying whether a trajectory matches an input prompt against the alternative of using a VLM (GPT-4.1 or GPT-5).

Figure 5: F1 scores across different numbers of sampled trajectories

N

. Decision criteria performances stabilize after

N=10

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

$W_{\text{rgd-ref}}$	$W_{\text{rgd}}$	$W_{\text{sim}}$	$W_{\text{sim-ref}}$	$W_{\text{sim-ani}}$	$W_{\text{aff}}$	Total
105	44	38	23	12	2	224

Appendix

Table S1 : Number of prompts per warp in our benchmark.

Method	Decision Criteria	GPT-4.1	GPT-5
LLM-Direct	—	62.1	79.1
Ours	Majority-Consensus	68.0	83.3
Ours	Most-Restrictive	66.5	82.5
Ours	Hierarchical-Consistency	66.7	82.6
Ours	Least-Restrictive	64.6	82.1
Ours	Oracle	68.5	83.5

Appendix

Table S2 : Motion trajectory generation experiment with two additional baseline decision criteria of always selecting the most or least restrictive warps. We observe that they perform strictly worse than the two methods proposed in the main paper.

Method	Decision Criteria	Precision	Recall	F1 Score
GPT-4.1	—	62.0	96.9	75.6
GPT-5	—	74.0	84.7	79.0
Ours	Majority-Consensus	85.8	66.1	74.6
Ours	Most-Restrictive	86.1	42.1	56.5
Ours	Hierarchical-Consistency	80.5	89.0	84.6
Ours	Least-Restrictive	65.0	91.8	76.1

Appendix

Table S3 : Motion trajectory verification experiment (Table 2 of the main paper) with additional decision criteria of always selecting the most or least restrictive warps.

Figure S2 : We demonstrate a limitation of our self-consistency approach in terms of the definition of a shape family. In text prompt 1, “heptagram” has two distinct prototypes: heptagram {7/2} and heptagram {7/3}. Under the warp $W_{\text{sim-ref}}$ , cluster 1 contains correct {7/3} trajectories, while cluster 2 has {7/2}. While, for generation, our approach would produce a correct interpretation of heptagram, we would mark all heptagram {7/2} as false for verification. Similarly, for the letter Ms in text prompt 2, our approach would generate a correct form of the letter, but verify all other letter Ms that are not an affine warp of the shapes in cluster 1 as false.

Figure S3 : When a shape is aligned with the screen space’s coordinate frame, applying an anisotropic scale of 2 along the x-axis does not introduce skews to the shapes (top). However, once a shape is rotated out of alignment, non-uniform scales effectively skew the shapes out of their respective shape families (bottom).

Decision Criteria	More Restrictive	Match	Less Restrictive
Majority-Consensus	48.2%	49.6%	1.8%
Hierarchical-Consistency	4.0%	72.3%	22.3%

Appendix

Table S4 : Our decision criteria has different failure modes when not selecting the ground truth warp. Majority-Consensus selects more restrictive distance metrics for 108 prompts (95.6% of the time when it is not matching with the ground truth), while Hierarchical-Consistency is more conservative for 50 prompts (80.6% of the time). Note that both decision criteria choose distance metrics that are on the same level of the hierarchy but different from the ground truth for 1 and 3 prompts respectively.

Figure S4 : We demonstrate a case where Hierarchical-Consistency chooses the correct warp but Majority-Consensus fails by selecting a more restrictive warp. The prompt (top) asks for a rhombus-shaped path. We show clustering results of all warps in our hierarchy, with largest clusters under each warp highlighted in green. Each trajectory has a ground truth correctness label on the upper right.

Figure S5 : An example case where Majority-Consensus correctly chooses the ground truth warp but Hierarchical-Consistency fails by selecting a less restrictive warp. The prompt (top) asks for a path shaped like an equilateral triangle traversed in a counterclockwise manner. We show clustering results of all warps in our hierarchy.

Figure S6 : More results of applying our self-consistency approach with the oracle warps. The top half shows two success cases where the largest clusters do correspond to the correct shape families (figure 8 and 4-petal rose). The bottom half shows two failure cases where the LLM did not produce more members of the ground truth shape families than non-members (pentagram outline and reuleaux triangle).

	$\tau=0.25$	$\tau=0.5$	$\tau=1.0$	$\tau=2.0$	$\tau=4.0$	$\tau=8.0$
F1 score	84.0	84.6	83.8	83.1	80.3	76.8
Clusters (avg $\pm$ std)	4.0 $\pm$ 4.9	3.7 $\pm$ 4.8	3.5 $\pm$ 4.5	3.1 $\pm$ 4.0	2.6 $\pm$ 3.1	2.1 $\pm$ 2.1

Appendix

Table S5 : F1 scores of verification by Hierarchical-Consistency and the average number of DBSCAN clusters across the distance threshold

\tau

Figure S7 : We prompt a VLM with a scatter plot representation of a motion trajectory. Each dot is the center position of the animated element and the color indicates time (transition from blue to red).

Figure S8 : Motion trajectory benchmark: prompt 1–36 with ground truth shape families.

Figure S9 : Motion trajectory benchmark: prompt 37–72 with ground truth shape families.

Figure S10 : Motion trajectory benchmark: prompt 73–108 with ground truth shape families.

Figure S11 : Motion trajectory benchmark: prompt 109–144 with ground truth shape families.

Figure S12 : Motion trajectory benchmark: prompt 145–180 with ground truth shape families.

Figure S13 : Motion trajectory benchmark: prompt 181–216 with ground truth shape families.

Figure S14 : Motion trajectory benchmark: prompt 217–224 with ground truth shape families.

Trajectory Generation

Trajectory generation creates plausible, goal-directed paths and motion sequences for people, vehicles, and robots.

See all 182 papers

LAMP: Language-Assisted Motion Planning for Controllable Video Generation
Dec 3, 2025Muhammed Burak Kizil, Enes Sanli, Niloy J. Mitra +3Camera-Controlled Video Generation Trajectory Generation
Koç University · University College London · Adobe Research +1
IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation
Dec 11, 2025Yuan-Ming Li, Qize Yang, Nan Lei +5Text-to-Motion Generation
Sun Yat-sen University · Tongyi Lab, Alibaba Group · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China +1
Re $^2$ MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement
Apr 20, 2026Jiakun Zheng, Ting Xiao, Shiqin Cao +3Text-to-Motion Generation Human Motion Generation
1East China University of Science and Technology · 2Hong Kong University of Science and Technology · Institute of Artificial Intelligence (TeleAI), China Telecom

Dec 3, 2025J/K move · Enter open · S save

LAMP: Language-Assisted Motion Planning for Controllable Video Generation

Muhammed Burak Kizil, Enes Sanli, Niloy J. Mitra, Erkut Erdem+2

Koç University · University College London · Adobe Research+1

Camera-Controlled Video Generation Trajectory Generation Language Model-Based Control Motion Planning Controllable Video Generation Language-Conditioned Robot Planning

Open paper AskPDF arXiv

Video generation has achieved remarkable progress in visual fidelity and controllability, enabling conditioning on text, layout, or motion. Among these, motion control - specifying object dynamics and camera trajectories - is essential for composing complex, cinematic scenes, yet existing interfaces remain limited. We introduce LAMP that leverages large language models (LLMs) as motion planners to translate natural language descriptions into explicit 3D trajectories for dynamic objects and (relatively defined) cameras. LAMP defines a motion domain-specific language (DSL), inspired by cinematography conventions. By harnessing program synthesis capabilities of LLMs, LAMP generates structured motion programs from natural language, which are deterministically mapped to 3D trajectories. We construct a large-scale procedural dataset pairing natural text descriptions with corresponding motion programs and 3D trajectories. Experiments demonstrate LAMP's improved performance in motion controllability and alignment with user intent compared to state-of-the-art alternatives establishing the first framework for generating both object and camera motions directly from natural language specifications. Code, models and data are available on our project page.

Abstract

Trajectory Generation

Explore similar work

LAMP: Language-Assisted Motion Planning for Controllable Video Generation

IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation

Re2^22MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement

Re $^2$ MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement