AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models

Teng Wang; Yanting Lu; Ruize Wang

cs.CVMar 9, 2026

AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models

Authors: Teng Wang, Yanting Lu, Ruize Wang

Abstract

We present AutoTraces, an autoregressive vision-language-trajectory model for robot trajectory forecasting in humam-populated environments, which harnesses the inherent reasoning capabilities of large language models (LLMs) to model complex human behaviors. In contrast to prior works that rely solely on textual representations, our key innovation lies in a novel trajectory tokenization scheme, which represents waypoints with point tokens as categorical and positional markers while encoding waypoint numerical values as corresponding point embeddings, seamlessly integrated into the LLM's space through a lightweight encoder-decoder architecture. This design preserves the LLM's native autoregressive generation mechanism while extending it to physical coordinate spaces, facilitates modeling of long-term interactions in trajectory data. We further introduce an automated chain-of-thought (CoT) generation mechanism that leverages a multimodal LLM to infer spatio-temporal relationships from visual observations and trajectory data, eliminating reliance on manual annotation. Through a two-stage training strategy, our AutoTraces achieves SOTA forecasting accuracy, particularly in long-horizon prediction, while exhibiting strong cross-scene generalization and supporting flexible-length forecasting.

Figures & tables

Figure 1: Comparison between our AutoTraces and previous LLM solutions. Our method introduces point tokens and embeddings for waypoint representation, enabling autoregressive prediction.

Figure 2: The overall framework of our AutoTraces built upon LLaVa-Video [ 47 ] for socially compliant trajectory forecasting. In the first stage, the model is pre-trained on video-text pairs with reasoning prompts to acquire thinking knowledge. The second stage fine-tunes the model using trajectory waypoints (via a Point Encoder), aligning visual observations (via a Vision Encoder) and goal locations (via a Point Encoder). During inference, the model generates future trajectory waypoints as $<$ point $>$ tokens, enabling autoregressive predictions.

Figure 3: Illustration of the CoT generation for AutoTraces, incorporating visual observations and trajectory analysis. Red points and lines denote the historical trajectory, while blue points and lines denote the ground-truth trajectory. Action annotations (R: right, L: left, S: straight) are marked along the trajectory.

Method	Single Model	T=5		T=8		T=10
Method	Single Model	L2 (m) $\downarrow$	L1 (m) $\downarrow$	L2 (m) $\downarrow$	L1 (m) $\downarrow$	L2 (m) $\downarrow$	L1 (m) $\downarrow$
GNM [ 41 ]	✗	0.855	1.034	1.908	2.362	1.708	2.164
GNM [ 41 ]	✓	0.895	1.122	1.456	1.826	1.708	2.164
ViNT [ 42 ]	✗	0.973	1.168	1.430	1.754	1.714	2.132
ViNT [ 42 ]	✓	0.908	1.106	1.435	1.762	1.714	2.132
NoMad [ 43 ]	✗	1.167	1.440	1.625	2.102	–	–

Table 1: Performance evaluation of trajectory prediction over 5 to 10 steps on SCAND dataset [ 20 ] , with an interval of 1s per step. Baselines with gray background are specifically trained and tested on fixed trajectory lengths. In contrast, “Single Model” variants are predominantly trained on T=10 sequences (except NoMad trained on T=8) and evaluated across all horizons under truncated-length settings.

Methods	Benchmark	T=5		T=8		T=10
Methods	Benchmark	L2 (m) $\downarrow$	L1 (m) $\downarrow$	L2 (m) $\downarrow$	L1 (m) $\downarrow$	L2 (m) $\downarrow$	L1 (m) $\downarrow$
GNM [ 41 ]	GoStanford	3.375	4.164	5.018	6.186	6.022	7.413
GNM [ 41 ]	RECON	1.754	2.328	3.141	4.168	4.117	5.448
ViNT [ 42 ]	GoStanford	2.980	3.688	4.320	5.348	5.192	6.378
ViNT [ 42 ]	RECON	1.804	2.406	3.158	4.206	4.151	5.510
NoMad [ 43 ]	GoStanford	3.470	4.252	5.108	6.224	–	–

Table 2: Performance evaluation of cross-scene trajectory prediction on unseen GoStanford dataset [ 17 ] for indoor and RECON dataset [ 39 ] for outdoor scenarios with prediction horizon ranging from 5 to 10 steps, with an interval of 1s per step.

Horizon	LLaVa-Video [ 47 ]				AutoTraces (Ours)
Horizon	L2@S (m) $\downarrow$	L1@S (m) $\downarrow$	IEAcc	TPR	L2@S (m) $\downarrow$	L1@S (m) $\downarrow$	IEAcc	TPR
12	1.653	2.086	40.34%	375.64	1.611	2.046	99.92%	25.00
15	2.057	2.597			1.857	2.354
18	2.426	3.055			2.124	2.692
20	2.584	3.244			2.324	2.950

Table 3: Performance Comparison of Extended Trajectory Prediction on SCAND [ 20 ] with prediction horizon over 12-20 steps (1s/step).

Figure 4: Ablation study of our learnable waypoint tokenization scheme and CoT reasoning on SCAND [ 20 ] dataset.

Figure 5: Visualizations of predicted trajectory from our AutoTraces and other baselines on SCAND [ 20 ] dataset.

Figure 6: Comparison between autoregressive and one-pass predictions on SCAND [ 20 ] and RECON [ 39 ] datasets.

Robot Motion Prediction

See all 9 papers

TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understanding
Feb 23, 2026Fan Yang, Shurong Zheng, Hongyin Zhao +5Visual Grounding VLM Interpretability
Foundation Model Research Center, Institute of Automation · Peng Cheng Laboratory, Shenzhen, China · School of Artificial Intelligence, University of Chinese Academy of Science, Beijing, China +2
Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics
Dec 15, 2025Enshen Zhou, Yibo Li, Jingkun An +123D Spatial Reasoning VLMs for Robotics
School of Software, Beihang University · Beijing Academy of Artificial Intelligence · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University +2
MoCAR: Motion-code Coordinate-aware AutoRegression for Continuous Trajectory Forecasting
Oct 5, 2026Yiming Xu, Hao Cheng, Monika SesterTrajectory Generation Trajectory Representation Learning
Institute of Cartography and Geoinformatics Leibniz University Hannover, 30167 Hannover, Germany · Faculty of Geo-Information Science and Earth Observation University of Twente, 7522 NH Enschede, The Netherlands

Feb 23, 2026J/K move · Enter open · S save

TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understanding

Fan Yang, Shurong Zheng, Hongyin Zhao, Yufei Zhan+4

Foundation Model Research Center, Institute of Automation · Peng Cheng Laboratory, Shenzhen, China · School of Artificial Intelligence, University of Chinese Academy of Science, Beijing, China+2

Visual Grounding VLM Interpretability

Open paper AskPDF arXiv

Recent Large Vision-Language Models (LVLMs) demonstrate remarkable capabilities in image understanding and natural language generation. However, current approaches focus predominantly on global image understanding, struggling to simulate human visual attention trajectories and explain associations between descriptions and specific regions. We propose TraceVision, a unified vision-language model integrating trajectory-aware spatial understanding in an end-to-end framework. TraceVision employs a Trajectory-aware Visual Perception (TVP) module for bidirectional fusion of visual features and trajectory information. We design geometric simplification to extract semantic keypoints from raw trajectories and propose a three-stage training pipeline where trajectories guide description generation and region localization. We extend TraceVision to trajectory-guided segmentation and video scene understanding, enabling cross-frame tracking and temporal attention analysis. We construct the Reasoning-based Interactive Localized Narratives (RILN) dataset to enhance logical reasoning and interpretability. Extensive experiments on trajectory-guided captioning, text-guided trajectory prediction, understanding, and segmentation demonstrate that TraceVision achieves state-of-the-art performance, establishing a foundation for intuitive spatial interaction and interpretable visual understanding.

Abstract

Robot Motion Prediction

Explore similar work

TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understanding

Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics

MoCAR: Motion-code Coordinate-aware AutoRegression for Continuous Trajectory Forecasting