TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understanding

Fan Yang; Shurong Zheng; Hongyin Zhao; Yufei Zhan; Xin Li; Yousong Zhu; Chaoyang Zhao Ming Tang; Jinqiao Wang

cs.CVFeb 23, 2026

TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understanding

Authors: Fan Yang, Shurong Zheng, Hongyin Zhao, Yufei Zhan, Xin Li, Yousong Zhu, Chaoyang Zhao Ming Tang, Jinqiao Wang

Organizations: Foundation Model Research Center, Institute of Automation · Peng Cheng Laboratory, Shenzhen, China · School of Artificial Intelligence, University of Chinese Academy of Science, Beijing, China · School of Artificial Intelligence, China University of Mining and Technology-Beijing, Beijing, China ·

Abstract

Recent Large Vision-Language Models (LVLMs) demonstrate remarkable capabilities in image understanding and natural language generation. However, current approaches focus predominantly on global image understanding, struggling to simulate human visual attention trajectories and explain associations between descriptions and specific regions. We propose TraceVision, a unified vision-language model integrating trajectory-aware spatial understanding in an end-to-end framework. TraceVision employs a Trajectory-aware Visual Perception (TVP) module for bidirectional fusion of visual features and trajectory information. We design geometric simplification to extract semantic keypoints from raw trajectories and propose a three-stage training pipeline where trajectories guide description generation and region localization. We extend TraceVision to trajectory-guided segmentation and video scene understanding, enabling cross-frame tracking and temporal attention analysis. We construct the Reasoning-based Interactive Localized Narratives (RILN) dataset to enhance logical reasoning and interpretability. Extensive experiments on trajectory-guided captioning, text-guided trajectory prediction, understanding, and segmentation demonstrate that TraceVision achieves state-of-the-art performance, establishing a foundation for intuitive spatial interaction and interpretable visual understanding.

Figures & tables

Figure 1: Multi-modal capabilities of TraceVision across image, video, and segmentation tasks. The model processes traditional captioning, trajectory-guided interpretation and grounding, video sequence analysis, and precise segmentation, demonstrating its versatility in handling diverse visual understanding scenarios with trajectory-based spatial reasoning.

Figure 2: TraceVision architecture overview. The model processes trajectory, image, and text inputs through a unified framework. The TVP module performs bidirectional fusion between visual and trajectory features via cross-attention, enabling trajectory-conditioned captioning, text-guided trajectory prediction tasks.

Figure 3: Trajectory simplification: Geometric Simplification algorithm reduces 410 original points to 37 keypoints while preserving spatial structure.

Figure 4: RILN dataset construction pipeline showing the generation of diverse trajectory-based tasks from image-trajectory pairs. The pipeline creates four main task types: referential trajectory interpretation, grounding, interactive reasoning Q&A, and multi-turn dialogue synthesis, with hierarchical reasoning trees spanning from global scene understanding to fine-grained object-level spatial reasoning.

	Contrlled Caption Generation						Controlled Trace Generation
Method	BLUE-1↑	BLUE-4↑	METEOR↑	ROUGE_L↑	CIDEr↑	SPICE↑	LBM(k=0)↓	LBM(k=1)↓
LN ( Pont-Tuset et al., 2020 )	0.522	0.246	-	0.483	1.065	0.365	-	-
MITR ( Meng et al., 2021 )	0.607	0.292	0.263	0.487	1.485	0.317	0.163	0.154
PIxelLLM ( Xu et al., )	-	-	-	-	-	-	0.153	-
LLaVA 1.5-13B ( Liu et al., 2023b )	0.590	0.280	0.250	0.480	1.450	0.310	-	-
Ferret-13B ( You et al., 2023 )	0.600	0.283	0.255	0.482	1.470	0.315	0.160	0.170

Table 1: Quantitative results for trajectory-aware vision-language tasks on the COCO test set of Localized Narratives dataset. Controlled Caption Generation refers to generating descriptions given trajectory input, while Controlled Trajectory Generation refers to predicting trajectories given text input. Note: smaller values of LBM are better. Best results are in bold and second-best results are underlined.

Model	VG		RefCOCOg		Ref-L4			Ferret Bench	MDVP Bench
Model	METEOR	CIDEr	METEOR	CIDEr	ROUGE-L	METEOR	CIDEr	Refer. Desc.	Avg.
PixelLLM	19.9	148.9	14.3	82.3	-	-	-	-	-
GLaMM-7B ( Rasheed et al., 2024 )	17.0	127.0	15.7	104.0	23.8	10.1	51.1	-	-
Osprey-7B ( Yuan et al., 2024 )	-	-	16.6	108.3	-	-	-	72.2	44.3
Ferret-7B ( You et al., 2023 )	-	-	-	-	22.3	10.7	39.7	68.7	47.6
VP-LLaVA-8B ( Lin et al., 2024 )	-	-	22.4	153.6	-	-	-	75.2	70.6

Table 2: Performance comparison on trajectory-aware regional captioning across multiple benchmarks. TraceVision demonstrates competitive performance on Visual Genome (VG), RefCOCOg, and Ref-L4 datasets, as well as benchmarks including Ferret-Bench and MDVP-Bench for spatial understanding evaluation.

Models	RefCOCO			RefCOCO+			RefCOCOg
Models	val	testA	testB	val	testA	testB	val	test
bounding box P @ 0.5
VisionLLM ( Wang et al., 2023 )	86.7	-	-	-	-	-	-	-
Shikra-7B ( Chen et al., 2023 )	87.0	90.6	80.2	81.6	87.4	71.1	82.3	82.2
Ferret-7B	87.5	91.4	82.5	80.8	87.4	73.1	83.9	84.8
PixelLLM-7B	89.8	92.2	86.4	83.2	87.0	78.9	84.6	86.0

Table 3: Performance comparison on referring localization and segmentation across RefCOCO datasets. TraceVision achieves state-of-the-art performance on both bounding box localization ([email protected]) and segmentation (cIoU) tasks. ∗ indicates zero-shot results without task-specific fine-tuning.

Figure 5: Performance analysis visualization comparing trajectory-aware methods across different evaluation metrics and datasets.

Table 9

Figure 6: Qualitative trajectory visualization results showing original images (left), baseline Qwen2.5-VL-7B heatmaps (center), and TraceVision heatmaps (right). Compared to the baseline, TraceVision generates broader, semantically aligned attention patterns that accurately focus on salient objects mentioned in descriptions and better capture human visual attention mechanisms.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Overview of the TraceVision framework. The model takes images or multi-frame videos as input and supports multiple interaction modalities (query, caption, click, trajectory) to perform various tasks including trajectory-guided caption generation, trajectory prediction, trajectory-guided segmentation, referential trajectory interpretation, referential trajectory grounding, and interactive trajectory reasoning through QA and multi-turn dialogue.

Stage	Number	Task	Source
Stage 1: Trajectory-Aware Pretraining	252K	Localized Narratives	COCO
Stage 1.5: End-to-End Joint Training	885K	Localized Narratives	COCO, ADE20K
	885K	Localized Narratives	Flickr30k, OpenImage
	50K	Localized Narrative Video	OVIS, UVO
	50K	Localized Narrative Video	Oops
	118K	Panoptic Narrative Grounding	COCO

Appendix

Table 7: Dataset distribution across training stages in TraceVision

Figure 8: Dataset distribution across the three training stages of TraceVision. Stage 1 (0.25M samples) focuses solely on trajectory-aware pretraining with COCO Localized Narratives. Stage 1.5 (1.33M samples) incorporates diverse multimodal tasks including localized narratives (66%), video understanding (4%), panoptic grounding (9%), video narrative grounding (3%), video QA (1%), and REC/RES (17%). Stage 2 (1.3M samples) emphasizes instruction fine-tuning with reasoning-based interactive tasks (25%), reasoning segmentation (37%), and interactive editing (38%).

Model	HC-STVG ( Tang et al., 2021 )		VideoRefer-Bench-D ( Yuan et al., 2025 )
Model	METEOR	CIDEr	SC	AD	TD	HD	Avg.
Elysium-7B ( Wang et al., 2024 )	–	–	2.35	0.30	0.02	3.59	1.57
Merlin-7B ( Yu et al., 2024 )	11.3	10.5	–	–	–	–	–
Artemis-7B ( Qiu et al., 2024 )	18.0	53.2	3.42	1.34	1.39	2.90	2.26
VideoRefer-7B ( Yuan et al., 2025 )	18.7	68.6	4.44	3.27	3.10	3.04	3.46
DAM-8B	21.0	91.0	4.69	3.61	3.34	3.09	3.68

Appendix

Table 8: Performance comparison on video understanding benchmarks. TraceVision demonstrates competitive performance across HC-STVG and VideoRefer-Bench-D datasets.

(a) Cross-Backbone Validation			(b) TVP Attention Mechanisms			(c) Stage-wise Training
Method	RefCOCO	LBM $\downarrow$	Configuration	RefCOCO	LBM $\downarrow$	Training Stage	RefCOCO	LBM $\downarrow$
Baseline (Qwen2.5-VL)	87.2	0.147	Baseline (no TVP)	90.0	0.147	Stage 1: Pretrain	89.2	0.165
TraceVision (Qwen2.5-VL)	88.3	0.117	+ Traj $\rightarrow$ Vis only	90.2	0.135	Stage 1.5: Joint Train	90.1	0.128
TraceVision (InternVL2)	88.1	0.121	+ Vis $\rightarrow$ Traj only	90.1	0.139	Stage 2: Instruct FT	90.4	0.117
Average Gain	+1.1	-0.030	+ Traj $\leftrightarrow$ Vis (full)	90.4	0.117	Total Gain	+1.2	-0.048

Appendix

Table 9: Comprehensive ablation studies on TraceVision. (a) Cross-backbone validation demonstrates architecture-agnostic effectiveness. (b) Bidirectional attention analysis validates mutually reinforcing synergy. (c) Stage-wise training analysis shows progressive performance gains.

Compression	Key Points	RefCOCO	LBM(k=0) $\downarrow$	Latency (ms)	Reduction
0% (all points)	410	90.6	0.108	312	—
75%	103	90.5 (-0.1)	0.112 (+0.004)	156	-50%
91% (ours)	37	90.4 (-0.2)	0.117 (+0.009)	107	-66%
95%	21	89.8 (-0.8)	0.135 (+0.027)	89	-71%

Appendix

Table 10: Ablation study on trajectory compression rate. Our semantic-guided DP at 91% compression achieves optimal balance between performance and efficiency.

Method	RefCOCO	LBM(k=0) $\downarrow$
Uniform Sampling	88.5	0.145
Standard DP (geometric only)	89.7	0.128
Semantic-Guided DP (ours)	90.4	0.117

Appendix

Table 11: Comparison of trajectory simplification methods with identical compression rate (37 keypoints). Semantic-guided DP significantly outperforms geometric-only baselines.

Figure 9: Overview of TraceVision architecture. The model consists of three main components: (1) Vision Encoder that processes images/videos, (2) Trajectory Visual Perceiver (TVP) that integrates trajectory information through bidirectional cross-attention with visual features, and (3) Large Language Model (Qwen 2.5) that performs joint trajectory-text generation. The Trajectory Visual Perceiver Module uses bidirectional attention mechanisms for Visually-Informed Trajectory Refinement (VTR) and Trajectory-Aware Visual Feature Enhancement (TVF). A lightweight Mask Decoder generates segmentation masks from trajectory-conditioned tokens.

Decoder Type	Parameters	RefCOCO IoU	Latency (ms)
SAM decoder	636M	84.2	245
Mask2Former	223M	82.7	198
Ours (2-layer conv)	12M	83.4	107

Appendix

Table 12: Comparison of segmentation decoder architectures. Our lightweight 2-layer decoder achieves competitive performance with dramatically reduced parameters and latency.

# Tokens	RefCOCO IoU	Parameters
3	80.8	6M
6 (ours)	83.4	12M
12	83.9	24M

Appendix

Table 13: Ablation study on segmentation codebook size. 6 tokens provide optimal balance between performance and parameter efficiency.

Figure 10: Qualitative segmentation and description results. Examples showing color-coded segmentation masks and corresponding natural language descriptions across diverse scenarios including outdoor scenes, indoor workspaces, urban settings, transportation, architecture, and action scenes. The model accurately identifies objects (highlighted in different colors) and generates detailed descriptions with correct spatial relationships and object attributes.

Figure 11: Attention heatmap visualization. Visualization of the Trajectory Visual Perceiver’s attention mechanism across four examples. Each row shows the original image (left), attention heatmap with red-to-yellow gradient indicating attention intensity (middle), and overlay visualization (right). The heatmaps reveal that the model focuses on semantically salient objects and regions relevant for scene understanding.

Visual Grounding

Visual grounding connects language to the image regions, objects, or pixels that support a description, question, or action.

See all 144 papers

Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models
Jun 2, 2026Mahtab Bigverdi, Linjie Li, Weikai Huang +8Multimodal Understanding Spatial Reasoning
University of Washington · 2Allen Institute for AI · 3Microsoft +1
Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics
Dec 15, 2025Enshen Zhou, Yibo Li, Jingkun An +123D Spatial Reasoning VLMs for Robotics
School of Software, Beihang University · Beijing Academy of Artificial Intelligence · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University +2
Spatial-Interactor: Learning Spatial Reasoning through Interaction with the Observable Physical World
Sep 19, 2026Kaixiang Yao, Xu Wang, Miao Pan +10Visual Spatial Reasoning Spatial Reasoning
Zhejiang University · SAP

Jun 2, 2026J/K move · Enter open · S save

Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models

Mahtab Bigverdi, Linjie Li, Weikai Huang, Yiming Liu+7

University of Washington · 2Allen Institute for AI · 3Microsoft+1

Multimodal Understanding Spatial Reasoning Visual Spatial Reasoning Visual Reasoning Latent Visual Reasoning VLM Reasoning+3

Open paper AskPDF arXiv

Vision language models (VLMs) excel at many tasks but still struggle with spatial reasoning when critical information is not directly observable. Many such problems require imaginative perception: inferring what would be seen from an unseen viewpoint, tracing paths through occluded spaces, or integrating partial observations into a coherent spatial representation. We introduce Imaginative Perception Tokens (IPT), intermediate perceptual representations that externalize what a VLM would perceive under alternative spatial configurations while remaining consistent with the observed input. To study this capability, we formulate three tasks, Perspective Taking (PET), Path Tracing (PT), and Multiview Counting (MVC), and construct datasets of approximately 20K examples with ground truth imaginations, answers, and evaluation benchmarks. Using the unified VLM BAGEL as the backbone, IPT supervision consistently improves spatial reasoning and often outperforms textual chain of thought training, even without generating images at inference time. On MVC, IPT improves accuracy by 3.4% and achieves competitive performance with strong closed-source models on PT. We further find that combining IPT and label-only supervision yields additional gains, whereas textual chain of thought can substantially degrade performance, suggesting a modality mismatch when spatial computation is forced through language. Overall, IPT provides a principled supervision signal for reasoning about unobserved spatial structure, improving generalization while producing interpretable intermediate representations.

Abstract

Visual Grounding

Explore similar work

Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models

Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics

Spatial-Interactor: Learning Spatial Reasoning through Interaction with the Observable Physical World