VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings

Ruizhi Chen; Zining Fan; Yudi Guo; Jiang Liu; Zhiyu Lin; Zhongjiang He; Shouqian Shi; Hao Sun

cs.CVOct 16, 2026

VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings

Authors: Ruizhi Chen, Zining Fan, Yudi Guo, Jiang Liu, Zhiyu Lin, Zhongjiang He, Shouqian Shi, Hao Sun

Organizations: Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd. · Peking University · Nanjing University · Xi’an Jiaotong University

Abstract

Fine-grained alignment between text and video object trajectories is a core challenge in multimodal understanding. Existing embedding models excel at global retrieval, yet their single global representation entangles multiple objects and their spatiotemporal dynamics, limiting direct object-level grounding. To address this issue, we propose VTrajEmbed, an embedding model that decomposes a video into object trajectories and aligns each trajectory embedding with text for object-level grounding through retrieval. Our main contributions: (1)Slow-Fast Trajectory Encoding, a dual-branch architecture that establishes trajectory representations by densely sampling low-resolution frames for high-frequency motion cues and sparsely selecting high-resolution frames for scene context, aggregated by a shared LLM backbone; (2)Mask-Guided Contrastive Learning, a training objective built upon these representations that couples an intra-group loss and a batch-level bidirectional loss under a relation mask, excluding unverified relations and preventing confirmed matches from becoming false negatives; (3)Text2Traj-Bench, a trajectory-level text-video benchmark with model-assisted construction and human-verified query--trajectory relations. Experimental results show that VTrajEmbed-5B improves text-to-trajectory and trajectory-to-text R@1 over Qwen3-VL-Embedding-8B by 15.48 and 18.05 percentage points, respectively, and achieves 49.48% average R@1 across five global retrieval benchmarks, compared with 42.86% for Qwen3-VL-Embedding-8B.

Figures & tables

Figure 1: Overview of VTrajEmbed. (a) Model architecture. The Slow-Fast trajectory encoder combines dense low-resolution object observations with sparse high-resolution scene context and maps trajectories and text into a shared embedding space for similarity-based retrieval. (b) Training objective. Mask-guided contrastive learning jointly optimizes within-video trajectory discrimination and bidirectional trajectory-text retrieval using verified query-trajectory relations while excluding uncertain pairs. (c) Data. Text2Traj-Bench uses trajectory curation, verified description, and relation construction to produce fixed per-video candidate pools and human-verified annotations, including descriptions and within- and cross-video query-trajectory relations.

Variant	R@1	R@3	mAP
Full model	69.67	98.51	83.72
w/o Slow branch	64.58(5.09 $\downarrow$ )	95.49(3.02 $\downarrow$ )	79.86(3.86 $\downarrow$ )
w/o Fast branch	41.08(28.59 $\downarrow$ )	87.01(11.50 $\downarrow$ )	64.94(18.78 $\downarrow$ )
Fast observations: 32 $\rightarrow$ 16	64.31(5.36 $\downarrow$ )	95.05(3.46 $\downarrow$ )	79.81(3.91 $\downarrow$ )
w/o $L_{\mathrm{bi}}$ (group only)	49.29(20.38 $\downarrow$ )	93.55(4.96 $\downarrow$ )	71.10(12.62 $\downarrow$ )
w/o $L_{\mathrm{group}}$ (bidirectional only)	48.06(21.61 $\downarrow$ )	81.45(17.06 $\downarrow$ )	66.27(17.45 $\downarrow$ )

Table 3: Ablation study of Slow-Fast encoding and training objectives. Other settings follow Section 4.1 . Scores are percentages; parentheses show percentage-point drops from the full model.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
Task and candidate ranking
$V$ , $e$	Input video and natural-language query, respectively.
$r$ , $\mathcal{R}(V)$	A candidate object trajectory and the fixed candidate pool for video $V$ . Each trajectory stores boxes and original timestamps.
$\hat{r}$ , $r^{*}$	Highest-scoring retrieved trajectory and designated benchmark target, respectively.
$s(e,r)$	Retrieval logit: $\exp(\ell)q(e)^{\top}v(r)$ .
$M$	Number of candidates in an evaluation pool; uniform random selection succeeds with probability $1/M$ .

Appendix

Table 4: Notation used in the main text.

Multimodal Embedding

Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.

See all 159 papers

TraVEL: Trajectory-Guided Video Embedding Learning for Driving-Video Retrieval
Aug 13, 2026Yi-Chung Chen, Philip Jacobson, Tom Lampo +6VLMs for Autonomous Driving Multimodal Embedding
Uber AV Labs · Purdue University
BoxTuning: Directly Injecting the Object Box for Multimodal Model Fine-Tuning
Apr 13, 2026Zekun Qian, Ruize Han, Wei FengMultimodal Large Language Models Multimodal QA
School of Computer Science and Technology, Tianjin University, Tianjin, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China
Video-Text Temporal Localization via Multi-Scale Convolution and Dynamic Routing
Jul 6, 2026Gengtian Shi, Jinze Yu, Chenhao Wu +5Temporal Video Grounding Temporal Convolutional Networks
Graduate School of Fundamental Science and Engineering, Waseda University, Tokyo, Japan · Generative AI Innovation Center, Amazon Web Services, Japan · Amazon, Germany

Aug 13, 2026J/K move · Enter open · S save

TraVEL: Trajectory-Guided Video Embedding Learning for Driving-Video Retrieval

Yi-Chung Chen, Philip Jacobson, Tom Lampo, Yiren Lu+5

Uber AV Labs · Purdue University

VLMs for Autonomous Driving Multimodal Embedding Video Retrieval

Open paper AskPDF arXiv

Efficiently retrieving relevant clips from large-scale driving logs is essential for data curation, model development, and safety analysis. Structured and rule-based retrieval systems can explicitly target driving events, but typically require expert-defined rules, auxiliary data, and multi-stage perception pipelines. Multimodal embedding models offer a simpler and more efficient alternative by representing each video with a single searchable vector. However, general-purpose models often rely on shortcuts from static scene context and struggle to distinguish motion-centric events, such as turning left versus right or accelerating versus decelerating. In this work, we study how to adapt a general-purpose multimodal embedding model to driving-video retrieval. We first fine-tune Qwen3-VL-Embedding on paired clips and reasoning traces from nuReasoning using an InfoNCE objective. While this stage substantially improves overall retrieval, caption supervision alone remains insufficient for fine-grained motion understanding. We therefore introduce TraVEL (Trajectory-Guided Video Embedding Learning), a motion-aware fine-tuning framework that uses ego-trajectory similarity as a reward within Group Relative Policy Optimization. Trajectories serve only as privileged training supervision; retrieval still operates on single-vector video embeddings without ego poses, expert rules, or auxiliary perception outputs. We further construct a driving-video retrieval benchmark from nuReasoning. Experiments show that TraVEL improves motion-centric retrieval across model scales: relative to SFT, it raises longitudinal and lateral mAP by 9.8 and 4.7 points at 2B, with corresponding gains of 7.2 and 1.5 points at 8B. TraVEL thus combines physically grounded supervision with efficient embedding-based search.

Abstract

Multimodal Embedding

Explore similar work

TraVEL: Trajectory-Guided Video Embedding Learning for Driving-Video Retrieval

BoxTuning: Directly Injecting the Object Box for Multimodal Model Fine-Tuning

Video-Text Temporal Localization via Multi-Scale Convolution and Dynamic Routing