TrajTok: Learning Trajectory Tokens enables better Video Understanding

Chenhao Zheng; Jieyu Zhang; Jianing Zhang; Weikai Huang; Ashutosh Kumar; Quan Kong; Oncel Tuzel; Chun-Liang Li

cs.CVFeb 26, 2026

TrajTok: Learning Trajectory Tokens enables better Video Understanding

Authors: Chenhao Zheng, Jieyu Zhang, Jianing Zhang, Weikai Huang, Ashutosh Kumar, Quan Kong, Oncel Tuzel, Chun-Liang Li, +1 more

Organizations: University of Washington · Allen Institute for Artificial Intelligence · Woven by Toyota, Inc · Apple

Abstract

Tokenization in video models, typically through patchification, generates an excessive and redundant number of tokens. This severely limits video efficiency and scalability. While recent trajectory-based tokenizers offer a promising solution by decoupling video duration from token count, they rely on complex external segmentation and tracking pipelines that are slow and task-agnostic. We propose TrajTok, an end-to-end video tokenizer module that is fully integrated and co-trained with video models for a downstream objective, dynamically adapting its token granularity to semantic complexity, independent of video duration. TrajTok contains a unified segmenter that performs implicit clustering over pixels in both space and time to directly produce object trajectories in a single forward pass. By prioritizing downstream adaptability over pixel-perfect segmentation fidelity, TrajTok is lightweight and efficient, yet empirically improves video understanding performance. With TrajTok, we implement a video CLIP model trained from scratch (TrajViT2). It achieves the best accuracy at scale across both classification and retrieval benchmarks, while maintaining efficiency comparable to the best token-merging methods. TrajTok also proves to be a versatile component beyond its role as a tokenizer. We show that it can be seamlessly integrated as either a probing head for pretrained visual features (TrajAdapter) or an alignment connector in vision-language models (TrajVLM) with especially strong performance in long-video reasoning.

Figures & tables

Figure 2: Overview of the TrajTok architecture. TrajTok comprises a trajectory segmenter and a trajectory encoder . The segmenter proposes trajectory masks for all objects in an image or video within a single forward pass. The encoder then aggregates raw video pixels or encoded visual features (parameterized by $f$ in the figure) according to these masks to produce trajectory tokens. The number of tokens per trajectory can be flexibly adjusted based on the available compute budget.

Figure 3: Training with downstream understanding tasks reshapes the segmentation granularity. We visualize the trajectory masks produced by our segmenter when trained with only segmentation supervision versus jointly with segmentation and CLIP objectives. The CLIP objective reshapes the segmentation granularity, producing finer foreground object masks while merging background regions.

Figure 4: TrajTok is a versatile module applicable across pretraining, feature adaptation, and finetuning stages. We demonstrate its use in three scenarios: TrajViT2, which trains a visual encoder from scratch; TrajAdapter, which adapts pretrained features for downstream tasks; and TrajVLM, which uses TrajTok as a connector in LLaVA-style large vision–language models.

Table 4

	VideoMAE-v2 [ 27 ]		V-JEPA2 [ 2 ]
Probing Method	K400 [ 34 ]	SSv2 [ 29 ]	K400 [ 34 ]	SSv2 [ 29 ]
Linear probing	79.4	59.1	84.5	73.7
Attentive probing	80.2	59.7	85.1	74.2
Perceiver probing	79.9	59.8	84.7	74.2
TrajAdapter (1 token/traj)	82.0	60.4	87.2	74.6
TrajAdapter (2 token/traj)	82.4	60.8	87.8	74.9

Table 3: Probing results on different video backbones. Top-1 accuracy (%) on Kinetics-400 (K400) and Something-Something-V2 (SSv2) using various probing strategies. TrajAdapter consistently improves the downstream probing accuracy of pretrained backbone features, and performance further increases with more tokens per trajectory.

Figure 7: Trajectory masks produced by our segmenter vs. by TrajViT pipeline. While our segmenter produces coarser masks and may miss very small objects, it demonstrates strong semantic grouping ability that is sufficient for downstream understanding tasks.

Figure 8: VideoQA results for TrajTok applying to large vision-language model. VLM with TrajTok as connector (TrajVLM) notably outperforms patch pooling baseline (PatchVLM) in long-video benchmarks, while the performance is mixed for short-video benchmark.

Module	Variation	VEQ (%)	STQ (%)	Retrieval (R@5)
Default Architecture		42.3	70.1	22.1
Backbone	no hierarchical features	39.3 ( $\downarrow$ 3.0)	66.2 ( $\downarrow$ 3.9)	19.2 ( $\downarrow$ 2.9)
Output Res.	56 $\rightarrow$ 224	44.1 ( $\uparrow$ 1.8)	73.0 ( $\uparrow$ 2.9)	22.0 ( $\downarrow$ 0.1)
Perceiver	no detach gradient	34.1 ( $\downarrow$ 8.2)	59.3 ( $\downarrow$ 10.8)	18.3 ( $\downarrow$ 3.8)
Seg. Loss	− dice loss	39.0 ( $\downarrow$ 3.3)	68.9 ( $\downarrow$ 1.2)	16.7 ( $\downarrow$ 5.4)
Seg. Loss	− focal loss	41.2 ( $\downarrow$ 1.1)	67.4 ( $\downarrow$ 2.7)	22.1 (0.0)

Table 4: Ablations of segmenter design.

Module	Variation	1 token/traj	2 token/traj	4 token/traj
Default Architecture		22.1	23.0	23.2
Attention Mask	w/o mask	17.4 ( $\downarrow$ 4.7)	17.9 ( $\downarrow$ 5.1)	18.3 ( $\downarrow$ 4.9)
Query Init.	Fourier $\rightarrow$ random	22.1 (0.0)	22.5 ( $\downarrow$ 0.5)	21.9 ( $\downarrow$ 1.3)
Perceiver Depth	2 $\rightarrow$ 4	22.3 ( $\uparrow$ 0.2)	22.9 ( $\downarrow$ 0.1)	23.5 ( $\uparrow$ 0.3)

Table 5: Ablations of trajectory encoder design.

Hyperparameter	Value
Resolution	224
Frame sampling	uniform 16 frames (videos only)
Optimizer	AdamW
Base LR	$1\times 10^{-3}$
Weight decay	0.01
Optimizer momentum	$\beta_{1}=0.9,\ \beta_{2}=0.999$

Table 6: Hyperparameters used for segmenter pretraining.

Figure 9: Qualitative Examples of the trajectory masks produced by our segmenter.

VIPSeg Benchmark	VEQ-SQ	VEQ-RQ	STQ-EN
EntitySAM	84.7	64.5	43.3
SAM 2.1	83.1	36.7	41.7
Ours (scaled-up)	85.5	45.1	40.2

Table 7: Evaluation of a scaled-up version of our segmenter on the ViPEntitySeg benchmark. This model is not used in any main-paper experiments; it serves only as an isolated study of segmentation quality under increased capacity.

Category	Dataset Name(s)	Notes / Source
Academic VideoQA	llava_video_mc_academic	MC-style QA
	llava_video_oe_academic	Open-ended QA
	clevrer	Causal & counterfactual reasoning
	funqa	Fine-grained temporal QA
	star	Long-horizon procedural QA
	intent_qa	Human intent reasoning

Table 8: Datasets used for training TrajVLM under the “final” mixture. This mixture combines a large set of academic VideoQA datasets, temporal reasoning datasets, and synthetic captioning/QA corpora.

Hyperparameter	Value
Max video frames	128
Total training steps	10,000
Training stages	Pretraining + Instruction tuning
Sequence length	8192
Global batch size	32
Device batch size	4

Table 9: Key hyperparameters used for training TrajVLM. Values reflect the shared configuration across all VLM experiments.

Explore similar work

TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenization
Jun 16, 2026Weiliang Chen, Yuanhui Huang, Xuebo Wang +1Visual Tokenization Video Token Compression
Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China · Department of Automation, Tsinghua University, Beijing, 100084, China · Kuaishou Technology, Beijing, China
VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization
Apr 14, 2026Andrei Atanov, Jesse Allardice, Roman Bachmann +6Visual Tokenization Video Token Compression
Apple · Swiss Federal Institute of Technology Lausanne (EPFL)
GeoTrace: Geometry-Aware Trajectory Token Compression for Video Large Language Models
Jul 10, 2026Guohuan Xie, Mengqi Lei, Chuan Shi +3Efficient VLM Inference Video-Language Models
BNRist, THUIBCS, BLBCI, School of Software, Tsinghua University · Yangtze Delta Region Institute, Tsinghua University · Beijing University of Posts and Telecommunications, Beijing, China

Jun 16, 2026J/K move · Enter open · S save

TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenization

Weiliang Chen, Yuanhui Huang, Xuebo Wang, Yueqi Duan

Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China · Department of Automation, Tsinghua University, Beijing, 100084, China · Kuaishou Technology, Beijing, China

Visual Tokenization Video Token Compression Visual Token Compression

Open paper AskPDF arXiv

Video tokenization is fundamental to scalable video generation, as the number of tokens directly determines the computational cost and the length of videos that can be modeled. Existing tokenizers mainly improve scalability by compressing videos into fewer tokens, but they often continue to represent persistent content, such as static backgrounds and consistent object appearances, repeatedly across frames and chunks. In this paper, we propose TivTok (Time-Invariant Tokenizer), a reuse-aware video tokenizer that makes persistent information reusable across time. TivTok represents a clip with Time-Invariant (TIV) tokens that encode information shared across frames and Time-Variant (TV) tokens that encode frame-specific residuals. To obtain this factorization, we introduce Scope-Induced Factorization (SIF), which assigns different attention scopes to the two token groups: TIV tokens attend to the full clip, whereas each TV token only accesses its corresponding frame together with the TIV tokens. In the decoder, Invariant Broadcasting (IB) reuses the same TIV tokens across frames and chunks for parallel reconstruction and long-video tokenization. Experiments show that TivTok achieves an rFVD of 12.65 on the standard

16{\times}256{\times}256

benchmark and improves compression efficiency by 2.91

\times

for 128-frame videos compared with the evaluated baselines, while using only 1.1% of the tokens required by downsample-based tokenizers in our evaluation.

Abstract

Explore similar work

TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenization

VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

GeoTrace: Geometry-Aware Trajectory Token Compression for Video Large Language Models