Tokenization in video models, typically through patchification, generates an excessive and redundant number of tokens. This severely limits video efficiency and scalability. While recent trajectory-based tokenizers offer a promising solution by decoupling video duration from token count, they rely on complex external segmentation and tracking pipelines that are slow and task-agnostic. We propose TrajTok, an end-to-end video tokenizer module that is fully integrated and co-trained with video models for a downstream objective, dynamically adapting its token granularity to semantic complexity, independent of video duration. TrajTok contains a unified segmenter that performs implicit clustering over pixels in both space and time to directly produce object trajectories in a single forward pass. By prioritizing downstream adaptability over pixel-perfect segmentation fidelity, TrajTok is lightweight and efficient, yet empirically improves video understanding performance. With TrajTok, we implement a video CLIP model trained from scratch (TrajViT2). It achieves the best accuracy at scale across both classification and retrieval benchmarks, while maintaining efficiency comparable to the best token-merging methods. TrajTok also proves to be a versatile component beyond its role as a tokenizer. We show that it can be seamlessly integrated as either a probing head for pretrained visual features (TrajAdapter) or an alignment connector in vision-language models (TrajVLM) with especially strong performance in long-video reasoning.
Figures & tables
Figure 2: Overview of the TrajTok architecture. TrajTok comprises a trajectory segmenter and a trajectory encoder . The segmenter proposes trajectory masks for all objects in an image or video within a single forward pass. The encoder then aggregates raw video pixels or encoded visual features (parameterized by f in the figure) according to these masks to produce trajectory tokens. The number of tokens per trajectory can be flexibly adjusted based on the available compute budget.
Figure 3: Training with downstream understanding tasks reshapes the segmentation granularity. We visualize the trajectory masks produced by our segmenter when trained with only segmentation supervision versus jointly with segmentation and CLIP objectives. The CLIP objective reshapes the segmentation granularity, producing finer foreground object masks while merging background regions.
Figure 4: TrajTok is a versatile module applicable across pretraining, feature adaptation, and finetuning stages. We demonstrate its use in three scenarios: TrajViT2, which trains a visual encoder from scratch; TrajAdapter, which adapts pretrained features for downstream tasks; and TrajVLM, which uses TrajTok as a connector in LLaVA-style large vision–language models.
Table 4Figure 5
VideoMAE-v2 [ 27 ]
V-JEPA2 [ 2 ]
Probing Method
K400 [ 34 ]
SSv2 [ 29 ]
K400 [ 34 ]
SSv2 [ 29 ]
Linear probing
79.4
59.1
84.5
73.7
Attentive probing
80.2
59.7
85.1
74.2
Perceiver probing
79.9
59.8
84.7
74.2
TrajAdapter (1 token/traj)
82.0
60.4
87.2
74.6
TrajAdapter (2 token/traj)
82.4
60.8
87.8
74.9
Table 3: Probing results on different video backbones. Top-1 accuracy (%) on Kinetics-400 (K400) and Something-Something-V2 (SSv2) using various probing strategies. TrajAdapter consistently improves the downstream probing accuracy of pretrained backbone features, and performance further increases with more tokens per trajectory.
Figure 7: Trajectory masks produced by our segmenter vs. by TrajViT pipeline. While our segmenter produces coarser masks and may miss very small objects, it demonstrates strong semantic grouping ability that is sufficient for downstream understanding tasks.
Figure 8: VideoQA results for TrajTok applying to large vision-language model. VLM with TrajTok as connector (TrajVLM) notably outperforms patch pooling baseline (PatchVLM) in long-video benchmarks, while the performance is mixed for short-video benchmark.
Module
Variation
VEQ (%)
STQ (%)
Retrieval (R@5)
Default Architecture
42.3
70.1
22.1
Backbone
no hierarchical features
39.3 ( ↓ 3.0)
66.2 ( ↓ 3.9)
19.2 ( ↓ 2.9)
Output Res.
56 → 224
44.1 ( ↑ 1.8)
73.0 ( ↑ 2.9)
22.0 ( ↓ 0.1)
Perceiver
no detach gradient
34.1 ( ↓ 8.2)
59.3 ( ↓ 10.8)
18.3 ( ↓ 3.8)
Seg. Loss
− dice loss
39.0 ( ↓ 3.3)
68.9 ( ↓ 1.2)
16.7 ( ↓ 5.4)
Seg. Loss
− focal loss
41.2 ( ↓ 1.1)
67.4 ( ↓ 2.7)
22.1 (0.0)
Table 4: Ablations of segmenter design.
Module
Variation
1 token/traj
2 token/traj
4 token/traj
Default Architecture
22.1
23.0
23.2
Attention Mask
w/o mask
17.4 ( ↓ 4.7)
17.9 ( ↓ 5.1)
18.3 ( ↓ 4.9)
Query Init.
Fourier → random
22.1 (0.0)
22.5 ( ↓ 0.5)
21.9 ( ↓ 1.3)
Perceiver Depth
2 → 4
22.3 ( ↑ 0.2)
22.9 ( ↓ 0.1)
23.5 ( ↑ 0.3)
Table 5: Ablations of trajectory encoder design.
Hyperparameter
Value
Resolution
224
Frame sampling
uniform 16 frames (videos only)
Optimizer
AdamW
Base LR
1×10−3
Weight decay
0.01
Optimizer momentum
β1=0.9,β2=0.999
Table 6: Hyperparameters used for segmenter pretraining.
Figure 9: Qualitative Examples of the trajectory masks produced by our segmenter.
VIPSeg Benchmark
VEQ-SQ
VEQ-RQ
STQ-EN
EntitySAM
84.7
64.5
43.3
SAM 2.1
83.1
36.7
41.7
Ours (scaled-up)
85.5
45.1
40.2
Table 7: Evaluation of a scaled-up version of our segmenter on the ViPEntitySeg benchmark. This model is not used in any main-paper experiments; it serves only as an isolated study of segmentation quality under increased capacity.
Category
Dataset Name(s)
Notes / Source
Academic VideoQA
llava_video_mc_academic
MC-style QA
llava_video_oe_academic
Open-ended QA
clevrer
Causal & counterfactual reasoning
funqa
Fine-grained temporal QA
star
Long-horizon procedural QA
intent_qa
Human intent reasoning
Table 8: Datasets used for training TrajVLM under the “final” mixture. This mixture combines a large set of academic VideoQA datasets, temporal reasoning datasets, and synthetic captioning/QA corpora.
Hyperparameter
Value
Max video frames
128
Total training steps
10,000
Training stages
Pretraining + Instruction tuning
Sequence length
8192
Global batch size
32
Device batch size
4
Table 9: Key hyperparameters used for training TrajVLM. Values reflect the shared configuration across all VLM experiments.
Video tokenization is fundamental to scalable video generation, as the number of tokens directly determines the computational cost and the length of videos that can be modeled. Existing tokenizers mainly improve scalability by compressing videos into fewer tokens, but they often continue to represent persistent content, such as static backgrounds and consistent object appearances, repeatedly across frames and chunks. In this paper, we propose TivTok (Time-Invariant Tokenizer), a reuse-aware video tokenizer that makes persistent information reusable across time. TivTok represents a clip with Time-Invariant (TIV) tokens that encode information shared across frames and Time-Variant (TV) tokens that encode frame-specific residuals. To obtain this factorization, we introduce Scope-Induced Factorization (SIF), which assigns different attention scopes to the two token groups: TIV tokens attend to the full clip, whereas each TV token only accesses its corresponding frame together with the TIV tokens. In the decoder, Invariant Broadcasting (IB) reuses the same TIV tokens across frames and chunks for parallel reconstruction and long-video tokenization. Experiments show that TivTok achieves an rFVD of 12.65 on the standard 16×256×256 benchmark and improves compression efficiency by 2.91× for 128-frame videos compared with the evaluated baselines, while using only 1.1% of the tokens required by downsample-based tokenizers in our evaluation.
Weiliang Chen, Yuanhui Huang, Xuebo Wang +1
Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China · Department of Automation, Tsinghua University, Beijing, 100084, China · Kuaishou Technology, Beijing, China
Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video's inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner -- where the first tokens (emergently) capture abstract information, such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count according to downstream needs and encoding videos longer than the baselines with the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.
Andrei Atanov, Jesse Allardice, Roman Bachmann +6
Apple · Swiss Federal Institute of Technology Lausanne (EPFL)
Although Video Large Language Models (Video LLMs) have shown strong performance in video understanding, their efficiency is still limited by the large number of visual tokens. Existing video token compression methods typically rely on frame-wise saliency or heuristic token merging, which can over-focus on locally salient regions and produce ambiguous fused features. To address these issues, we propose GeoTrace, a training-free spatiotemporal token compression framework that decomposes video evidence into exact skeleton tokens and traceable residual event tokens. Specifically, Contextual Farthest-Point Anchoring (CFPA) preserves salient, context-consistent, and high-coverage skeleton tokens, while Trajectory-Constrained Residual Condensation (TCRC) compresses residual tokens through one-to-one temporal trajectories and constrained near-manifold condensation, producing traceable event tokens with reduced ambiguity. We evaluate GeoTrace on four Video LLMs across four video understanding benchmarks, and the results demonstrate its effectiveness and generalization across different model architectures and scenarios. On LLaVA-OneVision, with only 10% visual tokens retained, GeoTrace achieves a 12.99× TFLOPs reduction while preserving 99.1% of the vanilla performance. Overall, GeoTrace offers a compact and traceable token representation for efficient and robust Video LLM inference. Code is available at Code.
Guohuan Xie, Mengqi Lei, Chuan Shi +3
BNRist, THUIBCS, BLBCI, School of Software, Tsinghua University · Yangtze Delta Region Institute, Tsinghua University · Beijing University of Posts and Telecommunications, Beijing, China