We introduce TrackTok, an efficient video tokenization framework that produces object-centric tokens, i.e., tokens that consistently encode the same semantic objects across time, even as it moves, deforms, or undergoes partial occlusion. Conventional patch-based or existing holistic tokenizers produce token identities often tied to fixed image locations. In contrast, our approach learns temporally persistent object-centric representations that bind visual features across frames into stable semantic units. This enables substantially more compact, continuous latent video representations, reducing the number of tokens required for downstream generative modeling with state-of-the-art flow models, while preserving the semantic structure needed for high-quality synthesis. Our tokenizer offers a favorable quality-efficiency trade-off. TrackTok achieves comparable class-conditional generative FVD on UCF-101 dataset at 8 times fewer latent tokens and better latency, and outperforms existing 1D tokenization baselines on frame prediction task on Kinetics-600 across low token budgets. These results suggest that object-aligned and temporally persistent tokenization is a promising direction for scalable video generation, where reducing latent token counts is critical for efficient modeling. Our code for this work is available at https://github.com/phamleyennhi/tracktok.
Figures & tables
Figure 2: Method Overview. TrackTok encodes an input video V into a 1D set of object-centric, semantically persistent, latent tokens Z . Its encoder E is composed of semantic feature extraction with V-JEPA 2.1 Mur-Labadia et al. (2026) , the construction of input-dependent queries Q using centroids of k -Means clusters C , and a ViT performing attention based on semantic similarity instead of spatial position. The decoder D reconstructs the three-dimensional RGB video from the 1D latent set Z .
Figure 3: Frame Prediction on Kinetics-600 , lower is better (gFVD). Ours outperforms baselines at low token regime.
Method
#Tokens
T/P (%)
Data Size
#Params
gFVD ↓
Tokenizer
Generator
UCF-101
CV-VAE + DiT/XL Zhao et al. (2024)
4096
0.391
>610 M
182M
674M
262
OmniTok Wang et al. (2024a)
4096
0.391
1.4 M
82.2M
650M
191
CogVideo Hong et al. (2022)
2065
3.151
5.4 M
–
9.4B
626
MAGVIT-v2-AR Yu et al. (2024a)
1280
0.488
1.8 M
–
840M
109
TATS Ge et al. (2022)
1024
0.391
<0.5 M
32M
321M
332
Table 1: Class-Conditional Video Generation on UCF-101. We report generation FVD (gFVD) across methods under comparable token budgets, token-to-pixel (T/P) ratios, training data size, and number of trainable parameters. While prior methods achieve lower gFVD at moderate compression rates, TrackTok remains competitive even under extreme compression (128 tokens, 0.012% T/P), highlighting a favorable efficiency–quality trade-off in the low-token regime. † denotes models trained using the same setup as ours. Best and second best are highlighted per token budget.
Figure 4: Left: UCF-101 reconstruction quality . Our rFVD remains stable while LARP and AdapTok significantly degrade at low token count. Right: Generation quality vs. latency , lower-is-better. Ours at 32 tokens achieves 8× lower latency and comparable quality to LARP at 1024 tokens.
Figure 5: Class-Conditional Generation on UCF-101 Across Token Budgets. We show samples for diverse action categories, including hand motions (rows 1–2), dynamic natural scenes (row 3), and human body pose sequences (rows 4–5). Even at 32 tokens, our generated videos retain recognizable object structure, motion coherence, and semantic consistency across frames, comparable to LARP with 32× as many tokens. We include more qualitative examples in Appendix E .
Figure 6: Token Impact Across Time. We sample four frames from the input clip together with reconstructed frames and token impact visualizations (c.f. Sec. 4.1 ). For each sample, we apply PCA on token features and map the top three components to RGB as color for the corresponding token impact mask. The visualization highlights the semantic object alignment and temporal tracking of tokens learned by TrackTok in contrast to AdapTok’s or LARP’s tokens, which are dominated by high-frequency textures and edges. Both baselines also show degraded reconstruction quality at 128 tokens. Furthermore, leveraging only V-JEPA 2.1 without our cluster queries (CQ) still results in tokens encoding fixed regions of the video. Refer to Appendix E for more uncurated examples.
Obj. Discovery (per frame)
Tracking (across frames)
Purity ↑
FG-ARI ↑
mBO ↑
FG-ARI ↑
mBO ↑
J&F↑
AdapTok
83.1
6.2
10.5
2.7
8.0
7.6
LARP
84.5
6.9
12.3
3.3
8.5
7.7
Ours w/o CQ
84.9
9.9
13.7
6.3
10.7
10.5
Ours
93.8
21.3
25.2
16.7
21.1
19.1
Table 2: Object Discovery and Tracking . For equal token counts, TrackTok’s tokens are more strongly aligned with individual objects as well as more temporally consistent than those of baselines.
Table A2: Denoiser (LightningDiT-XL) Training Configuration. Settings spanning both columns are shared by class-conditional generation (UCF-101) and frame prediction (Kinetics-600). The recipe is the same for every token budget; only the sequence length N and latent dimension d (Tab. A1 ) differ, plus the batch size and number of steps for N=1024 .
Hyperparameter
Search grid
AdapTok
LARP
Ours w/o CQ
Ours
Token linking (across clips)
Context length L (clips)
{1,2,4,8}
1
1
1
1
Neighbour radius r
{0.2,0.35,–}
0.2
0.2
0.35
0.35
Top- k votes
{1,3,5}
3
5
5
5
Temperature τ
{0.05,0.1,0.2}
0.05
0.2
0.1
0.1
Link threshold θlink
{0.3,0.45,0.6,0.75,0.9}
0.3
0.3
0.9
0.9
Appendix
Table B1 : Token-linking hyperparameters used for the object discovery and tracking evaluation. Tokens are linked across consecutive 4-frame clips by cosine similarity of their bottleneck embeddings, then tracks are merged agglomeratively into object-level clusters. For every method the full joint grid (540 linking × 54 merge combinations) is swept on the YTVIS-2021 training split at 128 px, ranked by whole-video FG-ARI, and the best configuration is evaluated once on the 210 validation videos at 518 px. “Ours w/o CQ” reuses the configuration selected for the full model. “–” denotes a disabled component.
Setting
Value
Dataset / split (hyperparameter selection)
YTVIS-2021 train, 128 px, without J&F
Dataset / split (evaluation)
YTVIS-2021 val (210 videos), 518 px
Clip length
4 frames
Tokens per clip / token dim.
28 / 64
Token → pixel assignment
Single-token ablation, per-pixel arg-max of reconstruction error
Linking feature
Bottleneck token embedding (cosine similarity)
Appendix
Table B2 : Evaluation settings shared by all tokenizers in the object discovery and tracking benchmark.
Figure C1 : Average Token Heatmap Videos. We visualize the impact of individual tokens on video pixels, averaged over the UCF-101 validation split. Each row shows a randomly selected token, independently per method. AdapTok and LARP (left) learns mostly tokens that encode individual video patches. Encoding V-JEPA 2.1 features instead of RGB, ours without cluster queries (middle) learns to encode the same spatial patches for all frames into shared tokens. This encoder behavior changes drastically by adding cluster queries as additional inductive bias, resulting in TrackTok (right) to assign tokens to semantic units depending on the input. Full visualizations in below.
Figure C2 : Average token heatmaps for LARP [ 1 ] . Most heatmaps are sharp blobs that stay anchored to the same image region across frames and across videos, recovering an obvious tile pattern that tiles the 128×128 frame. This confirms that most of LARP’s tokens are positionally bound: token identity is tied to a fixed spatial location rather than to scene content.
Figure C3 : Average token heatmaps for AdapTok [ 2 ] . Every token collapses to one or two sharp, point-like blobs that stay fixed to the same pixels across frames and across videos, i.e. AdapTok’s tokens are positionally bound like LARP’s.
Figure C4 : Average token heatmaps for TrackTok without cluster queries. Without the inductive bias, the encoder reverts to a positional tokenization scheme: each token represents a sharp, spatially-confined blob anchored to a fixed location in the frame, and the tokens jointly tile each frame. Comparing this figure to Fig. C5 isolates the effect of the semantic-clustering bottleneck, which breaks the position-token correspondence and lets token identity track scene content.
Figure C5 : Average token heatmaps for TrackTok . In contrast to the sharp, position-locked blobs produced by LARP (cf. Fig. C2 ), TrackTok ’s content-conditioned tokens yield aggregate maps that are nearly spatially uniform: no token consistently activates on a fixed image region across videos. This is the signature we expect from a content-allocated token set: the spatial support of token zi depends on the scene rather than on its index, and is consistent with the visualizations in Fig. 6 , where each token covers semantically coherent objects whose spatial extent varies between clips.
Method
#Tokens
#Dim
PSNR ↑
LPIPS ↓
SSIM ↑
rFVD ↓
gFVD ↓
Ours w/o PE in CQ
128
128
23.32
0.1381
0.7647
68.95
145.91
Ours w/ trilinear PE in CQ
128
128
23.37
0.1371
0.7661
65.99
133.02
Ours
128
128
23.36
0.1375
0.7657
66.04
125.52
Appendix
Table D1 : Ablation on PE of Cluster Queries
Method
#Tokens
#Dim
PSNR ↑
LPIPS ↓
SSIM ↑
rFVD ↓
gFVD ↓
Frozen V-JEPA2.1
512
1024
23.99
0.1273
0.7850
47.55
127.65
Ours w/ V-JEPA2.1
128
128
23.36
0.1375
0.7657
66.04
125.52
Frozen VideoMAE
1568
1024
34.08
0.0284
0.9722
3.80
533.60
Ours w/ VideoMAE
128
128
28.88
0.0636
0.9203
25.28
220.73
Frozen DINOv3
1024
1024
20.92
0.1889
0.6592
127.85
339.03
Ours w/ DINOv3
128
128
20.57
0.2007
0.6451
126.19
204.23
Appendix
Table D2 : RAE as Encoder Ablation
Config.
#Tokens
gFVD ↓
Frozen V-JEPA2.1
512 (fixed)
127.65
Frozen V-JEPA2.1 + Average Pooling
128
218.12
Frozen V-JEPA2.1 + Subsampling
128
209.81
Frozen V-JEPA2.1 + Clustering
128
218.30
TrackTok (ours)
128
125.52
Appendix
Table D3 : Comparison of tokenization strategies. Lower gFVD is better.
Method
#Tokens
#Dim
K
PSNR ↑
LPIPS ↓
SSIM ↑
rFVD ↓
gFVD ↓
Ours
1024
16
1
21.81
0.1919
0.6743
200.12
386.38
Ours
1024
16
2
22.57
0.1726
0.7049
135.22
318.87
Ours
1024
16
4
23.79
0.1292
0.7795
62.79
116.50
Ours
1024
16
8
23.85
0.1280
0.7814
59.31
124.11
Ours
1024
16
16
23.16
0.1429
0.7569
69.22
226.26
Ours
1024
16
32
22.57
0.1552
0.7354
84.90
209.18
Appendix
Table D4 : Ablation on #Clusters K at 1024 tokens
Method
#Tokens
#Dim
K
PSNR ↑
LPIPS ↓
SSIM ↑
rFVD ↓
gFVD ↓
Ours
256
64
1
23.28
0.1550
0.7320
125.12
259.90
Ours
256
64
16
24.17
0.1391
0.7621
64.16
128.73
Ours
256
64
32
23.85
0.1454
0.7513
67.93
130.46
Ours
256
64
64
24.05
0.1418
0.7577
62.72
131.01
Appendix
Table D5 : Ablation on #Clusters K at 256 tokens
Method
#Tokens
#Dim
PSNR ↑
LPIPS ↓
SSIM ↑
rFVD ↓
gFVD ↓
Ours
128
128
23.36
0.1375
0.7657
66.04
125.52
Ours
128
64
22.90
0.1469
0.7493
79.25
146.78
Ours
128
32
22.35
0.1592
0.7282
103.03
184.45
Ours
128
16
21.88
0.1710
0.7077
134.77
250.48
Appendix
Table D6 : Ablation on Token Dimension
Method
Representation
rFVD ↓
LPIPS ↓
Qwen2.5-VL + Wan2.1-T2V-1.3B (oracle-4)
Video caption
3033.90
0.7999
Qwen2.5-VL + Wan2.1-T2V-1.3B
Video caption
3097.57
0.8459
Ours (128 tokens)
Continuous tokens
63.89
0.1495
Appendix
Table D7 : Text-Based Baseline. Using a video-captioning model (Qwen2.5-VL) together with a text-to-video generator (Wan2.1-T2V-1.3B) as a video autoencoder fails to faithfully reconstruct the input video, in contrast to TrackTok.
Figure E1 : Unsupervised object tracking on a YouTube-VIS 2021 validation video ( 62 frames, 4 objects), one fixed colour per track. TrackTok keeps each runner in a single track for all frames, while the baselines fragment objects into many short-lived tracks and our TrackTok without cluster queries (CQ) binds tracks to image regions that are not semantically-aligned.
Figure E2 : TrackTok mostly keeps each object in one track for all frames, while the baselines fragment objects into many short-lived tracks.
Figure E3 : TrackTok mostly keeps each object in one track for all frames, while the baselines fragment objects into many short-lived tracks.
Figure E4 : Random qualitative reconstruction and token impact from TrackTok at 32 tokens.
Figure E5 : Random qualitative reconstruction and token impact from TrackTok at 128 tokens
Figure E6 : Random qualitative reconstruction and token impact from TrackTok at 256 tokens.
Figure E7 : Generated video clips of 16×128×128 resolution across randomly selected action classes in UCF-101, using LightningDiT trained on TrackTok latents at 32 tokens.
Figure E8 : Generated video clips of 16×128×128 resolution across randomly selected action classes in UCF-101, using LightningDiT trained on TrackTok latents at 128 tokens.
Figure E9 : Generated video clips of 16×128×128 resolution across randomly selected action classes in UCF-101, using LightningDiT trained on TrackTok latents at 256 tokens.
Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video's inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner -- where the first tokens (emergently) capture abstract information, such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count according to downstream needs and encoding videos longer than the baselines with the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.
Andrei Atanov, Jesse Allardice, Roman Bachmann +6
Apple · Swiss Federal Institute of Technology Lausanne (EPFL)
Video tokenization is fundamental to scalable video generation, as the number of tokens directly determines the computational cost and the length of videos that can be modeled. Existing tokenizers mainly improve scalability by compressing videos into fewer tokens, but they often continue to represent persistent content, such as static backgrounds and consistent object appearances, repeatedly across frames and chunks. In this paper, we propose TivTok (Time-Invariant Tokenizer), a reuse-aware video tokenizer that makes persistent information reusable across time. TivTok represents a clip with Time-Invariant (TIV) tokens that encode information shared across frames and Time-Variant (TV) tokens that encode frame-specific residuals. To obtain this factorization, we introduce Scope-Induced Factorization (SIF), which assigns different attention scopes to the two token groups: TIV tokens attend to the full clip, whereas each TV token only accesses its corresponding frame together with the TIV tokens. In the decoder, Invariant Broadcasting (IB) reuses the same TIV tokens across frames and chunks for parallel reconstruction and long-video tokenization. Experiments show that TivTok achieves an rFVD of 12.65 on the standard 16×256×256 benchmark and improves compression efficiency by 2.91× for 128-frame videos compared with the evaluated baselines, while using only 1.1% of the tokens required by downsample-based tokenizers in our evaluation.
Weiliang Chen, Yuanhui Huang, Xuebo Wang +1
Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China · Department of Automation, Tsinghua University, Beijing, 100084, China · Kuaishou Technology, Beijing, China
Tokenization in video models, typically through patchification, generates an excessive and redundant number of tokens. This severely limits video efficiency and scalability. While recent trajectory-based tokenizers offer a promising solution by decoupling video duration from token count, they rely on complex external segmentation and tracking pipelines that are slow and task-agnostic. We propose TrajTok, an end-to-end video tokenizer module that is fully integrated and co-trained with video models for a downstream objective, dynamically adapting its token granularity to semantic complexity, independent of video duration. TrajTok contains a unified segmenter that performs implicit clustering over pixels in both space and time to directly produce object trajectories in a single forward pass. By prioritizing downstream adaptability over pixel-perfect segmentation fidelity, TrajTok is lightweight and efficient, yet empirically improves video understanding performance. With TrajTok, we implement a video CLIP model trained from scratch (TrajViT2). It achieves the best accuracy at scale across both classification and retrieval benchmarks, while maintaining efficiency comparable to the best token-merging methods. TrajTok also proves to be a versatile component beyond its role as a tokenizer. We show that it can be seamlessly integrated as either a probing head for pretrained visual features (TrajAdapter) or an alignment connector in vision-language models (TrajVLM) with especially strong performance in long-video reasoning.
Chenhao Zheng, Jieyu Zhang, Jianing Zhang +6
University of Washington · Allen Institute for Artificial Intelligence · Woven by Toyota, Inc +1