TrackTok: Object-Centric Video Tokenization with Semantically Persistent Tokens

Nhi Pham; Christopher Wewer; Bernt Schiele; Jonas Fischer; Jan Eric Lenssen

cs.CVOct 24, 2026

TrackTok: Object-Centric Video Tokenization with Semantically Persistent Tokens

Authors: Nhi Pham, Christopher Wewer, Bernt Schiele, Jonas Fischer, Jan Eric Lenssen

Organizations: Max Planck Institute for Informatics, Saarland Informatics Campus, Germany · Saarland University, Saarland Informatics Campus, Germany

Abstract

We introduce TrackTok, an efficient video tokenization framework that produces object-centric tokens, i.e., tokens that consistently encode the same semantic objects across time, even as it moves, deforms, or undergoes partial occlusion. Conventional patch-based or existing holistic tokenizers produce token identities often tied to fixed image locations. In contrast, our approach learns temporally persistent object-centric representations that bind visual features across frames into stable semantic units. This enables substantially more compact, continuous latent video representations, reducing the number of tokens required for downstream generative modeling with state-of-the-art flow models, while preserving the semantic structure needed for high-quality synthesis. Our tokenizer offers a favorable quality-efficiency trade-off. TrackTok achieves comparable class-conditional generative FVD on UCF-101 dataset at 8 times fewer latent tokens and better latency, and outperforms existing 1D tokenization baselines on frame prediction task on Kinetics-600 across low token budgets. These results suggest that object-aligned and temporally persistent tokenization is a promising direction for scalable video generation, where reducing latent token counts is critical for efficient modeling. Our code for this work is available at https://github.com/phamleyennhi/tracktok.

Figures & tables

Figure 2: Method Overview. TrackTok encodes an input video $\mathbf{V}$ into a 1D set of object-centric, semantically persistent, latent tokens $\mathcal{Z}$ . Its encoder $\mathcal{E}$ is composed of semantic feature extraction with V-JEPA 2.1 Mur-Labadia et al. (2026) , the construction of input-dependent queries $\mathcal{Q}$ using centroids of $k$ -Means clusters $\mathcal{C}$ , and a ViT performing attention based on semantic similarity instead of spatial position. The decoder $\mathcal{D}$ reconstructs the three-dimensional RGB video from the 1D latent set $\mathcal{Z}$ .

Figure 3: Frame Prediction on Kinetics-600 , lower is better (gFVD). Ours outperforms baselines at low token regime.

Method	#Tokens	T/P (%)	Data Size	#Params		gFVD $\downarrow$
				Tokenizer	Generator	UCF-101
CV-VAE + DiT/XL Zhao et al. (2024)	4096	0.391	$>610$ M	182M	674M	262
OmniTok Wang et al. (2024a)	4096	0.391	$1.4$ M	82.2M	650M	191
CogVideo Hong et al. (2022)	2065	3.151	$5.4$ M	–	9.4B	626
MAGVIT-v2-AR Yu et al. (2024a)	1280	0.488	$1.8$ M	–	840M	109
TATS Ge et al. (2022)	1024	0.391	$<0.5$ M	32M	321M	332

Table 1: Class-Conditional Video Generation on UCF-101. We report generation FVD (gFVD) across methods under comparable token budgets, token-to-pixel (T/P) ratios, training data size, and number of trainable parameters. While prior methods achieve lower gFVD at moderate compression rates, TrackTok remains competitive even under extreme compression (128 tokens,

0.012\%

T/P), highlighting a favorable efficiency–quality trade-off in the low-token regime.

\dagger

denotes models trained using the same setup as ours. Best and second best are highlighted per token budget.

Figure 4: Left: UCF-101 reconstruction quality . Our rFVD remains stable while LARP and AdapTok significantly degrade at low token count. Right: Generation quality vs. latency , lower-is-better. Ours at $32$ tokens achieves $8\times$ lower latency and comparable quality to LARP at $1024$ tokens.

Figure 5: Class-Conditional Generation on UCF-101 Across Token Budgets. We show samples for diverse action categories, including hand motions (rows 1–2), dynamic natural scenes (row 3), and human body pose sequences (rows 4–5). Even at 32 tokens, our generated videos retain recognizable object structure, motion coherence, and semantic consistency across frames, comparable to LARP with $32\times$ as many tokens. We include more qualitative examples in Appendix E .

Figure 6: Token Impact Across Time. We sample four frames from the input clip together with reconstructed frames and token impact visualizations (c.f. Sec. 4.1 ). For each sample, we apply PCA on token features and map the top three components to RGB as color for the corresponding token impact mask. The visualization highlights the semantic object alignment and temporal tracking of tokens learned by TrackTok in contrast to AdapTok’s or LARP’s tokens, which are dominated by high-frequency textures and edges. Both baselines also show degraded reconstruction quality at 128 tokens. Furthermore, leveraging only V-JEPA 2.1 without our cluster queries (CQ) still results in tokens encoding fixed regions of the video. Refer to Appendix E for more uncurated examples.

	Obj. Discovery (per frame)			Tracking (across frames)
	Purity $\uparrow$	FG-ARI $\uparrow$	mBO $\uparrow$	FG-ARI $\uparrow$	mBO $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$
AdapTok	83.1	6.2	10.5	2.7	8.0	7.6
LARP	84.5	6.9	12.3	3.3	8.5	7.7
Ours w/o CQ	84.9	9.9	13.7	6.3	10.7	10.5
Ours	93.8	21.3	25.2	16.7	21.1	19.1

Table 2: Object Discovery and Tracking . For equal token counts, TrackTok’s tokens are more strongly aligned with individual objects as well as more temporally consistent than those of baselines.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
Architecture
Backbone (frozen)	V-JEPA2.1 ViT-L/16, patch $2{\times}16{\times}16$ , input $128{\times}128$
Encoder / decoder	Transformer, hidden size $768$ , $12$ layers, $12$ heads
Decoder output patch	$4{\times}8{\times}8$
Pooling head	shared-identity multi-query; query $=[\text{centroid}\,\\|\,\text{PE}\,\\|\,\text{identity}]$
K-means iterations	$20$

Appendix

Table A1: Autoencoder Training Configuration. Values are shared across all six token budgets unless listed per budget.

Component	Class-conditional	Frame prediction
Architecture
Backbone	LightningDiT-XL: hidden size $1152$ , $28$ layers, $16$ heads
Block	SwiGLU MLP (ratio $8/3$ ), RMSNorm, adaLN-Zero
Token embedding	linear projection of each latent token ( $d\to 1152$ )
Positional embedding	fixed sinusoidal, absolute (no RoPE)
Timestep embedding	sinusoidal, $256$ -dim

Appendix

Table A2: Denoiser (LightningDiT-XL) Training Configuration. Settings spanning both columns are shared by class-conditional generation (UCF-101) and frame prediction (Kinetics-600). The recipe is the same for every token budget; only the sequence length

N

and latent dimension

d

(Tab. A1 ) differ, plus the batch size and number of steps for

N{=}1024

Hyperparameter	Search grid	AdapTok	LARP	Ours w/o CQ	Ours
Token linking (across clips)
Context length $L$ (clips)	$\{1,2,4,8\}$	1	1	1	1
Neighbour radius $r$	$\{0.2,0.35,\text{--}\}$	0.2	0.2	0.35	0.35
Top- $k$ votes	$\{1,3,5\}$	3	5	5	5
Temperature $\tau$	$\{0.05,0.1,0.2\}$	0.05	0.2	0.1	0.1
Link threshold $\theta_{\mathrm{link}}$	$\{0.3,0.45,0.6,0.75,0.9\}$	0.3	0.3	0.9	0.9

Appendix

Table B1 : Token-linking hyperparameters used for the object discovery and tracking evaluation. Tokens are linked across consecutive 4-frame clips by cosine similarity of their bottleneck embeddings, then tracks are merged agglomeratively into object-level clusters. For every method the full joint grid (540 linking

\times

54 merge combinations) is swept on the YTVIS-2021 training split at

128\,

px, ranked by whole-video FG-ARI, and the best configuration is evaluated once on the 210 validation videos at

518\,

px. “Ours w/o CQ” reuses the configuration selected for the full model. “–” denotes a disabled component.

Setting	Value
Dataset / split (hyperparameter selection)	YTVIS-2021 train, $128\,$ px, without $\mathcal{J}\&\mathcal{F}$
Dataset / split (evaluation)	YTVIS-2021 val (210 videos), $518\,$ px
Clip length	4 frames
Tokens per clip / token dim.	28 / 64
Token $\to$ pixel assignment	Single-token ablation, per-pixel arg-max of reconstruction error
Linking feature	Bottleneck token embedding (cosine similarity)

Appendix

Table B2 : Evaluation settings shared by all tokenizers in the object discovery and tracking benchmark.

Figure C1 : Average Token Heatmap Videos. We visualize the impact of individual tokens on video pixels, averaged over the UCF-101 validation split. Each row shows a randomly selected token, independently per method. AdapTok and LARP (left) learns mostly tokens that encode individual video patches. Encoding V-JEPA 2.1 features instead of RGB, ours without cluster queries (middle) learns to encode the same spatial patches for all frames into shared tokens. This encoder behavior changes drastically by adding cluster queries as additional inductive bias, resulting in TrackTok (right) to assign tokens to semantic units depending on the input. Full visualizations in below.

Figure C2 : Average token heatmaps for LARP [ 1 ] . Most heatmaps are sharp blobs that stay anchored to the same image region across frames and across videos, recovering an obvious tile pattern that tiles the $128{\times}128$ frame. This confirms that most of LARP’s tokens are positionally bound: token identity is tied to a fixed spatial location rather than to scene content.

Figure C3 : Average token heatmaps for AdapTok [ 2 ] . Every token collapses to one or two sharp, point-like blobs that stay fixed to the same pixels across frames and across videos, i.e. AdapTok’s tokens are positionally bound like LARP’s.

Figure C4 : Average token heatmaps for TrackTok without cluster queries. Without the inductive bias, the encoder reverts to a positional tokenization scheme: each token represents a sharp, spatially-confined blob anchored to a fixed location in the frame, and the tokens jointly tile each frame. Comparing this figure to Fig. C5 isolates the effect of the semantic-clustering bottleneck, which breaks the position-token correspondence and lets token identity track scene content.

Figure C5 : Average token heatmaps for TrackTok . In contrast to the sharp, position-locked blobs produced by LARP (cf. Fig. C2 ), TrackTok ’s content-conditioned tokens yield aggregate maps that are nearly spatially uniform: no token consistently activates on a fixed image region across videos. This is the signature we expect from a content-allocated token set: the spatial support of token $\mathbf{z}_{i}$ depends on the scene rather than on its index, and is consistent with the visualizations in Fig. 6 , where each token covers semantically coherent objects whose spatial extent varies between clips.

Method	#Tokens	#Dim	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	rFVD $\downarrow$	gFVD $\downarrow$
Ours w/o PE in CQ	128	128	23.32	0.1381	0.7647	68.95	145.91
Ours w/ trilinear PE in CQ	128	128	23.37	0.1371	0.7661	65.99	133.02
Ours	128	128	23.36	0.1375	0.7657	66.04	125.52

Appendix

Table D1 : Ablation on PE of Cluster Queries

Method	#Tokens	#Dim	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	rFVD $\downarrow$	gFVD $\downarrow$
Frozen V-JEPA2.1	512	1024	23.99	0.1273	0.7850	47.55	127.65
Ours w/ V-JEPA2.1	128	128	23.36	0.1375	0.7657	66.04	125.52
Frozen VideoMAE	1568	1024	34.08	0.0284	0.9722	3.80	533.60
Ours w/ VideoMAE	128	128	28.88	0.0636	0.9203	25.28	220.73
Frozen DINOv3	1024	1024	20.92	0.1889	0.6592	127.85	339.03
Ours w/ DINOv3	128	128	20.57	0.2007	0.6451	126.19	204.23

Appendix

Table D2 : RAE as Encoder Ablation

Config.	#Tokens	gFVD $\downarrow$
Frozen V-JEPA2.1	512 (fixed)	127.65
Frozen V-JEPA2.1 + Average Pooling	128	218.12
Frozen V-JEPA2.1 + Subsampling	128	209.81
Frozen V-JEPA2.1 + Clustering	128	218.30
TrackTok (ours)	128	125.52

Appendix

Table D3 : Comparison of tokenization strategies. Lower gFVD is better.

Method	#Tokens	#Dim	$K$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	rFVD $\downarrow$	gFVD $\downarrow$
Ours	1024	16	1	21.81	0.1919	0.6743	200.12	386.38
Ours	1024	16	2	22.57	0.1726	0.7049	135.22	318.87
Ours	1024	16	4	23.79	0.1292	0.7795	62.79	116.50
Ours	1024	16	8	23.85	0.1280	0.7814	59.31	124.11
Ours	1024	16	16	23.16	0.1429	0.7569	69.22	226.26
Ours	1024	16	32	22.57	0.1552	0.7354	84.90	209.18

Appendix

Table D4 : Ablation on #Clusters

K

at 1024 tokens

Method	#Tokens	#Dim	$K$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	rFVD $\downarrow$	gFVD $\downarrow$
Ours	256	64	1	23.28	0.1550	0.7320	125.12	259.90
Ours	256	64	16	24.17	0.1391	0.7621	64.16	128.73
Ours	256	64	32	23.85	0.1454	0.7513	67.93	130.46
Ours	256	64	64	24.05	0.1418	0.7577	62.72	131.01

Appendix

Table D5 : Ablation on #Clusters

K

at 256 tokens

Method	#Tokens	#Dim	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	rFVD $\downarrow$	gFVD $\downarrow$
Ours	128	128	23.36	0.1375	0.7657	66.04	125.52
Ours	128	64	22.90	0.1469	0.7493	79.25	146.78
Ours	128	32	22.35	0.1592	0.7282	103.03	184.45
Ours	128	16	21.88	0.1710	0.7077	134.77	250.48

Appendix

Table D6 : Ablation on Token Dimension

Method	Representation	rFVD $\downarrow$	LPIPS $\downarrow$
Qwen2.5-VL + Wan2.1-T2V-1.3B (oracle-4)	Video caption	3033.90	0.7999
Qwen2.5-VL + Wan2.1-T2V-1.3B	Video caption	3097.57	0.8459
Ours (128 tokens)	Continuous tokens	63.89	0.1495

Appendix

Table D7 : Text-Based Baseline. Using a video-captioning model (Qwen2.5-VL) together with a text-to-video generator (Wan2.1-T2V-1.3B) as a video autoencoder fails to faithfully reconstruct the input video, in contrast to TrackTok.

Figure E1 : Unsupervised object tracking on a YouTube-VIS 2021 validation video ( $62$ frames, $4$ objects), one fixed colour per track. TrackTok keeps each runner in a single track for all frames, while the baselines fragment objects into many short-lived tracks and our TrackTok without cluster queries (CQ) binds tracks to image regions that are not semantically-aligned.

Figure E2 : TrackTok mostly keeps each object in one track for all frames, while the baselines fragment objects into many short-lived tracks.

Figure E3 : TrackTok mostly keeps each object in one track for all frames, while the baselines fragment objects into many short-lived tracks.

Figure E4 : Random qualitative reconstruction and token impact from TrackTok at $32$ tokens.

Figure E5 : Random qualitative reconstruction and token impact from TrackTok at $128$ tokens

Figure E6 : Random qualitative reconstruction and token impact from TrackTok at $256$ tokens.

Figure E7 : Generated video clips of $16\times 128\times 128$ resolution across randomly selected action classes in UCF-101, using LightningDiT trained on TrackTok latents at $32$ tokens.

Figure E8 : Generated video clips of $16\times 128\times 128$ resolution across randomly selected action classes in UCF-101, using LightningDiT trained on TrackTok latents at $128$ tokens.

Figure E9 : Generated video clips of $16\times 128\times 128$ resolution across randomly selected action classes in UCF-101, using LightningDiT trained on TrackTok latents at $256$ tokens.

Visual Tokenization

Visual tokenization turns images and video into compact, structured tokens for efficient perception, compression, and generation.

See all 70 papers

VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization
Apr 14, 2026Andrei Atanov, Jesse Allardice, Roman Bachmann +6Visual Tokenization Video Token Compression
Apple · Swiss Federal Institute of Technology Lausanne (EPFL)
TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenization
Jun 16, 2026Weiliang Chen, Yuanhui Huang, Xuebo Wang +1Visual Tokenization Video Token Compression
Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China · Department of Automation, Tsinghua University, Beijing, 100084, China · Kuaishou Technology, Beijing, China
TrajTok: Learning Trajectory Tokens enables better Video Understanding
Feb 26, 2026Chenhao Zheng, Jieyu Zhang, Jianing Zhang +6Visual Tokenization Video Representation Learning
University of Washington · Allen Institute for Artificial Intelligence · Woven by Toyota, Inc +1

Apr 14, 2026J/K move · Enter open · S save

VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

Andrei Atanov, Jesse Allardice, Roman Bachmann, Oğuzhan Fatih Kar+5

Apple · Swiss Federal Institute of Technology Lausanne (EPFL)

Visual Tokenization Video Token Compression Long Video Generation

Open paper AskPDF arXiv

Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video's inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner -- where the first tokens (emergently) capture abstract information, such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count according to downstream needs and encoding videos longer than the baselines with the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.

Abstract

Visual Tokenization

Explore similar work

VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenization

TrajTok: Learning Trajectory Tokens enables better Video Understanding