Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video's inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner -- where the first tokens (emergently) capture abstract information, such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count according to downstream needs and encoding videos longer than the baselines with the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.
Figures & tables
Figure 2: VideoFlexTok reconstructions from a variable number of tokens. We find that just a few VideoFlexTok tokens capture information such as the semantic identities (e.g., a woman in the right example), scene geometry (the “arch” ), camera motion (moving forward), and object motion (rotation).
Figure 3: VideoFlexTok overview. The encoder takes the spatiotemporal VAE video latents, interleaves them with learnable register tokens across the time dimension, and passes them through the Transformer with a time-causal attention pattern. This results in a 2D representation with the temporal and coarse-to-fine dimensions . Nested dropout randomly drops a random number of last register tokens along the 2nd dimension, inducing the coarse-to-fine structure. The decoder is a flow-based generative model that temporally interleaves masked register tokens with noisy VAE latents and passes them through a time-causal Transformer. The losses are: 1) the rectified flow reconstructive loss that predicts clean patches from their noised version and tokens, and 2) the representation alignment loss ( Yu et al., 2025 ) between the decoder and DINOv2 ( Oquab et al., 2023 ) features, which distills semantic information into VideoFlexTok tokens.
Figure 4: Probing the first VideoFlexTok tokens. We design the following probing experiment to analyze the information contained in the first VideoFlexTok tokens. Given a source video, we keep only one or two tokens per latent frame and make an isolated change to its first frame, e.g., changing an orange to an apple, using Nano Banana ( Google, 2025 ) . We then condition the decoder on both the original tokens and the new edited frame for reconstruction. We find that, in most cases, VideoFlexTok preserves the motion pattern from the original video and visual appearance from the edited frame throughout the reconstructed video, suggesting that the first tokens primarily capture the motion information.
Figure 5: Flexible-length autoregressive text-to-video generation. A text-to-video generative model using VideoFlexTok tokens can generate token sequences of varying length for a given conditioning. All token budgets lead to plausible generations, with 2-4 tokens/frame capturing the overall scene details and motion described in the text conditioning well (e.g., the balloon movement), while generating more tokens can express more fine-grained details (e.g., the number of floors).
Figure 6: Compute-efficient AR training with VideoFlexTok . We show how the fidelity (top) and alignment (bottom) metrics change across three complementary scaling axes. Scaling the model size (left). We show how the fidelity (top, gFVD) and alignment (bottom, Classification Score) metrics change as we scale the size of the class-to-video autoregressive model. Using VideoFlexTok maintains good fidelity across a wider range of model sizes while solving the conditioning task well, achieving a much higher alignment score with smaller models. This implies that we can train much smaller models to solve the class-to-video downstream task. Scaling the number of training tokens (middle). We show how the fidelity (top, gFVD) and alignment (bottom, Classification Score) metrics evolve during training of the class-to-video downstream model. We use the 1.3B AR model size for this experiment. Using VideoFlexTok enables having good reconstructions throughout the whole training, and achieves similar or better alignment using 5–10 times fewer training tokens than the 3D Grid tokenizer. Text-to-video FLOPs efficiency (right). We show how the fidelity (top, gFVD) and alignment (bottom, ViCLIP Score) metrics change as we scale both the model size and the number of training tokens in the compute-optimal way. We vary the model size from 400M to 5.2B with the estimated ratio of D/N=20 ( Hoffmann et al., 2022 ) . In addition to models trained on full-length VideoFlexTok token sequences, we train a model on shorter 32-token sequences per latent frame while keeping the number of steps the same, resulting in much lower computational cost (purple line). Overall, we find that autoregressive generative modeling over VideoFlexTok tokens can achieve similar performance using an order of magnitude less compute.
Figure 7: Flexible-length generation. We measure fidelity (top, gFVD), and alignment (bottom, classification score and ViCLIP similarity, see Section 4.1 ) for VideoFlexTok and 3D grid tokenizers on class-to-video (left) and text-to-video (right) tasks. Using much fewer tokens, VideoFlexTok maintains fidelity comparable to or better than the 3D tokenizer, while achieving higher alignment, i.e., better solving the corresponding conditional generation task.
Figure 8: Long text-to-video generation. We show an exemplar generation of a 10-second 81-frame video using only 672 tokens (32 tokens per frame).
Tokenizer
# Tokens
rFVD ( ↓ )
gFVD ( ↓ )
Cls. Score ( ↑ )
VidTok FSQ ( Tang et al., 2024 )
1280
84.1
131.7
0.799
Cosmos-DV ( NVIDIA et al., 2025 )
1280
220.5
187.6
0.825
Omnitokenizer ( Junke et al., 2024 )
1280
63.6
102.6
0.858
LARP ( Wang et al., 2024a )
1024
42.1
87.5
0.739
VideoFlexTok
5-1280
48.7 †
80.0 †
0.833 †
Table 1: System-level comparison on Kinetics-600 class-to-video generation. For each tokenizer, we train a 2.2B class-conditional AR model and evaluate their reconstruction (rFVD) and generation quality in terms of fidelity (gFVD) and alignment (Cls. Score). † indicates VideoFlexTok results for a sequence of 160 tokens.
Register Structure
rFVD ( ↓ )
gFVD ( ↓ )
Flat (1D)
48.9
352.1
Time-causal (2D)
69.9
287.6
Table 2: 1D vs 2D registers structure.
Decoder Attention
rFVD ( ↓ )
gFVD ( ↓ ) 32 Tok
Full
58.3
211.5
Time-Causal
80.9
175.1
Table 3: Decoder self-attention pattern.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: REPA ( Yu et al., 2025 ) loss ablation. We compare tokenizers trained with and without the REPA loss on the class-to-video downstream task. We find that REPA inductive bias loss improves both the fidelity of the generated samples and the alignment with the class conditioning.
Figure 10: VideoFlexTok decoder attention ablation We ablate the decoder attention pattern by comparing the alignment score (Classification Score) on the class-to-video generative task. We find that causal attention leads to a higher alignment score with fewer tokens, suggesting the early tokens capture more semantic information in this case (see Section C.2 for discussion).
AR order
#Tokens
gFVD ( ↓ )
time-first
1
187.1
4
151.1
16
152.7
64
210.1
256
246.6
depth-first
256
242.6
Appendix
Table 4: Ablating different autoregressive orders over VideoFlexTok tokens. We compare autoregressive orders on the class-to-video downstream task. We use depth-first and time-first orders for the same underlying tokenizer. We find no significant difference when using all tokens. The time-first order allows adjusting the number of tokens during evaluation leading to better performance.
Figure 11: Hierarchical vs. raster-order generation. We compare the VideoFlexTok and 3D grid tokenizers’ performance at the same sequence length (1280 tokens). We find that hierarchical generation with VideoFlexTok leads to 1) better alignment (ViCLIP score) and 2) much better fidelity (gFVD) when not using classifier-free guidance.
Figure 12: Text-to-video inference cost analysis. We compare the inference cost of various configurations of the number of AR-generated tokens and the number of VideoFlexTok flow decoder steps. For each AR model size and the number of generated tokens, we perform {1,2,5,10,20,40} denoising steps and plot the corresponding lines. We find that for all considered AR sizes and inference budgets, the best performance is achieved by generating less than 256 tokens per frame and performing multiple denoising steps.
Figure 13: VideoFlexTok reconstruction example . From top to bottom, each row corresponds to a video reconstructed using 1, 2, 4, …, 256 tokens. The last row shows the original video.
Figure 14: VideoFlexTok reconstruction example . From top to bottom each row corresponds to a video reconstructed using 1, 2, 4, …, 256 tokens. The last row shows the original video.
Figure 15: VideoFlexTok reconstruction example . From top to bottom each row corresponds to a video reconstructed using 1, 2, 4, …, 256 tokens. The last row shows the original video.
Figure 16: VideoFlexTok reconstruction example . From top to bottom each row corresponds to a video reconstructed using 1, 2, 4, …, 256 tokens. The last row shows the original video.
VideoFlexTok configuration
d12-d12 (ablation setting)
d18-d18
d18-d28
Encoder depth denc
12
18
18
Decoder depth ddec
12
18
28
Encoder dim. wenc=64denc
768
1152
1152
Decoder dim. wdec=64ddec
768
1152
1792
Encoder Transformer parameters
84.9M
286.7M
286.7M
Decoder Transformer parameters
84.9M
286.7M
1.1B
Appendix
Table 5: VideoFlexTok training settings. Model and training configuration for different autoregressive Transformer sizes.
Configuration
AR 49M
AR 85M
AR 201M
AR 393M
AR 679M
AR 1.33B
AR 2.29B
Num. non-embedding parameters
49M
85M
201M
393M
679M
1.33B
2.29B
Decoder depth ddec
10
12
16
20
24
30
36
Decoder dim. wdec
640
768
1024
1280
1536
1920
2304
Cross-attention dim.
n/a
MLP ratio
4
Max. sequence length
1280
Appendix
Table 6: Class-conditioned AR training settings. Model and training configuration for different autoregressive Transformer sizes.
Configuration
AR 400M
AR 1.1B
AR 2.0B
AR 5.2B
AR 5.2B (4 × C)
Num. total parameters
400M
110M
2.02B
5.20B
5.20B
Decoder depth ddec
12
20
30
42
42
Decoder dim. wdec
768
1280
1920
2688
2688
Cross-attention dim.
12
20
30
42
42
MLP ratio
4
Max. sequence length
1280
Appendix
Table 7: Text-conditioned AR training settings. Model and training configuration for different autoregressive Transformer sizes.
Tokenizer
# Tok.
MAE ( ↓ )
MSE ( ↓ )
PSNR ( ↑ )
LPIPS ( ↓ )
rFID ( ↓ )
rFVD ( ↓ )
VidTok FSQ ( Tang et al., 2024 )
1280
0.0276
0.00270
25.64
0.0967
5.26
35.47
LARP ( Wang et al., 2024a )
1024
0.0476 ‡
0.00760 ‡
21.21 ‡
0.1111 ‡
5.50 ‡
28.33 ‡
Omnitokenizer ( Junke et al., 2024 )
1280
0.0206
0.00100
30.01
0.1199
7.43
38.67
Cosmos-DV ( NVIDIA et al., 2025 )
1280
0.0287
0.00284
25.46
0.122
8.51
47.20
VideoFlexTok d18-d28
5–1280
0.0607 †
0.01079 †
19.67 †
0.18545 †
4.97 †
20.53 †
Appendix
Table 8: Reconstruction metrics on MSR-VTT. Evaluation is performed on 5k MSR-VTT videos (17 frames, 4 FPS, 256 × 256). All tokenizers use the same decoder architecture. † indicates VideoFlexTok results using 1280 tokens per video. ‡ marks results obtained from models evaluated at 128×128 input resolution.
Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse tokens carry the clip's global semantics while later tokens further specify details. Existing flexible tokenizers only apply a representation-alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. SemanTok achieves high semantic alignment and video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model 3.4× its size, and larger SemanTok AR models further improve fidelity. It keeps semantic alignment on out-of-distribution classes and gives the decoder higher semantic alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later tokens.
Mikhail Dereviannykh, Vikram Voleti, Simon Donne +3
Video tokenization is fundamental to scalable video generation, as the number of tokens directly determines the computational cost and the length of videos that can be modeled. Existing tokenizers mainly improve scalability by compressing videos into fewer tokens, but they often continue to represent persistent content, such as static backgrounds and consistent object appearances, repeatedly across frames and chunks. In this paper, we propose TivTok (Time-Invariant Tokenizer), a reuse-aware video tokenizer that makes persistent information reusable across time. TivTok represents a clip with Time-Invariant (TIV) tokens that encode information shared across frames and Time-Variant (TV) tokens that encode frame-specific residuals. To obtain this factorization, we introduce Scope-Induced Factorization (SIF), which assigns different attention scopes to the two token groups: TIV tokens attend to the full clip, whereas each TV token only accesses its corresponding frame together with the TIV tokens. In the decoder, Invariant Broadcasting (IB) reuses the same TIV tokens across frames and chunks for parallel reconstruction and long-video tokenization. Experiments show that TivTok achieves an rFVD of 12.65 on the standard 16×256×256 benchmark and improves compression efficiency by 2.91× for 128-frame videos compared with the evaluated baselines, while using only 1.1% of the tokens required by downsample-based tokenizers in our evaluation.
Weiliang Chen, Yuanhui Huang, Xuebo Wang +1
Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China · Department of Automation, Tsinghua University, Beijing, 100084, China · Kuaishou Technology, Beijing, China
Autoregressive (AR) video generative models rely on video tokenizers that compress pixels into discrete token sequences. The length of these token sequences is crucial for balancing reconstruction quality against downstream generation computational cost. Traditional video tokenizers apply a uniform token assignment across temporal blocks of different videos, often wasting tokens on simple, static, or repetitive segments while underserving dynamic or complex ones. To address this inefficiency, we introduce EVATok, a framework to produce Efficient Video Adaptive Tokenizers. Our framework estimates optimal token assignments for each video to achieve the best quality-cost trade-off, develops lightweight routers for fast prediction of these optimal assignments, and trains adaptive tokenizers that encode videos based on the assignments predicted by routers. We demonstrate that EVATok delivers substantial improvements in efficiency and overall quality for video reconstruction and downstream AR generation. Enhanced by our advanced training recipe that integrates video semantic encoders, EVATok achieves superior reconstruction and state-of-the-art class-to-video generation on UCF-101, with at least 24.4% savings in average token usage compared to the prior state-of-the-art LARP and our fixed-length baseline.