cs.CVApr 14, 2026
SaveVideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization
Organizations: Apple · Swiss Federal Institute of Technology Lausanne (EPFL)
Abstract
Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video's inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner -- where the first tokens (emergently) capture abstract information, such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count according to downstream needs and encoding videos longer than the baselines with the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.
Figures & tables
Figure 2: VideoFlexTok reconstructions from a variable number of tokens. We find that just a few VideoFlexTok tokens capture information such as the semantic identities (e.g., a woman in the right example), scene geometry (the “arch” ), camera motion (moving forward), and object motion (rotation).
Figure 3: VideoFlexTok overview. The encoder takes the spatiotemporal VAE video latents, interleaves them with learnable register tokens across the time dimension, and passes them through the Transformer with a time-causal attention pattern. This results in a 2D representation with the temporal and coarse-to-fine dimensions . Nested dropout randomly drops a random number of last register tokens along the 2nd dimension, inducing the coarse-to-fine structure. The decoder is a flow-based generative model that temporally interleaves masked register tokens with noisy VAE latents and passes them through a time-causal Transformer. The losses are: 1) the rectified flow reconstructive loss that predicts clean patches from their noised version and tokens, and 2) the representation alignment loss ( Yu et al., 2025 ) between the decoder and DINOv2 ( Oquab et al., 2023 ) features, which distills semantic information into VideoFlexTok tokens.
Figure 4: Probing the first VideoFlexTok tokens. We design the following probing experiment to analyze the information contained in the first VideoFlexTok tokens. Given a source video, we keep only one or two tokens per latent frame and make an isolated change to its first frame, e.g., changing an orange to an apple, using Nano Banana ( Google, 2025 ) . We then condition the decoder on both the original tokens and the new edited frame for reconstruction. We find that, in most cases, VideoFlexTok preserves the motion pattern from the original video and visual appearance from the edited frame throughout the reconstructed video, suggesting that the first tokens primarily capture the motion information.
Figure 5: Flexible-length autoregressive text-to-video generation. A text-to-video generative model using VideoFlexTok tokens can generate token sequences of varying length for a given conditioning. All token budgets lead to plausible generations, with 2-4 tokens/frame capturing the overall scene details and motion described in the text conditioning well (e.g., the balloon movement), while generating more tokens can express more fine-grained details (e.g., the number of floors).
Figure 6: Compute-efficient AR training with VideoFlexTok . We show how the fidelity (top) and alignment (bottom) metrics change across three complementary scaling axes. Scaling the model size (left). We show how the fidelity (top, gFVD) and alignment (bottom, Classification Score) metrics change as we scale the size of the class-to-video autoregressive model. Using VideoFlexTok maintains good fidelity across a wider range of model sizes while solving the conditioning task well, achieving a much higher alignment score with smaller models. This implies that we can train much smaller models to solve the class-to-video downstream task. Scaling the number of training tokens (middle). We show how the fidelity (top, gFVD) and alignment (bottom, Classification Score) metrics evolve during training of the class-to-video downstream model. We use the 1.3B AR model size for this experiment. Using VideoFlexTok enables having good reconstructions throughout the whole training, and achieves similar or better alignment using 5–10 times fewer training tokens than the 3D Grid tokenizer. Text-to-video FLOPs efficiency (right). We show how the fidelity (top, gFVD) and alignment (bottom, ViCLIP Score) metrics change as we scale both the model size and the number of training tokens in the compute-optimal way. We vary the model size from 400M to 5.2B with the estimated ratio of ( Hoffmann et al., 2022 ) . In addition to models trained on full-length VideoFlexTok token sequences, we train a model on shorter 32-token sequences per latent frame while keeping the number of steps the same, resulting in much lower computational cost (purple line). Overall, we find that autoregressive generative modeling over VideoFlexTok tokens can achieve similar performance using an order of magnitude less compute.
Figure 7: Flexible-length generation. We measure fidelity (top, gFVD), and alignment (bottom, classification score and ViCLIP similarity, see Section 4.1 ) for VideoFlexTok and 3D grid tokenizers on class-to-video (left) and text-to-video (right) tasks. Using much fewer tokens, VideoFlexTok maintains fidelity comparable to or better than the 3D tokenizer, while achieving higher alignment, i.e., better solving the corresponding conditional generation task.
Figure 8: Long text-to-video generation. We show an exemplar generation of a 10-second 81-frame video using only 672 tokens (32 tokens per frame).
| Tokenizer | # Tokens | rFVD ( ) | gFVD ( ) | Cls. Score ( ) |
|---|---|---|---|---|
| VidTok FSQ ( Tang et al., 2024 ) | 1280 | 84.1 | 131.7 | 0.799 |
| Cosmos-DV ( NVIDIA et al., 2025 ) | 1280 | 220.5 | 187.6 | 0.825 |
| Omnitokenizer ( Junke et al., 2024 ) | 1280 | 63.6 | 102.6 | 0.858 |
| LARP ( Wang et al., 2024a ) | 1024 | 42.1 | 87.5 | 0.739 |
| VideoFlexTok | 5-1280 | 48.7 † | 80.0 † | 0.833 † |
Table 1: System-level comparison on Kinetics-600 class-to-video generation. For each tokenizer, we train a 2.2B class-conditional AR model and evaluate their reconstruction (rFVD) and generation quality in terms of fidelity (gFVD) and alignment (Cls. Score). † indicates VideoFlexTok results for a sequence of 160 tokens.
| Register Structure | rFVD ( ) | gFVD ( ) |
|---|---|---|
| Flat (1D) | 48.9 | 352.1 |
| Time-causal (2D) | 69.9 | 287.6 |
Table 2: 1D vs 2D registers structure.
| Decoder Attention | rFVD ( ) | gFVD ( ) 32 Tok |
|---|---|---|
| Full | 58.3 | 211.5 |
| Time-Causal | 80.9 | 175.1 |
Table 3: Decoder self-attention pattern.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: REPA ( Yu et al., 2025 ) loss ablation. We compare tokenizers trained with and without the REPA loss on the class-to-video downstream task. We find that REPA inductive bias loss improves both the fidelity of the generated samples and the alignment with the class conditioning.
Figure 10: VideoFlexTok decoder attention ablation We ablate the decoder attention pattern by comparing the alignment score (Classification Score) on the class-to-video generative task. We find that causal attention leads to a higher alignment score with fewer tokens, suggesting the early tokens capture more semantic information in this case (see Section C.2 for discussion).
| AR order | #Tokens | gFVD ( ) |
|---|---|---|
| time-first | 1 | 187.1 |
| 4 | 151.1 | |
| 16 | 152.7 | |
| 64 | 210.1 | |
| 256 | 246.6 | |
| depth-first | 256 | 242.6 |
Table 4: Ablating different autoregressive orders over VideoFlexTok tokens. We compare autoregressive orders on the class-to-video downstream task. We use depth-first and time-first orders for the same underlying tokenizer. We find no significant difference when using all tokens. The time-first order allows adjusting the number of tokens during evaluation leading to better performance.
Figure 11: Hierarchical vs. raster-order generation. We compare the VideoFlexTok and 3D grid tokenizers’ performance at the same sequence length (1280 tokens). We find that hierarchical generation with VideoFlexTok leads to 1) better alignment (ViCLIP score) and 2) much better fidelity (gFVD) when not using classifier-free guidance.
Figure 12: Text-to-video inference cost analysis. We compare the inference cost of various configurations of the number of AR-generated tokens and the number of VideoFlexTok flow decoder steps. For each AR model size and the number of generated tokens, we perform denoising steps and plot the corresponding lines. We find that for all considered AR sizes and inference budgets, the best performance is achieved by generating less than 256 tokens per frame and performing multiple denoising steps.
Figure 13: VideoFlexTok reconstruction example . From top to bottom, each row corresponds to a video reconstructed using 1, 2, 4, …, 256 tokens. The last row shows the original video.
Figure 14: VideoFlexTok reconstruction example . From top to bottom each row corresponds to a video reconstructed using 1, 2, 4, …, 256 tokens. The last row shows the original video.
Figure 15: VideoFlexTok reconstruction example . From top to bottom each row corresponds to a video reconstructed using 1, 2, 4, …, 256 tokens. The last row shows the original video.
Figure 16: VideoFlexTok reconstruction example . From top to bottom each row corresponds to a video reconstructed using 1, 2, 4, …, 256 tokens. The last row shows the original video.
| VideoFlexTok configuration | d12-d12 (ablation setting) | d18-d18 | d18-d28 |
|---|---|---|---|
| Encoder depth | 12 | 18 | 18 |
| Decoder depth | 12 | 18 | 28 |
| Encoder dim. | 768 | 1152 | 1152 |
| Decoder dim. | 768 | 1152 | 1792 |
| Encoder Transformer parameters | 84.9M | 286.7M | 286.7M |
| Decoder Transformer parameters | 84.9M | 286.7M | 1.1B |
Table 5: VideoFlexTok training settings. Model and training configuration for different autoregressive Transformer sizes.
| Configuration | AR 49M | AR 85M | AR 201M | AR 393M | AR 679M | AR 1.33B | AR 2.29B |
|---|---|---|---|---|---|---|---|
| Num. non-embedding parameters | 49M | 85M | 201M | 393M | 679M | 1.33B | 2.29B |
| Decoder depth | 10 | 12 | 16 | 20 | 24 | 30 | 36 |
| Decoder dim. | 640 | 768 | 1024 | 1280 | 1536 | 1920 | 2304 |
| Cross-attention dim. | n/a | ||||||
| MLP ratio | 4 | ||||||
| Max. sequence length | 1280 | ||||||
Table 6: Class-conditioned AR training settings. Model and training configuration for different autoregressive Transformer sizes.
| Configuration | AR 400M | AR 1.1B | AR 2.0B | AR 5.2B | AR 5.2B (4 C) |
|---|---|---|---|---|---|
| Num. total parameters | 400M | 110M | 2.02B | 5.20B | 5.20B |
| Decoder depth | 12 | 20 | 30 | 42 | 42 |
| Decoder dim. | 768 | 1280 | 1920 | 2688 | 2688 |
| Cross-attention dim. | 12 | 20 | 30 | 42 | 42 |
| MLP ratio | 4 | ||||
| Max. sequence length | 1280 | ||||
Table 7: Text-conditioned AR training settings. Model and training configuration for different autoregressive Transformer sizes.
| Tokenizer | # Tok. | MAE ( ) | MSE ( ) | PSNR ( ) | LPIPS ( ) | rFID ( ) | rFVD ( ) |
|---|---|---|---|---|---|---|---|
| VidTok FSQ ( Tang et al., 2024 ) | 1280 | 0.0276 | 0.00270 | 25.64 | 0.0967 | 5.26 | 35.47 |
| LARP ( Wang et al., 2024a ) | 1024 | 0.0476 ‡ | 0.00760 ‡ | 21.21 ‡ | 0.1111 ‡ | 5.50 ‡ | 28.33 ‡ |
| Omnitokenizer ( Junke et al., 2024 ) | 1280 | 0.0206 | 0.00100 | 30.01 | 0.1199 | 7.43 | 38.67 |
| Cosmos-DV ( NVIDIA et al., 2025 ) | 1280 | 0.0287 | 0.00284 | 25.46 | 0.122 | 8.51 | 47.20 |
| VideoFlexTok d18-d28 | 5–1280 | 0.0607 † | 0.01079 † | 19.67 † | 0.18545 † | 4.97 † | 20.53 † |
Table 8: Reconstruction metrics on MSR-VTT. Evaluation is performed on 5k MSR-VTT videos (17 frames, 4 FPS, 256 256). All tokenizers use the same decoder architecture. † indicates VideoFlexTok results using 1280 tokens per video. ‡ marks results obtained from models evaluated at 128×128 input resolution.