Autoregressive (AR) video generative models rely on video tokenizers that compress pixels into discrete token sequences. The length of these token sequences is crucial for balancing reconstruction quality against downstream generation computational cost. Traditional video tokenizers apply a uniform token assignment across temporal blocks of different videos, often wasting tokens on simple, static, or repetitive segments while underserving dynamic or complex ones. To address this inefficiency, we introduce EVATok, a framework to produce Efficient Video Adaptive Tokenizers. Our framework estimates optimal token assignments for each video to achieve the best quality-cost trade-off, develops lightweight routers for fast prediction of these optimal assignments, and trains adaptive tokenizers that encode videos based on the assignments predicted by routers. We demonstrate that EVATok delivers substantial improvements in efficiency and overall quality for video reconstruction and downstream AR generation. Enhanced by our advanced training recipe that integrates video semantic encoders, EVATok achieves superior reconstruction and state-of-the-art class-to-video generation on UCF-101, with at least 24.4% savings in average token usage compared to the prior state-of-the-art LARP and our fixed-length baseline.
Figures & tables
Figure 1: EVATok highlights. Top : EVATok achieves superior video reconstruction and downstream generation quality with significant savings in token usage. Bottom : EVATok assigns tokens in an intuitive way. Clips with dynamic motion or complex layout will be encoded with more tokens, while clips that are repetitive or simple will be assigned fewer tokens.
Figure 2: Four-stage framework for adaptive video tokenizer training. Stage 1 trains a proxy tokenizer to reconstruct videos under all candidate assignments. Stage 2 applies the proxy tokenizer to compute proxy rewards for all candidate assignments across videos from a dataset. It identifies the assignments with maximum proxy rewards to curate a classification dataset of videos and their optimal assignments. Stage 3 trains a router on the curated dataset to predict the optimal assignments for videos. Stage 4 trains the final tokenizer from scratch, with the router determining the assignment for each input video during training.
Figure 3: Architecture of 1D variable-length video tokenizer for EVATok. The input video is spatio-temporally patchified into 3D embeddings. According to a given assignment a , 1D variable-length query embeddings are initialized from these 3D embeddings. After Q-Former encoding and quantization, 1D discrete tokens are produced. Finally, 3D queries are initialized to reconstruct the video frames from the 1D tokens.
Figure 4: Quality-cost trade-off curves for different assignment strategies. By adaptively assigning token budgets to different temporal blocks across various videos, our max-proxy-reward strategy (green series) achieves superior performance under various overall budgets compared to the typical fixed uniform token assignment approach (red series). The router-based assignment (blue series) delivers performance close to that of the max-proxy-reward strategy on both WebVid and UCF datasets (the latter unseen during router training).
Settings
PSNR ↑
LPIPS ↓
rFVD ↓
#rTokens ↓
A1. Uniform (Proxy Tok.)
27.26
0.1178
73
1024
A2. Uniform (Final Tok.)
27.77
0.1056
63
1024
A2 + VideoMAE Disc.
26.68
0.1197
13
1024
B1. Router (Proxy Tok.)
27.05
0.1182
50
721 (-29.6%)
B2. Router (Final Tok.)
27.68
0.1068
33
721 (-29.6%)
B2 + VideoMAE Disc.
26.90
0.1144
9.2
721 (-29.6%)
Table 1: Final tokenizer validation on WebVid. The tokenizers are trained for 400k iterations. With the router, final tokenizers achieve comparable LPIPS and better rFVD with 29.6% saving in token length (row A2 vs . B2 and row A2+ vs . B2+). Final tokenizers outperform proxy tokenizers with the same training efforts (row A2 vs . A1, B2 vs . row B1), showing the importance of bridging the training-inference gap for variable-length tokenizers.
Settings
LPIPS ↓
rFVD ↓
#rTokens ↓
gFVD ↓
#gTokens ↓
Uniform (Final)
0.1303
13
1024
98
1024
Router (Final)
0.1212
13
774 (-24.4%)
96
740 (-27.7%)
Table 2: Final tokenizer validation on UCF. The final tokenizer with router beats the fixed uniform assignment baseline in both reconstruction and downstream AR generation, while saving 24.4% and 27.7% token length separately.
Method
#Params
#rTokens
rFVD ↓
gFVD ↓
#gTokens
Tokenizer
Generator
K600
UCF
UCF
Diffusion-based generative models with continuous video tokenizers
VideoFusion [ 37 ]
-
2B
-
-
-
173
-
HPDM [ 48 ]
-
725M
-
-
-
66
-
W.A.L.T-L [ 20 ]
-
313M
-
-
3.3
46
-
MLM generative models with discrete video tokenizers
Table 3: System-level comparison for tokenizers and downstream generation models. EVATok achieves superior performances in UCF-101 video reconstruction, downstream class-to-video generation and K600 frame prediction, while saving 24.4% tokens in reconstruction and 26.2% tokens in UCF class-to-video generation.
Methods
Tok. Param.
#gTokens
AR Param.
gFVD ↓
AdapTok [ 33 ]
195M
1024
633M
11
LARP [ 59 ]
173M
1024
632M
5.1
EVATok (ours)
145M
862 (-15.8%)
633M
4.0
Table 4: K600 frame prediction comparison. In similar settings, EVATok performs the best with 15.8% less generated tokens.
Figure 5: Quality-cost curve: threshold based vs . max-proxy-reward vs . uniform assignment. While threshold-based assignment improves rFVD against uniform assignment, it underperforms our max-proxy-reward strategy.
Configuration
PSNR ↑
LPIPS ↓
rFVD ↓
gFVD ↓
Final Recipe (Uniform)
25.05
0.1303
13
98
- VideoMAE [ 53 , 63 ] Disc.
26.21
0.1097
65
155
- V-JEPA2 [ 2 ] Align.
25.30
0.1253
18
144
- Both
26.41
0.1095
80
230
Table 5: Ablation study for video representation alignment and video semantic discriminator. Removing either design will lead to degradation in rFVD and downstream gFVD.
Settings / Dataset
PSNR ↑
LPIPS ↓
rFVD ↓
#rTokens ↓
Final w/ VideoMAE Disc. (WebVid)
27.37
0.1063
7.3
721
Final w/o VideoMAE Disc. (WebVid)
28.18
0.0983
32
721
Final w/ VideoMAE Disc. (UCF&K600)
25.75
0.1140
9.7
774
Table 6: The detailed performances of final tokenizers reconstruction results. All models are trained for the full 1000k iterations. The tokenizers trained on WebVid-10M are evaluated on the WebVid validation set, and the tokenizers trained on UCF-101 and K600 are evaluated on UCF-101 training set.
Figure 6: Adaptive reconstruction results on WebVid. We downsample 16 frames into 8 frames for visualization, and each two frames represent a 4-frame temporal block. The router typically assigns more tokens to the initial temporal block, allowing the reconstruction of subsequent frames to also benefit from more precise information encoded for the initial block. Content with simple layouts receives fewer tokens (first example vs . other examples). If later frames largely repeat previous ones, they are assigned the minimum number of tokens. Video clips that vary constantly and intensely are allocated more tokens.
Figure 7: Adaptive reconstruction results on UCF-101. We downsample 16 frames into 8 frames for visualization, and each two frames represent a 4-frame temporal block.
Figure 8: Qualitative comparison for using VideoMAE discriminator or not. Using VideoMAE discriminator can degrade the PSNR/LPIPS, but in actual perceptual checking, we find that this degradation is traded for alleviated blurriness and artifact patterns. Zoom in to check the visual details.
Figure 9: Adaptive generation results on UCF-101. We use the 633M GPT model trained on EVATok. We use a constant 3.0 CFG for sampling.
Figure 10: Adaptive generation results on K600 frame prediction. We use the 633M GPT model trained on EVATok. We don’t use CFG for sampling, following typical approaches. The 1, 3, 5 frames from the 5 condition frames are plotted as the condition part, and the rest 11 frames are downsampled into 6 frames for visualization.
Stage / Task
Model Size
Dataset
Bsz
Iters / Epochs
GPUs
Time
Stage 1: Proxy tokenizer training
145M
WebVid (or UCF& K600)
128
400k iters
64×V100
116 h
Stage 2: Data curation
–
WebVid 100k-Subset
–
–
4×64×V100
12.5 h
Stage 3: Router training
20M
WebVid 100k-Subset
128
50k iters
32×V100
5 h
Stage 4: Final adaptive tokenizer training
145M
WebVid (or UCF& K600)
128
1000k iters
64×V100
347 h
AR training: UCF-101 class-to-video
633M
UCF-101
128
3000 epochs
64×V100
88 h
AR training: K600 frame prediction
633M
Kinetics-600
128
75 epochs
64×V100
140 h
Table 7: Summary of compute and time for the four-stage tokenizer pipeline and subsequent AR model training.
Figure 11: Example for the attention masks in our Q-Former style adaptive tokenizer.
Dataset
Method
Val top1/top5 acc.
Proxy Reward Percentile
WebVid
Best-Uniform
-
84.88%
Router
11.72% / 35.03%
96.96%
UCf-101
Best-Uniform
-
88.46%
Router
5.77% / 23.68%
96.19%
Table 8: Accuracy vs . proxy reward percentile for the router assignment. In terms of accuracy. the assignments predicted by the router do not usually hit the top1 or top5 highest proxy reward assignments. However, in terms of proxy reward percentile, the router assignments achieve good results, and generalize to unseen dataset (UCF-101) well.
Figure 12: Image tokenization quality-cost trade-off curve. On ImageNet 256×256 reconstruction, the improvements of max-proxy-reward assignment can be marginal compared to uniform assignment.
LPIPS ↓
rFID ↓
#rTokens ↓
gFID ↓
#gTokens ↓
Uniform (Final)
0.2205
1.22
256
4.72
256
Router (Final)
0.2455
1.46
205 (-19.9%)
4.51
197 (-23.0%)
Table 9: Image final tokenizer validation. For ImageNet 256×256 , saving 19.9% tokens by adaptive tokenization inevitably leads to worse rFID, but the performance and efficiency of downstream AR generation models can still benefit from our router. The AR generation models use a constant 1.5 CFG during inference.
Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video's inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner -- where the first tokens (emergently) capture abstract information, such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count according to downstream needs and encoding videos longer than the baselines with the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.
Andrei Atanov, Jesse Allardice, Roman Bachmann +6
Apple · Swiss Federal Institute of Technology Lausanne (EPFL)
Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse tokens carry the clip's global semantics while later tokens further specify details. Existing flexible tokenizers only apply a representation-alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. SemanTok achieves high semantic alignment and video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model 3.4× its size, and larger SemanTok AR models further improve fidelity. It keeps semantic alignment on out-of-distribution classes and gives the decoder higher semantic alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later tokens.
Mikhail Dereviannykh, Vikram Voleti, Simon Donne +3
Latent diffusion models have emerged as a dominant framework for high-fidelity image and video synthesis, operating in compact latent spaces with variational autoencoders (VAEs) to enhance computational efficiency without compromising visual quality. However, conventional VAEs are suboptimal for video data as they employ fixed compression ratios that cannot adapt to the varying complexity of spatio-temporal content. We present KATok (Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation), a transformer-based VAE that incorporates an adaptive token selector which is jointly learned with latent tokens. By evaluating each token's content-richness as keep-or-drop probability, the token selector effectively discards uninformative tokens, naturally allowing data-dependent compression. Applying adaptive tokenization to diffusion models may cause spatial misalignment, as token dropping can disturb the original spatio-temporal structure. To alleviate this issue, we propose two position-prediction strategies: cascaded and joint generation, to ensure spatial consistency. We empirically show that our model achieves strong reconstruction and generation quality at a state-of-the-art compression ratio. Further analysis on video data reveals that this improvement is primarily achieved by reducing spatio-temporal redundancy and removing uninformative tokens, as supported by both quantitative and qualitative results.