EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation

Tianwei Xiong; Jun Hao Liew; Zilong Huang; Zhijie Lin; Jiashi Feng; Xihui Liu

cs.CVMar 12, 2026

EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation

Authors: Tianwei Xiong, Jun Hao Liew, Zilong Huang, Zhijie Lin, Jiashi Feng, Xihui Liu

Organizations: The University of Hong Kong · ByteDance Seed

Abstract

Autoregressive (AR) video generative models rely on video tokenizers that compress pixels into discrete token sequences. The length of these token sequences is crucial for balancing reconstruction quality against downstream generation computational cost. Traditional video tokenizers apply a uniform token assignment across temporal blocks of different videos, often wasting tokens on simple, static, or repetitive segments while underserving dynamic or complex ones. To address this inefficiency, we introduce $\textbf{EVATok}$ , a framework to produce $\textbf{E}$ fficient $\textbf{V}$ ideo $\textbf{A}$ daptive $\textbf{Tok}$ enizers. Our framework estimates optimal token assignments for each video to achieve the best quality-cost trade-off, develops lightweight routers for fast prediction of these optimal assignments, and trains adaptive tokenizers that encode videos based on the assignments predicted by routers. We demonstrate that EVATok delivers substantial improvements in efficiency and overall quality for video reconstruction and downstream AR generation. Enhanced by our advanced training recipe that integrates video semantic encoders, EVATok achieves superior reconstruction and state-of-the-art class-to-video generation on UCF-101, with at least 24.4% savings in average token usage compared to the prior state-of-the-art LARP and our fixed-length baseline.

Figures & tables

Figure 1: EVATok highlights. Top : EVATok achieves superior video reconstruction and downstream generation quality with significant savings in token usage. Bottom : EVATok assigns tokens in an intuitive way. Clips with dynamic motion or complex layout will be encoded with more tokens, while clips that are repetitive or simple will be assigned fewer tokens.

Figure 2: Four-stage framework for adaptive video tokenizer training. Stage 1 trains a proxy tokenizer to reconstruct videos under all candidate assignments. Stage 2 applies the proxy tokenizer to compute proxy rewards for all candidate assignments across videos from a dataset. It identifies the assignments with maximum proxy rewards to curate a classification dataset of videos and their optimal assignments. Stage 3 trains a router on the curated dataset to predict the optimal assignments for videos. Stage 4 trains the final tokenizer from scratch, with the router determining the assignment for each input video during training.

Figure 3: Architecture of 1D variable-length video tokenizer for EVATok. The input video is spatio-temporally patchified into 3D embeddings. According to a given assignment $a$ , 1D variable-length query embeddings are initialized from these 3D embeddings. After Q-Former encoding and quantization, 1D discrete tokens are produced. Finally, 3D queries are initialized to reconstruct the video frames from the 1D tokens.

Figure 4: Quality-cost trade-off curves for different assignment strategies. By adaptively assigning token budgets to different temporal blocks across various videos, our max-proxy-reward strategy (green series) achieves superior performance under various overall budgets compared to the typical fixed uniform token assignment approach (red series). The router-based assignment (blue series) delivers performance close to that of the max-proxy-reward strategy on both WebVid and UCF datasets (the latter unseen during router training).

Settings	PSNR $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$	#rTokens $\downarrow$
A1. Uniform (Proxy Tok.)	27.26	0.1178	73	1024
A2. Uniform (Final Tok.)	27.77	0.1056	63	1024
A2 + VideoMAE Disc.	26.68	0.1197	13	1024
B1. Router (Proxy Tok.)	27.05	0.1182	50	721 (-29.6%)
B2. Router (Final Tok.)	27.68	0.1068	33	721 (-29.6%)
B2 + VideoMAE Disc.	26.90	0.1144	9.2	721 (-29.6%)

Table 1: Final tokenizer validation on WebVid. The tokenizers are trained for 400k iterations. With the router, final tokenizers achieve comparable LPIPS and better rFVD with 29.6% saving in token length (row A2 vs . B2 and row A2+ vs . B2+). Final tokenizers outperform proxy tokenizers with the same training efforts (row A2 vs . A1, B2 vs . row B1), showing the importance of bridging the training-inference gap for variable-length tokenizers.

Settings	LPIPS $\downarrow$	rFVD $\downarrow$	#rTokens $\downarrow$	gFVD $\downarrow$	#gTokens $\downarrow$
Uniform (Final)	0.1303	13	1024	98	1024
Router (Final)	0.1212	13	774 (-24.4%)	96	740 (-27.7%)

Table 2: Final tokenizer validation on UCF. The final tokenizer with router beats the fixed uniform assignment baseline in both reconstruction and downstream AR generation, while saving 24.4% and 27.7% token length separately.

Method	#Params		#rTokens	rFVD $\downarrow$	gFVD $\downarrow$		#gTokens
	Tokenizer	Generator			K600	UCF	UCF
Diffusion-based generative models with continuous video tokenizers
VideoFusion [ 37 ]	-	2B	-	-	-	173	-
HPDM [ 48 ]	-	725M	-	-	-	66	-
W.A.L.T-L [ 20 ]	-	313M	-	-	3.3	46	-
MLM generative models with discrete video tokenizers

Table 3: System-level comparison for tokenizers and downstream generation models. EVATok achieves superior performances in UCF-101 video reconstruction, downstream class-to-video generation and K600 frame prediction, while saving 24.4% tokens in reconstruction and 26.2% tokens in UCF class-to-video generation.

Methods	Tok. Param.	#gTokens	AR Param.	gFVD $\downarrow$
AdapTok [ 33 ]	195M	1024	633M	11
LARP [ 59 ]	173M	1024	632M	5.1
EVATok (ours)	145M	862 (-15.8%)	633M	4.0

Table 4: K600 frame prediction comparison. In similar settings, EVATok performs the best with 15.8% less generated tokens.

Figure 5: Quality-cost curve: threshold based vs . max-proxy-reward vs . uniform assignment. While threshold-based assignment improves rFVD against uniform assignment, it underperforms our max-proxy-reward strategy.

Configuration	PSNR $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$	gFVD $\downarrow$
Final Recipe (Uniform)	25.05	0.1303	13	98
- VideoMAE [ 53 , 63 ] Disc.	26.21	0.1097	65	155
- V-JEPA2 [ 2 ] Align.	25.30	0.1253	18	144
- Both	26.41	0.1095	80	230

Table 5: Ablation study for video representation alignment and video semantic discriminator. Removing either design will lead to degradation in rFVD and downstream gFVD.

Settings / Dataset	PSNR $\uparrow$	LPIPS $\downarrow$	rFVD $\downarrow$	#rTokens $\downarrow$
Final w/ VideoMAE Disc. (WebVid)	27.37	0.1063	7.3	721
Final w/o VideoMAE Disc. (WebVid)	28.18	0.0983	32	721
Final w/ VideoMAE Disc. (UCF&K600)	25.75	0.1140	9.7	774

Table 6: The detailed performances of final tokenizers reconstruction results. All models are trained for the full 1000k iterations. The tokenizers trained on WebVid-10M are evaluated on the WebVid validation set, and the tokenizers trained on UCF-101 and K600 are evaluated on UCF-101 training set.

Figure 6: Adaptive reconstruction results on WebVid. We downsample 16 frames into 8 frames for visualization, and each two frames represent a 4-frame temporal block. The router typically assigns more tokens to the initial temporal block, allowing the reconstruction of subsequent frames to also benefit from more precise information encoded for the initial block. Content with simple layouts receives fewer tokens (first example vs . other examples). If later frames largely repeat previous ones, they are assigned the minimum number of tokens. Video clips that vary constantly and intensely are allocated more tokens.

Figure 7: Adaptive reconstruction results on UCF-101. We downsample 16 frames into 8 frames for visualization, and each two frames represent a 4-frame temporal block.

Figure 8: Qualitative comparison for using VideoMAE discriminator or not. Using VideoMAE discriminator can degrade the PSNR/LPIPS, but in actual perceptual checking, we find that this degradation is traded for alleviated blurriness and artifact patterns. Zoom in to check the visual details.

Figure 9: Adaptive generation results on UCF-101. We use the 633M GPT model trained on EVATok. We use a constant 3.0 CFG for sampling.

Figure 10: Adaptive generation results on K600 frame prediction. We use the 633M GPT model trained on EVATok. We don’t use CFG for sampling, following typical approaches. The 1, 3, 5 frames from the 5 condition frames are plotted as the condition part, and the rest 11 frames are downsampled into 6 frames for visualization.

Stage / Task	Model Size	Dataset	Bsz	Iters / Epochs	GPUs	Time
Stage 1: Proxy tokenizer training	145M	WebVid (or UCF& K600)	128	400k iters	64×V100	116 h
Stage 2: Data curation	–	WebVid 100k-Subset	–	–	4×64×V100	12.5 h
Stage 3: Router training	20M	WebVid 100k-Subset	128	50k iters	32×V100	5 h
Stage 4: Final adaptive tokenizer training	145M	WebVid (or UCF& K600)	128	1000k iters	64×V100	347 h
AR training: UCF-101 class-to-video	633M	UCF-101	128	3000 epochs	64×V100	88 h
AR training: K600 frame prediction	633M	Kinetics-600	128	75 epochs	64×V100	140 h

Table 7: Summary of compute and time for the four-stage tokenizer pipeline and subsequent AR model training.

Figure 11: Example for the attention masks in our Q-Former style adaptive tokenizer.

Dataset	Method	Val top1/top5 acc.	Proxy Reward Percentile
WebVid	Best-Uniform	-	84.88%
WebVid	Router	11.72% / 35.03%	96.96%
UCf-101	Best-Uniform	-	88.46%
UCf-101	Router	5.77% / 23.68%	96.19%

Table 8: Accuracy vs . proxy reward percentile for the router assignment. In terms of accuracy. the assignments predicted by the router do not usually hit the top1 or top5 highest proxy reward assignments. However, in terms of proxy reward percentile, the router assignments achieve good results, and generalize to unseen dataset (UCF-101) well.

Figure 12: Image tokenization quality-cost trade-off curve. On ImageNet

256\times 256

reconstruction, the improvements of max-proxy-reward assignment can be marginal compared to uniform assignment.

	LPIPS $\downarrow$	rFID $\downarrow$	#rTokens $\downarrow$	gFID $\downarrow$	#gTokens $\downarrow$
Uniform (Final)	0.2205	1.22	256	4.72	256
Router (Final)	0.2455	1.46	205 (-19.9%)	4.51	197 (-23.0%)

Table 9: Image final tokenizer validation. For ImageNet

256\times 256

, saving 19.9% tokens by adaptive tokenization inevitably leads to worse rFID, but the performance and efficiency of downstream AR generation models can still benefit from our router. The AR generation models use a constant 1.5 CFG during inference.

Explore similar work

VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization
Apr 14, 2026Andrei Atanov, Jesse Allardice, Roman Bachmann +6Visual Tokenization Video Token Compression
Apple · Swiss Federal Institute of Technology Lausanne (EPFL)
SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation
Sep 30, 2026Mikhail Dereviannykh, Vikram Voleti, Simon Donne +3Visual Tokenization Autoregressive Video Generation
Stability AI · Karlsruhe Institut für Technologie
Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation
Aug 25, 2026Yeonkyeong Lee, Hyunsung Go, Jongmin Kim +2Video Token Compression Variational Autoencoders
Kakao Corp., Republic of Korea

Apr 14, 2026J/K move · Enter open · S save

VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

Andrei Atanov, Jesse Allardice, Roman Bachmann, Oğuzhan Fatih Kar+5

Apple · Swiss Federal Institute of Technology Lausanne (EPFL)

Visual Tokenization Video Token Compression Long Video Generation

Open paper AskPDF arXiv

Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video's inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner -- where the first tokens (emergently) capture abstract information, such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count according to downstream needs and encoding videos longer than the baselines with the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.

Abstract

Explore similar work

VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation

Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation