Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation

Yeonkyeong Lee; Hyunsung Go; Jongmin Kim; Sewoong Lim; Donghoon Lee

cs.CVAug 25, 2026

Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation

Authors: Yeonkyeong Lee, Hyunsung Go, Jongmin Kim, Sewoong Lim, Donghoon Lee

Organizations: Kakao Corp., Republic of Korea

Abstract

Latent diffusion models have emerged as a dominant framework for high-fidelity image and video synthesis, operating in compact latent spaces with variational autoencoders (VAEs) to enhance computational efficiency without compromising visual quality. However, conventional VAEs are suboptimal for video data as they employ fixed compression ratios that cannot adapt to the varying complexity of spatio-temporal content. We present KATok (Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation), a transformer-based VAE that incorporates an adaptive token selector which is jointly learned with latent tokens. By evaluating each token's content-richness as keep-or-drop probability, the token selector effectively discards uninformative tokens, naturally allowing data-dependent compression. Applying adaptive tokenization to diffusion models may cause spatial misalignment, as token dropping can disturb the original spatio-temporal structure. To alleviate this issue, we propose two position-prediction strategies: cascaded and joint generation, to ensure spatial consistency. We empirically show that our model achieves strong reconstruction and generation quality at a state-of-the-art compression ratio. Further analysis on video data reveals that this improvement is primarily achieved by reducing spatio-temporal redundancy and removing uninformative tokens, as supported by both quantitative and qualitative results.

Figures & tables

Panel of Figure 3 — Figure 3: Two variants of sparse content generation. (a) Diffusion training with joint content–position generation. During training, content and position tokens are modeled jointly but follow decoupled noise schedules— $\sigma_{\text{content}}$ for latent denoising and $\sigma_{\text{pos}}$ for spatio-temporal coordinate prediction—stabilizing training and improving spatial alignment. (b) Cascaded generation of mask and content. While ground-truth token masks and positions are used during training, at inference the mask prior model generates a plausible token mask, which is then thresholded to extract positions for positional encoding in the sparse content generation model.

Figure 6: Qualitative reconstruction results. Our results are shown at $256^{2}\times 16$ and $512^{2}\times 16$ , while ElasticTok is at $256^{2}\times 16$ . The learned token mask $m_{i}$ is visualized by mapping token $i$ back to its spatial patch; black tiles denote suppressed tokens. Our tokenizer attains better perceptual quality with far fewer tokens (e.g., 1104 $\rightarrow$ 836 on the ocean clip; 1712 $\rightarrow$ 330 indoors); the top-right example is a solid-white video, an extreme case where our method needs only 28 tokens.

Figure 7: Token count vs. content entropy. Scatter plots show the relationship between spatial, temporal, and spatio–temporal entropy (x-axis) and the number of tokens used by our tokenizer (y-axis). Each panel reports Pearson correlations. Highlighted examples illustrate the trend: orange — simple, low-motion video uses few tokens; cyan — complex scene with strong motion uses many tokens; blue — complex but mostly static scene uses fewer tokens despite high spatial entropy. Overall, token count correlates much more strongly with temporal entropy than with spatial entropy (pearson $r\!\approx\!0.87$ vs. $0.62$ ).

Method	Resolution	#Tokens	Channels	Comp. $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	rFVD $\downarrow$
Omni - VAE	$256^{2}\times 17\textsuperscript{*}$	5120	8	96	28.10	0.05	0.88	7.84
Omni - VAE	$512^{2}\times 33$ *	36864	8	96	24.07	0.06	0.80	16.85
Elastic - KL	$256^{2}\times 16$	3845.56	8	102.25	30.52	0.06	0.91	12.37
Elastic - KL	$512^{2}\times 32$	–	–	–	–	–	–	–
Ours	$256^{2}\times 16$	366.24	64	134.21	31.24	0.04	0.94	5.12
Ours	$512^{2}\times 32$	1554.24	64	253.00	33.23	0.05	0.95	6.40

Table 1: Reconstruction on the panda70m validation set. #Tokens denotes the token count; for Ours and ElasticTok-KL, it indicates the average number of tokens used. Comp.

\uparrow

is the compression ratio,

\mathrm{Comp.}=\frac{H\times W\times T\times 3}{(\#\text{tokens})\times(\text{token channels})}

where “token channels” equals the Channels column. Our tokenizer achieves better reconstruction quality with substantially fewer tokens. Note: ElasticTok is available only at

256^{2}

spatial resolution, thus results at

512^{2}\times 32

are not supported (cells marked “–”). * Metrics are evaluated on the first 16/32 frames for fair comparison.

Configuration	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	Tokens $\downarrow$
Full model	30.85	0.07	0.93	365.57
$-$ latent reg.	31.26	0.07	0.94	361.70
$-$ asymmetric decoding	29.61	0.11	0.92	377.80
$-$ Video-LPIPS	29.28	0.11	0.91	404.00
$-$ register tokens	29.17	0.12	0.91	410.20
$-$ soft attention mask †	19.00	0.51	0.54	2.00

Table 2: Ablations on VAE components (100K training). Higher PSNR/SSIM indicate better reconstruction and lower LPIPS is better. Tokens reports the average number of active tokens (max = 514). † indicates training collapse (only register tokens remain active).

Table 8

Figure 8: Generation quality across datasets. Qualitative results on UCF-101, Sky-Timelapse, and Kinetics-600.

Figure 9: Effect of joint content–position and cascaded generation. The naive model for generating sparse latents often leads to content–position misalignment, as highlighted by the red boxes, while the proposed joint content–position and cascaded generation strategies mitigate this issue and improve overall generation quality.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Stage 1	Stage 2	Stage 3
Model Architecture
(All resolution tuples are [H, W, T].)
Encoder patch size	[16,16,8]	[16,16,8]	[16,16,8]
Decoder patch size	[8, 8, 4]	[8, 8, 4]	[8, 8, 4]
Hidden dimension	768	768	768
Number of heads	16	16	16

Appendix

Table S1 : Training configurations for the three-stage training pipeline of KATok. Stage 1 performs VAE pretraining with pixel- and perceptual-level supervision. Stage 2 extends training to multi-resolution video data for enhanced spatial–temporal generalization. Stage 3 introduces adversarial fine-tuning with non-saturating GAN loss to improve perceptual sharpness and realism.

Shared content model (SiT-XL) — used by both variants
Architecture / Parameters	SiT-XL / 686.29M
Hidden dim / Depth / Heads	1152 / 28 / 16
Input channels	64 (latent)
QK-normalization	✓
Positional embedding	Variant-dependent (see text)
Optimizer	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.999$ ), wd $=0$

Appendix

Table S2 : Diffusion training configurations. Both generation variants share the same SiT-XL content backbone (top). The joint variant concatenates 3D position coordinates to the content channels; the cascaded variant instead adds a lightweight mask prior module that is trained jointly with the content model.

Figure S1 : Cross-paradigm reconstruction comparison. (a,b) Continuous-token VAEs on Kinetics-600 and Epic-Kitchens at $256^{2}/512^{2}$ ; transformer-based (KATok, OmniTokenizer) and CNN-based (LTX-Video, Cosmos) baselines all use fixed compression, while KATok adapts per-sample. PSNR is plotted against total latent size ( $\#\text{tokens}\times\text{channels}$ ) to account for latent-channel differences. (c,d) Adaptive VQ tokenizers (EVATok, AdapTok) at $128^{2}$ . KATok remains Pareto-competitive across paradigms, and at $128^{2}$ uses 7 $\times$ fewer tokens while improving PSNR by 2.6–4.0 dB.

Method	Tok. Type	Gen.	Gen. Params	#Tokens	gFVD $\downarrow$
128 2 resolution (as reported in prior work)
MAGVIT-v2 [ 21 ]	VQ, fixed	Masked Language Model	307M	1,280	58
EVATok [ 19 ]	VQ, adaptive	AR	633M	758	48
AdapTok [ 12 ]	VQ, adaptive	AR	633M	1,024	67
256 2 resolution (as reported in prior work)
OmniTokenizer [ 18 ]	VQ, fixed	AR	650M	5,120	191

Appendix

Table S3 : System-level comparison on UCF-101 class-conditional generation. Methods grouped by spatial resolution. Bottom group: controlled comparison with identical SiT-XL backbone and training protocol, isolating the effect of tokenization design. “Tok. Type” describes the tokenizer (VQ/KL) and adaptive nature; “Gen.” denotes generator type.

Figure S2 : Qualitative reconstruction results. Examples are sampled from the Panda70M validation set. Each sample shows reconstruction of $512^{2}\times 16$ videos visualized with a 2-frame skip. For each example, we visualize the ground truth (GT), reconstructed frames (Recon), and the corresponding mask visualization (Mask Vis.) along with the number of tokens used. The maximum token budget is 2050. The results illustrate that the model preserves fine details while adaptively allocating tokens according to motion and texture complexity.

Figure S3 : Qualitative video generation results with mask visualization on UCF-101. Each row shows a sample generated by the cascaded generation pipeline at $256^{2}\times 16$ resolution, visualized with a 2-frame skip. The left column displays the token selection mask generated by the mask prior (white = active, black = dropped), and the right column shows the corresponding generated video frames. The cascaded model adaptively allocates more tokens to regions with complex motion and fine-grained details, while suppressing tokens in static or homogeneous areas.

Figure S4 : Qualitative video generation results with mask visualization on SkyTimelapse. Same setup as Fig. S3 . Compared to UCF-101, the generated masks tend to be sparser, reflecting the relatively static nature of sky scenes. The model uses fewer tokens for uniform regions (e.g., clear skies) and allocates more tokens where cloud textures or horizon details require finer representation.

Figure S5 : Qualitative video generation results with mask visualization on Kinetics-600. Same setup as Fig. S3 . Kinetics-600 contains diverse real-world activities, resulting in highly varied mask patterns across samples. Scenes with large camera or object motion retain more tokens, while static backgrounds are aggressively pruned.

Figure S6 : Effect of token budget on video generation (UCF-101). Qualitative comparison of videos generated with 200 tokens (left) and 400 tokens (right) on the UCF-101 dataset. All samples are generated at $256^{2}\times 16$ resolution and visualized with a 2-frame skip. With fewer tokens, the model generates simpler scenes with less motion, while increasing the token budget yields richer motion dynamics and finer visual details.

Figure S7 : Effect of token budget on video generation (SkyTimelapse). Qualitative comparison of videos generated with 200 tokens (left) and 400 tokens (right) on the SkyTimelapse dataset. All samples are generated at $256^{2}\times 16$ resolution and visualized with a 2-frame skip. With more tokens, the model produces more complex cloud formations and richer atmospheric variations.

Figure S8 : Effect of token budget on video generation (Kinetics-600). Qualitative comparison of videos generated with 200 tokens (left) and 400 tokens (right) on the Kinetics-600 dataset. All samples are generated at $256^{2}\times 16$ resolution and visualized with a 2-frame skip. Higher token budgets enable the model to capture more complex activities and scene details characteristic of this diverse dataset.

Mask source	GT	Predicted	$1\%$ -flipped GT	Random
Mask FVD $\downarrow$	53.08	91.80	139.58	15,405.39

Appendix

Table S4 : Mask quality measured by FVD on mask sequences. Mask sequences are treated as videos; FVD is computed on two disjoint Kinetics-600 validation halves (

2{,}048

clips each), with one half fixed to ground-truth masks. Lower is better.

Figure S9 : Failure cases on clips with high-frequency detail and large motion. Each example shows the ground truth (top row, GT) and the reconstruction (bottom row, Recon), visualized as $16$ frames sampled with a stride of $2$ . On clips that contain abundant high-frequency detail together with large motion, the fast-moving fine texture is hard to reconstruct faithfully, and the reconstruction becomes blurry precisely in the moving high-frequency regions. These samples attain low fidelity (PSNR $\approx 19$ – $20$ ) on the Panda-70M validation set.

Method	#Tokens	Ch.	Comp. $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	rFID $\downarrow$
TiTok-L-VAE	32	16	384	19.51	0.17	0.49	1.62
TiTok-B-VAE	64	16	192	20.74	0.14	0.55	1.25
TiTok-S-VAE	128	16	96	22.51	0.09	0.64	0.84
FlexTok d12-d12 †	256	–	–	18.47	0.22	0.47	3.18
FlexTok d18-d18 †	256	–	–	18.54	0.21	0.47	1.18
FlexTok d18-d28 †	256	–	–	18.41	0.21	0.47	1.05

Appendix

Table S6 : Image reconstruction on the ImageNet validation set. #Tokens denotes the average token count. Comp.

\uparrow

=\frac{H\times W\times C}{\text{\#Tokens}\times\text{Channels}}

; not applicable for VQ models ( – ). Ours-Img32 and Ours-Img16 use patch sizes

32^{2}

and

16^{2}

, respectively. † FlexTok supports variable-length tokenization (1–256) but lacks content-adaptive token selection; we evaluate at its maximum budget of 256 for best performance.

Figure S10 : ImageNet reconstructions with content-adaptive tokenization. Columns show GT, reconstructions with $32{\times}32$ and $16{\times}16$ patches, and the learned token mask for $16{\times}16$ (black tiles = dropped tokens). Reducing the patch size increases redundancy between neighboring patches, prompting the model to drop more tokens, whereas larger $32{\times}32$ patches retain most tokens.

$\lambda_{\text{sparse}}$	0.005	0.010	0.020	0.050	0.100
PSNR $\uparrow$	28.67	28.78	22.46	21.14	19.62
rFVD $\downarrow$	50.16	50.38	462.27	680.45	863.51
#Tokens (Avg.)	401.5	372.3	34.9	16.2	6.9

Appendix

Table S7 : Effect of sparsity regularization weight

\lambda_{\text{sparse}}

. All models are trained for 70K steps (Stage 1) on Panda-70M with symmetric

16^{2}\times 8

encoder–decoder patches and evaluated on the Panda-70M validation set at

256^{2}\times 16

Figure S11 : Token count distributions on SkyTimelapse, UCF-101, and Kinetics-600. Histograms of the number of tokens per video across the entire dataset, with mean token counts indicated by dashed red lines. SkyTimelapse uses the fewest tokens on average (316.9), consistent with its predominantly static sky scenes. UCF-101 exhibits a higher mean (368.2) due to its greater motion and scene variability. Kinetics-600, which contains diverse real-world activities with complex motion, uses the most tokens on average (399.2) and exhibits a wide spread across the full token range with a notable concentration near the maximum budget (512). These distributions confirm that the adaptive tokenizer allocates tokens in proportion to content complexity.

Figure S12 : Step-wise visualization of $x_{0}$ prediction during generation. Three samples generated on UCF-101 are shown, each visualized at flow-matching timesteps $t\in\{0.0,0.25,0.5,0.75,1.0\}$ (top to bottom) with frames 0, 4, 8, 12 (left to right). At $t=0.0$ (pure noise), the decoded $x_{0}$ prediction shows only coarse color blobs. By $t=0.25$ , global scene layout and dominant motion emerge. Subsequent steps progressively sharpen spatial details and temporal coherence, with the final output at $t=1.0$ producing a clean, temporally consistent video.

Explore similar work

Adaptive Tokenisation Via Temporal Redundancy Masking And Latent Inpainting
Jun 4, 2026Kevin Dave, Sai Aditya Patkuri, Chhaya Kumar Das +3Visual Tokenization Video Token Compression
Phronetic AI · IISc Bangalore · IIT Bombay
EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation
Mar 12, 2026Tianwei Xiong, Jun Hao Liew, Zilong Huang +3Video Token Compression Visual Token Compression
The University of Hong Kong · ByteDance Seed
KVAE: Family of Tokenizers for Multimodal Generative Models
Aug 6, 2026Andrey Shutkin, Denis Parkhomenko, Ivan Kirillov +11Variational Autoencoders Audio Token Compression
Kandinsky Lab

Jun 4, 2026J/K move · Enter open · S save

Adaptive Tokenisation Via Temporal Redundancy Masking And Latent Inpainting

Kevin Dave, Sai Aditya Patkuri, Chhaya Kumar Das, Gouranga Bala+2

Phronetic AI · IISc Bangalore · IIT Bombay

Visual Tokenization Video Token Compression Visual Token Pruning Visual Token Compression

Open paper AskPDF arXiv

Adaptive video tokenisation seeks to dynamically allocate token budgets based on the underlying visual complexity of a sequence. Current continuous-regime approaches achieve this via iterative binarised searches or trained neural regressors, while discrete methods often require a full-rate decoder pass to estimate information content. We demonstrate that such computational overheads are not strictly necessary. We show that the latent space of a frozen continuous video tokeniser inherently encodes temporal redundancy that can be exploited directly: spatial positions whose latent representations change minimally between consecutive frames carry near-zero additional information. We introduce a parameter-free adaptive token allocation mechanism that applies a fixed threshold to per-position temporal-L1 differences, identifying and dropping redundant latent positions. Consequently, the compression rate emerges naturally from the input content rather than being enforced top-down: static scenes get compressed aggressively, while highly dynamic sequences retain more tokens. To reconstruct the dropped positions, we propose the Latent Inpainting Transformer (LIT), a lightweight factorised spatial-temporal attention architecture. The resulting inference pipeline is highly efficient, requiring only a single encoder pass and one LIT forward pass, eliminating the need for auxiliary routing networks. Evaluations across TokenBench and DAVIS, which are the standard benchmarks used by recent tokenisers~\cite{infotok, agarwal2025cosmos}, indicate that our framework yields meaningful, content-driven token allocation while maintaining competitive reconstruction fidelity, and delivers a

31\times

inference-time speedup over the continuous adaptive baseline (ElasticTok-CV) and an

\approx2\times

speedup over the discrete information-theoretic baseline (InfoTok)

Abstract

Explore similar work

Adaptive Tokenisation Via Temporal Redundancy Masking And Latent Inpainting

EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation

KVAE: Family of Tokenizers for Multimodal Generative Models