Visual Tokenization

Follow this topic Ask about this topic

Visual tokenization studies how to represent images and video as sequences or sets of units that models can process efficiently. Approaches range from fixed patch grids and discrete or continuous autoencoder codes to semantic, object-centric, hierarchical, and variable-length tokens; video methods also encode motion and reuse information across frames. These representations support image generation, video understanding, multimodal language models, and robotic control, where they can reduce computation while retaining useful visual detail. The main challenge is balancing compactness and reconstruction fidelity with semantic and spatial information that remains useful across tasks and downstream models.

Latest papers

70 papers8 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens
Apr 6, 2026Tommie Kerssies, Gabriele Berton, Ju He +5Video World Models World Model Learning
Amazon · Eindhoven University of Technology · Johns Hopkins University
Semantic-Aware Prefix Learning for Token-Efficient Image Generation
Mar 26, 2026Qingfeng Li, Haoxian Zhang, Xu He +4Visual Tokenization Visual Token Compression
Institute of Automation, Chinese Academy of Sciences · Kling Team, Kuaishou Technology · Tsinghua University
Mitigating Premature Discretization with Progressive Quantization for Robust Vector Tokenization
Mar 17, 2026Wenhao Zhao, Qiran Zou, Zhouhan Lin +1Vector Quantization Visual Tokenization
National University of Singapore, Singapore · Shanghai Jiao Tong University, Shanghai, China
GST-VLA: Structured Gaussian Spatial Tokens for 3D Depth-Aware Vision-Language-Action Models
Mar 10, 2026Md Selim Sarowar, Omer Tariq, Sungho KimVision-Language-Action Models VLMs for Robotics
Yeungnam University · Korea Advanced Institute of Science and Technology, KAIST
CaTok: Taming Mean Flows for One-Dimensional Causal Image Tokenization
Mar 6, 2026Yitong Chen, Zuxuan Wu, Xipeng Qiu +1Image Tokenization Visual Tokenization
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Innovation Institute · Shanghai Key Laboratory of Multimodal Embodied AI
OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens
Mar 2, 2026Yiying Yang, Wei Cheng, Sijin Chen +5Multimodal Generation Visual Tokenization
Fudan University · StepFun · HKU MMLab +1
TrajTok: Learning Trajectory Tokens enables better Video Understanding
Feb 26, 2026Chenhao Zheng, Jieyu Zhang, Jianing Zhang +6Visual Tokenization Video Representation Learning
University of Washington · Allen Institute for Artificial Intelligence · Woven by Toyota, Inc +1
COMiT: Learning Structured Visual Tokens through Sequential Communication
Feb 24, 2026Aram Davtyan, Yusuf Sahin, Yasaman Haghighi +4Image Tokenization Visual Tokenization
Computer Vision Group, University of Bern, Switzerland · VITA Lab, EPFL, Switzerland
QuickGrasp: Responsive Video-Language Querying Service via Accelerated Tokenization and Edge-Augmented Inference
Feb 23, 2026Miao Zhang, Ruixiao Zhang, Jianxin Shi +3Efficient VLM Inference VLM Inference
School of Computing Science, Simon Fraser University, Canada · Computer Science Department, University of Illinois Urbana-Champaign, United States · College of Cryptology and Cyber Science, Nankai University, China +1
Spiking Patches: Asynchronous, Sparse, and Efficient Tokens for Event Cameras
Oct 30, 2025Christoffer Koo Øhrstrøm, Ronja Güldenring, Lazaros NalpantidisEvent-Based Vision Visual Tokenization
Technical University of Denmark

Apr 6, 2026J/K move · Enter open · S save

A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens

Tommie Kerssies, Gabriele Berton, Ju He, Qihang Yu+4

Amazon · Eindhoven University of Technology · Johns Hopkins University

Video World Models World Model Learning Video Prediction Visual Tokenization

Open paper AskPDF arXiv

Anticipating diverse future states is a central challenge in video world modeling. Discriminative world models produce a deterministic prediction that implicitly averages over possible futures, while existing generative world models remain computationally expensive. Recent work demonstrates that predicting the future in the feature space of a vision foundation model (VFM), rather than a latent space optimized for pixel reconstruction, requires significantly fewer world model parameters. However, most such approaches remain discriminative. In this work, we introduce DeltaTok, a tokenizer that encodes the VFM feature difference between consecutive frames into a single continuous "delta" token, and DeltaWorld, a generative world model operating on these tokens to efficiently generate diverse plausible futures. Delta tokens reduce video from a three-dimensional spatio-temporal representation to a one-dimensional temporal sequence, for example yielding a 1,024x token reduction with 512x512 frames. This compact representation enables tractable multi-hypothesis training, where many futures are generated in parallel and only the best is supervised. At inference, this leads to diverse predictions in a single forward pass. Experiments on dense forecasting tasks demonstrate that DeltaWorld forecasts futures that more closely align with real-world outcomes, while having over 35x fewer parameters and using 2,000x fewer FLOPs than existing generative world models. Code and weights: https://deltatok.github.io.