Visual Token Compression

Follow this topic Ask about this topic

Vision encoders represent images and video as patch or frame tokens, and long token sequences can make multimodal models costly to run, especially during prefill and memory-intensive video reasoning. Methods include training-free pruning and merging, learned resamplers, quantization and codecs, and adaptive token budgets guided by text queries, spatial structure, or temporal redundancy. They are evaluated on tasks such as visual question answering, video understanding, OCR, segmentation, retrieval, generation, and robot control. The central tension is that fewer tokens do not always mean faster end-to-end inference, and aggressive compression can discard details needed for a particular question or task.

Latest papers

271 papers46 papers in the last four weeks, up 229% on the four weeks before. 0.3% of all new papers.

PerCoV2: Ultra-Low Bit-Rate Perceptual Image Compression via Query-Based 1D Multimodal Image Tokens
Mar 12, 2025Nikolai Körber, Eduard Kromer, Andreas Siebert +3Learned Image Compression Generative Image Compression
Technical University of Munich · University of Applied Sciences Landshut · TU Wien

Mar 12, 2025J/K move · Enter open · S save

PerCoV2: Ultra-Low Bit-Rate Perceptual Image Compression via Query-Based 1D Multimodal Image Tokens

Nikolai Körber, Eduard Kromer, Andreas Siebert, Sascha Hauke+2

Technical University of Munich · University of Applied Sciences Landshut · TU Wien

Learned Image Compression Generative Image Compression Visual Token Compression

Open paper AskPDF arXiv

Despite recent progress in learned image compression, current image codecs still struggle to maintain realistic reconstructions at low bit-rates, often producing structured artifacts such as grid patterns or repetitive textures, even when trained with perceptual or adversarial losses. We introduce PerCoV2, an ultra-low bit-rate perceptual image compression system that unifies semantic tokenization, flow-based generation, and learned entropy modeling within a single framework. Building on the fully open flow-based SANA architecture, PerCoV2 introduces a novel resolution-adaptive 1D query-based tokenizer that produces compact semantic image tokens with a dual role in flow matching: providing a data-dependent reconstruction prior for initialization and a conditioning signal for flow-based refinement. By explicitly decoupling semantic representation from perceptual generation, our dual representation simplifies the flow-based learning objective, leading to more stable optimization and improved perceptual compression performance. PerCoV2 further introduces a dedicated 1D masked entropy model to improve rate efficiency and optional decoder-side multimodal enhancement via a vision-language model (Molmo) without increasing the transmitted bit budget. On MSCOCO-30k, PerCoV2 achieves state-of-the-art statistical fidelity, measured by FID and KID, across ultra-low and extreme bit-rates (0.0015-0.025 bpp). When trained solely on the general-purpose SA-1B dataset, PerCoV2 further demonstrates strong zero-shot generalization to widely adopted high-resolution benchmarks, including DIV2K and CLIC 2020, achieving competitive statistical fidelity with the current leading method, AEIC-ME. Finally, we introduce PerCoV2-distilled, a practical single-step variant derived from multi-step flow matching that accelerates decoding by 5.37x over PerCoV1, while preserving perceptual compression performance.