VidToC: Rethinking Video Token Compression for Multimodal Large Language Models
Organizations: AIM3 Lab, Renmin University of China · MiLM Plus, Xiaomi Inc · MiLM Plus, Xiaomi Inc.
Abstract
Extending multimodal large language models (MLLMs) from images to videos faces substantial memory and latency challenges due to the massive number of visual tokens. These challenges have driven the development of numerous visual token compression methods and more efficient LLM architectures. However, existing compression methods are often evaluated with different training setups, tasks, and models, making fair comparison difficult. Meanwhile, the growing use of hybrid Mamba-Transformer architectures raises questions about whether compression strategies transfer across backbones. We present VidToC, a controlled evaluation framework for representative compression algorithms, LLM backbones, and time encoding strategies, covering training-free and fine-tuned settings on three complementary video understanding tasks. Our results reveal task-dependent compression behavior and characterize the effects of compression placement, time encoding, and backbone choice. These findings provide empirical guidance for designing and selecting video token compression strategies across tasks and training settings.
Figures & tables
| Method | Type | Core idea |
| Simple methods | ||
| Uniform Prune | P | Keep tokens at fixed intervals. |
| Window Prune | P | Keep the center token in each spatial grid. |
| Spatial Pool | M | Average-pool spatial tokens within each frame. |
| Spatiotemporal Pool | M | Average-pool tokens jointly over space and time. |
| Pixel Unshuffle Chen et al. (2024b) | M | Fold spatial dimensions into channels. |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Video Token Compression
Video token compression reduces redundant spatial and temporal video representations to make understanding and generation more efficient.
See all 89 papers