Large Multimodal Models (LMMs) uniformly perceive video frames, creating computational inefficiency for videos with inherently varying temporal redundancy. This paper present Quicksviewer, an LMM with new perceiving paradigm that partitions a nonuniform redundant video into varying cubes using Gumbel Softmax, followed by a unified resampling for each cube to achieve efficient video understanding. We further propose a Gumbel-noise annealing strategy that enables stable end-to-end LMM training with the discrete cubing network. This simple and intuitive approach dynamically compress video online based on the momentum of frame differences, significantly reducing spatiotemporal complexity (overall 45× compression rate), while enabling efficient training with large receptive field. We train the model from a language backbone through three progressive stages, each incorporating lengthy videos on average of 420s/1fps thanks to the perceiving efficiency. With only 0.8M total video samples, our model outperforms contemporary LMMs with the same backbone and comparable parameters amount. An experiment in A/B pretraining with same backbone and training data shows that, w/ cubing achieves lower training loss and higher benchmark accuracy (+2.1) than the fixed partitioning baseline, strictly validating the benefits. Moreover, the front-loaded compression in encoding substantially reduces latency in the computationally expensive prefill and decoding, making it suited to mainstream EPD disaggregated inference deployment. Code: https://quicksviewer.github.io.
Figures & tables
Figure 2: Left: Three-stage training. Middle and right: Controlled A/B pretraining with the same backbone (Qwen2.5 + SigLIP) and training data (150K captions). Cubing yields lower training loss and improves benchmark accuracy (+2.1) over fixed partitioning at the same input token budget.
Model
Visual tokens / original frame
Compression
Max train frames
Eval frames
D600 (%)
Video-MME w/o subs (%)
Trained compression models
VideoLLaMA2 7B
72
8x
16
16
97.33
47.9
PLLaVA 7B
144
4x
16
16
97.33
-
Kangaroo 8B
128
8x
160
64
89.33
56.0
TimeMarker 8B
36*
16x*
128
120*
80.00*
57.3
LongVU 7B
≤ 144
Adaptive
-
1 FP
-
-
Table 1: Comparison of Quicksviewer with representative video LMMs and compression methods. Metrics are compiled from official implementations or reported results; see Appendix C for definitions and details. Quicksviewer achieves 45 × compression and 62.3% accuracy on Video-MME.
Figure 3: Left: The network architecture of Quicksviewer, that performs unified understanding of videos and images through visual tokens from cascaded modules. Right: The cubing network, that partitions an online video into nonuniform cubes based on Gumbel Softmax.
Figure 4: (a) Domain-specific results on VideoMME. (b) Inference time comparison with LLaVA-Next-Video. (c) Distribution of cube lengths on VideoMME.
Figure 5: The "Visual Lag" phenomenon occurring during the model’s cube-based segmental comprehension, where current cubes incorporate terminal frames from preceding event scenes to enable retrospective understanding.
Figure 6: (a) Left: Gumbel noise progressively anneals to 0.001 following the decaying learning rate with cosine scheduler. (b) Right: Compared to non-annealed training (cyan curve), adding Gumbel noise annealing (purple curve) yields more stable and superior loss convergence.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Stage-1
Stage-2
Stage-3
Vision
Resolution
384×384
384×384
384×384
FPS,#Frames
1, Max 420
1, Max 420
1, Max 420
Data
Image-Text #samples
LCS, OBELICS
LLaVAOV-SingleImage
LLaVAOV-MultiImages
558K, 20K
2.99M
100K
Video-Text #samples
FineVideo, ANetCaptions
FineVideo, ShareGPT4Video, ANetCaptions
Sec. 3. 4
87K, 8K
118K
599K
Appendix
Table 4: Detailed configuration for each training stage.
Modality
Task
# Samples
Dataset
Image-Text
Interleaved Pairs
20K
OBELICS
Single-Image Captioning
558K
LCS
Video-Text
Captioning
95K
FineVideoCaptions, ANetCaptions
Appendix
Table 5: Training data statistics for the alignment stage.
Modality
Task
# Samples
Dataset
Text
Instruction
93K
Evo-Instruct
Image-Text
Interleaved Pairs
20K
OBELICS
Single-Image Captioning
50K
LCS
Single-Image Tasks
2.8M
LLaVAOneVision
Video-Text
Captioning
5K
FineVideoCaptions, AnetCaptions
VQA
75K
FineVideoQAs
Appendix
Table 6: Training data statistics for the pre-training stage.
Modality
Task
# Samples
Dataset
Text
Instruction
9K
Evo-Instruct
Image-Text
Single-Image Tasks
40K
LLaVA-OneVision-SingleImage
Multi-Images Tasks
100K
LLaVA-OneVision-MultiImages
Video-Text
Captioning
52K
TextVR, MovieChat, YouCook2
Dense Captioning
4K
ShareGPT4Video
Classification
1K
Kinetics-710
Appendix
Table 7: Training data statistics for the supervised fine-tuning stage.
Model
Visual tokens / original frame
Compression
Max train frames
Eval frames
D600 (%)
Video-MME w/o subs (%)
Trained compression models
LLaMA-VID 7B
2
128x
-
1 FPS
conditional 0
-
VideoLLaMA2 7B
72
8x
16
16
97.33
47.9
VideoLLaMA3 7B
Dynamic
Dynamic
180
≤ 180
≥ 70.00
66.2
PLLaVA 7B
144
4x
16
16
97.33
-
Chat-UniVi 7B
7 at F=64
36.57x
64
64
89.33
-
Appendix
Table 8: Comparison of representative efficient video LMMs publicly available by June 30, 2025. Video-MME results use the no-subtitle setting. BIMBA follows the v1 128-frame setup (training cap assumed). Video-XL’s count of 144 tokens/frame excludes additional visual summarization tokens (VSTs). - denotes unavailable or unresolved values, including unverified subtitle protocols; N/A denotes not applicable. D600=100[1−min(600,Feval)/600] is the sampling-budget omission for 600 input frames, not an architectural limit or an observed adaptive frame-drop rate. Capacity0 means the reported frame cap is at least 600; “+ adaptive” indicates additional content-dependent pruning. Formatting tokens are excluded. Asterisks mark conditional settings: TimeSuite’s ratio uses the raw spatial patch grid; TimeMarker’s values use a 600-second video; MiniCPM-V’s ratio assumes a single square 448-pixel view; ReTaKe’s capacity does not account for adaptive frame selection.
Figure 7: Qualitative analysis showns that Quicksviewer effectively understands lengthy documentary and sports videos, as well as informative single and multiple images.
Adapting Multimodal Large Language Models (MLLMs) for hour-long videos is bottlenecked by context limits. Dense visual streams saturate token budgets and exacerbate the lost-in-the-middle phenomenon. Existing heuristics, like sparse sampling or uniform pooling, blindly sacrifice fidelity by discarding decisive moments and wasting bandwidth on irrelevant backgrounds. We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding. Tempo leverages a Small Vision-Language Model (SVLM) as a local temporal compressor, casting token reduction as an early cross-modal distillation process to generate compact, intent-aligned representations in a single forward pass. To enforce strict budgets without breaking causality, we introduce Adaptive Token Allocation (ATA). Exploiting the SVLM's zero-shot relevance prior and semantic front-loading, ATA acts as a training-free O(1) dynamic router. It allocates dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to maintain the global storyline. Extensive experiments show our 6B architecture achieves state-of-the-art performance with aggressive dynamic compression (0.5-16 tokens/frame). On the extreme-long LVBench (4101s), Tempo scores 52.3 under a strict 8K visual budget, outperforming GPT-4o and Gemini 1.5 Pro. Scaling to 2048 frames reaches 53.7. Crucially, Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency rather than greedily padded context windows.
Junjie Fei, Jun Chen, Zechun Liu +13
Meta AI · King Abdullah University of Science and Technology (KAUST)
Multimodal Large Language Models (MLLMs) have demonstrated strong performance in video understanding, yet efficiently processing long, high-resolution videos remains challenging. Such videos often contain substantial spatiotemporal redundancy, and processing redundant visual tokens can incur avoidable computational overhead. Many existing methods prune visual tokens during or after vision transformer (ViT) encoding, leaving much of the encoding cost unaddressed. Some approaches prune patches before encoding but rely on learned auxiliary networks for patch selection, incurring additional training and inference overhead. To address these limitations, we propose FlashGaze, a training-free method that reduces spatiotemporal redundancy before ViT encoding without introducing auxiliary networks. FlashGaze uses pixel-space differences as a proxy for information loss and employs Quadtree Dynamic Programming to jointly optimize patch dropping, merging, and keeping under a fixed budget. Experiments on two MLLM backbones across multiple benchmarks demonstrate substantial efficiency gains while largely preserving accuracy. On Qwen3-VL-8B, FlashGaze retains 98% of the full-input baseline accuracy on LongVideoBench while achieving up to 5.4x and 17x speedups in ViT encoding and MLLM prefill, respectively, and reducing peak GPU memory usage by a factor of 1.8. These efficiency gains enable the model to process videos with more frames and higher resolutions on the same GPU hardware, unlocking video understanding at scales previously out of reach.
Ziye Zhu, Yanghao Zhou, Lixing Tan +3
Peking University · Canva Research · Sun Yat-sen University
Video large language models (Video-LLMs) have demonstrated strong capabilities in video understanding tasks. However, their practical deployment is still hindered by the inefficiency introduced by processing massive amounts of visual tokens. Although recent approaches achieve extremely low token retention ratios while maintaining accuracy comparable to full-token baselines, most of them perform compression only at the late stage of prefilling, leaving the efficiency of the vision encoder unoptimized. In this paper, we first show that vision encoding contributes a large portion to the time-to-first-token (TTFT). Therefore, instead of compressing visual tokens only after the vision encoder, performing compression inside the encoder still leaves substantial room for exploration. Based on this insight, we propose EarlyTom, a training-free token compression framework that performs early-stage visual token compression inside the vision encoder, enabling significantly better TTFT reduction and higher throughput. In addition, we introduce a decoupled spatial token selection strategy that improves the overall compression effectiveness. EarlyTom reduces TTFT by up to 2.65x and FLOPs by up to 61% on a single NVIDIA A100 GPU for the LLaVA-OneVision-7B model, while maintaining accuracy comparable to the full-token baseline. These improvements substantially enhance the practicality of deploying Video-LLMs in real-world production scenarios.
Hesong Wang, Xin Jin, Lu Lu +4
Zhejiang University · Westlake University · Alibaba Cloud Computing