Small Vision-Language Models are Smart Compressors for Long Video Understanding

Junjie Fei; Jun Chen; Zechun Liu; Yunyang Xiong; Chong Zhou; Wei Wen; Junlin Han; Mingchen Zhuge

cs.CVApr 9, 2026

Small Vision-Language Models are Smart Compressors for Long Video Understanding

Authors: Junjie Fei, Jun Chen, Zechun Liu, Yunyang Xiong, Chong Zhou, Wei Wen, Junlin Han, Mingchen Zhuge, +8 more

Organizations: Meta AI · King Abdullah University of Science and Technology (KAUST)

Abstract

Adapting Multimodal Large Language Models (MLLMs) for hour-long videos is bottlenecked by context limits. Dense visual streams saturate token budgets and exacerbate the lost-in-the-middle phenomenon. Existing heuristics, like sparse sampling or uniform pooling, blindly sacrifice fidelity by discarding decisive moments and wasting bandwidth on irrelevant backgrounds. We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding. Tempo leverages a Small Vision-Language Model (SVLM) as a local temporal compressor, casting token reduction as an early cross-modal distillation process to generate compact, intent-aligned representations in a single forward pass. To enforce strict budgets without breaking causality, we introduce Adaptive Token Allocation (ATA). Exploiting the SVLM's zero-shot relevance prior and semantic front-loading, ATA acts as a training-free $O(1)$ dynamic router. It allocates dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to maintain the global storyline. Extensive experiments show our 6B architecture achieves state-of-the-art performance with aggressive dynamic compression (0.5-16 tokens/frame). On the extreme-long LVBench (4101s), Tempo scores 52.3 under a strict 8K visual budget, outperforming GPT-4o and Gemini 1.5 Pro. Scaling to 2048 frames reaches 53.7. Crucially, Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency rather than greedily padded context windows.

Figures & tables

Figure 1: Tempo achieves SOTA long video understanding via query-aware Adaptive Token Allocation (ATA). (a) Motivation: Query-agnostic methods either miss transient moments (sparse sampling) or blur details (uniform pooling). Tempo instead utilizes a small vision-language model as a smart compressor for query-aware cross-modal distillation. (b) Mechanism: ATA dynamically allocates high bandwidth (16 tokens/frame) to relevant segment for fine-grained details, while compressing redundant contexts into minimal temporal anchors ( $\sim$ 0.5 tokens/frame) to maintain causality. (c) Result: Leading performance on LVBench. Tempo-6B achieves superior accuracy at extreme compression rates ( i.e. , 4 or 6 tokens/frame), outperforming open-source models and proprietary baselines with a fraction of the context budget.

Figure 2: Overview of the Tempo framework. Our unified architecture casts long video understanding as an end-to-end, query-aware compression process. The Local Compressor (Left). For each segment, a Small Vision-Language Model (SVLM) acts as a semantic temporal compressor. Under causal attention, learnable memory tokens $\mathbf{M}$ inherently distill the preceding visual tokens $\mathbf{X}_{i}$ and user query $Q$ . Inference-Only Bypass (Middle). During a single forward pass, an Adaptive Token Allocation (ATA) controller intercepts the hidden state $\mathbf{h}_{i}^{\mathrm{rel}}$ to compute a zero-shot relevance score $s_{i}$ . This enables an $\mathcal{O}(1)$ dynamic head truncation, allocating dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to strictly satisfy a global budget $B_{\max}$ . The Global Decoder (Right). The compressed memory tokens are assembled into a highly sparse, time-aware sequence using explicit temporal tags ( e.g. , <t=2.0s> ). A global LLM synthesizes this condensed multimodal context to generate the final response.

Proprietary Models
Model	Size	Tokens	LongVideoBench	MLVU	Video-MME (w/o sub.)		LVBench
Model	Size	per frame	(473s)	(651s)	Overall (1010s)	Long (2386s)	(4101s)
GPT-4o Hurst et al. (2024)	-	-	66.7	64.6	71.9	65.3	30.8
Gemini 1.5 Pro Team et al. (2024)	-	-	64.0	-	75.0	67.4	33.1
General Open-Source MLLMs
VideoChat2-HD Li et al. (2025)	7B	72	-	47.9	45.3	39.8	-

Table 1: Comparison with state-of-the-art MLLMs on long video benchmarks, highlighting Tempo’s superior accuracy and extreme token efficiency. Bold and underline denote the best and second-best results among specialized long video MLLMs. “-” indicates unavailable results. * indicates the average tokens per frame are dynamically adjusted. For our model, we report the theoretical dynamic range (0.5–16) alongside the actual empirical average tokens per frame ( gray rows ), demonstrating that Tempo inherently operates substantially below the maximum budget limits in practice.

Ablation Setting	LongVB	MLVU	Video-MME (w/o sub.)		LVBench
Ablation Setting	(473s)	(651s)	Overall (1010s)	Long (2386s)	(4101s)
A. Progressive Training Curriculum
w/o Stage 3 - Long-Context SFT (16K Budget)	61.4	67.2	66.1	56.3	47.3
w/o Adaptive Token Allocation (16K Budget)	62.8	73.5	67.0	56.2	51.1
Tempo Default (8K Budget)	65.1	75.2	67.7	57.0	52.3
B. Segment-Level Budget Allocation

Table 2: Ablation studies on Tempo’s core components. We decompose our framework across five dimensions: (A) progressive training curriculum, (B) segment-level budget allocation, (C) intra-segment token reduction scheme, (D) relevance scoring source, and (E) temporal continuity. Unless otherwise specified, all variants process videos uniformly sampled at 2 FPS up to a maximum of 1024 frames, strictly bounded by an 8K visual token budget for fair comparison. The default Tempo configuration is highlighted in gray . LongVB denotes LongVideoBench.

Figure 3: Scaling behavior of Tempo. We investigate the interplay between maximum frame capacity (

f_{\max}

) and total visual token budgets. (Left) On Video-MME (Long), a strict 4K budget acts as an optimal sweet spot by aggressively filtering redundancy, whereas larger budgets (8K/12K) yield marginal noise. (Right) On the extreme-long LVBench, restrictive budgets eventually limit the achievable performance, whereas expansive capacities ( e.g. , 12K) monotonically unlock new peaks at higher frame densities, proving the necessity of scaled context for hour-long video understanding.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A: Distribution of allocated tokens per segment. (Top) 4K budget (

B=4096

); (Bottom) 8K budget (

B=8192

). Across four benchmarks, ATA consistently exhibits a strongly right-skewed, long-tailed allocation pattern. The majority of segments are compressed into very low-token representations, while a small fraction receives substantially higher allocations for query-aligned segments. Notably, this distribution pattern remains stable under different global budgets.

Figure B: Macro-level budget utilization and adaptation efficiency. (Top) 4K budget; (Bottom) 8K budget. Each red dot denotes the average token consumption per segment for a video sample. The dashed green line indicates the dataset-level average theoretical capacity . Points above the line correspond to shorter videos whose per-segment capacity is higher than the dataset-wide average. Adaptability: On datasets with diverse video lengths ( e.g. , LongVideoBench, Video-MME), the consumption distribution remains well below the theoretical capacity, indicating query-driven compression. Reliability: On extremely long videos (LVBench), the token consumption forms a clear ceiling at the theoretical limit, demonstrating strict adherence to the global budget constraint.

Figure C: Qualitative examples of query-aware Adaptive Token Allocation. We visualize the token allocation distribution across three distinct types of video QA. The red-bordered key frames correspond to segments with high token allocation, while grey-bordered frames represent redundant contexts compressed to minimal tokens. (Top) Precise Action Retrieval: ATA isolates the exact moment the target action (lassoing the yak) occurs. (Middle) Targeted Object Grounding: ATA highlights specific scenes containing cooking machinery relevant to the query concepts (“machine”, “cook”), while effectively suppressing general manual food preparation. (Bottom) Global Summarization: ATA maintains a dense allocation distribution to capture the overarching theme of the diverse food vlog.

Trainable Modules
Configuration	Stage 0: Alignment	Stage 1: Pre-training	Stage 2: Broad SFT	Stage 3: Long-Context SFT
Local Compressor (SVLM)	✗ Frozen	✓ Unfrozen	✓ Unfrozen	✗ Frozen
Memory Tokens	✓ Unfrozen	✓ Unfrozen	✓ Unfrozen	✗ Frozen
Linear Projector	✓ Unfrozen	✓ Unfrozen	✓ Unfrozen	✗ Frozen
Global Decoder (LLM)	✗ Frozen	✓ Unfrozen	✓ Unfrozen	✓ Unfrozen
Optimization Hyper-parameters

Appendix

Table A: Hyper-parameter configurations across the four-stage training curriculum of Tempo. ✓ indicates that the module is trainable, while ✗ indicates that it is frozen during training.

Modality	Stage 0: Alignment	Stage 1: Pre-training	Stage 2: Broad SFT	Stage 3: Long-Context SFT
Image Samples	$\sim$ 558K	$\sim$ 2.00M	$\sim$ 0.93M	$\sim$ 384K (Mixed)
Video Samples	-	$\sim$ 1.38M	$\sim$ 2.25M
Text Samples	-	$\sim$ 143K	$\sim$ 71K
Total Size	$\sim$ 558K	$\sim$ 3.52M	$\sim$ 3.25M	$\sim$ 384K

Appendix

Table B: Statistics of training data mixtures. Approximate number of samples per modality across the four progressive training stages.

Explore similar work

One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
Apr 15, 2026Zheyu Zhang, Ziqi Pang, Shixing Chen +3Long-Video Understanding Video Token Compression
University of Illinois Urbana-Champaign · Amazon Prime Video
Learning Compact Video Representations for Efficient Long-form Video Understanding in Large Multimodal Models
Feb 19, 2026Yuxiao Chen, Jue Wang, Zhikang Zhang +8Long-Video Understanding Video Token Compression
Amazon AGI
KiToke: Distribution-Aware Token Compression for Video Large Language Models
Apr 3, 2026Haifeng Huang, Biyin Xu, Yang LiVideo Token Compression Visual Token Compression
Department of Computer Science, Iowa State University.

Apr 15, 2026J/K move · Enter open · S save

One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

Zheyu Zhang, Ziqi Pang, Shixing Chen, Xiang Hao+2

University of Illinois Urbana-Champaign · Amazon Prime Video

Long-Video Understanding Video Token Compression Long-Context VLMs Query-Aware Video Frame Selection

Open paper AskPDF arXiv

Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames. With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information. To address this, we explore extreme video token compression towards one token per frame at the final LLM layer. Our key insight is that heuristic-based compression, widely adopted by previous methods, is prone to information loss, and this necessitates supervising LLM layers into learnable and progressive modules for token-level compression (LP-Comp). Such compression enables our VLM to digest 2x-4x more frames with improved performance. To further increase the token efficiency, we investigate frame-level compression, which selects the frames most relevant to the queries via the internal attention scores of the LLM layers, named question-conditioned compression (QC-Comp). As a notable distinction from previous studies, we mitigate the position bias of LLM attention in long contexts, i.e., the over-concentration on the beginning and end of a sequence, by splitting long videos into short segments and employing local attention. Collectively, our combined token-level and frame-level leads to an extreme compression model for long video understanding, named XComp, achieving a significantly larger compression ratio and enabling denser frame sampling. Our XComp is finetuned from VideoChat-Flash with a data-efficient supervised compression tuning stage that only requires 2.5% of the supervised fine-tuning data, yet boosts the accuracy from 42.9% to 46.2% on LVBench and enhances multiple other long video benchmarks.

Abstract

Explore similar work

One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

Learning Compact Video Representations for Efficient Long-form Video Understanding in Large Multimodal Models

KiToke: Distribution-Aware Token Compression for Video Large Language Models