Adapting Multimodal Large Language Models (MLLMs) for hour-long videos is bottlenecked by context limits. Dense visual streams saturate token budgets and exacerbate the lost-in-the-middle phenomenon. Existing heuristics, like sparse sampling or uniform pooling, blindly sacrifice fidelity by discarding decisive moments and wasting bandwidth on irrelevant backgrounds. We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding. Tempo leverages a Small Vision-Language Model (SVLM) as a local temporal compressor, casting token reduction as an early cross-modal distillation process to generate compact, intent-aligned representations in a single forward pass. To enforce strict budgets without breaking causality, we introduce Adaptive Token Allocation (ATA). Exploiting the SVLM's zero-shot relevance prior and semantic front-loading, ATA acts as a training-free O(1) dynamic router. It allocates dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to maintain the global storyline. Extensive experiments show our 6B architecture achieves state-of-the-art performance with aggressive dynamic compression (0.5-16 tokens/frame). On the extreme-long LVBench (4101s), Tempo scores 52.3 under a strict 8K visual budget, outperforming GPT-4o and Gemini 1.5 Pro. Scaling to 2048 frames reaches 53.7. Crucially, Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency rather than greedily padded context windows.
Figures & tables
Figure 1: Tempo achieves SOTA long video understanding via query-aware Adaptive Token Allocation (ATA). (a) Motivation: Query-agnostic methods either miss transient moments (sparse sampling) or blur details (uniform pooling). Tempo instead utilizes a small vision-language model as a smart compressor for query-aware cross-modal distillation. (b) Mechanism: ATA dynamically allocates high bandwidth (16 tokens/frame) to relevant segment for fine-grained details, while compressing redundant contexts into minimal temporal anchors ( ∼ 0.5 tokens/frame) to maintain causality. (c) Result: Leading performance on LVBench. Tempo-6B achieves superior accuracy at extreme compression rates ( i.e. , 4 or 6 tokens/frame), outperforming open-source models and proprietary baselines with a fraction of the context budget.
Figure 2: Overview of the Tempo framework. Our unified architecture casts long video understanding as an end-to-end, query-aware compression process. The Local Compressor (Left). For each segment, a Small Vision-Language Model (SVLM) acts as a semantic temporal compressor. Under causal attention, learnable memory tokens M inherently distill the preceding visual tokens Xi and user query Q . Inference-Only Bypass (Middle). During a single forward pass, an Adaptive Token Allocation (ATA) controller intercepts the hidden state hirel to compute a zero-shot relevance score si . This enables an O(1) dynamic head truncation, allocating dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to strictly satisfy a global budget Bmax . The Global Decoder (Right). The compressed memory tokens are assembled into a highly sparse, time-aware sequence using explicit temporal tags ( e.g. , <t=2.0s> ). A global LLM synthesizes this condensed multimodal context to generate the final response.
Model
Size
Tokens
LongVideoBench
MLVU
Video-MME (w/o sub.)
LVBench
per frame
(473s)
(651s)
Overall (1010s)
Long (2386s)
(4101s)
Proprietary Models
GPT-4o Hurst et al. (2024)
-
-
66.7
64.6
71.9
65.3
30.8
Gemini 1.5 Pro Team et al. (2024)
-
-
64.0
-
75.0
67.4
33.1
General Open-Source MLLMs
VideoChat2-HD Li et al. (2025)
7B
72
-
47.9
45.3
39.8
-
Table 1: Comparison with state-of-the-art MLLMs on long video benchmarks, highlighting Tempo’s superior accuracy and extreme token efficiency. Bold and underline denote the best and second-best results among specialized long video MLLMs. “-” indicates unavailable results. * indicates the average tokens per frame are dynamically adjusted. For our model, we report the theoretical dynamic range (0.5–16) alongside the actual empirical average tokens per frame ( gray rows ), demonstrating that Tempo inherently operates substantially below the maximum budget limits in practice.
Ablation Setting
LongVB
MLVU
Video-MME (w/o sub.)
LVBench
(473s)
(651s)
Overall (1010s)
Long (2386s)
(4101s)
A. Progressive Training Curriculum
w/o Stage 3 - Long-Context SFT (16K Budget)
61.4
67.2
66.1
56.3
47.3
w/o Adaptive Token Allocation (16K Budget)
62.8
73.5
67.0
56.2
51.1
Tempo Default (8K Budget)
65.1
75.2
67.7
57.0
52.3
B. Segment-Level Budget Allocation
Table 2: Ablation studies on Tempo’s core components. We decompose our framework across five dimensions: (A) progressive training curriculum, (B) segment-level budget allocation, (C) intra-segment token reduction scheme, (D) relevance scoring source, and (E) temporal continuity. Unless otherwise specified, all variants process videos uniformly sampled at 2 FPS up to a maximum of 1024 frames, strictly bounded by an 8K visual token budget for fair comparison. The default Tempo configuration is highlighted in gray . LongVB denotes LongVideoBench.
Figure 3: Scaling behavior of Tempo. We investigate the interplay between maximum frame capacity ( fmax ) and total visual token budgets. (Left) On Video-MME (Long), a strict 4K budget acts as an optimal sweet spot by aggressively filtering redundancy, whereas larger budgets (8K/12K) yield marginal noise. (Right) On the extreme-long LVBench, restrictive budgets eventually limit the achievable performance, whereas expansive capacities ( e.g. , 12K) monotonically unlock new peaks at higher frame densities, proving the necessity of scaled context for hour-long video understanding.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A: Distribution of allocated tokens per segment. (Top) 4K budget ( B=4096 ); (Bottom) 8K budget ( B=8192 ). Across four benchmarks, ATA consistently exhibits a strongly right-skewed, long-tailed allocation pattern. The majority of segments are compressed into very low-token representations, while a small fraction receives substantially higher allocations for query-aligned segments. Notably, this distribution pattern remains stable under different global budgets.
Figure B: Macro-level budget utilization and adaptation efficiency. (Top) 4K budget; (Bottom) 8K budget. Each red dot denotes the average token consumption per segment for a video sample. The dashed green line indicates the dataset-level average theoretical capacity . Points above the line correspond to shorter videos whose per-segment capacity is higher than the dataset-wide average. Adaptability: On datasets with diverse video lengths ( e.g. , LongVideoBench, Video-MME), the consumption distribution remains well below the theoretical capacity, indicating query-driven compression. Reliability: On extremely long videos (LVBench), the token consumption forms a clear ceiling at the theoretical limit, demonstrating strict adherence to the global budget constraint.
Figure C: Qualitative examples of query-aware Adaptive Token Allocation. We visualize the token allocation distribution across three distinct types of video QA. The red-bordered key frames correspond to segments with high token allocation, while grey-bordered frames represent redundant contexts compressed to minimal tokens. (Top) Precise Action Retrieval: ATA isolates the exact moment the target action (lassoing the yak) occurs. (Middle) Targeted Object Grounding: ATA highlights specific scenes containing cooking machinery relevant to the query concepts (“machine”, “cook”), while effectively suppressing general manual food preparation. (Bottom) Global Summarization: ATA maintains a dense allocation distribution to capture the overarching theme of the diverse food vlog.
Configuration
Stage 0: Alignment
Stage 1: Pre-training
Stage 2: Broad SFT
Stage 3: Long-Context SFT
Trainable Modules
Local Compressor (SVLM)
✗ Frozen
✓ Unfrozen
✓ Unfrozen
✗ Frozen
Memory Tokens
✓ Unfrozen
✓ Unfrozen
✓ Unfrozen
✗ Frozen
Linear Projector
✓ Unfrozen
✓ Unfrozen
✓ Unfrozen
✗ Frozen
Global Decoder (LLM)
✗ Frozen
✓ Unfrozen
✓ Unfrozen
✓ Unfrozen
Optimization Hyper-parameters
Appendix
Table A: Hyper-parameter configurations across the four-stage training curriculum of Tempo. ✓ indicates that the module is trainable, while ✗ indicates that it is frozen during training.
Modality
Stage 0: Alignment
Stage 1: Pre-training
Stage 2: Broad SFT
Stage 3: Long-Context SFT
Image Samples
∼ 558K
∼ 2.00M
∼ 0.93M
∼ 384K (Mixed)
Video Samples
-
∼ 1.38M
∼ 2.25M
Text Samples
-
∼ 143K
∼ 71K
Total Size
∼ 558K
∼ 3.52M
∼ 3.25M
∼ 384K
Appendix
Table B: Statistics of training data mixtures. Approximate number of samples per modality across the four progressive training stages.
Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames. With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information. To address this, we explore extreme video token compression towards one token per frame at the final LLM layer. Our key insight is that heuristic-based compression, widely adopted by previous methods, is prone to information loss, and this necessitates supervising LLM layers into learnable and progressive modules for token-level compression (LP-Comp). Such compression enables our VLM to digest 2x-4x more frames with improved performance. To further increase the token efficiency, we investigate frame-level compression, which selects the frames most relevant to the queries via the internal attention scores of the LLM layers, named question-conditioned compression (QC-Comp). As a notable distinction from previous studies, we mitigate the position bias of LLM attention in long contexts, i.e., the over-concentration on the beginning and end of a sequence, by splitting long videos into short segments and employing local attention. Collectively, our combined token-level and frame-level leads to an extreme compression model for long video understanding, named XComp, achieving a significantly larger compression ratio and enabling denser frame sampling. Our XComp is finetuned from VideoChat-Flash with a data-efficient supervised compression tuning stage that only requires 2.5% of the supervised fine-tuning data, yet boosts the accuracy from 42.9% to 46.2% on LVBench and enhances multiple other long video benchmarks.
Zheyu Zhang, Ziqi Pang, Shixing Chen +3
University of Illinois Urbana-Champaign · Amazon Prime Video
With recent advancements in video backbone architectures, combined with the remarkable achievements of large language models (LLMs), the analysis of long-form videos spanning tens of minutes has become both feasible and increasingly prevalent. However, the inherently redundant nature of video sequences poses significant challenges for contemporary state-of-the-art models. These challenges stem from two primary aspects: 1) efficiently incorporating a larger number of frames within memory constraints, and 2) extracting discriminative information from the vast volume of input data. In this paper, we introduce a novel end-to-end schema for long-form video understanding, which includes an information-density-based adaptive video sampler (AVS) and an autoencoder-based spatiotemporal video compressor (SVC) integrated with a multimodal large language model (MLLM). Our proposed system offers two major advantages: it adaptively and effectively captures essential information from video sequences of varying durations, and it achieves high compression rates while preserving crucial discriminative information. The proposed framework demonstrates promising performance across various benchmarks, excelling in both long-form video understanding tasks and standard video understanding benchmarks. These results underscore the versatility and efficacy of our approach, particularly in managing the complexities of prolonged video sequences.
Video Large Language Models (Video LLMs) achieve strong performance on video understanding tasks but suffer from high inference costs due to the large number of visual tokens. We study training-free, query-agnostic video token compression, where heuristic importance rules can over-represent large redundant visual groups and miss less frequent visual patterns, particularly under aggressive token budgets. We propose KiToke, a distribution-aware compression framework that treats visual tokens as samples from the video's spatiotemporal token distribution. KiToke uses global embedding density as a proxy for representational redundancy and inverse-density sampling to balance token allocation across visual patterns of different frequencies. It further aggregates unselected tokens into retained anchors within detected temporal intervals to reduce cross-transition mixing. Across evaluated backbones, KiToke achieves the highest average accuracy among compared training-free methods and remains robust down to 1% retention, where most evaluated baselines underperform uniform random token sampling. On LLaVA-OneVision, it preserves 98.5% and 89.2% of the uncompressed model's four-benchmark mean accuracy at 10% and 1% token retention, with 6.8x and 12.5x prefill speedups, respectively, including compression overhead.
Haifeng Huang, Biyin Xu, Yang Li
Department of Computer Science, Iowa State University.