One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

Zheyu Zhang; Ziqi Pang; Shixing Chen; Xiang Hao; Vimal Bhat; Yu-Xiong Wang

cs.CVApr 15, 2026

One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

Authors: Zheyu Zhang, Ziqi Pang, Shixing Chen, Xiang Hao, Vimal Bhat, Yu-Xiong Wang

Organizations: University of Illinois Urbana-Champaign · Amazon Prime Video

Abstract

Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames. With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information. To address this, we explore extreme video token compression towards one token per frame at the final LLM layer. Our key insight is that heuristic-based compression, widely adopted by previous methods, is prone to information loss, and this necessitates supervising LLM layers into learnable and progressive modules for token-level compression (LP-Comp). Such compression enables our VLM to digest 2x-4x more frames with improved performance. To further increase the token efficiency, we investigate frame-level compression, which selects the frames most relevant to the queries via the internal attention scores of the LLM layers, named question-conditioned compression (QC-Comp). As a notable distinction from previous studies, we mitigate the position bias of LLM attention in long contexts, i.e., the over-concentration on the beginning and end of a sequence, by splitting long videos into short segments and employing local attention. Collectively, our combined token-level and frame-level leads to an extreme compression model for long video understanding, named XComp, achieving a significantly larger compression ratio and enabling denser frame sampling. Our XComp is finetuned from VideoChat-Flash with a data-efficient supervised compression tuning stage that only requires 2.5% of the supervised fine-tuning data, yet boosts the accuracy from 42.9% to 46.2% on LVBench and enhances multiple other long video benchmarks.

Figures & tables

Figure 1: Left: We present XComp that explores using the LLM layers to progressively compress the video tokens towards the extreme of one token per frame . Right: Such capabilities enable the model to better improve itself with denser input frames without significantly increasing video tokens.

Figure 2: Overview. Our XComp comprises of two parts to achieve extreme compression in long video understanding. (1) At the token level , we propose the supervised compression tuning that enables the LLM to compress every video frame into one compact token in a learnable and progressive manner, namely, LP-Comp (Sec. 3.2 ). (2) At the frame level , we utilize the internal attention mechanism to select the frames relevant to the questions, which is question-conditioned compression, namely, QC-Comp (Sec. 3.3 ). Both aspects combined lead to our objective of extreme compression in long video understanding: one token per highly selective frame.

Figure 3: Learnable and Progressive Compression (LP-Comp). With supervised compression tuning , the LLM layers learn to condense the visual tokens progressively into a concise set of tokens until reaching the extreme of one token per frame.

Figure 4: Question-Conditioned Compression. To reduce the visual redundancy and improve long video understanding performance, we split a video into individual segments and assign question-conditioned relevance scores to video frames. The frames with lower relevance scores are discarded so that the LLM only concentrates on the informative frames.

Proprietary Models
	Size	LongVideoBench	MLVU	VideoMME (Long)	LVBench
Average Duration		473s	651s	2386s	4101s
GPT-4V [ 48 ]	-	59.1	49.2	53.5	–
GPT-4o [ 49 ]	-	66.7	64.6	65.3	30.8
Gemini-1.5-Pro [ 58 ]	-	64.0	–	67.4	33.1
3 $\sim$ 9B-Scale VLMs

Table 1: Long video understanding comparison. Our method XComp built upon VideoChat-Flash-2B achieves state-of-the-art results among 2B-scale models, demonstrating the effectiveness of extreme compression in long video understanding. Note that the performance of proprietary models and 3

\sim

9B-scale VLMs are provided for reference.

Figure 5: LongVideoBench case analysis. XComp leverages QC-Comp to divide a long video into short segments, mitigating attention bias. Local attention highlights key frames relevant to the question and correct answer, enabling effective filtering of irrelevant content.

Method	Training-free	Learnable
Baseline (w/o compression)	41.8	-
+ Heuristic	41.1	-
+ Step-wise	38.4	42.3
+ Progressive	39.7	44.3

Table 2: Ablation study using LVBench: progressive v.s. step-wise non-heuristic token compression v.s. heuristic compression from VideoChat-Flash, and learnable v.s. training-free token compression. All the evaluations are conducted with 1024 frames. Both progressive compression and staged compression would gradually compress tokens to 1 token per frame with the same amount of average tokens per frame.

Strategy	LVBench	MLVU
Uniform	43.6	64.1
Suffix	44.3	65.2

Table 3: Comparison of token compression strategies on LVBench and MLVU. Both Uniform Drop and Suffix-Preservation follow the same Cosine compression schedule and efficient continual training.

Attention	LongVideoBench	MLVU	VideoMME (long)
Global	59.5	66.3	44.8
Segment Local (Ours)	59.7	66.7	45.7

Table 4: Comparison of attention strategies in QC-drop. Segment-wise attention consistently outperforms global attention across benchmarks.

	LongVideoBench
Baseline (w/o compression)	58.3
XComp w/o QC-Comp	58.8
XComp	59.7

Table 5: Ablation results showing the individual and combined impact of LP-Comp and QC-Comp on LongVideoBench. Both components contribute to the overall performance gains of XComp.

Model Name	LP-Comp	Variants	QC-Comp	Variants	LongVideoBench	LVBench
VideoChat-Flash	No	-	No	-	58.3	42.9
XComp	Yes	Uniform	No	-	58.2	43.6
XComp	Yes	Suffix	No	-	58.8	44.3
XComp	Yes	Suffix	Yes	Global	59.5	45.6
XComp	Yes	Suffix	Yes	Segment Local	59.7	46.2

Table 6: The complete table of all ablation studies on LongVideoBench and LVBench.

Model Version	VideoMME (Long)	LVBench
LLaVA-Next-Video	62.7	40.6
LLaVA-Next-Video + FT	61.4	41.4
LLaVA-Next-Video + XComp	63.2	43.1

Table 7: Generalize to LLaVA-Next-Video, a different model from VideoChat-Flash. XComp consistently improves the efficiency and accuracy of LLaVA-Next-Video on the two long video benchmarks, outperforming both the baseline and fine-tuning.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value / Description
tunable_parts	Large Langauge Model
learning_rate	$1\times 10^{-5}$
weight_decay	$0.0$
warmup_ratio	0.03
lr_scheduler_type	Cosine
dataloader_num_workers	1

Appendix

Table A : The hyperparameters used in fine-tuning.

Hyperparameter	Value / Description
temperature	0.0
do_sample	False
num_beams	1
n_repeat	2 (each frame in 2 chunks)
n_selected_frames	256 (Long), 512 (MME), 1,024 (MLVU), 2,048 (LVB)

Appendix

Table B : The hyperparameters used in evaluation.

Image datasets	Video datasets
LLaVA - OneVision [ 32 ] , LLaVA - NeXT [ 39 ] , M4 - Instruct [ 33 ]	Kinetics - 400 [ 27 ] , Something - Something [ 20 ] , TGIF - QA [ 25 ] , TVQA [ 30 ] , CLEVRER [ 74 ] , NExT - QA [ 67 ] , FAVD [ 54 ] , MovieChat - 1K [ 57 ] , TextVR [ 65 ] , ShareGPT - Video [ 9 ] , ShareGPT - 4o [ 14 ] , Oops [ 15 ] , OVIS [ 51 ] , UVO [ 60 ] , GUI - World [ 7 ] , Vript [ 72 ] , HT - Step [ 1 ] , Ego4D [ 21 ] , LLaVA - Video - 178K [ 78 ] , VideoChat-Flash [ 35 ]

Appendix

Table C : Datasets used for supervised fine - tuning (71,927 instances).

	Size	LongVideoBench	MLVU	VideoMME (Long)	LVBench
Average Duration		473s	651s	2386s	4101s
VideoChat-Flash-2B [ 35 ]	2B	58.3	65.7	44.9	42.9
VideoChat-Flash-2B+FT	2B	57.4	65.6	44.7	43.2
XComp	2B	59.7	66.7	45.6	46.2

Appendix

Table D : Ablation study of fine-tuning.

Frames	Model	TFLOPs		Latency
1,024	VideoChat-Flash	43		0.22 s
1,024	Ours	20	( 53% $\downarrow$ )	0.12 s	( 45% $\downarrow$ )
2,048	VideoChat-Flash	132		0.54 s
2,048	Ours	58	( 56% $\downarrow$ )	0.25 s	( 54% $\downarrow$ )
4,096	VideoChat-Flash	448		1.56 s
4,096	Ours	187	( 58% $\downarrow$ )	0.66 s	( 58% $\downarrow$ )

Appendix

Table E : Efficiency Comparison on LVBench Queries at Inference Stage.

	CLEVRER						VDC
	Expl (Opt)	Expl (Q)	Pred (Opt)	Pred (Q)	Cntrf (Opt)	Cntrf (Q)	BLEU@1	BLEU@4
VideoChat-Flash-2B	0.9463	0.8497	0.7789	0.5738	0.7775	0.4007	7.1	1.6
XComp	0.9398	0.8406	0.7697	0.5600	0.7625	0.3652	7.0	1.6

Appendix

Table F : CLEVRER and VDC results.

Model	MME-VideoOCR (Overall)	MME-VideoOCR ( $>$ 30s)
VideoChat-Flash-2B	37.1	49.1
VideoChat-Flash-2B + FT	34.9	46.0
XComp	35.4	46.4

Appendix

Table G : MME-VideoOCR results.

Figure A : Multi-Hop Needle-in-a-Haystack QA Performance.

Explore similar work

Small Vision-Language Models are Smart Compressors for Long Video Understanding
Apr 9, 2026Junjie Fei, Jun Chen, Zechun Liu +13Long-Video Understanding Video Token Compression
Meta AI · King Abdullah University of Science and Technology (KAUST)
Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding
Oct 12, 2026Shuo Gao, Chenhao Zheng, Zhongzheng RenLong-Video Understanding Efficient VLM Inference
Renmin University of China · University of Washington · UNC-Chapel Hill
Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs
Sep 3, 2026Prakhar KhatriVideo-Language Model Evaluation Long-Video Understanding
Independent Researcher

Apr 9, 2026J/K move · Enter open · S save

Small Vision-Language Models are Smart Compressors for Long Video Understanding

Junjie Fei, Jun Chen, Zechun Liu, Yunyang Xiong+12

Meta AI · King Abdullah University of Science and Technology (KAUST)

Long-Video Understanding Video Token Compression Vision-Language Models Video Compression

Open paper AskPDF arXiv

Adapting Multimodal Large Language Models (MLLMs) for hour-long videos is bottlenecked by context limits. Dense visual streams saturate token budgets and exacerbate the lost-in-the-middle phenomenon. Existing heuristics, like sparse sampling or uniform pooling, blindly sacrifice fidelity by discarding decisive moments and wasting bandwidth on irrelevant backgrounds. We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding. Tempo leverages a Small Vision-Language Model (SVLM) as a local temporal compressor, casting token reduction as an early cross-modal distillation process to generate compact, intent-aligned representations in a single forward pass. To enforce strict budgets without breaking causality, we introduce Adaptive Token Allocation (ATA). Exploiting the SVLM's zero-shot relevance prior and semantic front-loading, ATA acts as a training-free

O(1)

dynamic router. It allocates dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to maintain the global storyline. Extensive experiments show our 6B architecture achieves state-of-the-art performance with aggressive dynamic compression (0.5-16 tokens/frame). On the extreme-long LVBench (4101s), Tempo scores 52.3 under a strict 8K visual budget, outperforming GPT-4o and Gemini 1.5 Pro. Scaling to 2048 frames reaches 53.7. Crucially, Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency rather than greedily padded context windows.

Abstract

Explore similar work

Small Vision-Language Models are Smart Compressors for Long Video Understanding

Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding

Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs