Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency

Sixun Dong; Wei Li; Andong Deng; Qi Qian; Victor Zhu; Zhengping Ji; Chen Chen

cs.CVOct 3, 2026

Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency

Authors: Sixun Dong, Wei Li, Andong Deng, Qi Qian, Victor Zhu, Zhengping Ji, Chen Chen

Organizations: University of Central Florida · Meta Reality Labs · Axon

Abstract

Efficient long-video understanding with vision-language models (VLMs) is often framed as selecting informative frames or visual tokens at a fixed native resolution. We show that per-frame resolution can instead be traded for denser temporal coverage, while front-end decoding latency depends on the size of the candidate pool rather than the final token budget. An empirical study across multiple VLMs and long-video benchmarks yields three findings: dense low-resolution sampling outperforms sparse native-resolution sampling at matched token budgets; resolution-sensitive tasks benefit from selected high-resolution frames; and front-end decoding dominates wall time for hour-long videos. Motivated by these findings, we introduce LoHi, a training-free, single-pass framework that combines a dense low-resolution video stream with sparse high-resolution image frames through the VLM's native video and image pathways. LoHi-Anchor selects high-resolution frames using codec I-frame metadata, while LoHi-SemDiv uses query relevance and visual diversity over CLIP features. Across three long-video benchmarks, LoHi improves average accuracy by 10.6 percentage points over the native-resolution baseline at a matched token budget and by 5.2 percentage points over the strongest prior efficiency method. It also reduces front-end decoding latency by up to 7x on hour-long videos. Project page: https://sixundong.com/projects/lohi

Figures & tables

Panel of Figure 1 — Figure 1: Three practical lessons on long-video understanding. (a) At matched token budgets, dense low-resolution sampling substantially outperforms the default baseline. (b) Resolution-sensitive tasks (OCR, Attribute Perception) significantly drop under aggressive down-scaling. Resolution-insensitive tasks (Action Recognition, Temporal Reasoning) remain robust. (c) Decode-then-select(D) scale linearly with video length; while uniform fixed-budget(U) keeps front-end latency constant.

	Token-efficiency at fixed resolution					Resolution-aware		Frames-Res
Axis	FastV [ 3 ]	VisionZip [ 38 ]	VideoLLaMA3 [ 41 ]	AKS [ 29 ]	BOLT [ 17 ]	MMTok [ 6 ]	VisionThink [ 39 ]	LoHi (ours)
Stage	Post-encoder	Post-encoder	Post-encoder	Post-decode	Post-decode	Post-encoder	Multi-turn	Pre-decode
Selection signal	Attn.	Sim.	Training.	Sim	Sim.	Coverage	LLM judgment	Uniform
Per-frame cost	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$
Front-end latency	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$
Modality	Video	Video	Video	Video	Video	Image	Image	V+I

Table 1: Positioning LoHi in long-video efficiency. Per-frame cost and Front-end latency are graded as low (

\bullet

), medium (

\bullet

), or high (

\bullet

Panel of Figure 2 — Figure 2: Three lessons across frames, resolution, and front-end latency. (a) At fixed low resolution ( $r{=}0.25$ ), accuracy on VideoMME rises monotonically with frame count $N$ grows from $16$ to $256$ for all four task categories ( Sec. 3.1 ). (b) Under a matched token budget, resolution-sensitive tasks (R-Sens.) favor higher resolution over more frames, while resolution-insensitive tasks (R-Non-Sens.) prefer denser low-resolution sampling ( Sec. 3.2 ). (c) Video decoder requires group-of-pictures (GoP) seeking, so constructing a dense candidate pool costs far more than uniform sampling( Sec. 3.3 ).

Setting	Qwen3-VL-4B			Qwen3-VL-8B			VideoLLaMA3-7B
Setting	VideoMME	MLVU	LVBench	VideoMME	MLVU	LVBench	VideoMME	MLVU †	LVBench
16F ${@}1.0$	57.78	58.89	38.61	60.04	58.95	38.22	46.81	35.88	32.09
64F ${@}0.5$	62.30	65.40	40.93	65.96	66.41	41.83	64.37	38.97	33.83
256F ${@}0.25$	64.44	69.36	43.71	67.04	71.29	44.80	50.04	40.30	36.99

Table 2: Comparison of diverse frontier VLMs on different long-video benchmarks.

Figure 3: Overview of LoHi framework. Compared with previous work (a) which prunes or selects within a native resolution pathway, LoHi (b) decomposes one decoded video into two complementary streams: dense low-resolution Lo-V via the video pathway and sparse high-resolution Hi-I via the image pathway. (c) shows the LoHi-SemDiv selector, which picks Hi-I indices via greedy MAP.

The new recipe: dense low-resolution video
Method	#Frames	Setting	Tokens	VideoMME (%)				MLVU	LVBench	Avg.
Method	(Decoded)	( $N$ F ${@}r$ )	(Budget)	Short	Med.	Long	All	(%)	(%)	(%)
Qwen3-VL-4B	16	$16$ F ${@}1.0$	5,760	68.44	55.67	49.22	57.78	58.89	38.61	51.76
Low-Res-Base	256	$256$ F ${@}0.25$	5,760	75.78	62.33	55.22	64.44	69.36	43.71	59.17
Pre-encoder keyframe selection
CLIP-TopK	256 $\rightarrow$ 16	$16$ F ${@}1.0$	5,760	65.67	56.33	49.56	57.19	69.36	44.74	57.10

Table 3: Performance Comparison on Qwen3-VL-4B. Avg. is the mean over three benchmarks.

Table 7

$25\%$ default budget ( $\sim$ 1,440 tokens)
Method	Setting	VideoMME	LVBench	MLVU	Avg.
Qwen3-VL-4B	16F $@1.0$	57.78	38.61	58.89	51.76
VisionZip [ 38 ]	16F $@1.0$	55.56	34.47	53.75	47.93
MMTok [ 6 ]	16F $@1.0$	55.19	34.80	54.82	48.27
FlashVid [ 8 ]	16F $@1.0$	55.11	35.44	54.54	48.36
Resize	16F $@0.5$	56.26	35.51	57.95	49.91

Table 6: Token pruning vs Resize.

Table 9

Figure 4: Visualization of selected high-resolution images by our proposed Hi-I Selector.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Selector	$K$	Short	Medium	Long	All
LoHi-SemDiv	2	75.67	65.78	55.33	65.59
LoHi-SemDiv-Anchor	2	75.67	65.67	55.22	65.52
LoHi-SemDiv	4	76.22	65.22	57.22	66.22
LoHi-SemDiv-Anchor	4	76.22	66.00	57.00	66.41
LoHi-SemDiv	8	77.11	65.78	57.56	66.81
LoHi-SemDiv-Anchor	8	77.11	66.00	57.56	66.89

Appendix

Table 9: LoHi-SemDiv-Anchor on VideoMME (Qwen3-VL-4B,

128

F Lo-V, interleaved cue).

	VideoMME
Variant	Short	Medium	Long	All	MLVU	LVBench
LoHi-SemDiv (reference)	77.11	65.78	57.56	66.81	74.38	45.90
LoHi-SemDiv++ (frame-PCA)	77.22	66.22	57.33	66.93	75.02	45.77
$\Delta$	$+$ 0.11	$+$ 0.44	$-$ 0.23	$+$ 0.12	$\mathbf{+0.64}$	$-$ 0.13

Appendix

Table 10: LoHi-SemDiv++ on Qwen3-VL-4B (

128

@0.25

K{=}8

Frames	$N_{\text{CPU}}$	CPU mean (s)	$N_{\text{GPU}}$	GPU mean (s)	GPU / CPU
16	900	0.60	900	1.04	$1.73\times$
64	900	1.47	$45^{\dagger}$	$3.38^{\dagger}$	-
128	900	2.45	—	—	—
256	900	3.94	—	—	—

Appendix

Table 11: Decode wall time on VideoMME with uniform sampling. CPU decord is profiled on the full 900-clip set at every frame count; NVDEC is profiled at

N{=}16

on the same set and at

N{=}64

on a 45-clip subsample (

\dagger

Method	#Frames	Setting	Tokens	VideoMME (%)				MLVU	LVBench	Avg.
Method	(Decoded)	$N$ F ${@}r$	(Budget)	Short	Med.	Long	All	(%)	(%)	(%)
Qwen3-VL-8B
Vanilla	16	$16$ F ${@}1.0$	5,760	69.89	57.56	52.67	60.04	58.95	38.22	52.40
CLIP-TopK	256 $\rightarrow$ 16	$16$ F ${@}1.0$	5,760	68.44	58.44	53.33	60.07	69.73	44.09	57.97
AKS [ 29 ]	256 $\rightarrow$ 16	$16$ F ${@}1.0$	5,760	68.56	58.22	53.33	60.04	69.88	44.09	58.00
BOLT [ 17 ]	256 $\rightarrow$ 16	$16$ F ${@}1.0$	5,760	72.89	59.67	54.00	62.19	69.34	43.83	58.45

Appendix

Table 12: Cross-backbone transfer of LoHi . All LoHi rows use a

128

F Lo-V at

r_{\ell}{=}0.25

paired with

K{=}4

Hi-I frames at

r_{h}{=}1.0

. Avg. is the mean over VideoMME-All, MLVU, and LVBench.

Setting	VideoMME	MLVU	LVBench
Qwen3.5-4B
$16$ F $@1.0$	60.74	60.48	38.54
$256$ F $@0.25$	67.41	69.87	41.45
$\Delta$	$+6.67$	$+9.39$	$+2.91$

Appendix

Table 13: Frame–resolution trade-off on Qwen3.5-4B at the same token budget.

Configuration	VideoMME	MLVU	LVBench
Qwen3.5-4B
$128$ F $@0.25$	64.93	66.80	39.51
$256$ F $@0.25$	67.41	69.87	41.45
LoHi-Uniform	67.52	67.48	44.61
LoHi-Anchor	67.70	66.70	44.74
LoHi-SemDiv	68.15	70.42	46.68

Appendix

Table 14: LoHi selectors on Qwen3.5-4B (

128

F Lo-V at

r_{\ell}{=}0.25

K{=}8

Hi-I).

Method	VideoMME	MLVU	LVBench
Qwen2.5-VL-7B
Vanilla [email protected]	58.78	58.00	37.83
Low-Res-Base [email protected]	63.78	64.77	41.06
LoHi-Uniform $K{=}8$	63.26	63.80	40.35
LoHi-SemDiv $K{=}8$	64.41	68.03	43.51
VideoChat3-4B

Appendix

Table 15: Generalization to additional VLM backbones. Both LoHi variants use 128 Lo-V frames at

r_{\ell}=0.25

and Hi-I frames at

r_{h}=1.0

\dagger

For VideoLLaMA3, MLVU scores use the output normalization described in Sec. C.7 .

Configuration	Lo-V	Hi-I	Short	Medium	Long	All
Qwen3-VL-4B
$256$ F $@0.25$	$256$ F	—	75.78	62.33	55.22	64.44
LoHi-Uniform	$128$ F	$K{=}8$	75.67	66.44	55.22	65.78
LoHi-SemDiv	$128$ F	$K{=}8$	77.11	65.78	57.56	66.81
LoHi-Uniform	$256$ F	$K{=}4$	77.67	65.33	56.78	66.59
LoHi-Anchor	$256$ F	$K{=}4$	77.00	65.89	57.78	66.89

Appendix

Table 16: Pushing the Lo-V to

N{=}256

on VideoMME (Qwen3-VL-4B). All rows use

r_{\ell}{=}0.25

for Lo-V and

r_{h}{=}1.0

for Hi-I.

Lo-V $+$ Hi-I	Visual tokens	Acc. (%)
Qwen3.5-27B
[email protected], $K=0$	25,088	82.14
[email protected], $K=8$	30,848	82.86
[email protected], $K=16$	36,608	83.71
[email protected], $K=32$	48,128	85.00
[email protected], $K=64$	71,168	85.86

Appendix

Table 17: Evaluation under larger visual-token budgets on EgoLongQA with Qwen3.5-27B. All configurations use 512 sampled frames.

Trigger	2nd-pass rate	Acc. (%)	$\Delta$ vs. Lo
Lo-V only	$0\%$	63.00	0.00
Entropy ( $\theta$ tuned)	$37\%$	65.71	$+2.71$
Always-Hi	$100\%$	65.78	$+2.78$

Appendix

Table 18: Two-pass triggers on VideoMME (Qwen3-VL-4B,

128

F Lo-V).

Configuration	Raw (%)	Normalized (%)	No-paren rate
$16$ F $@1.0$	35.88	52.38	$38.5\%$
$64$ F $@0.5$	38.97	55.27	$30.9\%$
$256$ F $@0.25$	40.30	57.73	$30.8\%$
$\Delta$ ( $256$ F $-\,16$ F)	$+4.42$	$+5.35$	—

Appendix

Table 19: VideoLLaMA3-7B on the MLVU dev split, with and without output normalization. “No-paren rate” is the fraction of outputs lacking a closing parenthesis.

Benchmark	Mean (range)	Std (max)	Min (per video)
LVBench	$0.990$ – $0.998$	$0.032$	$0.806$
MLVU	$0.977$ – $0.999$	$0.069$	$0.639$
VideoMME	$0.959$ – $0.991$	$0.064$	$0.674$

Appendix

Table 20: Keep ratio of DiffFP in VideoLLaMA3-7B,

45

configurations,

60

videos each. The lowest mean is

0.959

(VideoMME,

256

@1.0

Selector	$K{=}4$ (%)		$K{=}8$ (%)
Selector	Appended	Interleaved	Appended	Interleaved
LoHi-Uniform	64.78	65.52 $(+0.74)$	65.30	65.78 $(+0.48)$
LoHi-SemDiv	65.70	66.22 $(+0.52)$	66.37	66.81 $(+0.44)$
LoHi-Anchor	65.96	65.85 $(-0.11)$	65.85	65.89 $(+0.04)$

Appendix

Table 21: Prompt template ablation on VideoMME (Qwen3-VL-4B,

128

F Lo-V at

r{=}0.25

SemDiv $-$ Uniform	$\Delta$ (%)	95% CI	McNemar $p$	W/L
VideoMME	$+1.04$	$[-0.15,+2.22]$	$0.094$	144/116
MLVU	$\mathbf{+6.76}$	$[+5.17,+8.41]$	$\mathbf{<10^{-16}}$	233/86
LVBench	$+1.81$	$[-0.13,+3.76]$	$0.063$	120/92

Appendix

Table 22: Paired comparison of LoHi-SemDiv and LoHi-Uniform. Confidence intervals are computed using video-level paired bootstrap. W/L denotes questions answered correctly only by SemDiv/Uniform, respectively.

Query-aware $-$ uniform selection (%)	VideoMME	MLVU	LVBench
CLIP-TopK $-$ Vanilla ([email protected])	$-0.6$	$+10.5$	$+6.1$
AKS $-$ Vanilla ([email protected])	$-0.5$	$+10.3$	$+6.1$
LoHi-SemDiv $-$ LoHi-Uniform	$+1.0$	$+6.8$	$+1.8$

Appendix

Table 23: Accuracy gains of query-aware selection over the corresponding uniform baselines at matched budgets.

MLVU action-count	Lo-V only	LoHi-Uniform	LoHi-SemDiv
Accuracy (%)	41.75	29.13	53.40

Appendix

Table 24: Effect of Hi-I selection on MLVU action counting with Qwen3-VL-4B (

n=206

Query type	$n$	Base	Uniform	SemDiv
Static location ( where was X )	111	68.47	70.27	73.87
Transient action ( put in )	48	64.58	66.67	83.33
Action destination	45	71.11	75.56	77.78
Yes/no ( did I … )	43	58.14	48.84	65.12
Static entity count	27	18.52	37.04	37.04

Appendix

Table 25: Accuracy (%) by question template within MLVU’s ego category, using Qwen3-VL-4B. Base denotes Low-Res-Base; Uniform and SemDiv use

K=8

Hi-I frames.

Qwen3-VL-4B	Subtitled (2,232 Qs)	No subtitle (468 Qs)
[email protected]	56.18	65.38
[email protected]	63.13	70.73
Text-only	61.56	–
Question-only	–	30.56

Appendix

Table 26: Accuracy (%) of visual-only and text-only inputs on VideoMME using Qwen3-VL-4B. Subsets are defined by subtitle availability. Visual inputs exclude subtitles; text-only inputs contain the transcript and question.

Subtitle length	Tokens	% of visual budget
Median	1,539	27%
Mean	3,018	52%
P90	7,799	135%

Appendix

Table 27: VideoMME subtitle lengths measured with the Qwen3-VL tokenizer, relative to the visual-token budget of 5,760 tokens.

Subtitled subset	w/o subs	w/ subs	Subtitle gain (%)
Low-Res-Base [email protected]	63.13	73.12	$+9.99$
LoHi-SemDiv $K=8$	65.28	72.13	$+6.85$
Gap (LoHi $-$ Base)	$+2.15$	$-0.99$	–
$p$ -value	0.011	0.117	–

Appendix

Table 28: Accuracy (%) with and without subtitles on the 2,232 subtitle-available questions in VideoMME, using Qwen3-VL-4B. Reported

p

-values compare LoHi-SemDiv with Low-Res-Base in each setting.

Explore similar work

LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs
May 17, 2026Jihwan Kim, Nikhil Parthasarathy, Danfeng Qin +5Efficient VLM Inference Video-Language Models
1Google DeepMind · 2Seoul National University
One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
Apr 15, 2026Zheyu Zhang, Ziqi Pang, Shixing Chen +3Long-Video Understanding Video Token Compression
University of Illinois Urbana-Champaign · Amazon Prime Video
Small Vision-Language Models are Smart Compressors for Long Video Understanding
Apr 9, 2026Junjie Fei, Jun Chen, Zechun Liu +13Long-Video Understanding Video Token Compression
Meta AI · King Abdullah University of Science and Technology (KAUST)

May 17, 2026J/K move · Enter open · S save

LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs

Jihwan Kim, Nikhil Parthasarathy, Danfeng Qin, Junhwa Hur+4

1Google DeepMind · 2Seoul National University

Efficient VLM Inference Video-Language Models Long-Video Understanding Knowledge Distillation Visual Token Compression

Open paper AskPDF arXiv

The fundamental challenge in scaling Video Large Language Models (Video LLMs) to long-form video lies in managing the explosion of visual-token context length. Existing strategies predominantly focus on "post-hoc" token reduction -- reducing visual tokens after feature extraction to alleviate the LLM's computational overhead. While these methods effectively reduce the number of visual tokens, we observe that the primary latency bottleneck then shifts from the LLM to the expensive per-frame processing of the vision encoder. To address this, we introduce LiteFrame, a strong, yet highly efficient video encoder backbone for Video LLMs. To train LiteFrame, we propose Compressed Token Distillation (CTD), a novel training framework that teaches a compact student vision encoder to directly predict information-dense, spatio-temporally compressed representations produced by a large teacher vision model, effectively bypassing redundant computation. When coupled with further Language Model Adaptation (LMA), this approach results in a new latency-accuracy Pareto frontier -- compared with InternVL3-8B, LiteFrame provides a 35% reduction in end-to-end latency while processing 8

\times

more frames and improves average video understanding accuracy across multiple benchmarks. Our results demonstrate a new potential path to unlocking longer-form video understanding under fixed compute budgets.

Abstract

Explore similar work

LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs

One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

Small Vision-Language Models are Smart Compressors for Long Video Understanding