FrameScout: Scouting Query-Relevant Frames for Long Video Understanding

Haonan Hu; Shuhao Chen; Weisen Jiang; Lizhao Gao; James Kwok; Yu Zhang

cs.CVOct 13, 2026

FrameScout: Scouting Query-Relevant Frames for Long Video Understanding

Authors: Haonan Hu, Shuhao Chen, Weisen Jiang, Lizhao Gao, James Kwok, Yu Zhang

Organizations: Southern University of Science and Technology · The Hong Kong University of Science and Technology · South China University of Technology · Independent Researcher

Abstract

Long video understanding requires VideoLLMs to answer queries about videos spanning tens of minutes to hours, but such long videos produce far more visual tokens than current VideoLLMs can process. To address this issue, keyframe selectors are proposed to select the most query-relevant frames by computing the similarity between frame and query embeddings. Existing selectors either encode frames independently without temporal context or process chunks in isolation with ranking objectives that are difficult to optimize. To alleviate those limitations, we propose FrameScout with two core designs: (i) a streaming selector architecture that leverages a sliding-window KV cache and successor aggregation to produce temporally-contextualized frame embeddings; (ii) a frame-query contrastive objective that aligns frame and query embeddings in a shared space, directly training the selector to distinguish relevant frames from irrelevant ones. Extensive experiments on three long-video benchmarks and six base VideoLLMs demonstrate that FrameScout consistently achieves state-of-the-art performance. Code is available at https://github.com/Hubert-1102/FrameScout.

Figures & tables

Figure 2: Illustration of the sliding-window KV Cache and successor aggregation mechanisms.

\rectanglecolor Gray6-36-7 \rectanglecolor Gray10-310-7 \rectanglecolor Gray14-314-7 \rectanglecolor Gray18-318-7 \rectanglecolor Gray22-322-7 \rectanglecolor Gray26-326-7	VideoLLM	Method	VideoMME		MLVU	LongVB
	VideoLLM	Method	Long	Overall	M-Avg	Val
Closed-Source	GPT-5.4-mini [ 43 ]	Uniform	46.9	56.1	57.8	52.9
		FOCUS [ 75 ]	42.1 (-4.8)	53.2 (-2.9)	62.3 (+4.5)	54.2 (+1.3)
		FlexSelect [ 73 ]	50.4 (+3.5)	59.3 (+3.2)	61.2 (+3.4)	58.8 (+5.9)
		FrameScout	52.3 (+5.4)	60.8 (+4.7)	62.7 (+4.9)	58.9 (+6.0)
	GPT-5.4 [ 43 ]	Uniform	60.9	66.9	68.5	63.6

Table 1: Accuracy (%) on long-video understanding benchmarks. Relative improvements over uniform sampling are shown in parentheses.

Table 5

Figure 5: Distribution of similarities between two strategies.

Training context	Training loss	VideoMME	LongVB
Streaming KV context	0.3731	64.2	62.1
Independent chunks (ours)	0.2057	65.6	63.3

Table 4: Comparison of training context strategies under the same wall-clock budget using Qwen2.5-VL-7B.

Figure 6: t-SNE visualization of query and frame embeddings.

Figure 7: Qualitative comparison with FlexSelect on a temporal reasoning query.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Chunk size $B$	Long	Overall
8	60.3	68.8
12	61.2	69.3
16	61.1	68.9

Appendix

Table 5: Sensitivity to chunk size

B

on VideoMME using Qwen3-VL-8B.

$\beta$	Long	Overall
0.0	55.5	65.3
0.2	55.1	65.5
0.3	56.1	65.7
0.5	54.6	64.8
0.8	51.4	63.2

Appendix

Table 6: Sensitivity to discount factor

\beta

on VideoMME using LLaVA-Video-7B.

$\mathcal{M}$	Method	VideoMME	LongVB
Qwen2.5-VL	F2C [ 53 ]	62.3	60.6
	BOLT [ 38 ]	63.5	61.6
	HICom [ 40 ]	57.8	–
	VTM [ 30 ]	61.1	–
	FrameScout	65.6	63.3
Qwen2	M-LLM [ 22 ]	58.7	57.0

Appendix

Table 7: Comparison with recent training-free selectors (F2C, BOLT), token compression methods (HICom, VTM), and the training-based selector M-LLM. VideoMME reports Overall accuracy. “–”: not evaluated.

Variant	VideoMME	LongVB
w/o sliding-window KV cache (Separate Windows)	65.0	61.3
w/o successor aggregation	65.3	62.7
w/o contrastive objective (Spearman)	61.1	60.7
Full FrameScout	65.6	63.3

Appendix

Table 8: Effect of components in FrameScout using Qwen2.5-VL-7B.

	Overall				Long
Method	$K{=}4$	$8$	$16$	$60$	$K{=}4$	$8$	$16$	$60$
Uniform	49.2	53.9	57.3	62.6	44.4	47.8	48.3	51.4
FOCUS	54.4	57.0	57.7	63.4	47.4	47.3	49.0	54.3
FlexSelect	52.7	55.0	59.0	64.9	47.6	48.3	51.8	55.4
FrameScout	55.1	57.4	60.3	65.6	48.1	51.4	52.7	57.3

Appendix

Table 9: Accuracy under different frame budgets

K

on VideoMME using Qwen2.5-VL-7B.

	Uniform	FOCUS	FlexSelect	FrameScout
Accuracy	64.9	61.4	65.3	66.2

Appendix

Table 10: Accuracy (%) under a 1 fps frame budget on the VideoMME subset with videos shorter than 10 minutes, using Qwen2.5-VL-7B.

Top- $K$	10	20	30	40
GT span recall	73.6	86.0	91.7	99.2

Appendix

Table 11: Recall (%) of human-annotated GT temporal spans on NExT-GQA by the top-

K

teacher-attention pseudo-labeled frames.

Projector	VideoMME Overall	VideoMME Long	LongVB
Identity	61.7	56.4	60.3
Linear	63.0	56.0	59.9
Two-layer MLP (ours)	65.6	57.3	63.3

Appendix

Table 12: Ablation on the projection head

\phi

using Qwen2.5-VL-7B.

Future aggregation	VideoMME Overall	LongVB
Backward temporal smoothing	65.0	61.3
Non-causal temporal aggregation	65.2	62.4
Successor aggregation (ours)	65.6	63.3

Appendix

Table 13: Comparison of future context aggregation strategies using Qwen2.5-VL-7B.

Method	VideoMME Long	VideoMME Overall	LongVB
FlexSelect [ 73 ]	55.4	64.9	63.0
FrameScout (teacher: InternVL2.5-8B)	57.3	65.6	63.3
FrameScout (teacher: Qwen2.5-VL-7B)	57.0	65.2	63.1

Appendix

Table 14: Sensitivity to the teacher model used for pseudo-label generation using Qwen2.5-VL-7B.

Selection rule	Accuracy	Avg. #Frames
Fixed $K$	61.7	30
Adaptive ( $\operatorname{sim}>\delta$ )	62.0	26.5

Appendix

Table 15: Fixed versus threshold-based adaptive frame selection on LongVB using Qwen2.5-VL-7B.

Prompt	Method	$K{=}8$	$K{=}16$	$K{=}32$	$K{=}60$
Ours	BOLT [ 38 ]	55.63	59.44	62.07	63.52
Ours	FrameScout	57.44	60.30	63.11	65.59
BOLT official	BOLT [ 38 ]	57.44	61.07	64.19	64.67
BOLT official	FrameScout	59.07	61.26	64.67	66.78

Appendix

Table 16: BOLT and FrameScout under our evaluation prompt and BOLT’s official prompt on VideoMME Overall using Qwen2.5-VL-7B.

Figure 8: Comparison with the frame selection results of the FOCUS [ 75 ] .

Figure 9: Comparison with the frame selection results of the FlexSelect [ 73 ] .

Query-Aware Video Frame Selection

Query-aware video frame selection finds the frames most useful for answering a question while limiting visual tokens and computation.

See all 70 papers

QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding
Jul 1, 2026Jun Peng, Baiyang Song, Jie Li +4Long-Video Understanding Video Frame Selection
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, P.R. China · Peng Cheng Laboratory, Shenzhen, P.R. China · School of Electronic and Computer Engineering, Peking University, P.R. China
Query-aligned video frame selection for long video understanding
Sep 15, 2026Md. Safayet Islam, Dilip Sarkar, Liang LiangQuery-Aware Video Frame Selection Long-Video QA
Department of Computer Science University of Miami Coral Gables, FL 33146
RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding
Aug 30, 2026Shanqing Xu, Meng Luo, Mengchen Qian +7Large Vision-Language Models Long-Video Understanding
Huazhong University of Science and Technology · National University of Singapore · Fudan University

Jul 1, 2026J/K move · Enter open · S save

QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding

Jun Peng, Baiyang Song, Jie Li, Hui Li+3

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, P.R. China · Peng Cheng Laboratory, Shenzhen, P.R. China · School of Electronic and Computer Engineering, Peking University, P.R. China

Long-Video Understanding Video Frame Selection Long-Video QA Query-Aware Video Frame Selection

Open paper AskPDF arXiv

Video understanding is often plagued by severe temporal redundancy, where processing dense frame sequences is both semantically inefficient and computationally expensive. This challenge is further amplified when only a small subset of frames is truly relevant to the given query. In this paper, we propose a Query- and Content-Aware (QCA) keyframe selection framework that can select a compact yet information-rich set of frames from long videos. QCA first partitions the video into temporal segments and estimates the information contribution of each segment by jointly modeling query relevance and content deviation, and dynamically allocates keyframe budget to each segment. Within each segment, QCA anchors on the most query-relevant frame and iteratively incorporates additional frames to maximize diversity while maintaining high semantic relevance to the query. Crucially, our method requires no additional training and can be seamlessly integrated into existing Video-LLMs. Extensive experiments across multiple long video understanding benchmarks demonstrate that our proposed approach achieves state-of-the-art performance and has strong generalization ability. For instance, QCA achieves 67.8% on LongVideoBench using 128 frames, while GPT-4o achieves 66.7% using 256 frames. Our codes are available in GitHub.

Abstract

Query-Aware Video Frame Selection

Explore similar work

QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding

Query-aligned video frame selection for long video understanding

RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding