Keyframe selection (KFS) has long produced compact video summaries for browsing and retrieval, and representative frames for thumbnails. More recently, when conditioned on a question, KFS provides an alternative to uniform sampling for long-video question answering by selecting frames that are more relevant to the question. Most methods rank frames by similarity to the question. Yet a relevant frame may score poorly when the question combines subjects or moments that no single frame shows, or requires implicit information absent from its wording. Other methods try to break down the question into subqueries, but suffer from inaccurate decomposition due to their static initial context. Therefore, we propose CueKFS, a training-free method that reformulates question--frame matching as comparing frames against a set of dynamically generated visual cues. From an initial set of salient frames, we decompose the question into cues. Each cue concurrently probes the video to navigate to its own evidence. A reasoning VLM then agentically revises the cue set against its evidence to re-explore the video. CueKFS then allocates the budget across the surviving cues. Across three benchmarks, CueKFS establishes state-of-the-art results in all 27 evaluated settings with available prior results, achieving budget-averaged gains of up to +4.54% over the previous baseline and a median of only two VLM calls. We further provide a detailed behavioral analysis of CueKFS, showing that agentic cue refinement drives active re-exploration of the video, yielding relative similarity gains of up to 92% over the initial context.
Figures & tables
Figure 1: Overview of CueKFS . Cue Decomposition builds a clip-wise video overview and applies three rules to produce initial visual cues. Cue-Wave-Guided Refinement computes one wave per cue, samples evidence from plateaus and peaks, refines cues in batches with a reasoning VLM, and re-explores the video. Cue Budget Allocation ranks cues by peak similarity, removes similar nearby frames in favor of higher-ranked cues, and allocates the budget by remaining evidence-frame counts. The displayed cue set and frame counts are illustrative.
Table 1: Accuracy (%) on three long-video QA benchmarks at three frame budgets with three downstream VLMs. Bold denotes the best result in each block. For Qwen2.5-VL and InternVL3, baseline results at k=32 are from Chen et al. (2026) .
Table 2: Step-by-step ablations on Video-MME with GPT-5.5 as the downstream VLM.
Hyperparameter
Values evaluated
Avg
Reasoning VLM
Qwen3.6-27B / GPT-5.5
81.04 / 81.52
Overview frames Nrepr
16 / 24 / 32 / 48 / 64
80.50 / 81.52 / 81.32 / 82.35 / 82.35
Cues per batch B
3 / 5 / 10 / 20 / all
82.06 / 82.35 / 82.30 / 82.47 / 82.42
Review image budget M
16 / 24 / 32 / 48 / 64
82.42 / 82.47 / 82.55 / 82.20 / 82.13
VLM frame edge (px)
448 / 768 / 1024 / 1536
82.55 / 81.75 / 81.95 / 82.38
Wave smoothing window h
1 / 3 / 5
82.29 / 82.55 / 81.29
Table 3: Hyperparameter ablations on Video-MME. Values and their corresponding average accuracies are listed in the same order. Bold marks the configuration used in Table 1 and its accuracy.
Table 4: Refinement behavior on Video-MME. Panel (a) reports action frequencies by video length and overall. Counts are averaged over questions where the action occurs. Panel (b) reports refinement gains for overlapping question groups defined by whether each action occurred.
Table 5: SigLIP2 score on Video-MME. For each cue set, we average per-cue maximum scores within each question and report the median across questions. The Question row reports the median maximum score for the question text. Candidate frames are 1-fps video frames (“Full video”) or the overview frames from Cue Decomposition (“Overview”). Final cues comprise kept initial cues, rewritten cues, and split cues. Since Overview is a subset of Full video, the relative gain Full / Overview -1 is nonnegative. Larger gains indicate stronger matches beyond the overview.
Table 6: Cue coverage on Video-MME. A frame is covered if any cue has a SigLIP2 score of at least 0.5 . Panel (a) reports median Jaccard overlap before and after refinement. Panel (b) reports QA accuracy (%) with GPT-5.5 as the downstream VLM, given frames selected by CueKFS , sampled outside CueKFS ’s final cue coverage, or sampled uniformly. Δ is the overall accuracy difference from CueKFS .
Figure 3: Qualitative illustration of CueKFS : successful refinement (top) and failure (bottom). Correct options are green. We display selected overview frames and up to two evidence frames per cue per stage. The top panel is a curated mechanism illustration using real video frames; the bottom pairs recorded cues and actions with illustrative evidence frames.
Video understanding is often plagued by severe temporal redundancy, where processing dense frame sequences is both semantically inefficient and computationally expensive. This challenge is further amplified when only a small subset of frames is truly relevant to the given query. In this paper, we propose a Query- and Content-Aware (QCA) keyframe selection framework that can select a compact yet information-rich set of frames from long videos. QCA first partitions the video into temporal segments and estimates the information contribution of each segment by jointly modeling query relevance and content deviation, and dynamically allocates keyframe budget to each segment. Within each segment, QCA anchors on the most query-relevant frame and iteratively incorporates additional frames to maximize diversity while maintaining high semantic relevance to the query. Crucially, our method requires no additional training and can be seamlessly integrated into existing Video-LLMs. Extensive experiments across multiple long video understanding benchmarks demonstrate that our proposed approach achieves state-of-the-art performance and has strong generalization ability. For instance, QCA achieves 67.8% on LongVideoBench using 128 frames, while GPT-4o achieves 66.7% using 256 frames. Our codes are available in GitHub.
Jun Peng, Baiyang Song, Jie Li +4
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, P.R. China · Peng Cheng Laboratory, Shenzhen, P.R. China · School of Electronic and Computer Engineering, Peking University, P.R. China
Query-aware keyframe selection enables multimodal large language models (MLLMs) to process long videos using only a small set of question-relevant frames. Existing score-based methods, however, typically search within a fixed, uniformly sampled candidate pool, preventing evidence outside this pool from ever being selected. Given a limited relevance-scoring budget, the key challenge is to allocate evaluations adaptively to promising frames while continuing to explore underrepresented temporal regions. We introduce FORTE, a training-free framework that addresses this challenge through two stages: adaptive relevance scoring and global keyframe optimization. Starting from sparse, uniformly distributed observations, our efficient Gaussian-process relevance predictor estimates relevance for unscored frames, exploiting temporal locality and the approximately banded kernel structure to reduce the core computation from cubic to linear time in the number of frames for fixed bandwidth. The scoring stage then selects which frames to score next by balancing predicted relevance with temporal coverage, prioritizing promising regions while also exploring less-represented parts of the video. The optimization stage selects the final keyframes by maximizing an objective that jointly captures measured relevance and temporal coverage. We derive an exact algorithm that leverages the logarithmic coverage structure to identify the optimal subset of the scored candidate pool in time linear in the pool size, for a fixed final-frame budget. Experiments on four long-video question-answering benchmarks show that FORTE achieves the highest observed mean accuracy among the compared selectors under every tested scoring budget. Further evaluations demonstrate its consistent effectiveness across different relevance scorers and downstream MLLMs.
Recent multimodal large language models (MLLMs) have substantially advanced video understanding, yet long-form video QA remains challenging under fixed input token budgets, where uniform sampling can be inefficient for evidence localization. We propose ReQuest , an uncertainty-driven, question-adaptive keyframe selection pipeline that aligns question intent with relevant video content through selective computation. ReQuest integrates (i) a lightweight question-aware selector distilled from MLLM-generated supervision, (ii) Re-thinking Routing that triggers additional inference only when the model is uncertain with a length-adaptive criterion, and (iii) uncertainty-guided adaptive non-maximum suppression that selects temporally diverse frames while adjusting spacing based on question difficulty. As a plug-andplay method, ReQuest improves long-video QA without modifying or fine-tuning the underlying MLLM. Experiments on Video-MME, MLVU, and LongVideoBench demonstrate consistent accuracy gains with competitive computational cost, with particularly strong improvements in medium and long video regimes.
Minkuk Kim, Suyong Yun, Young Tae Kim +3
Kyung Hee University, Republic of Korea · Electronics and Telecommunications Research Institute (ETRI), Republic of Korea