Large multimodal models (LMMs) have recently demonstrated remarkable performance in video question answering (VideoQA), yet reasoning over video remains challenging due to high inference cost and diluted information. Keyframe selection offers efficiency and sharper reasoning but suffers from sparse supervision and redundant frame choices when relying only on image-text similarity. We present a question-aware keyframe selection framework with two components: pseudo keyframe labels derived from LMMs that provide informative supervision and a coverage regularization that promotes diverse, complementary evidence across time. Experiments on NExT-QA show that our method significantly improves accuracy, especially for temporal and causal question types, establishing keyframe selection as an effective and learnable module for VideoQA.
Figures & tables
Figure 1: An overview of our proposed question-aware keyframe selection framework. The model processes the video with an image encoder and the question with a text encoder. The question embedding is then used in two parallel streams: (1) it is fed into a Gaussian Generator to create an initial keyframe distribution, and (2) it is used to generate synthetic keyframes via CLIP similarity and VLM ranking, which serve as weak supervision. To adapt the selection process, the LMM’s own backbone is re-purposed with a prompt to determine the question type. This information guides the Question-Conditioned Coverage Regularization (QCCR) module, which refines the initial distribution. Finally, the predicted keyframes are passed to the downstream LMM to generate the final answer.
NExT-QA
Descriptive
Temporal
Causal
Average
Co-Mem
54.4
50.0
45.9
48.5
HCRN
54.0
49.3
47.1
48.9
HGA
57.8
49.1
48.1
50.0
IGV
59.6
51.7
48.6
51.3
HQGA
59.4
52.3
49.0
51.8
Table 1: Performance comparison with state-of-the-art methods on the NEXT-QA benchmark. All methods are evaluated on accuracy across three question categories. Our method achieves the highest average accuracy, demonstrating its overall effectiveness.
Figure 2: A qualitative example illustrating the effectiveness of VLM Ranking over CLIP similarity. For the question “What does the child do after ending the call?”, frames with high CLIP scores are often redundant while the VLM Ranking correctly identifies the temporally relevant frames that occur after the call has ended.
NExT-QA
Descriptive
Temporal
Causal
Average
Ours
77.8
71.5
71.6
73.3
w/o QCCR
75.9
67.9
70.9
71.5
w/o SKS
69.0
74.4
67.8
70.4
Table 2: Ablation study of our core components on the NEXT-QA dataset. The performance drop in both cases confirms that each component makes a crucial contribution to the final performance.
Recent multimodal large language models (MLLMs) have substantially advanced video understanding, yet long-form video QA remains challenging under fixed input token budgets, where uniform sampling can be inefficient for evidence localization. We propose ReQuest , an uncertainty-driven, question-adaptive keyframe selection pipeline that aligns question intent with relevant video content through selective computation. ReQuest integrates (i) a lightweight question-aware selector distilled from MLLM-generated supervision, (ii) Re-thinking Routing that triggers additional inference only when the model is uncertain with a length-adaptive criterion, and (iii) uncertainty-guided adaptive non-maximum suppression that selects temporally diverse frames while adjusting spacing based on question difficulty. As a plug-andplay method, ReQuest improves long-video QA without modifying or fine-tuning the underlying MLLM. Experiments on Video-MME, MLVU, and LongVideoBench demonstrate consistent accuracy gains with competitive computational cost, with particularly strong improvements in medium and long video regimes.
Minkuk Kim, Suyong Yun, Young Tae Kim +3
Kyung Hee University, Republic of Korea · Electronics and Telecommunications Research Institute (ETRI), Republic of Korea
Multimodal large language models (MLLMs) process multimodal inputs by converting text, images, and videos into token sequences that are subsequently processed by a backbone language model. While MLLMs have achieved excellent performance in understanding the content of individual images, video understanding remains significantly more difficult because videos contain large number of video frames. MLLMs typically process only a subset of these frames, usually ranging from 8 to 64. MLLMs usually sample frames uniformly, regardless of their relevance to the question being answered. To address this limitation, several training-free, model-agnostic methods for selecting question-relevant frames have recently been proposed. In this work, we introduce a frame-selection method designed specifically for multiple-choice questions. We extend the query text by appending semantic cues derived from the answer choices and employ a direct query-frame alignment scoring mechanism. To the best of our knowledge, our method is the first to directly utilize answer choices as inference-time cues for selecting frames relevant to answering a question. The method first constructs a compact candidate pool by subsampling video frames at a fixed rate. The frames are then scored according to their maximum cosine similarity across all question-answer pairs to identify the most relevant frames for a given query. This approach preserves a fixed token budget while improving the relevance of the visual evidence provided to the downstream MLLM. We evaluate the effectiveness of our frame-selection method on the MLVU, Video-MME, and LongVideoBench benchmarks using three MLLMs: LLaVA-Mini, Qwen2-VL, and LLaVA-Video. Experimental results demonstrate that answer-aware frame selection generally outperforms uniform sampling and existing training-free frame-selection methods under the same frame budget.
Md. Safayet Islam, Dilip Sarkar, Liang Liang
Department of Computer Science University of Miami Coral Gables, FL 33146
Query-aware keyframe selection enables multimodal large language models (MLLMs) to process long videos using only a small set of question-relevant frames. Existing score-based methods, however, typically search within a fixed, uniformly sampled candidate pool, preventing evidence outside this pool from ever being selected. Given a limited relevance-scoring budget, the key challenge is to allocate evaluations adaptively to promising frames while continuing to explore underrepresented temporal regions. We introduce FORTE, a training-free framework that addresses this challenge through two stages: adaptive relevance scoring and global keyframe optimization. Starting from sparse, uniformly distributed observations, our efficient Gaussian-process relevance predictor estimates relevance for unscored frames, exploiting temporal locality and the approximately banded kernel structure to reduce the core computation from cubic to linear time in the number of frames for fixed bandwidth. The scoring stage then selects which frames to score next by balancing predicted relevance with temporal coverage, prioritizing promising regions while also exploring less-represented parts of the video. The optimization stage selects the final keyframes by maximizing an objective that jointly captures measured relevance and temporal coverage. We derive an exact algorithm that leverages the logarithmic coverage structure to identify the optimal subset of the scored candidate pool in time linear in the pool size, for a fixed final-frame budget. Experiments on four long-video question-answering benchmarks show that FORTE achieves the highest observed mean accuracy among the compared selectors under every tested scoring budget. Further evaluations demonstrate its consistent effectiveness across different relevance scorers and downstream MLLMs.