cs.CVNov 7, 2026

Reason First, Retrieve What Matters: Hypothesis-Guided Keyframe Selection for Long Video Understanding

Authors: Yisong Hu, Junbo Wang, Yuke Li, Jiangbin Zheng

Organizations: All authors are with the School of Software, Northwestern Polytechnical University, Xi’an 710072, China

Abstract

Long video understanding requires models to identify and reason over sparse, temporally distributed evidence within limited frame budgets. Existing keyframe selection methods typically retrieve query-relevant frames from a densely sampled candidate set before inference. However, global image-text matching incurs unnecessary computation when a small set of uniformly sampled frames already provides sufficient evidence. Query relevance alone may also overlook evidence that distinguishes competing answers, while one-shot selection offers no opportunity to correct earlier selection errors. We propose HypoKFS, a training-free, plug-and-play framework that couples keyframe selection with iterative inference by multimodal large language models (MLLMs). HypoKFS first reasons over sparsely sampled frames to construct an initial hypothesis space. The resulting answer distribution determines whether additional frame selection is needed. When needed, the retained hypotheses guide the selection of discriminative frames, followed by renewed inference and hypothesis refinement. This iterative process directs additional visual computation toward unresolved hypotheses while avoiding unnecessary global frame matching. Experiments on Video-MME, LongVideoBench, and MLVU demonstrate consistent improvements across multiple MLLM backbones. HypoKFS improves accuracy by up to 6.7 percentage points and achieves 72.9% accuracy on Video-MME at only 1.3% of the reference frame-hypothesis matching cost.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. MarKey: Marginal Utility Guided Greedy Keyframe Selection for Long Video Understanding

    Sep 14, 2026Hongchang Shi, Jinpeng Hu, Ao Wang +4Long-Video UnderstandingVideo Frame Selection

  2. QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding

    Jul 1, 2026Jun Peng, Baiyang Song, Jie Li +4Long-Video UnderstandingVideo Frame Selection

  3. Query-aligned video frame selection for long video understanding

    Sep 15, 2026Md. Safayet Islam, Dilip Sarkar, Liang LiangQuery-Aware Video Frame SelectionLong-Video QA