Query-Aware Video Frame Selection

Overview

This field studies how video-language models can answer questions about long or streaming videos by processing a small set of relevant frames instead of every frame. Methods score frames against the query, often combining semantic relevance with temporal coverage, diversity, event boundaries, or evidence from audio and model uncertainty; approaches range from training-free retrieval and greedy selection to learned policies and agentic search. Evaluations commonly use long-video question-answering benchmarks such as VideoMME, LongVideoBench, and MLVU, with growing attention to streaming settings and inference cost. The central challenge is selecting enough complementary evidence to preserve temporal context without wasting a limited frame or token budget, while ensuring methods generalize across queries, models, and videos.

Latest papers

CardsList

77 papers9 papers in the last four weeks, down 25% on the four weeks before. 0.1% of all new papers.

  1. VTG-RAG: Long-Video Reasoning via Query-Adaptive Coarse-to-Fine Sampling and Temporal Graph Retrieval

    Nov 11, 2026Yao Lu, Yunhao Wang, Yaxin Gao +7Long-Video QALong-Video Understanding

  2. SAVE: Less is More in Agentic Video Reasoning

    Nov 11, 2026Tymoteusz Wilk, Sergey Prokudin, Xi WangVideo ReasoningActive Perception for Video Understanding

  3. Reason First, Retrieve What Matters: Hypothesis-Guided Keyframe Selection for Long Video Understanding

    Nov 7, 2026Yisong Hu, Junbo Wang, Yuke Li +1Long-Video UnderstandingQuery-Aware Video Frame Selection

  4. BEV3SR: Evidence-Aware 3D Spatial Reasoning from Video

    Nov 6, 2026Fan Zhu, Rui Yu3D Spatial Reasoning3D Visual Question Answering

  5. From Label Priors to Task Evidence: Efficient Long-Video Frame Selection via Bayesian GRPO

    Oct 18, 2026Shuochen Chang, Bingjie Gao, Qingyang Liu +7Long-Video UnderstandingQuery-Aware Video Frame Selection

  6. DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding

    Oct 14, 2026Boyuan Sun, Deshui Miao, Wenzhao Gao +8Streaming Video UnderstandingVideo Question Answering

  7. FrameScout: Scouting Query-Relevant Frames for Long Video Understanding

    Oct 13, 2026Haonan Hu, Shuhao Chen, Weisen Jiang +3Query-Aware Video Frame SelectionLong-Video Understanding

  8. Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding

    Oct 12, 2026Shuo Gao, Chenhao Zheng, Zhongzheng RenLong-Video UnderstandingEfficient VLM Inference

  9. RACER: Reflective Agent Coupling Query Interpretation and Tool-Based Retrieval for Frame Selection in Long Video Understanding

    Oct 6, 2026Yiyang Huang, Yitian Zhang, Yizhou Wang +6Long-Video UnderstandingQuery-Aware Video Frame Selection

  10. Enhancing Long-Video VLM Embeddings with Query-Aware Streaming Latent Reasoning

    Oct 4, 2026Haozhe Chi, Song Jin, Yang Jin +1Multimodal EmbeddingVision-Language Retrieval

  11. Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency

    Oct 3, 2026Sixun Dong, Wei Li, Andong Deng +4Efficient VLM InferenceLong-Video Understanding

  12. Less Is More: Genetic Frame Selection for Efficient Novel View Synthesis

    Sep 28, 2026Diego E. Farchione, Ramzi Idoughi, Alberto Jaspe-Villanueva +1Novel View SynthesisFeed-Forward 3D Reconstruction

  13. CueKFS: Agentic Cue-Driven Keyframe Selection for Long Video Understanding

    Sep 25, 2026Weitai Kang, Hanieh Deilamsalehy, Yumo Xu +3Query-Aware Video Frame SelectionLong-Video Understanding

  14. Query-aligned video frame selection for long video understanding

    Sep 15, 2026Md. Safayet Islam, Dilip Sarkar, Liang LiangQuery-Aware Video Frame SelectionLong-Video QA

  15. MarKey: Marginal Utility Guided Greedy Keyframe Selection for Long Video Understanding

    Sep 14, 2026Hongchang Shi, Jinpeng Hu, Ao Wang +4Long-Video UnderstandingVideo Frame Selection

  16. One Skill Does Not Fit All: Automatic Discovery and Taxonomy-Guided Routing of Frame-Selection Skills for Long-Video Question Answering

    Sep 11, 2026Jian Hu, Zixu Cheng, Da Li +3LLM Agent Skill LearningLong-Video QA

  17. Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding

    Sep 10, 2026Weitong Cai, Hang Zhang, Yukai Huang +6Memory-Augmented VLMsEfficient VLM Inference

  18. Evaluation of MLLM-Agnostic Plug-and-Play Keyframe Selection Methods for Long Video Understanding

    Sep 4, 2026Dilip Sarkar, Md. Safayet Islam, Liang LiangLong-Video UnderstandingVideo Frame Selection

  19. CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding

    Sep 3, 2026Jing Jiang, Yiran Ling, Ruonan Li +2Streaming Video UnderstandingEfficient VLM Inference

  20. RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding

    Aug 30, 2026Shanqing Xu, Meng Luo, Mengchen Qian +7Large Vision-Language ModelsLong-Video Understanding