Query-Aware Video Frame Selection

This field studies how video-language models can answer questions about long or streaming videos by processing a small set of relevant frames instead of every frame. Methods score frames against the query, often combining semantic relevance with temporal coverage, diversity, event boundaries, or evidence from audio and model uncertainty; approaches range from training-free retrieval and greedy selection to learned policies and agentic search. Evaluations commonly use long-video question-answering benchmarks such as VideoMME, LongVideoBench, and MLVU, with growing attention to streaming settings and inference cost. The central challenge is selecting enough complementary evidence to preserve temporal context without wasting a limited frame or token budget, while ensuring methods generalize across queries, models, and videos.