Video question answering (VQA) with vision-language models (VLMs) depends critically on which frames are selected from the input video, yet most systems rely on uniform or heuristic sampling that cannot be optimized for downstream answering quality. We introduce HORNet, a lightweight frame selection policy trained with Group Relative Policy Optimization (GRPO) to learn which frames a frozen VLM needs to answer questions correctly. With fewer than 1M trainable parameters, HORNet reduces input frames by up to 99% and VLM processing time by up to 93%, while improving answer quality on short-form benchmarks (+1.7% F1 on MSVD-QA) and achieving strong performance on temporal reasoning tasks (+7.3 points over uniform sampling on NExT-QA). We formalize this as Select Any Frames (SAF), a task that decouples visual input curation from VLM reasoning, and show that GRPO-trained selection generalizes better out-of-distribution than supervised and PPO alternatives. HORNet's policy further transfers across VLM answerers without retraining, yielding an additional 8.5% relative gain when paired with a stronger model. Evaluated across six benchmarks spanning 341,877 QA pairs and 114.2 hours of video, our results demonstrate that optimizing what a VLM sees is a practical and complementary alternative to optimizing what it generates while improving efficiency. Code is available at https://github.com/ostadabbas/HORNet.
Figures & tables
Figure 1: HORNet pipeline. Given a video V={v1,v2,…,vT} with T uniformly sampled frames, our TimeSFormer-based video encoder E extracts per-frame features F∈RT×D . A lightweight trainable MLP policy πθ scores each frame independently, producing keep probabilities pt∈[0,1] and a binary selection mask b∈{0,1}T . Only the frames selected by the mask ( V′⊆V ) are passed to the frozen Qwen3-VL answerer. For example, to answer “How does the boy in black react while the boy on the green disc goes down?” , HORNet selects only the frames capturing the key interaction moment, discarding irrelevant context; correctly predicting “Look at him” . At training time, GRPO samples K candidate subsets, evaluates each via String F1 reward against the ground-truth answer, and updates πθ through group-normalized policy gradients. The VLM answerer remain frozen throughout while the encoder and MLP policy are trainable.
Method
Learned Selection
Reward Optimized
Frozen VLM
Param. Efficient
Uniform Sampling
×
×
✓
✓
SeViLA [ 36 ]
✓
×
×
×
Frame-Voyager [ 37 ]
✓
×
×
×
F2C [ 30 ]
×
×
✓
✓
ReFoCUS [ 19 ]
✓
✓
∼
×
ViaRL [ 35 ]
✓
✓
×
×
Table 1: Research gap. Existing frame selection methods satisfy at most two of four desirable properties simultaneously. HORNet is the first to achieve all four: learned selection, reward-based optimization, a fully frozen VLM, and parameter efficiency ( < 1M parameters). ✓ fully supported, × not supported, ∼ partial.
Figure 2: HORNet encoder E . Input frames are patchified with a P×P convolution, processed by spatial self-attention within each frame, and then by temporal self-attention across frames at each patch location. The resulting temporally contextualized patch tokens are pooled to yield per-frame video representations used by HORNet for frame selection. B is batch size, T is frame count and D is hidden dimension. We set P =16, T =32 and D =768 in our training.
Dataset
Model
F1-Lev ↑
Frame Sel. (s) ↓
Qwen Proc. (s) ↓
Avg. Frames ↓
MSVD [ 34 ]
Qwen3-VL-2B (Baseline)
0.3483
–
0.28
11.65
HORNet+Qwen3-VL-2B (Ours)
0.3543 +1.7%
0.12
0.10 ↓ 64%
4.00 ↓ 66%
MSRVTT [ 33 ]
Qwen3-VL-2B (Baseline)
0.3209
–
0.58
47.52
HORNet+Qwen3-VL-2B (Ours)
0.3029 -5.6%
0.09
0.09 ↓ 84%
4.00 ↓ 92%
NextOE [ 32 ]
Qwen3-VL-2B (Baseline)
0.3045
–
1.01
1157.88
HORNet+Qwen3-VL-2B (Ours)
0.2738 -10.1%
0.52
0.19 ↓ 81%
8.00 ↓ 99%
Table 2: Performance comparison across open-ended QA datasets. We adopt the aforementioned F1-Lev metric to measure model performance, which is a weighted combination of token-level F1 and normalized edit similarity on lemmatized texts. We also report efficiency measured in runtime and average frames passed to Qwen. Qwen’s baseline processing time includes uniform sampling, video encoding, and answer generation. In our setup, frame selection replaces the sampling step and is reported separately. Even under this accounting, the combined runtime of HORNet still yields a notable overall speedup. Additionally, when comparing generation speed, we follow Qwen’s default sampling rate (fps=2). Under the assumption of a 24-fps source video, the baseline effectively processes roughly 1/12 of all frames-still substantially more input than HORNet requires. We highlight best performance for each benchmark in bold and mark performance gain or loss as percentages.
Dataset
Model
Accuracy(%) ↑
Frame Sel. (s) ↓
Qwen Proc. (s) ↓
Avg. Frames ↓
VideoMME [ 11 ]
Qwen3-VL-2B (Baseline)
68.30
–
2.53
3066.73
HORNet+Qwen3-VL-2B (Ours)
52.10 -16.2%
1.51
0.18 ↓ 93%
8.00 ↓ 99%
ActivityNetQA [ 38 ]
Qwen3-VL-2B (Baseline)
75.00
–
2.37
3152.49
HORNet+Qwen3-VL-2B (Ours)
68.80 -6.2%
1.64
0.17 ↓ 93%
8.00 ↓ 99%
NextQA [ 32 ]
Qwen3-VL-2B (Baseline)
76.80
–
0.98
1157.88
HORNet+Qwen3-VL-2B (Ours)
71.50 -5.3%
0.53
0.25 ↓ 74%
8.00 ↓ 99%
Table 3: Performance comparison across selection-based MCQ datasets. We adopt selection-accuracy as our metric and report efficiency gain of HORNet similar to Table 2 . For each of these dataset we randomly sampled 1,000 QA pairs.
Figure 3: Qualitative example of HORNet’s frame-selection behavior on an MCQ and open-ended sample from the NExT-QA dataset [ 32 ] . Given fixed 8-frame input, uniform sampling in Qwen-VL captures frames of the child crawling instead of the slide following the discard of the cart (left), and a frame of a person working at a computer while missing the eating frames (right), leading the model to produce an incorrect answer. With a dense initial sampling ( T =256 frames), HORNet selects the full 8-frame sequence of action-relevant frames while discarding distractors, enabling the model to recover the correct prediction.
Training
MSVD (F1-Lev ↑ )
MSRVTT (F1-Lev ↑ )
No training (baseline)
0.3483
0.3209
SFT (weighted BCE)
0.3495
0.2882
PPO (clipped surrogate)
0.3585
0.2948
GRPO (Ours)
0.3543
0.3029
Table 4: Training objective ablation. All variants use the same TimeSformer-Tiny encoder, MLP policy ( < 1M params), and frozen Qwen3-VL-2B answerer. Trained on MSVD-QA only. MSRVTT-QA results show out-of-distribution generalization.
Strategy
MSVD (F1-Lev ↑ )
MSRVTT (F1-Lev ↑ )
NExT-QA (Acc. ↑ )
Random
0.3527
0.3027
65.88
Uniform
0.3493
0.3058
64.24
HORNet
0.3543
0.3029
71.50
Table 5: Frame selection strategy ablation. All methods select 4 frames and pass them to a frozen Qwen3-VL-2B answerer. MSVD-QA and MSRVTT-QA report F1-Lev; NExT-QA reports MCQ accuracy (%).
VLM Answerer
Size
F1-Lev ↑
Qwen3-VL-Instruct (baseline)
2B
0.3483
Qwen3-VL-Instruct + HORNet
2B
0.3543
Qwen2.5-VL-Instruct + HORNet
3B
0.3846
Table 6: VLM answerer ablation on MSVD-QA. The same HORNet policy (trained with GRPO) selects frames. Only the frozen answerer is swapped; frame selection is identical.
Frame selection is an important component of long-video question answering (VQA) with Multimodal Large Language Models (MLLMs). Existing frame-selection methods improve over simple top-k embedding retrieval and uniform sampling, but are typically applied under a fixed global selection budget. We introduce MetaSampling, a training-free, plug-and-play sampling strategy that can be applied on top of existing frame selectors. MetaSampling improves downstream VQA efficiency by dynamically reducing the number of frames passed to the MLLM while preserving, and in some cases improving, answer accuracy. We evaluate MetaSampling across 36 paired frame-selector--MLLM-backbone--VQA-benchmark configurations. MetaSampling reduces the number of selected frames in all 36 configurations and improves accuracy in 25 of them, yielding an average frame reduction of 8.9% while slightly improving accuracy overall.
Ashim Dahal, Bikramjit Banerjee
University of Southern Mississippi Hattiesburg, MS, USA
Vision-language models (VLMs) can ingest only a limited number of video frames, making frame selection a practical necessity. But do current Video QA benchmarks genuinely require temporal frame selection, or can most questions be answered regardless of which frames are shown? We introduce Frame Selection Sensitivity (FSS), a per-sample diagnostic that measures how much VLM accuracy changes when the most relevant frames are replaced with the least relevant ones. Across six benchmarks and eight VLMs, we find that a large majority of samples are frame-agnostic: only a minority are genuinely sensitive to frame choice. Combining FSS with a Language Independence Score (LIS) reveals that merely 5.5--31% of samples are Temporally Sensitive. We construct TempCore, compact evaluation subsets that isolate these temporal samples from existing benchmarks, and will release code and per-sample annotations upon publication.
Multimodal large language models (MLLMs) process multimodal inputs by converting text, images, and videos into token sequences that are subsequently processed by a backbone language model. While MLLMs have achieved excellent performance in understanding the content of individual images, video understanding remains significantly more difficult because videos contain large number of video frames. MLLMs typically process only a subset of these frames, usually ranging from 8 to 64. MLLMs usually sample frames uniformly, regardless of their relevance to the question being answered. To address this limitation, several training-free, model-agnostic methods for selecting question-relevant frames have recently been proposed. In this work, we introduce a frame-selection method designed specifically for multiple-choice questions. We extend the query text by appending semantic cues derived from the answer choices and employ a direct query-frame alignment scoring mechanism. To the best of our knowledge, our method is the first to directly utilize answer choices as inference-time cues for selecting frames relevant to answering a question. The method first constructs a compact candidate pool by subsampling video frames at a fixed rate. The frames are then scored according to their maximum cosine similarity across all question-answer pairs to identify the most relevant frames for a given query. This approach preserves a fixed token budget while improving the relevance of the visual evidence provided to the downstream MLLM. We evaluate the effectiveness of our frame-selection method on the MLVU, Video-MME, and LongVideoBench benchmarks using three MLLMs: LLaVA-Mini, Qwen2-VL, and LLaVA-Video. Experimental results demonstrate that answer-aware frame selection generally outperforms uniform sampling and existing training-free frame-selection methods under the same frame budget.
Md. Safayet Islam, Dilip Sarkar, Liang Liang
Department of Computer Science University of Miami Coral Gables, FL 33146