From Label Priors to Task Evidence: Efficient Long-Video Frame Selection via Bayesian GRPO

Shuochen Chang; Bingjie Gao; Qingyang Liu; Qianli Ma; Yibo Miao; Haonan Zhao; Zhaohe Liao; Xiaofeng Zhang

cs.CVOct 18, 2026

From Label Priors to Task Evidence: Efficient Long-Video Frame Selection via Bayesian GRPO

Authors: Shuochen Chang, Bingjie Gao, Qingyang Liu, Qianli Ma, Yibo Miao, Haonan Zhao, Zhaohe Liao, Xiaofeng Zhang,

Organizations: Shanghai Jiao Tong University · Tongji University

Abstract

Multimodal Large Language Models (MLLMs) face significant scalability challenges in long-video understanding due to quadratic attention complexity and context window constraints. Existing retrieval-based approaches typically rely on supervised learning from static keyframe annotations. However, these label priors often fail to align with the actual information required by downstream MLLMs for complex reasoning, creating a misalignment between the training target and inference needs. To bridge this gap, we propose a probabilistic framework that reformulates frame selection as latent-variable inference. We first initialize a lightweight policy using supervised learning to capture general visual saliency. Then, we treat a frozen MLLM as a task environment and use its answer likelihood as dense task evidence to refine the policy via Bayesian Group Relative Policy Optimization (GRPO). Within this formulation, the negative log-likelihood (NLL) over ground-truth answer tokens provides a principled dense reward, while a prior-matching KL term preserves semantic consistency with the elicited label prior. This process transitions the selection principle from static label priors to task evidence. Extensive experiments on Video-MME, MLVU, and LongVideoBench demonstrate that our selector achieves competitive accuracy with MLLM-based scorers while retaining the inference efficiency of lightweight methods. The selector is trained once with a frozen MLLM environment and transfers directly to multiple downstream MLLMs without retraining. Code is available at github.com/Miraclecsc/Frame_Selection_GRPO.

Figures & tables

Figure 1: The overall framework of our methods. (a) Keyframe Selector. The lightweight selector ( $f_{\theta}$ ) comprises stacked Video-Text Fusion Transformers. It modulates frame features with query semantics to output importance logits, then parameterizing a Plackett-Luce distribution. (b) Stage I: Label Prior Elicitation. We utilize keyframe label annotations to construct a Label Prior . The selector is trained via pairwise ranking loss to align its predicted logits with the label logits, establishing a robust initialization. (c) Stage II:Task-Evidence Posterior Inference. We freeze the MLLM and treat it as a frozen task environment. The selector performs Group Sampling to propose discrete frame subsets ( $\mathbf{z}$ ). The MLLM evaluates these subsets to provide dense NLL rewards ( $R$ ) and advantages ( $A$ ). The policy is refined via Bayesian GRPO to maximize task evidence while maintaining KL-regularization towards the label prior.

Uniform Sampling Baseline
Method	# Frames	Video-MME				LVB	MLVU
Method	# Frames	Short	Medium	Long	Overall	LVB	Needle-QA	Avg
Avg. Video Duration		1.3min	9min	41min	17min	12min	12min
Uniform	8	61.7	50.6	45.7	52.6	52.6	62.8	56.1
Uniform	32	71.4	60.8	49.2	60.5	56.3	72.4	62.2
Lightweight Scoring Methods

Table 1: Comparison of frame selection methods on Qwen2.5-VL-7B . Methods are categorized by type and grouped by frame count. Bold / underline denote the best/second-best performance.

Uniform Sampling Baseline
Method	# Frames	Video-MME				LVB	MLVU
Method	# Frames	Short	Medium	Long	Overall	LVB	Needle-QA	Avg
Avg. Video Duration		1.3min	9min	41min	17min	12min	12min
InternVL-3-8B
Uniform	8	68.7	58.3	48.8	58.6	53.4	77.2	64.4
Uniform	32	74.2	62.8	51.7	62.9	57.9	83.7	70.3

Table 2: Comparison of frame selection methods on InternVL-3-8B and LLaVA-OneVision-7B . Methods are categorized by type, grouped by model, and then by frame count. Bold / underline denote the best/second-best performance.

Qwen3-VL-8B
Method	# Frames	Video-MME				LVB	MLVU
Method	# Frames	Short	Medium	Long	Overall	LVB	Needle-QA	Avg
Avg. Video Duration		1.3min	9min	41min	17min	12min	12min
Uniform	8	64.8	51.8	47.7	54.7	53.3	60.4	56.9
Ours	8	67.8	56.2	50.8	58.3	58.2	79.8	66.7
Uniform	32	76.1	61.6	52.8	63.5	58.9	71.6	65.3

Table 3: Transferability analysis on state-of-the-art MLLMs ( Qwen3-VL-8B and InternVL-3.5-8B ). We compare the original model performance (Uniform) with our selector. Bold denotes the best performance.

Evaluation: InternVL-3-8B
Stage II Env.	Video-MME		LVB		MLVU
Stage II Env.	$K=8$	$K=32$	$K=8$	$K=32$	$K=8$	$K=32$
Qwen2.5-VL-7B	62.4	66.8	58.7	62.1	69.0	74.3
InternVL-3-8B	62.4	66.5	59.0	61.9	69.2	74.2

Table 4: Cross-environment Stage II training. Both selectors are evaluated on InternVL-3-8B with

K=8

and

K=32

. The selector trained with Qwen2.5-VL transfers directly to InternVL-3 without additional posterior training and achieves comparable performance to the selector trained with InternVL-3 itself. Bold denotes the best performance.

Method	Training Components			K = 8			K = 32
Method	Stage I	Stage II	Reward	V-MME	LVB	MLVU	V-MME	LVB	MLVU
SigLIP2	✗	✗	-	54.7	56.6	61.9	61.1	59.7	66.3
Label Prior	✓	✗	-	58.5	57.4	65.2	63.6	60.4	68.2
+ Label Reward	✓	✓	label	58.3	57.6	64.9	63.8	60.2	67.6
+ Task Evidence (Ours)	✓	✓	NLL	61.5	58.7	66.1	65.2	61.1	70.3

Table 5: Ablation study on training strategies. All results are evaluated using the Qwen2.5-VL-7B model. We investigate the impact of the Label Prior (Stage I) and the Bayesian GRPO update (Stage II) across different frame budgets (

K=8,32

). Reward Source indicates the reward signal used during GRPO.

Reward Type	Reward Properties		$K=8$			$K=32$
Reward Type	Granularity	Bias	V-MME	LVB	MLVU	V-MME	LVB	MLVU
Stage II reward signal
Label	Frame-level / medium	Annotation-biased	58.3	57.6	64.9	63.8	60.2	67.6
RLVR-style correctness	Sample-level / sparse	Task-level	57.3	56.5	65.3	62.1	58.9	66.7
NLL	Token-level / dense	Task-level	61.5	58.7	66.1	65.2	61.1	70.3

Table 6: Comparison of reward types for Stage II. Results are evaluated on Qwen2.5-VL-7B with

K=8

and

K=32

. Label reward provides medium-granularity but annotation-biased supervision. RLVR-style correctness reward is task-level but sparse. NLL provides dense task evidence and performs best across benchmarks. Bold denotes the best performance.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Stage I: Label Prior	Stage II: Task Evidence
Batch Size	32	1
Learning Rate	$1e^{-4}$	$5e^{-6}$
Optimizer	AdamW	AdamW
Weight Decay	$0.01$	$0.01$
Epochs	10	1
KL Coefficient ( $\beta$ )	-	0.2

Appendix

Table 7: Hyperparameter settings for the two-stage training process.

Method	Encoder / Scorer	GFLOPs	Mem.
			(GB)
Lightweight feature-based scoring
BOLT	SigLIP2-SO400M	240	0.80
F2C	SigLIP2-SO400M	241	0.80
Ours	SigLIP2-SO400M + selector	264	0.93
MLLM-based scoring

Appendix

Table 8: Selector-side efficiency comparison. GFLOPs and peak memory are measured on a 300-second video sampled at 1 FPS with

K=32

. BOLT, F2C, and ours use the same SigLIP2-SO400M feature extractor.

Method	Latency (s/sample)
BOLT	1.29
Ours	1.38
Q-Frame	1.53
GenS	214.6

Appendix

Table 9: Average per-sample inference latency on Video-MME. Latency is measured on a single NVIDIA H100 GPU with

K=32

and 1 FPS candidate sampling.

Method	Training Components			K = 8			K = 32
Method	Stage I	Stage II	Reward	V-MME	LVB	MLVU	V-MME	LVB	MLVU
SigLIP2	✗	✗	-	55.4	56.7	61.0	62.1	59.8	69.2
Label Prior	✓	✗	-	57.3	58.1	64.3	63.9	61.4	71.0
+ Label Reward	✓	✓	Label	57.1	57.4	64.9	64.1	61.0	71.4
+ Task Evidence	✓	✓	NLL	58.3	58.2	66.7	65.3	62.7	72.7

Appendix

Table 10: Extended Ablation Study on Qwen3-VL-8B. Comparison of training strategies across different frame budgets (

K=8,32

Method	Training Components			K = 8			K = 32
Method	Stage I	Stage II	Reward	V-MME	LVB	MLVU	V-MME	LVB	MLVU
SigLIP2	✗	✗	-	56.7	58.1	64.8	60.9	60.6	69.7
Label Prior	✓	✗	-	57.6	58.9	66.2	62.4	62.3	70.7
+ Label Reward	✓	✓	Label	57.3	58.1	66.5	62.8	62.7	70.3
+ Task Evidence	✓	✓	NLL	58.3	59.6	67.6	64.1	63.4	71.4

Appendix

Table 11: Extended Ablation Study on InternVL-3.5-8B. Comparison of training strategies across different frame budgets (

K=8,32

Metric	$K=8$	$K=16$	$K=32$	$K=64$	$K=128$
Video-MME	61.5	63.7	65.2	66.2	66.9
LVB	58.2	59.8	61.1	62.4	63.3
MLVU (Avg)	66.1	68.7	70.3	73.3	73.4

Appendix

Table 12: Performance with varying frame budgets (

K

) on Video-MME, LongVideoBench, and MLVU. All results are evaluated using Qwen2.5-VL .

Figure 4: Qualitative Examples 1 of Frame Selection. From top to bottom, we visualize the frames selected by our method compared to the baseline.

Panel of Figure 5 — Figure 5: Qualitative Examples 2 and 3 of Frame Selection. From top to bottom, we visualize the frames selected by our method compared to the baseline.

Long-Video Understanding

Long-video understanding enables models to retrieve, remember, and reason about events across extended visual and audio-visual sequences.

See all 153 papers

Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors
Jul 17, 2026Yilin Wang, Xiangxi Zheng, Dongxing Mao +6Efficient VLM Inference Multimodal Large Language Models
ZJU · NJU · CSU +2
Query-aligned video frame selection for long video understanding
Sep 15, 2026Md. Safayet Islam, Dilip Sarkar, Liang LiangQuery-Aware Video Frame Selection Long-Video QA
Department of Computer Science University of Miami Coral Gables, FL 33146
Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding
Aug 6, 2026Bo Zhang, Wenxin Wang, Feng Chen +4Efficient VLM Inference Long-Video Understanding
1Sichuan University · 2Adelaide University · 3Beijing Institute of Technology

Jul 17, 2026J/K move · Enter open · S save

Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors

Yilin Wang, Xiangxi Zheng, Dongxing Mao, Linjie Li+5

ZJU · NJU · CSU+2

Efficient VLM Inference Multimodal Large Language Models Video Frame Selection Query-Aware Video Frame Selection Adaptive Video Frame Sampling

Open paper AskPDF arXiv

Understanding long videos with multimodal large language models (MLLMs) requires selecting a compact set of frames from thousands of candidates, yet identifying the right frames seemingly requires understanding the video first. We resolve this circular dependency with a simple observation: cross-modal attention at validation-selected extraction layers in MLLMs already provides query-relevant frame evidence without requiring autoregressive generation. We exploit this property to build DAFS (Dynamic Attention-based Budget-aware Frame Selection), a training-free frame selector. A lightweight MLLM selector, even with only 2B parameters, can extract frame-level evidence by converting selected-layer attention into relevance scores through query-conditioned aggregation. This enables cross-frame comparison without autoregressive decoding. To handle the selector's own context constraint, we formulate the joint allocation of candidate pool size and per-frame token budget as a discrete optimization problem solved by dynamic programming. Under a 32-frame budget, our selector improves over uniform sampling by up to 6.4 points on Video-MME and outperforms prior training-based selectors under matched frame budgets, while generalizing across selector and answerer backbones, and across tasks, without retraining.

Abstract

Long-Video Understanding

Explore similar work

Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors

Query-aligned video frame selection for long video understanding

Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding