FrameGRPO: Incentivizing Video Reasoning via Multi-Frame Rollouts and Temporal Credit

Jiaxuan Sun; Biao Wang; Shuai Li

cs.CVOct 19, 2026

FrameGRPO: Incentivizing Video Reasoning via Multi-Frame Rollouts and Temporal Credit

Authors: Jiaxuan Sun, Biao Wang, Shuai Li

Organizations: Shanghai Jiao Tong University · Shanghai Innovation Institute · Tencent Hy

Abstract

Reinforcement learning with verifiable rewards, particularly Group Relative Policy Optimization (GRPO), has improved video reasoning in multimodal large language models. However, standard video GRPO varies only linguistic trajectories while conditioning every rollout in a group on the same sampled frames, which can produce homogeneous rewards and weaken group-relative learning when the shared visual evidence is insufficient or distracting. We propose FrameGRPO, a simple and effective reinforcement-learning framework that requires neither auxiliary networks nor changes to the model architecture. Specifically, FrameGRPO combines two lightweight components: (1) Multi-Frame Rollouts, which use only uniform temporal subsampling of a single maximal-frame decode to construct aligned views that diversify visual evidence and reasoning trajectories; and (2) Gated Temporal Efficiency Credit, which selectively reinforces a correct lower-ratio trajectory when all rollouts in a fixed full-ratio reference subset fail, without imposing an unconditional preference for fewer frames. Extensive experiments across diverse video understanding and reasoning benchmarks demonstrate robust overall improvements over standard GRPO, highlighting temporal-view diversification as an effective mechanism for exploration and credit assignment in video MLLMs.

Figures & tables

Figure 1: Training dynamics of FrameGRPO and GRPO. The four panels track VSI-Bench accuracy, within-group reward standard deviation, and the prevalence of uniformly correct and uniformly incorrect rollout groups during optimization. Exposing each group to multiple frame sampling ratios yields more informative reward contrasts, mitigates group-level saturation at either outcome extreme, and translates into stronger evaluation performance. Shaded bands visualize the variability of each metric.

Figure 2: Overview of FrameGRPO. Given a video–question pair, we assign the full-ratio input to a fixed subset of rollout slots and independently sample $\rho_{i}\sim\mathcal{U}[0,1]$ for the remaining slots to construct temporally aligned views $V_{\rho_{i}}$ . A shared policy generates $\{o_{i}\}_{i=1}^{G}$ , and a verifier assigns rewards $\{r_{i}\}_{i=1}^{G}$ . The outcomes of the fixed full-ratio subset jointly determine the gate for Temporal Efficiency Credit, which produces the modified advantages $\{\widetilde{A}_{i}\}_{i=1}^{G}$ . Policy optimization is conditioned only on the full-ratio input.

Figure 3: Complementarity across frame sampling ratios. Left: the lower-ratio 16-frame view captures the relevant condiment-spreading action, whereas the full-ratio 32-frame view is distracted by a subsequent event. Right: the higher-ratio view preserves the evidence required to associate the girl’s motion with the kite, whereas the lower-ratio view misses this critical evidence.

Method	Video General Benchmark			Video Reasoning Benchmark
	Video-MME	TempCompass	VideoEval-Pro	MMVU	VSI-Bench	NExT-GQA	Video-MMMU
GPT-4o	71.9	75.1	34.2	75.4	34.0	-	61.2
Gemini 1.5 Pro	75.0	70.5	39.3	71.2	48.8	-	53.4
LLaMA-VID	-	45.6	-	-	-	-	-
LLaVA-OneVision-7B	58.2	69.5	-	49.2	32.4	-	33.8
Kangaroo-8B	56.0	62.5	-	-	-	-	-

Table 1: Performance Comparison on Video Understanding and Reasoning Benchmarks. All results are reported as accuracy (%). The best result within each backbone is highlighted in bold.

Modules		Video General Benchmark			Video Reasoning Benchmark
MFRoll	TempCred	Video-MME	TempCompass	VideoEval-Pro	MMVU	VSI-Bench	NExT-GQA	Video-MMMU
✗	✗	60.0	74.4	41.0	65.6	34.1	79.5	52.0
✓	✗	60.7	74.4	41.8	66.3	34.7	79.8	52.2
✓	✓	61.3	74.9	42.4	66.7	37.9	79.6	54.2

Table 2: Ablation Study of FrameGRPO Modules. MFRoll and TempCred denote Multi-Frame Rollouts and Temporal Credit, respectively. All results are reported as accuracy (%).

Optimization Input	Video General Benchmark			Video Reasoning Benchmark
	Video-MME	TempCompass	VideoEval-Pro	MMVU	VSI-Bench	NExT-GQA	Video-MMMU
Partial-Ratio Input	60.2	74.0	41.0	66.7	32.9	79.4	50.1
Full-Ratio Input (Ours)	61.3	74.9	42.4	66.7	37.9	79.6	54.2

Table 3: Ablation Study of Policy Optimization Input. We compare conditioning policy optimization only on the partial-ratio input with conditioning only on the full-ratio input. All results are reported as accuracy (%).

Setting	Video General Benchmark			Video Reasoning Benchmark
	Video-MME	TempCompass	VideoEval-Pro	MMVU	VSI-Bench	NExT-GQA	Video-MMMU
Vanilla GRPO	60.0	74.4	41.0	65.6	34.1	79.5	52.0
FrameGRPO with Different $\alpha$
$\alpha=0.1$	61.3	75.3	43.1	67.7	35.8	79.6	52.6
$\alpha=0.2$	61.3	74.9	42.4	66.7	37.9	79.6	54.2
$\alpha=0.3$	61.0	75.0	41.5	65.4	34.3	79.6	52.0

Table 4: Sensitivity Analysis of Hyperparameter

\alpha

. Results under varying

\alpha

settings are reported on seven video benchmarks. The best performance for each benchmark is highlighted in bold. The configuration highlighted in blue corresponds to the setting used in the main results.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration		Video General Benchmark			Video Reasoning Benchmark
Method	Seed	Video-MME	TempCompass	VideoEval-Pro	MMVU	VSI-Bench	NExT-GQA	Video-MMMU
GRPO FrameGRPO	42	60.0 61.3	74.4 74.9	41.0 42.4	65.6 66.7	34.1 37.9	79.5 79.6	52.0 54.2
GRPO FrameGRPO	43	59.8 61.3	73.8 75.0	39.4 42.1	66.4 68.0	34.0 35.1	80.0 79.8	50.2 51.4
GRPO FrameGRPO	44	60.2 61.5	73.8 74.9	41.8 42.2	65.6 67.7	31.7 34.3	79.4 79.6	52.4 52.7

Appendix

Table 5: Robustness Across Random Seeds. Results for GRPO and FrameGRPO with Qwen2.5-VL-7B under three random seeds. All results are reported as accuracy (%). The best result within each seed is highlighted in bold.

Configuration		Video General Benchmark			Video Reasoning Benchmark
Method	Data Size	Video-MME	TempCompass	VideoEval-Pro	MMVU	VSI-Bench	NExT-GQA	Video-MMMU
GRPO FrameGRPO	2K	59.6 60.6	73.7 74.0	41.9 42.6	66.2 66.9	31.0 33.1	79.7 78.9	51.6 51.7
GRPO FrameGRPO	4K	60.6 61.4	74.0 75.2	42.5 42.6	65.3 66.7	33.4 37.6	79.6 79.5	51.2 52.4
GRPO FrameGRPO	8K	60.0 61.3	74.4 74.9	41.0 42.4	65.6 66.7	34.1 37.9	79.5 79.6	52.0 54.2

Appendix

Table 6: Robustness Across Training-Set Sizes. Results for GRPO and FrameGRPO with Qwen2.5-VL-7B using different numbers of training examples. All results are reported as accuracy (%). The best result within each data size is highlighted in bold.

Credit Design	Video General Benchmark			Video Reasoning Benchmark
	Video-MME	TempCompass	VideoEval-Pro	MMVU	VSI-Bench	NExT-GQA	Video-MMMU
Uniform Credit ( $0.2$ )	60.7	74.9	42.0	65.1	34.7	79.3	51.3
TEC w/o Gate	60.1	74.6	41.1	65.9	34.9	79.5	51.9
Gated TEC (Ours)	61.3	74.9	42.4	66.7	37.9	79.6	54.2

Appendix

Table 7: Ablation Study of Temporal Efficiency Credit. We compare a uniform credit with a fixed weight of

0.2

, TEC without the full-ratio gate, and the proposed gated TEC. All results are reported as accuracy (%). The best result for each benchmark is highlighted in bold.

Subset Size	Video General Benchmark			Video Reasoning Benchmark
	Video-MME	TempCompass	VideoEval-Pro	MMVU	VSI-Bench	NExT-GQA	Video-MMMU
$K_{\star}=1$	60.7	74.6	41.1	65.1	35.3	78.0	49.3
$K_{\star}=2$	61.3	74.9	42.4	66.7	37.9	79.6	54.2
$K_{\star}=4$	60.6	75.0	41.4	65.0	36.6	78.9	53.4

Appendix

Table 8: Ablation Study of Full-Ratio Reference Subset Size. We vary

K_{\star}

while keeping the group size fixed. All results are reported as accuracy (%). The best result for each benchmark is highlighted in bold, and the configuration used in our main experiments is highlighted in blue.

Method	Wall-Clock Time	GPU-Hours	Peak GPU Memory
Qwen2.5-VL-7B
GRPO	21h 24min	171.2	94.95 GiB
FrameGRPO	21h 19min	170.5	57.46 GiB
Qwen2-VL-7B
GRPO	17h 01min	136.1	94.63 GiB
FrameGRPO	19h 51min	158.8	63.80 GiB

Appendix

Table 9: Training Efficiency Comparison. All runs use eight NVIDIA H20 GPUs. GPU-hours are aggregated across all GPUs.

Hyperparameter	Value
Sampling
Group size $G$	8 rollouts per prompt
Training sampling temperature	1.0
Maximum prompt / completion length	8,192 / 768 tokens
Optimization
Optimizer	AdamW

Appendix

Table 10: Training Hyperparameters for FrameGRPO. The table reports the default configuration used in our experiments.

Figure 4: Prompt template used for both training and inference.

Figure 5: Reasoning example on NExT-GQA.

Figure 6: Reasoning example on Video-MME.

RL for Video Reasoning

See all 26 papers

Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs
Jun 16, 2026Chengwen Liu, Zhe Huang, Jisheng Dang +3Multimodal Reward Modeling Video Reasoning
School of Information Science and Engineering, Lanzhou University, Lanzhou, China · Beijing University of Posts and Telecommunications, Beijing, China · Cloud and AI BU, Huawei, Shenzhen, China +1
STRIVE: Structured Spatiotemporal Exploration for Reinforcement Learning in Video Question Answering
Apr 2, 2026Emad Bahrami, Olga Zatsarynna, Parth Pathak +3Video QA RL for Video Reasoning
University of Bonn · Lamarr Institute for Machine Learning and Artificial Intelligence · Meta +1
Frame Differential On-Policy Self-Distillation for Video Reasoning
Sep 30, 2026Haiying He, Xin Zheng, Shaoli Hu +4On-Policy Self-Distillation Video Reasoning
HKUST · NKU · SEU +1

Jun 16, 2026J/K move · Enter open · S save

Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs

Chengwen Liu, Zhe Huang, Jisheng Dang, Hong Peng+2

School of Information Science and Engineering, Lanzhou University, Lanzhou, China · Beijing University of Posts and Telecommunications, Beijing, China · Cloud and AI BU, Huawei, Shenzhen, China+1

Multimodal Reward Modeling Video Reasoning Visually Grounded Reasoning RL for VLMs RL for Video Reasoning

Open paper AskPDF arXiv

Reinforcement learning has improved the reasoning ability of large language models, but applying outcome-only rewards to video multimodal large language models (Video-MLLMs) provides limited guidance on which visual evidence should support the answer. Inspired by multisensory integration, where consistent cues can enhance the salience and reliability of perceptual estimates, we introduce Consensus Frame GRPO (CF-GRPO), a temporal-annotation-free process-level reward framework for evidence-aware video reasoning. CF-GRPO constructs a consensus frame prior from intrinsic video cues, including temporal coverage, scene-transition cues, and query-conditioned visual relevance. It then computes a model-side frame-use score from visual and response representations and optimizes their agreement through the Consensus Frame Reward (CFR). With salience-aware sparse aggregation and distribution sharpening, CFR provides a high-contrast reward signal without requiring human temporal annotations. Experiments show that VideoCFR achieves competitive performance across complex video reasoning benchmarks and improves several metrics over representative Video-MLLM and RL baselines, while the consensus prior provides an interpretable view of the evidence frames emphasized during training. The implementation is available at https://github.com/1Pansy/VideoCFR.

Abstract

RL for Video Reasoning

Explore similar work

Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs

STRIVE: Structured Spatiotemporal Exploration for Reinforcement Learning in Video Question Answering

Frame Differential On-Policy Self-Distillation for Video Reasoning