We introduce STRIVE (SpatioTemporal Reinforcement with Importance-aware Variant Exploration), a structured reinforcement learning framework for video question answering. While group-based policy optimization methods have shown promise in large multimodal models, they often suffer from low reward variance when responses exhibit similar correctness, leading to weak or unstable advantage estimates. STRIVE addresses this limitation by constructing multiple spatiotemporal variants of each input video and performing joint normalization across both textual generations and visual variants. By expanding group comparisons beyond linguistic diversity to structured visual perturbations, STRIVE enriches reward signals and promotes more stable and informative policy updates. To ensure exploration remains semantically grounded, we introduce an importance-aware sampling mechanism that prioritizes frames most relevant to the input question while preserving temporal coverage. This design encourages robust reasoning across complementary visual perspectives rather than overfitting to a single spatiotemporal configuration. Experiments on six challenging video reasoning benchmarks including VideoMME, TempCompass, VideoMMMU, MMVU, VSI-Bench, and PerceptionTest demonstrate consistent improvements over strong reinforcement learning baselines across multiple large multimodal models. Our results highlight the role of structured spatiotemporal exploration as a principled mechanism for stabilizing multimodal reinforcement learning and improving video reasoning performance.
Figures & tables
Figure 1: STRIVE prevents advantage collapse in video reinforcement learning. (a) Standard policy optimization samples multiple textual responses from a single, fixed video representation. When these generated responses lack diversity, the reward variance can drop to zero ( Var(r)=0 ), stalling the gradient of the objective J(θ) . (b) Our proposed STRIVE framework employs Variant Construction to systematically generate multiple spatiotemporal visual variants of the input video. By computing joint advantages across both textual generations and varied visual perspectives, STRIVE empirically maintains a consistently positive reward variance ( Var(r)>0 ), delivering informative updates to the policy J(θ) .
Figure 2: Overview of our STRIVE Framework . Given a video input v and question q , a set of transformations Ti is applied to generate video variants v~i . Each variant is paired with the question to form input tuples (v~i,q) . The policy model produces G textual responses for each of the M variants, resulting in a G×M matrix of responses oi,j . These are evaluated by a reward model to obtain ri,j . Advantages are then jointly normalized across the entire G×M pool, enriching reward variance.
Figure 3: Empirical analysis of optimization dynamics. (a) Standard deviation of rewards, demonstrating that STRIVE maintains consistently higher variance than the GRPO baseline. (b) Fraction of zero-advantage updates, illustrating STRIVE’s ability to prevent gradient starvation. (c) Gradient norm over training steps.
Model
VideoMME (wo sub)
VSI-Bench
PerceptionTest (mc)
TempCompass
VideoMMMU
MMVU (mc)
avg
GPT-4o [ 1 ]
71.9
34.0
-
75.1
61.2
75.4
-
Gemini-1.5-Pro [ 27 ]
75.0
48.8
-
70.5
-
-
-
VILA-1.5-8B [ 12 ]
-
28.9
-
-
20.8
-
-
LongVA-7B [ 34 ]
52.6
29.2
-
-
23.9
-
-
LLaVA-OV-7B [ 9 ]
58.2
32.4
57.1
69.5
-
64.7
-
Qwen2.5-VL-7B-SFT [ 4 ]
55.4
33.3
68.4
69.9
49.4
63.5
56.6
Table 1: Performance on six video understanding benchmarks. To ensure a fair comparison, models are evaluated with a maximum of 32 frames. Our proposed STRIVE framework outperforms the standard GRPO baseline across both Qwen2.5-VL-7B and Qwen3-VL-8B architectures. Results marked with * are evaluated using publicly available model weights.
Configuration
VideoMME
VSI-Bench
TempCompass
MMVU (mc)
Avg
(a) Variant Construction Strategies
GRPO Baseline
58.0
32.5
72.1
65.3
57.0
Deterministic Temporal
57.1
31.8
73.1
65.0
56.8
Stochastic
58.4
34.3
73.2
65.8
57.9
Importance-Based (Temporal Only)
59.0
34.2
72.8
66.2
58.1
Importance-Based (+ Spatial)
58.2
35.2
73.1
65.4
58.0
Table 2: Ablation Studies. We evaluate (a) the variant construction strategy, (b) the scoring mechanism used for importance-based grouping, (c) the trade-off between visual variants ( M ) and textual generations ( G ), and (d) the number of temporal segments ( K ). All ablation models are trained for 500 RL steps and evaluated on a subset of four core benchmarks.
Figure 4: Qualitative comparison between STRIVE and the standard GRPO baseline. (Top) STRIVE successfully isolates the relevant driving frames to correctly answer a question about a person’s mode of transport, whereas GRPO is distracted by irrelevant walking scenes. (Bottom) STRIVE accurately determines the strict temporal sequence of objects, demonstrating how question-guided variant construction improves fine-grained spatiotemporal reasoning and prevents reliance on flawed priors.
Reinforcement learning with verifiable rewards, particularly Group Relative Policy Optimization (GRPO), has improved video reasoning in multimodal large language models. However, standard video GRPO varies only linguistic trajectories while conditioning every rollout in a group on the same sampled frames, which can produce homogeneous rewards and weaken group-relative learning when the shared visual evidence is insufficient or distracting. We propose FrameGRPO, a simple and effective reinforcement-learning framework that requires neither auxiliary networks nor changes to the model architecture. Specifically, FrameGRPO combines two lightweight components: (1) Multi-Frame Rollouts, which use only uniform temporal subsampling of a single maximal-frame decode to construct aligned views that diversify visual evidence and reasoning trajectories; and (2) Gated Temporal Efficiency Credit, which selectively reinforces a correct lower-ratio trajectory when all rollouts in a fixed full-ratio reference subset fail, without imposing an unconditional preference for fewer frames. Extensive experiments across diverse video understanding and reasoning benchmarks demonstrate robust overall improvements over standard GRPO, highlighting temporal-view diversification as an effective mechanism for exploration and credit assignment in video MLLMs.
Jiaxuan Sun, Biao Wang, Shuai Li
Shanghai Jiao Tong University · Shanghai Innovation Institute · Tencent Hy
Reinforcement learning has advanced video reasoning in large multi-modal models, yet dominant pipelines either rely on on-policy self-exploration, which plateaus at the model's knowledge boundary, or hybrid replay that mixes policies and demands careful regularization. Dynamic context methods zoom into focused evidence but often require curated pretraining and two-stage tuning, and their context remains bounded by a small model's capability. In contrast, larger models excel at instruction following and multi-modal understanding, can supply richer context to smaller models, and rapidly zoom in on target regions via simple tools. Building on this capability, we introduce an observation-level intervention: a frozen, tool-integrated teacher identifies the missing spatiotemporal dependency and provides a minimal evidence patch (e.g., timestamps, regions etc.) from the original video while the question remains unchanged. The student answers again with the added context, and training updates with a chosen-rollout scheme integrated into Group Relative Policy Optimization (GRPO). We further propose a Robust Improvement Reward (RIR) that aligns optimization with two goals: outcome validity through correct answers and dependency alignment through rationales that reflect the cited evidence. Advantages are group-normalized across the batch, preserving on-policy exploration while directing it along causally meaningful directions with minimal changes to the training stack. Experiments on various related benchmarks show consistent accuracy gains and strong generalization. Code will be available at https://jethrojames.github.io/FFR/.
Haojian Huang, Chuanyu Qin, Yinchuan Li +1
The Hong Kong University of Science and Technology (Guangzhou) · Knowin AI · Institute of Information Engineering, Chinese Academy of Sciences
Reinforcement learning has improved the reasoning ability of large language models, but applying outcome-only rewards to video multimodal large language models (Video-MLLMs) provides limited guidance on which visual evidence should support the answer. Inspired by multisensory integration, where consistent cues can enhance the salience and reliability of perceptual estimates, we introduce Consensus Frame GRPO (CF-GRPO), a temporal-annotation-free process-level reward framework for evidence-aware video reasoning. CF-GRPO constructs a consensus frame prior from intrinsic video cues, including temporal coverage, scene-transition cues, and query-conditioned visual relevance. It then computes a model-side frame-use score from visual and response representations and optimizes their agreement through the Consensus Frame Reward (CFR). With salience-aware sparse aggregation and distribution sharpening, CFR provides a high-contrast reward signal without requiring human temporal annotations. Experiments show that VideoCFR achieves competitive performance across complex video reasoning benchmarks and improves several metrics over representative Video-MLLM and RL baselines, while the consensus prior provides an interpretable view of the evidence frames emphasized during training. The implementation is available at https://github.com/1Pansy/VideoCFR.
Chengwen Liu, Zhe Huang, Jisheng Dang +3
School of Information Science and Engineering, Lanzhou University, Lanzhou, China · Beijing University of Posts and Telecommunications, Beijing, China · Cloud and AI BU, Huawei, Shenzhen, China +1