STRIVE: Structured Spatiotemporal Exploration for Reinforcement Learning in Video Question Answering

Emad Bahrami; Olga Zatsarynna; Parth Pathak; Sunando Sengupta; Juergen Gall; Mohsen Fayyaz

cs.CVApr 2, 2026

STRIVE: Structured Spatiotemporal Exploration for Reinforcement Learning in Video Question Answering

Authors: Emad Bahrami, Olga Zatsarynna, Parth Pathak, Sunando Sengupta, Juergen Gall, Mohsen Fayyaz

Organizations: University of Bonn · Lamarr Institute for Machine Learning and Artificial Intelligence · Meta · Microsoft

Abstract

We introduce STRIVE (SpatioTemporal Reinforcement with Importance-aware Variant Exploration), a structured reinforcement learning framework for video question answering. While group-based policy optimization methods have shown promise in large multimodal models, they often suffer from low reward variance when responses exhibit similar correctness, leading to weak or unstable advantage estimates. STRIVE addresses this limitation by constructing multiple spatiotemporal variants of each input video and performing joint normalization across both textual generations and visual variants. By expanding group comparisons beyond linguistic diversity to structured visual perturbations, STRIVE enriches reward signals and promotes more stable and informative policy updates. To ensure exploration remains semantically grounded, we introduce an importance-aware sampling mechanism that prioritizes frames most relevant to the input question while preserving temporal coverage. This design encourages robust reasoning across complementary visual perspectives rather than overfitting to a single spatiotemporal configuration. Experiments on six challenging video reasoning benchmarks including VideoMME, TempCompass, VideoMMMU, MMVU, VSI-Bench, and PerceptionTest demonstrate consistent improvements over strong reinforcement learning baselines across multiple large multimodal models. Our results highlight the role of structured spatiotemporal exploration as a principled mechanism for stabilizing multimodal reinforcement learning and improving video reasoning performance.

Figures & tables

Figure 1: STRIVE prevents advantage collapse in video reinforcement learning. (a) Standard policy optimization samples multiple textual responses from a single, fixed video representation. When these generated responses lack diversity, the reward variance can drop to zero ( $\text{Var}(r)=0$ ), stalling the gradient of the objective $J(\theta)$ . (b) Our proposed STRIVE framework employs Variant Construction to systematically generate multiple spatiotemporal visual variants of the input video. By computing joint advantages across both textual generations and varied visual perspectives, STRIVE empirically maintains a consistently positive reward variance ( $\text{Var}(r)>0$ ), delivering informative updates to the policy $J(\theta)$ .

Figure 2: Overview of our STRIVE Framework . Given a video input

v

and question

q

, a set of transformations

\mathcal{T}_{i}

is applied to generate video variants

\tilde{v}_{i}

. Each variant is paired with the question to form input tuples

(\tilde{v}_{i},q)

. The policy model produces

G

textual responses for each of the

M

variants, resulting in a

G\times M

matrix of responses

o_{i,j}

. These are evaluated by a reward model to obtain

r_{i,j}

. Advantages are then jointly normalized across the entire

G\times M

pool, enriching reward variance.

Figure 3: Empirical analysis of optimization dynamics. (a) Standard deviation of rewards, demonstrating that STRIVE maintains consistently higher variance than the GRPO baseline. (b) Fraction of zero-advantage updates, illustrating STRIVE’s ability to prevent gradient starvation. (c) Gradient norm over training steps.

Model	VideoMME (wo sub)	VSI-Bench	PerceptionTest (mc)	TempCompass	VideoMMMU	MMVU (mc)	avg
GPT-4o [ 1 ]	71.9	34.0	-	75.1	61.2	75.4	-
Gemini-1.5-Pro [ 27 ]	75.0	48.8	-	70.5	-	-	-
VILA-1.5-8B [ 12 ]	-	28.9	-	-	20.8	-	-
LongVA-7B [ 34 ]	52.6	29.2	-	-	23.9	-	-
LLaVA-OV-7B [ 9 ]	58.2	32.4	57.1	69.5	-	64.7	-
Qwen2.5-VL-7B-SFT [ 4 ]	55.4	33.3	68.4	69.9	49.4	63.5	56.6

Table 1: Performance on six video understanding benchmarks. To ensure a fair comparison, models are evaluated with a maximum of 32 frames. Our proposed STRIVE framework outperforms the standard GRPO baseline across both Qwen2.5-VL-7B and Qwen3-VL-8B architectures. Results marked with * are evaluated using publicly available model weights.

(a) Variant Construction Strategies
Configuration	VideoMME	VSI-Bench	TempCompass	MMVU (mc)	Avg
GRPO Baseline	58.0	32.5	72.1	65.3	57.0
Deterministic Temporal	57.1	31.8	73.1	65.0	56.8
Stochastic	58.4	34.3	73.2	65.8	57.9
Importance-Based (Temporal Only)	59.0	34.2	72.8	66.2	58.1
Importance-Based (+ Spatial)	58.2	35.2	73.1	65.4	58.0

Table 2: Ablation Studies. We evaluate (a) the variant construction strategy, (b) the scoring mechanism used for importance-based grouping, (c) the trade-off between visual variants (

M

) and textual generations (

G

), and (d) the number of temporal segments (

K

). All ablation models are trained for 500 RL steps and evaluated on a subset of four core benchmarks.

Figure 4: Qualitative comparison between STRIVE and the standard GRPO baseline. (Top) STRIVE successfully isolates the relevant driving frames to correctly answer a question about a person’s mode of transport, whereas GRPO is distracted by irrelevant walking scenes. (Bottom) STRIVE accurately determines the strict temporal sequence of objects, demonstrating how question-guided variant construction improves fine-grained spatiotemporal reasoning and prevents reliance on flawed priors.

Video QA

Video question answering studies how AI systems use visual and audio evidence over time to answer questions about video.

See all 180 papers

FrameGRPO: Incentivizing Video Reasoning via Multi-Frame Rollouts and Temporal Credit
Oct 19, 2026Jiaxuan Sun, Biao Wang, Shuai LiRL for Video Reasoning Video Reasoning
Shanghai Jiao Tong University · Shanghai Innovation Institute · Tencent Hy
Find, Fix, Reason: Context Repair for Video Reasoning
Apr 17, 2026Haojian Huang, Chuanyu Qin, Yinchuan Li +1Video Reasoning RL for Video Reasoning
The Hong Kong University of Science and Technology (Guangzhou) · Knowin AI · Institute of Information Engineering, Chinese Academy of Sciences
Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs
Jun 16, 2026Chengwen Liu, Zhe Huang, Jisheng Dang +3Multimodal Reward Modeling Video Reasoning
School of Information Science and Engineering, Lanzhou University, Lanzhou, China · Beijing University of Posts and Telecommunications, Beijing, China · Cloud and AI BU, Huawei, Shenzhen, China +1

Oct 19, 2026J/K move · Enter open · S save

FrameGRPO: Incentivizing Video Reasoning via Multi-Frame Rollouts and Temporal Credit

Jiaxuan Sun, Biao Wang, Shuai Li

Shanghai Jiao Tong University · Shanghai Innovation Institute · Tencent Hy

RL for Video Reasoning Video Reasoning Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards Temporal Credit Assignment

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards, particularly Group Relative Policy Optimization (GRPO), has improved video reasoning in multimodal large language models. However, standard video GRPO varies only linguistic trajectories while conditioning every rollout in a group on the same sampled frames, which can produce homogeneous rewards and weaken group-relative learning when the shared visual evidence is insufficient or distracting. We propose FrameGRPO, a simple and effective reinforcement-learning framework that requires neither auxiliary networks nor changes to the model architecture. Specifically, FrameGRPO combines two lightweight components: (1) Multi-Frame Rollouts, which use only uniform temporal subsampling of a single maximal-frame decode to construct aligned views that diversify visual evidence and reasoning trajectories; and (2) Gated Temporal Efficiency Credit, which selectively reinforces a correct lower-ratio trajectory when all rollouts in a fixed full-ratio reference subset fail, without imposing an unconditional preference for fewer frames. Extensive experiments across diverse video understanding and reasoning benchmarks demonstrate robust overall improvements over standard GRPO, highlighting temporal-view diversification as an effective mechanism for exploration and credit assignment in video MLLMs.

Abstract

Video QA

Explore similar work

FrameGRPO: Incentivizing Video Reasoning via Multi-Frame Rollouts and Temporal Credit

Find, Fix, Reason: Context Repair for Video Reasoning

Reasoning as Intersection: Consensus-Frame Alignment for Visual Focus in Video-MLLMs