Human perception interprets video dynamics by constructing structured mental representations of entities, actions, and temporal boundaries before engaging in high-level reasoning. In contrast, current Video-LLMs typically rely on unstructured Chain-of-Thought (CoT), where crucial visual cues are obscured by verbose narration. While recent efforts ground reasoning in intermediate visual evidence, they rely on task-specific schemas that fail to generalize across diverse video understanding tasks. To address this, we introduce Structured Event Evidence, a unified, temporally ordered schema designed for general video reasoning. However, post-training a model to extract structured evidence while reasoning introduces a severe multi-objective optimization bottleneck, as candidate rollouts must simultaneously satisfy schema compliance, task accuracy, and reasoning budget. Standard reinforcement learning (RL) algorithms attempt to balance these objectives via static scalarization, enforcing a rigid accuracy-length trade-off across training and obscuring fine-grained reward signals. We resolve this by proposing Group-Relative Advantage Balancing (GRAB), which dynamically computes Pareto-optimal objective weights within each rollout group via a minimum-norm criterion over centered rewards. Operating strictly in reward space, GRAB incurs virtually zero computational overhead (<0.002% relative runtime to GRPO) and eliminates brittle weight tuning. Supported by our new PARSE-60K dataset (60K annotations across 35K videos) and a three-stage progressive curriculum, we train PARSE-4B. Across temporal grounding and video question-answering benchmarks, GRAB generates 13% to 39% fewer tokens while maintaining competitive accuracy with exhaustively hand-tuned static weights -- without requiring hyperparameter weight sweeps.
Figures & tables
Figure 2: Comparison of unstructured video CoT and our structured reasoning approach. Our method first extracts structured event evidence and then performs evidence-grounded reasoning over temporal dependencies, yielding clearer cues and more interpretable predictions. Video from ActivityNet-Captions ( Krishna et al., 2017 ) .
Figure 3: GRPO vs. GRAB on a rollout group. GRPO averages the four rewards, assigning identical scores and advantages to [1,1,0,0] and [1,0,0,1] . GRAB applies dynamic minimum-norm weights (Algorithm 1 ) to centered rewards before z -score normalization, correctly distinguishing response preferences.
Figure 4: How we build PARSE-60K. We first drop any video whose ID appears in an eval set. Stage 1 then cuts events, fills gaps, writes evidence, and judges quality. Stage 2 writes thinking traces and judges them again.
Model
Charades-TimeLens (R1@IoU)
ActivityNet-Captions (R1@IoU)
VideoMME
MLVU
@0.3
@0.5
@0.7
Len
@0.3
@0.5
@0.7
Len
Acc
Len
TR
Ego
Len
Baselines 1fps
Qwen3-VL-4B-Inst.* Bai et al. (2025a)
56.5
37.8
18.4
-
50.2
35.8
21.6
-
63.4
-
80.1
60.3
-
Qwen3-VL-4B-Think.* Bai et al. (2025a)
56.3
37.4
17.8
1439
47.8
31.7
18.8
1895
63.1
4016
79.5
59.0
5192
Ablations 1fps
w/o Evidence = 16
54.0
37.0
16.5
553
50.7
34.3
23.2
1158
60.8
2961
76.2
56.8
3128
Table 1: Ablation study at the 4B scale on temporal grounding and general video understanding. We remove evidence, structured thinking, or RL, and compare GRAB against GRPO, DAPO, GDPO, fixed reward weighting, and a Lagrangian length baseline. @ τ is Recall@1 at IoU threshold τ , VideoMME accuracy excludes subtitles, and TR and Ego are the Topic Reasoning and Ego Reasoning sub-tasks of MLVU. Len is the average number of generated tokens per response, so lower is better at comparable accuracy. A row ending in =8 or =16 gives the group size of its RL stage, and w/o RL has none. Entries marked “-” are not reported, since Qwen3-VL-4B-Instruct returns a bare answer whose length is not comparable with the reasoning rows.
Table 2: Experiments on the public video temporal understanding benchmarks. * means this model is reproduced under the same video settings of our model, otherwise, the results are from official reports with more video frames.
ETBench
VideoMME
MLVU
NExT-GQA
Metrics
RAR
ECA
RVQ
TVG
EPM
TAL
TEM
GVQ
Acc
TR (Acc)
Ego (Acc)
Acc
Acc
Acc
Acc
F1
F1
F1
Rec
Acc
Close-Source Models 2fps
GPT-4o ( Hurst et al., 2024 )
27.8
27.3
57.7
40.4
4.5
20.0
13.6
-
71.9
-
-
-
GPT-4v ( OpenAI, 2023 )
33.3
40.9
46.2
27.0
1.8
18.0
23.9
-
59.9
-
-
-
Open-Source Models
Table 3: Experiments on the public general video understanding benchmarks. TR and Ego mean Topic Reasoning and Ego Reasoning, respectively. Rows marked * are run by us at 1 fps, under the same video settings as PARSE-4B. The other rows are taken from official reports, which usually use more video frames.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Value
Lmax
hard maximum response length (tokens)
4096
Lbuffer
soft buffer below Lmax (tokens)
2500
Ltarget
Lmax−Lbuffer , start of the length penalty
1596
wacc
Fixed Weight (Accuracy), over (fmt, IoU, MC, len)
(1,2,2,1)
wdef
weight vector of the GRPO, DAPO and GDPO runs
(1,1,1,1)
Ltargetlag
length target of the Lagrangian baseline
2500
Appendix
Table 4: Reward and solver hyperparameters of the Stage-3 run. Weights are given over the four implemented rewards; since a prompt activates only one of the two task metrics, a group sees (wfmt,wtask,wlen) , which is (1,2,1) for Fixed Weight (Accuracy) and (1,1,1) for the default runs.
Figure 5: Distribution of task types in the RL training data. Temporal Grounding constitutes the largest share ( 53% ), providing precise IoU-based supervision, while Spatial VQA ( 21% ), Reasoning VQA ( 20% ) and Temporal VQA ( 6% ) broaden the training signal toward spatial, semantic and temporal understanding.
Model
TempCompass
EventBench
Overall
Event Order
Attr. Change
Overall
Temporal Reason.
Causal Reason.
Qwen3-VL-4B-Instruct
68.71
74.76
70.40
62.71
64.51
66.87
Qwen3-VL-4B-Thinking
68.60
73.20
69.00
60.44
62.57
65.22
PARSE-4B
70.64
76.96
73.85
63.48
65.42
69.47
Appendix
Table 5: Results on temporal and event reasoning benchmarks. TempCompass evaluates temporal perception; EventBench assesses event-level understanding including temporal and causal reasoning sub-tasks.
Model (8B)
VideoMME
CTL-mIoU
TempCompass
EventBench
8B-GRPO
64.81
43.73
65.79
64.87
8B-GRAB
66.98
46.81
66.23
66.10
Appendix
Table 6: GRAB vs. GRPO on the 8B model scale (LoRA fine-tuning). GRAB outperforms GRPO on all four benchmarks, so the margin is not confined to 4B. Both runs use half the RL data and a lower LoRA rank than the 4B experiments, so the absolute numbers sit below the 4B model and carry no scaling claim. CTL-mIoU is the mean IoU on Charades-TimeLens.
Group Size
GRAB (ms)
One Step (ms)
4
0.68
52,789
8
0.69
108,401
16
1.00
211,389
Appendix
Table 7: GRAB computational overhead. The weight computation is negligible ( <0.002% ) relative to the total training step time.
Model
F1
BLEU-4
ROUGE-L
CIDEr
Qwen3-VL-4B-Instruct
44.2
10.6
27.1
4.1
PARSE-4B
54.7
16.6
35.1
17.3
Appendix
Table 8: Evidence quality evaluation. PARSE-4B generates substantially higher-quality structured evidence compared to the base model.
Table 9: Static reward weights on Charades-TimeLens at G=16 , written as (wfmt,wtask,wlen) over the three objectives that can be live in a single group. Fixed Weight (Accuracy), GRPO and GRAB are the rows of the same name in Table 1 . The other two runs take the Fixed Weight (Accuracy) vector and set wlen to zero or to four times its tuned value, leaving wfmt and wtask alone, so the four static runs trace a single ratio. Output length falls monotonically in wlen/wtask across them, so the column orders the family.
Table 10: Spread across three random seeds on Charades-TimeLens. The means are the values reported in Table 1 , and the table adds the standard deviation over the seeds.
Table 11: Weights the solver returns, measured over the PARSE-4B run. The first three rows give the mean and the standard deviation of αq∗ across the groups where all three objectives are live; the means sum to one, and uniform weighting would put 1/3 on each. The fourth row is the value Section G.3 gives for two live objectives, measured on real groups rather than assumed. The last two rows compare the two runs checkpoint by checkpoint at the same training step, on Charades-TimeLens, with the range covering the checkpoints through step 100 .
Model
Overall
Halluc.
Logic.
Interp.
Qwen3-VL-4B-Thinking
4.00
4.0
4.25
3.75
PARSE-4B
4.41
4.0
4.50
4.75
Appendix
Table 12: Reasoning quality evaluation by Gemini-2.5-Pro. PARSE-4B produces more logical and interpretable reasoning chains.
Figure 6: Evidence Generation prompt used in the data curation pipeline to produce structured event evidence from videos.
Figure 7: Quality Judge prompt used in the data curation pipeline to validate generated evidence.
Figure 8: Thinking Generation prompt used in the data curation pipeline to produce evidence-grounded reasoning traces.
Figure 9: Prompts for Stage 1 evidence training.
Figure 10: Prompts for Stage 2 thinking warm-start. Orange parts mean these parts are different from the previous parts.
Video reasoning requires models to identify and verify temporally localized evidence within long video sequences. Recent Video Large Language Models (Video-LLMs) have shown promising reasoning abilities when aligned with reinforcement learning, yet existing approaches typically rely on outcome-based rewards that supervise only the final prediction. Such supervision provides limited guidance on how models should discover the relevant temporal evidence during intermediate reasoning. In this work, we propose TimeThink, a reinforcement learning framework that explicitly guides temporal evidence discovery in Video-LLMs. Our key idea is to treat temporal clue steps as the fundamental optimization primitive of video reasoning, where each reasoning step references a candidate time interval in the video. We introduce a step-wise temporal process reward that provides localized credit assignment for these clues and a joint process--outcome optimization objective that balances reasoning fidelity with task correctness. To enable scalable training, we construct TimeThink-RFT-20K, a dataset with automatically derived temporal evidence segments. Extensive experiments across video reasoning, temporal grounding, and general video understanding benchmarks show that TimeThink consistently improves both temporal localization and reasoning performance, achieving state-of-the-art results among open-source video RL models.
Handong Li, Longteng Guo, Zikang Liu +8
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.
Recent multimodal large language models (MLLMs) achieve strong results on video question answering, yet often fail to recover the temporal structure that makes a video coherent. They may merge neighboring events, miss single-event progression, or infer implausible relations across events. We study structured temporal understanding as the ability to recover event structure from visual evidence, including event boundaries, single-event progression, and multi-event relations. We argue that this gap arises from a mismatch between current post-training objectives and temporal reasoning, where caption-style supervision allows models to rely on language shortcuts rather than visual temporal grounding. To address this, we propose EventLens, a vision-centric reinforcement learning framework that learns video understanding through event-structure recovery. Instead of reconstructing captions, EventLens trains models to recover temporal structure under controlled perturbations, such as altered temporal granularity, reversed progression, and shuffled event order. We instantiate this framework with a three-level temporal ontology and derive three verifiable task families: event segmentation, progression discrimination, and multi-event relation reasoning. These tasks admit deterministic rewards, enabling task-specialized GRPO training without human preference labels or learned reward models. We further introduce multi-teacher on-policy distillation (MT-OPD) to consolidate specialized policies into a unified model. Experiments on Qwen3-VL backbones show consistent improvements on temporal grounding and event-sensitive reasoning benchmarks, while preserving general video-QA performance. Learning curves further demonstrate that EventLens achieves comparable downstream transfer with significantly reduced training cost compared to direct mixed RL optimization.
Video reasoning has advanced with large multimodal models (LMMs), yet their inference is often a single pass that returns an answer without verifying whether the reasoning is evidence-aligned. We introduce Reinforce to Learn, Elect to Reason (RLER), a dual paradigm that decouples learning to produce evidence from obtaining a reliable answer. In RLER-Training, we optimize the policy with group-relative reinforcement learning (RL) and 3 novel task-driven rewards: Frame-sensitive reward grounds reasoning on explicit key frames, Think-transparency reward shapes readable and parsable reasoning traces, and Anti-repetition reward boosts information density. These signals teach the model to emit structured, machine-checkable evidence and potentiate reasoning capabilities. In RLER-Inference, we apply a train-free orchestrator that generates a small set of diverse candidates, parses their answers and cited frames, scores them by evidence consistency, confidence, transparency, and non-redundancy, and then performs a robust evidence-weighted election. This closes the loop between producing and using evidence, improving reliability and interpretability without enlarging the model. We comprehensively evaluate RLER against various open-source and RL-based LMMs on 8 representative benchmarks. RLER achieves state of the art across all benchmarks and delivers an average improvement of 6.3% over base models, while using on average 3.1 candidates per question, indicating a favorable balance between compute and quality. The results support a simple thesis: making evidence explicit during learning and electing by evidence during inference is a robust path to trustworthy video reasoning.
Songyuan Yang, Weijiang Yu, Jilin Ma +5
National University of Defense Technology · Sun Yat-sen University