Recent multimodal large language models (MLLMs) achieve strong results on video question answering, yet often fail to recover the temporal structure that makes a video coherent. They may merge neighboring events, miss single-event progression, or infer implausible relations across events. We study structured temporal understanding as the ability to recover event structure from visual evidence, including event boundaries, single-event progression, and multi-event relations. We argue that this gap arises from a mismatch between current post-training objectives and temporal reasoning, where caption-style supervision allows models to rely on language shortcuts rather than visual temporal grounding. To address this, we propose EventLens, a vision-centric reinforcement learning framework that learns video understanding through event-structure recovery. Instead of reconstructing captions, EventLens trains models to recover temporal structure under controlled perturbations, such as altered temporal granularity, reversed progression, and shuffled event order. We instantiate this framework with a three-level temporal ontology and derive three verifiable task families: event segmentation, progression discrimination, and multi-event relation reasoning. These tasks admit deterministic rewards, enabling task-specialized GRPO training without human preference labels or learned reward models. We further introduce multi-teacher on-policy distillation (MT-OPD) to consolidate specialized policies into a unified model. Experiments on Qwen3-VL backbones show consistent improvements on temporal grounding and event-sensitive reasoning benchmarks, while preserving general video-QA performance. Learning curves further demonstrate that EventLens achieves comparable downstream transfer with significantly reduced training cost compared to direct mixed RL optimization.
Figures & tables
Figure 1: Motivation and task view of unified event-structure recovery. Existing video MLLMs often suffer from boundary ambiguity, progression ambiguity, and relation ambiguity when they must recover explicit temporal structure. We address these bottlenecks with three event-structure recovery tasks: event-boundary recovery, single-event progression discrimination, and multi-event relation reasoning through next-event prediction, fill-in, and narrative reordering.
Figure 2: Overview of EventLens. EventLens adopts a two-stage training pipeline for event-structure reinforcement learning. In Stage I, deterministic rewards train task-specialized GRPO teachers for event-boundary localization, single-event progression discrimination, and multi-event relation reasoning. In Stage II, MT-OPD routes each student trajectory to the corresponding teacher and distills dense token-level teacher logits into one deployable student. The teacher is selected by recovery family, so each student trajectory is supervised by the teacher trained with the matching verifier.
Figure 3: Scene-first construction of event structure for a curling news report. (a) Sampled frames and candidate shots. (b) KEEP retains a shot as one event, MERGE combines consecutive shots into one event, and SPLIT introduces event boundaries within a shot. (c) L1 phases contain L2 events, which contain L3 actions. Frames come from the historical report illustration; boundaries and event assignments are schematic.
Model
Temporal Grounding
Complex Reasoning
QV Highlights
Activity Net
Charades STA
Video Holmes
Video MMMU
Video TT
E.T. Bench
Qwen3-VL-4B Backbone
Qwen3-VL-4B [ 4 ]
54.31
41.99
43.64
39.96
55.44
36.90
34.55
+ SFT-Desc
55.63
42.18
43.52
41.26
55.00
41.70
32.57
+ SFT-DVC
57.13
42.36
40.18
41.97
56.22
41.20
37.78
+ EventLens
58.50
45.76
47.99
40.28
55.56
38.80
41.47
Table 1: Temporal grounding and complex reasoning under the shared 256-frame protocol. Temporal grounding uses mIoU; reasoning columns use official benchmark scores. Bold marks the best score within each backbone group.
Model
General Video QA
Long Video QA
MV Bench
Temp Compass
AoT Bench
Vino Ground
Video MME
MLVU
LongVideo Bench
Qwen3-VL-4B Backbone
Qwen3-VL-4B [ 4 ]
62.55
74.96
53.64
37.00
65.00
56.77
60.70
+ SFT-Desc
64.62
76.35
54.39
41.40
64.30
56.77
61.30
+ SFT-DVC
64.70
75.93
54.12
40.20
65.00
58.17
62.10
+ EventLens
63.90
75.12
53.99
45.00
65.30
58.37
61.60
Table 2: General and long-video QA under the shared 256-frame protocol. Columns use official benchmark scores. Bold marks the best score within each backbone group.
Model
Charades STA
QV Highlights
Vino Ground
E.T. Bench
MV Bench
LongVideo Bench
Qwen2.5-VL-7B [ 5 ]
38.34
27.66
29.40
30.96
63.38
59.20
Video-R1-7B [ 16 ]
28.80
22.33
31.20
29.50
60.27
56.20
EventLens-7B
42.70
43.79
41.00
35.85
64.00
58.60
Table 3: Comparison with existing video post-training methods on Qwen2.5-VL-7B. Charades-STA and QVHighlights report mIoU; VinoGround reports the Text score and E.T. Bench the average score.
Variant
TimeLens
E.T. Bench
Video MME
Vino Ground
Video TT
Holmes
MLVU
LongVideo Bench
avg mIoU
Avg
Avg
Score
Score
Acc
Avg
Avg
Base
46.65
34.55
65.00
37.00
36.90
39.96
56.77
60.70
R1 teacher
50.84
40.34
65.30
39.20
38.20
40.12
58.76
61.10
R2 teacher
50.76
40.94
65.40
45.40
39.50
40.61
59.76
61.80
R3 teacher
50.96
39.87
65.00
37.80
37.20
39.19
57.57
61.60
Single-teacher oracle
50.96
40.94
65.40
45.40
39.50
40.61
59.76
61.80
Table 4: Teacher consolidation and task-composition ablation on Qwen3-VL-4B. R1–R3 denote segmentation, progression, and relation teachers. Full MT-OPD is EventLens with all three teachers; the single-teacher oracle selects the best individual score per metric.
Variant
R1 Seg.
R2 Prog.
R3 Rel.
Base
0.399
0.550
0.555
R1+R2
0.589
0.680
0.574
R1+R3
0.592
0.590
0.708
R2+R3
0.444
0.700
0.705
Full MT-OPD
0.588
0.685
0.706
Table 5: Direct validation of recovery capabilities on Qwen3-VL-4B. R1–R3 denote segmentation, progression, and relation scores.
Figure 4: Wall-clock efficiency of EventLens versus direct Mixed-GRPO on four representative transfer metrics. The curves track MT-OPD student optimization and direct Mixed-GRPO training.
Training annotations
VinoGround Group
MVBench
LongVideo Bench
Charades mIoU
QVHighlights mIoU
E.T. Bench Avg.
Instruction base
37.00
62.55
60.70
43.64
54.31
34.55
Clean
39.00
63.38
61.20
45.70
58.16
38.20
10% jitter
38.60
63.20
61.30
45.61
57.99
37.92
20% jitter
38.60
63.40
61.00
45.42
57.58
37.71
Table 6: Robustness to timestamp perturbation in event supervision on Qwen3-VL-4B. Training uses 100 GRPO steps and 2,000 samples per recovery family; percentages denote perturbed chains. VinoGround reports the Group score.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Recovery view
Constructed recovery task
Deterministic target
Event-boundary recovery
Recover phase-, event-, or action-level time spans requested by the prompt.
Event intervals; balanced by level and domain
Single-event progression
Decide whether an event-level clip is played forward or backward.
Direction label; balanced by playback direction and domain
Multi-event relation
Recover the next event, a missing middle event, or the order of shuffled clips.
Choice or ordering; balanced by relation subtask and domain
Appendix
Table 7: Event-structure recovery views used for post-training.
Figure 5: Domain composition of the input videos used to construct proxy-task instances.
\fnum@figure : Additional E.T. Bench qualitative cases covering repeated action localization, temporal event matching, and grounded video QA. Across the three settings, the base model often recognizes the coarse visual category or plausible answer but misaligns it with the relevant temporal evidence, while EventLens with MT-OPD preserves finer event boundaries and support intervals.
\fnum@figure : Additional wall-clock transfer curves complementing the four representative metrics in Fig. 4 .
Understanding causal event relationships and achieving fine-grained temporal grounding in videos remain challenging for vision-language models (VLMs). We propose TEMPURA (Temporal Event Masked Prediction and Understanding for Reasoning in Action), a two-stage training framework that enhances the video temporal understanding of VLMs. Inspired by infilling techniques in language modeling, TEMPURA first performs masked event prediction, learning to reconstruct missing events and generate step-by-step causal explanations from dense event annotations. It then learns video segmentation and dense captioning, decomposing videos into non-overlapping events with detailed, timestamp-aligned descriptions. We train TEMPURA on VER, our large-scale dataset of 500K videos annotated with temporally aligned event descriptions and structured reasoning steps. Experiments on video temporal grounding and highlight detection benchmarks show that TEMPURA substantially improves strong base VLMs across model families and scales, confirming that combining event-level reasoning with fine-grained temporal segmentation is an effective recipe for video temporal understanding.
Jen-Hao Cheng, Yi-Hao Peng, Huapeng Zhou +11
University of Washington · Carnegie Mellon University · National Yang Ming Chiao Tung University +1
Human perception interprets video dynamics by constructing structured mental representations of entities, actions, and temporal boundaries before engaging in high-level reasoning. In contrast, current Video-LLMs typically rely on unstructured Chain-of-Thought (CoT), where crucial visual cues are obscured by verbose narration. While recent efforts ground reasoning in intermediate visual evidence, they rely on task-specific schemas that fail to generalize across diverse video understanding tasks. To address this, we introduce Structured Event Evidence, a unified, temporally ordered schema designed for general video reasoning. However, post-training a model to extract structured evidence while reasoning introduces a severe multi-objective optimization bottleneck, as candidate rollouts must simultaneously satisfy schema compliance, task accuracy, and reasoning budget. Standard reinforcement learning (RL) algorithms attempt to balance these objectives via static scalarization, enforcing a rigid accuracy-length trade-off across training and obscuring fine-grained reward signals. We resolve this by proposing Group-Relative Advantage Balancing (GRAB), which dynamically computes Pareto-optimal objective weights within each rollout group via a minimum-norm criterion over centered rewards. Operating strictly in reward space, GRAB incurs virtually zero computational overhead (<0.002% relative runtime to GRPO) and eliminates brittle weight tuning. Supported by our new PARSE-60K dataset (60K annotations across 35K videos) and a three-stage progressive curriculum, we train PARSE-4B. Across temporal grounding and video question-answering benchmarks, GRAB generates 13% to 39% fewer tokens while maintaining competitive accuracy with exhaustively hand-tuned static weights -- without requiring hyperparameter weight sweeps.
Zinuo Li, Yongxin Guo, Jun Liu +6
University of Western Australia · Tencent Youtu Lab · CUHKSZ +2
Video multimodal large language models (MLLMs) can describe what happens in a video, but rarely identify when the supporting evidence occurs. We study generalist video temporal grounding, in which one model predicts a variable-cardinality set of evidence intervals across video lengths, domains, query forms, and viewpoints. Existing training strategies are misaligned with this set-valued task: long-video labels often rely on brittle one-pass annotation, while reinforcement-learning rewards either fail to distinguish non-overlapping predictions or require fragile segment matching. TimeLens2 treats temporal evidence as an interval set throughout supervision and optimization. TimeLens2-93K constructs reliable multi-span supervision through caption-derived proposals, independent localization, cross-agent consensus, semantic verification, and boundary refinement. Our temporal Wasserstein reward computes exact one-dimensional W1 between uniform distributions over merged interval supports, providing dense, matching-free feedback under unequal cardinalities and equivalent fragmentation; temporal IoU complements it with precise-overlap feedback. Across seven benchmarks, TimeLens2-2B outperforms all size-matched baselines on every benchmark, while the 4B and 8B variants achieve state-of-the-art performance, surpassing open-source models with up to 397B parameters. The 2B, 4B, and 8B variants improve over their Qwen3-VL backbones by 14.2, 13.0, and 18.1 mIoU points, respectively.
Yuhan Zhu, Changlian Ma, Xiangyu Zeng +12
Nanjing University · Shanghai AI Laboratory · Shanghai Jiao Tong University +3