Recent multimodal large language models have advanced video understanding, but existing benchmarks mainly evaluate whether models can interpret observed visual content. In real-world scenarios, assistants must additionally anticipate what assistance is needed and decide when to provide it. To address this gap, we introduce \textsc{P3IBench}, a benchmark for proactive video understanding in real-life streaming scenarios. It evaluates three complementary capabilities: daily perception, risk prediction and intervention. We benchmark 12 representative open-source and proprietary video-language models, including streaming and adapted non-streaming systems. Our evaluation shows that current models struggle to make well-calibrated response decisions. Finally, we construct additional proactive training data and train \textsc{P3I-Assistant-4B}, achieving improvements on several tasks, though gains remain uneven across capabilities. Together, these findings show that effective proactive assistance requires not only visual understanding, but also precise response timing and appropriate restraint.
Figures & tables
Figure 1: Overview of P 3 IBench and model performance. (a) Three complementary proactive capabilities: Perception , recognizing relevant events as they occur; Prediction , anticipating risks before they materialize; and Intervention , proactively providing assistance or corrections when needed. (b) Overall ranking of evaluated models by average PRS-B (higher is better), together with task-wise PRS-B profiles across five subtasks. Detailed numerical results are reported in Table 2 .
Benchmark
Scale
Scenarios
Capabilities
Quality Control
#Videos
#Samples
Daily
Traffic
Crime
Pe
Pr
I
Anno
HR
StreamingBench
900
4,500
✓
✓
✓
LLM+Human
✓
OVO-Bench
644
2,814
✓
✓
✓
LLM+Human
✓
ProactiveVideoQA
1,377
1,427
✓
✓
✓
LLM+Rule+Human
ODV-Bench
1,190
6,348
✓
✓
✓
LLM+Rule+Human
✓
RIVER
1,067
4,278
✓
✓
LLM+Rule
✓
Table 1: Comparison of representative benchmarks for streaming video understanding and proactive assistance. Pe, Pr, and I indicate perception, prediction of not-yet-observed future events or risks, and proactive intervention, respectively. HR denotes human review.
Figure 2: Benchmark Statistics. (Left) Task distribution. (Top right) Video duration distribution across tasks. (Bottom right) Distribution of GT trigger positions across categories, normalized by video duration; dashed lines divide the video timeline into early, middle, and late thirds. For readability, the Risk Alert and Event Intervention scatterplots show GT events from videos up to 112s and 190s, respectively, covering 90.2% and 90.7% of their GT events.
Perception
Risk Prediction
Intervention
Overall Average
Model
Event
Object
Accident
Crime
PRS-U
PRS-B
PRS-U
PRS-B
PRS-U
PRS-B
PRS-U
PRS-B
PRS-U
PRS-B
PRS-U
PRS-B
Offline Models adapted for Proactive Tasks
GLM-4.1V-9B-Base
0.89
0.96
26.00
26.49
2.11
2.80
18.15
19.80
0.93
0.93
9.62
10.20
InternVL3.5-8B
5.43
9.06
28.02
28.08
23.19
38.79
43.81
52.15
5.89
5.89
21.27
26.79
Qwen2.5-VL-3B-Instruct
3.44
8.27
22.38
28.44
14.91
21.27
11.25
24.01
1.49
1.49
10.69
16.70
Table 2: Model Performance in P 3 IBench . PRS-U and PRS-B denote the proactive response score metrics evaluated in unidirectional and bidirectional setups, respectively. In each column, the best performance is highlighted in bold while the runner-up is underlined .
Figure 3: Response frequency distribution of different models across tasks. The response frequency is measured as the number of model responses within the ground-truth window of each target response, grouped into four categories: 0, 1, 2, and ≥3 .
Figure 4: Distribution of the first-response latency across different models and tasks. GT windows with no model response are excluded from the analysis. Within each distribution, the thin line, thick line, and circle denote the 10–90% percentile range, interquartile range, and median, respectively.
Figure 5: PRS-U of Qwen3-VL-4B-Instruct and JoyVL across different video durations (left) and ground-truth event times (right). Videos are separately binned by each dimension. The top row reports the median PRS-U among non-zero responses, while the bottom row reports the fraction of zero scores. Videos without ground-truth responses are excluded from the video-duration analysis.
Table 8
Model / Stage
Perception
Risk Prediction
Intervention
Event
Object
Accident
Crime
PRS-B Avg.
PRS-U
PRS-B
PRS-U
PRS-B
PRS-U
PRS-B
PRS-U
PRS-B
PRS-U
PRS-B
Qwen3-VL-4B-Instruct
14.84
21.35
32.67
33.82
30.93
41.78
35.29
37.03
9.13
9.13
28.62
+ SFT
5.84
18.10
30.72
48.62
2.96
61.78
16.84
47.04
8.05
8.05
36.72
+ RL (P 3 I-Assistant-4B)
7.47
22.77
32.35
40.03
32.88
87.59
29.67
51.21
9.02
9.02
42.12
JoyVL
9.94
14.78
29.96
36.40
19.66
50.62
43.15
52.19
4.10
4.10
31.62
Table 5: Effect of post-training on Qwen3-VL-4B-Instruct.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Atomic Event Monitoring. The model is asked to report the second occurrence of opening a door. The selected frames show a local excerpt around the target event; preceding occurrences are outside this excerpt.
Figure 7: Event Type Monitoring. The query covers throwing something into a bin without restricting the discarded object. The model must output 4 for the fourth matching occurrence, annotated at 209.546–211.046 s.
Figure 8: Compositional Process Monitoring. The queried process is folding and setting aside a T-shirt on the desk. Its annotated start and completion are 10.182 s and 20.759 s, respectively. The required responses are start and end .
Figure 9: Object Monitoring. The specified object is a pink car. The annotated disappearance times are 1.967 s and 7.5 s, each requiring Warning . The task requires maintaining the target object’s identity across the video and distinguishing separate disappearance events.
Figure 10: Accident Warning. The model monitors a driving scene for an imminent collision risk, with a GT warning target at 18.333 s. The 18.800 s and 20.0 s frames illustrate subsequent developments and do not provide evidence available at the annotated warning time.
Figure 11: Crime Alert. A surveillance-video example from UCF-Crime has a GT alert target at 6.0 s, with Warning as the required output. The surrounding frames provide context for the developing incident; later frames are not used to justify an earlier response.
Figure 12: Event Intervention. As the user approaches checkout without having bought tomatoes, the model should remind them of the missing item. The annotated response window is 355–365 s. The reference answer identifies both the attempted checkout and the unfulfilled shopping requirement.
Configuration
Value
Initialization
Qwen3-VL-4B-Instruct
Video sampling rate
1 FPS
Learning rate
1×10−5
Training epochs / steps
1 / 418
GPUs
4
Per-device batch size
1
Appendix
Table 6: Configuration for supervised fine-tuning.
Configuration
Value
Initialization
Final SFT checkpoint at step 418
Optimization algorithm
Group Relative Policy Optimization (GRPO)
Learning rate
5×10−7
Training epochs / steps
Approximately 1 / 143
Training GPUs / rollout GPUs
3 / 1
Per-device batch size
1
Appendix
Table 7: Configuration for reinforcement learning.
Figure 13: Failure cases from model response traces. (a) Repeated outputs for a single requested event occurrence. (b) A warning recorded 21.333 s after the annotated warning target. (c) A repeated floor-arrival notice that omits the required correction. Instructions and GT descriptions are abridged. Later frames provide context and do not justify earlier responses.
Perception
Risk Prediction
Intervention
Overall Average
Model
Event
Object
Accident
Crime
PRS-U’
PRS-B’
PRS-U’
PRS-B’
PRS-U’
PRS-B’
PRS-U’
PRS-B’
PRS-U’
PRS-B’
PRS-U’
PRS-B’
Offline Models adapted for Proactive Tasks
GLM-4.1V-9B-Base
0.89
0.96
26.00
26.49
7.38
8.08
21.76
23.41
1.32
1.32
11.47
12.05
InternVL3.5-8B
28.06
33.03
28.02
28.08
30.71
46.83
48.10
58.63
37.06
37.06
34.39
40.73
Qwen2.5-VL-3B-Instruct
31.70
38.52
63.20
69.27
63.74
71.39
46.25
60.82
16.98
16.98
44.37
51.40
Appendix
Table 8: Best-reply model performance in P 3 IBench . PRS-U’ and PRS-B’ retain the original matching and temporal decay rules but score each target by its highest-scoring matched reply, without dividing by the number of matched replies. The best and second-best available scores in each column are bold and underlined , respectively.
Streaming video understanding requires models to process continuous multimodal input while maintaining temporal context. Existing evaluations are predominantly reactive: they query a model at a selected timestamp and therefore do not assess when it should respond. Proactive interaction instead requires monitoring a standing request, responding within an appropriate interval after the target event, and otherwise remaining silent. We introduce LiveProBench, which evaluates models at one-second stream intervals without an explicit response cue. Its six subtasks vary trigger ambiguity and timing tolerance. Event Sensitivity geometrically combines response and silence rates on the same recording; four window-based subtasks distinguish early, in-window, and missed responses; and Duplicate Counting penalizes omissions and repetitions. Premature responses outnumber missed responses for half of the evaluated models, revealing a substantial gap in the temporal decision-making required for human-like interaction.
Kaixuan Du, Xin Wan, Hang Zhang +4
School of Automation Science and Electrical Engineering, Beihang University · Qwen Business Unit of Alibaba
Omni-proactive streaming video understanding, i.e., autonomously deciding when to speak and what to say from continuous audio-visual streams, is an emerging capability of omni-modal large language models. Existing benchmarks fall short in three key aspects: they rely primarily on visual signals, adopt polling or fixed-timestamp protocols instead of true proactive evaluation, and cover only a limited range of tasks, preventing reliable assessment and differentiation of omni-proactive streaming models. We present OmniPro, the first benchmark to jointly evaluate omni-modal perception, proactive responding, and diverse video understanding tasks. It comprises 2,700 human-verified samples spanning 9 sub-tasks and 3 cognitive levels, covering 6 basic video understanding capabilities. Notably, 84% of samples require audio signals (speech or non-speech), and each sample is annotated with modality-isolation labels to enable fine-grained multimodal analysis. We further introduce a dual-mode evaluation protocol: Probe mode assesses content understanding by querying the model before and after each ground-truth trigger, while Online mode evaluates full proactive ability by requiring models to autonomously decide when to respond in streaming input. Evaluating 11 representative models reveals three key findings: (1) audio provides consistent gains but with highly variable utilization across models, (2) performance degrades significantly over time, indicating limited long-horizon robustness, and (3) non-speech audio perception remains the weakest dimension.
Ruixiang Zhao, Jie Yang, Zijie Xin +4
Renmin University of China · WeChat Vision, Tencent Inc.
Recent advances in Streaming Video Understanding has enabled a new interaction paradigm where models respond proactively to user queries. Current proactive VideoLLMs rely on per-frame triggering decision making, which suffers from an efficiency-accuracy dilemma. We propose Em-Garde, a novel framework that decouples semantic understanding from streaming perception. At query time, the Instruction-Guided Proposal Parser transforms user queries into structured, perceptually grounded visual proposals; during streaming, a Lightweight Proposal Matching Module performs efficient embedding-based matching to trigger responses. Experiments on StreamingBench and OVO-Bench demonstrate consistent improvements over prior models in proactive response accuracy and efficiency, validating an effective solution for proactive video understanding under strict computational constraints.
Yikai Zheng, Xin Ding, Yifan Yang +6
Institute for AI Industry Research (AIR), Tsinghua University · University of Science and Technology of China · 4Microsoft Research Asia +1