cs.LGOct 15, 2026
SaveFrom Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios
Organizations: Fudan University · The Chinese University of Hong Kong · Shanghai Innovation Institute
Abstract
Recent multimodal large language models have advanced video understanding, but existing benchmarks mainly evaluate whether models can interpret observed visual content. In real-world scenarios, assistants must additionally anticipate what assistance is needed and decide when to provide it. To address this gap, we introduce \textsc{PIBench}, a benchmark for proactive video understanding in real-life streaming scenarios. It evaluates three complementary capabilities: daily perception, risk prediction and intervention. We benchmark 12 representative open-source and proprietary video-language models, including streaming and adapted non-streaming systems. Our evaluation shows that current models struggle to make well-calibrated response decisions. Finally, we construct additional proactive training data and train \textsc{PI-Assistant-4B}, achieving improvements on several tasks, though gains remain uneven across capabilities. Together, these findings show that effective proactive assistance requires not only visual understanding, but also precise response timing and appropriate restraint.
Figures & tables
Figure 1: Overview of P 3 IBench and model performance. (a) Three complementary proactive capabilities: Perception , recognizing relevant events as they occur; Prediction , anticipating risks before they materialize; and Intervention , proactively providing assistance or corrections when needed. (b) Overall ranking of evaluated models by average PRS-B (higher is better), together with task-wise PRS-B profiles across five subtasks. Detailed numerical results are reported in Table 2 .
| Benchmark | Scale | Scenarios | Capabilities | Quality Control | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| #Videos | #Samples | Daily | Traffic | Crime | Pe | Pr | I | Anno | HR | |
| StreamingBench | 900 | 4,500 | ✓ | ✓ | ✓ | LLM+Human | ✓ | |||
| OVO-Bench | 644 | 2,814 | ✓ | ✓ | ✓ | LLM+Human | ✓ | |||
| ProactiveVideoQA | 1,377 | 1,427 | ✓ | ✓ | ✓ | LLM+Rule+Human | ||||
| ODV-Bench | 1,190 | 6,348 | ✓ | ✓ | ✓ | LLM+Rule+Human | ✓ | |||
| RIVER | 1,067 | 4,278 | ✓ | ✓ | LLM+Rule | ✓ | ||||
Table 1: Comparison of representative benchmarks for streaming video understanding and proactive assistance. Pe, Pr, and I indicate perception, prediction of not-yet-observed future events or risks, and proactive intervention, respectively. HR denotes human review.
Figure 2: Benchmark Statistics. (Left) Task distribution. (Top right) Video duration distribution across tasks. (Bottom right) Distribution of GT trigger positions across categories, normalized by video duration; dashed lines divide the video timeline into early, middle, and late thirds. For readability, the Risk Alert and Event Intervention scatterplots show GT events from videos up to 112s and 190s, respectively, covering 90.2% and 90.7% of their GT events.
| Perception | Risk Prediction | Intervention | Overall Average | |||||||||
| Model | Event | Object | Accident | Crime | ||||||||
| PRS-U | PRS-B | PRS-U | PRS-B | PRS-U | PRS-B | PRS-U | PRS-B | PRS-U | PRS-B | PRS-U | PRS-B | |
| Offline Models adapted for Proactive Tasks | ||||||||||||
| GLM-4.1V-9B-Base | 0.89 | 0.96 | 26.00 | 26.49 | 2.11 | 2.80 | 18.15 | 19.80 | 0.93 | 0.93 | 9.62 | 10.20 |
| InternVL3.5-8B | 5.43 | 9.06 | 28.02 | 28.08 | 23.19 | 38.79 | 43.81 | 52.15 | 5.89 | 5.89 | 21.27 | 26.79 |
| Qwen2.5-VL-3B-Instruct | 3.44 | 8.27 | 22.38 | 28.44 | 14.91 | 21.27 | 11.25 | 24.01 | 1.49 | 1.49 | 10.69 | 16.70 |
Table 2: Model Performance in P IBench . PRS-U and PRS-B denote the proactive response score metrics evaluated in unidirectional and bidirectional setups, respectively. In each column, the best performance is highlighted in bold while the runner-up is underlined .
Figure 3: Response frequency distribution of different models across tasks. The response frequency is measured as the number of model responses within the ground-truth window of each target response, grouped into four categories: 0, 1, 2, and .
Figure 4: Distribution of the first-response latency across different models and tasks. GT windows with no model response are excluded from the analysis. Within each distribution, the thin line, thick line, and circle denote the 10–90% percentile range, interquartile range, and median, respectively.
Figure 5: PRS-U of Qwen3-VL-4B-Instruct and JoyVL across different video durations (left) and ground-truth event times (right). Videos are separately binned by each dimension. The top row reports the median PRS-U among non-zero responses, while the bottom row reports the fraction of zero scores. Videos without ground-truth responses are excluded from the video-duration analysis.
| Model / Stage | Perception | Risk Prediction | Intervention | ||||||||
| Event | Object | Accident | Crime | PRS-B Avg. | |||||||
| PRS-U | PRS-B | PRS-U | PRS-B | PRS-U | PRS-B | PRS-U | PRS-B | PRS-U | PRS-B | ||
| Qwen3-VL-4B-Instruct | 14.84 | 21.35 | 32.67 | 33.82 | 30.93 | 41.78 | 35.29 | 37.03 | 9.13 | 9.13 | 28.62 |
| + SFT | 5.84 | 18.10 | 30.72 | 48.62 | 2.96 | 61.78 | 16.84 | 47.04 | 8.05 | 8.05 | 36.72 |
| + RL (P 3 I-Assistant-4B) | 7.47 | 22.77 | 32.35 | 40.03 | 32.88 | 87.59 | 29.67 | 51.21 | 9.02 | 9.02 | 42.12 |
| JoyVL | 9.94 | 14.78 | 29.96 | 36.40 | 19.66 | 50.62 | 43.15 | 52.19 | 4.10 | 4.10 | 31.62 |
Table 5: Effect of post-training on Qwen3-VL-4B-Instruct.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Atomic Event Monitoring. The model is asked to report the second occurrence of opening a door. The selected frames show a local excerpt around the target event; preceding occurrences are outside this excerpt.
Figure 7: Event Type Monitoring. The query covers throwing something into a bin without restricting the discarded object. The model must output 4 for the fourth matching occurrence, annotated at 209.546–211.046 s.
Figure 8: Compositional Process Monitoring. The queried process is folding and setting aside a T-shirt on the desk. Its annotated start and completion are 10.182 s and 20.759 s, respectively. The required responses are start and end .
Figure 9: Object Monitoring. The specified object is a pink car. The annotated disappearance times are 1.967 s and 7.5 s, each requiring Warning . The task requires maintaining the target object’s identity across the video and distinguishing separate disappearance events.
Figure 10: Accident Warning. The model monitors a driving scene for an imminent collision risk, with a GT warning target at 18.333 s. The 18.800 s and 20.0 s frames illustrate subsequent developments and do not provide evidence available at the annotated warning time.
Figure 11: Crime Alert. A surveillance-video example from UCF-Crime has a GT alert target at 6.0 s, with Warning as the required output. The surrounding frames provide context for the developing incident; later frames are not used to justify an earlier response.
Figure 12: Event Intervention. As the user approaches checkout without having bought tomatoes, the model should remind them of the missing item. The annotated response window is 355–365 s. The reference answer identifies both the attempted checkout and the unfulfilled shopping requirement.
| Configuration | Value |
|---|---|
| Initialization | Qwen3-VL-4B-Instruct |
| Video sampling rate | 1 FPS |
| Learning rate | |
| Training epochs / steps | 1 / 418 |
| GPUs | 4 |
| Per-device batch size | 1 |
Table 6: Configuration for supervised fine-tuning.
| Configuration | Value |
|---|---|
| Initialization | Final SFT checkpoint at step 418 |
| Optimization algorithm | Group Relative Policy Optimization (GRPO) |
| Learning rate | |
| Training epochs / steps | Approximately 1 / 143 |
| Training GPUs / rollout GPUs | 3 / 1 |
| Per-device batch size | 1 |
Table 7: Configuration for reinforcement learning.
Figure 13: Failure cases from model response traces. (a) Repeated outputs for a single requested event occurrence. (b) A warning recorded 21.333 s after the annotated warning target. (c) A repeated floor-arrival notice that omits the required correction. Instructions and GT descriptions are abridged. Later frames provide context and do not justify earlier responses.
| Perception | Risk Prediction | Intervention | Overall Average | |||||||||
| Model | Event | Object | Accident | Crime | ||||||||
| PRS-U’ | PRS-B’ | PRS-U’ | PRS-B’ | PRS-U’ | PRS-B’ | PRS-U’ | PRS-B’ | PRS-U’ | PRS-B’ | PRS-U’ | PRS-B’ | |
| Offline Models adapted for Proactive Tasks | ||||||||||||
| GLM-4.1V-9B-Base | 0.89 | 0.96 | 26.00 | 26.49 | 7.38 | 8.08 | 21.76 | 23.41 | 1.32 | 1.32 | 11.47 | 12.05 |
| InternVL3.5-8B | 28.06 | 33.03 | 28.02 | 28.08 | 30.71 | 46.83 | 48.10 | 58.63 | 37.06 | 37.06 | 34.39 | 40.73 |
| Qwen2.5-VL-3B-Instruct | 31.70 | 38.52 | 63.20 | 69.27 | 63.74 | 71.39 | 46.25 | 60.82 | 16.98 | 16.98 | 44.37 | 51.40 |
Table 8: Best-reply model performance in P IBench . PRS-U’ and PRS-B’ retain the original matching and temporal decay rules but score each target by its highest-scoring matched reply, without dividing by the number of matched replies. The best and second-best available scores in each column are bold and underlined , respectively.
More in this topic
Video-Language Model Evaluation
Video-language model evaluation measures whether multimodal systems understand, ground, and reliably use visual and audio evidence over time.
See all 102 papers