Recent vision language models (VLMs) have achieved strong progress in video understanding. However, most existing video QA research and benchmarks still follow an offline, single-round paradigm, overlooking realistic interactions where users may interrupt the model during answer generation. To address this gap, we formulate the task of Online Video Question Answering under Interruption and introduce OVIBench, the first standardized benchmark for evaluating VLMs in this setting. OVIBench categorizes interruptions into three types: Cancellation, False Trigger, Correction and supports both open-ended and multiple-choice evaluations. To enable large-scale and reproducible testing, we develop an offline simulation protocol that reproduces interruption during generation under a unified temporal setup, together with a multi-dimensional metric suite for assessing interruption understanding and response generation. Experiments demonstrate that OVIBench effectively distinguishes models' interruption-handling abilities, especially in following correction requests. Finally, we construct a train set OVI-Train for interruption-aware fine-tuning. Models fine-tuned on this dataset achieve significant gains on OVIBench, validating the effectiveness of our benchmark and data design. OVIBench, OVI-Train, and the evaluation code will be released.
Figures & tables
Figure 1: Illustration of three interruption types in online video question answering: Cancellation, False Trigger, and Correction. In Online interactions, users may terminate the response, provide irrelevant signals, or introduce semantic corrections during generation.
Figure 2: (a) Distribution of video sources across five public datasets during data collection. (b) Sample size distribution of the benchmark under open-ended and multiple-choice evaluation settings. For each dataset, three bars are shown from left to right, corresponding to the False Trigger, Cancellation, and Correction interruption types.
Figure 3: Overview of OVIBench. (a) Benchmark data construction pipeline, including scene segmentation, VLM-based question generation and interruption insertion for both open-ended and multiple-choice formats. (b) Offline simulation of online video interruption, where the model generates responses autoregressively and reacts to interruption signals during generation.
Model
MAE IAC
MAE TFS
MAE IFA
MAE CCS
MAE CFS
MAE VES
Qwen3-VL-8B 4
0.2070
1.1445
0.1641
3.1289
2.0703
1.6016
Qwen3-VL-30B-A3B 4
0.1406
0.8984
0.1250
3.1602
2.3359
0.8477
Qwen3-VL-235B-A22B 4
0.0508
0.3672
0.0625
0.9453
0.5547
1.0273
Doubao-Seed-1.6 12
0.0859
1.0859
0.1484
2.6016
1.9766
2.1445
Gemini-2.5-Flash 9
0.0938
1.2656
0.1406
2.8984
1.8711
-
GPT-4.1-mini 1
0.0938
0.7500
0.1445
3.0938
1.6484
-
Table 1: MAE between different Judge models and human annotations on the validation set, and lower is better. The best value in each column is highlighted in bold.
Model/Metric
IAC ↑
TFS ↑
VES ↑
IFA ↑
CFS ↑
CCS ↑
CA ↑
Doubao-Seed-1.6 12
95.73%
9.61
6.23
79.42%
5.24
7.26
92.85 %
Gemini-2.5-Flash 9
76.45%
8.41
6.28
64.58%
4.42
6.97
73.39%
Qwen2.5-VL-7B 42
64.14%
7.55
5.34
50.11%
3.73
5.06
83.64%
Qwen3-VL-8B 4
93.63%
9.32
6.68
81.07%
4.91
6.75
88.61%
VideoChat-R1-7B 16
66.42%
7.69
5.36
51.03%
3.86
4.98
87.27%
VideoLLaMA3-7B 43
72.33%
7.50
5.25
57.92%
4.06
5.63
89.61%
Table 2: Experimental results on the open-ended subset of OVIBench. CA represents the accuracy of interruption-type classification. IAC and IFA are binary classification accuracy. TFS, VES, CFS, and CCS are scored on a 1-10 scale. Higher values indicate better performance ( ↑ ). The best results in each column are highlighted in bold.
Figure 4: Comparison of model performance across different interruption types on OVIBench. Each axis represents the recognition accuracy (%) for one interruption type, and larger areas indicate better overall interruption understanding capability.
Cancellaton
Correction
False Trigger
Total
ActivityNet
840
1616
799
3255
MovieChat
475
913
477
1865
QVHighlights
2736
5365
2744
10845
UCF_Crime
173
297
140
610
YouCook2
1041
2050
1099
4190
Table 3: Statistics of OVI-Train instances by video source and interruption type.
Model
All
False Trigger
Cancellation
Correction
MA
CA
MA
CA
MA
CA
MA
CA
Open-Source Models
Qwen2.5-VL-7B 42
65.54
81.39
43.46
98.69
45.69
42.74
86.99
92.79
Qwen2.5-VL-72B 42
71.46
93.36
95.37
99.16
30.35
98.49
65.93
77.24
VideoChat-R1-7B 16
71.92
83.67
56.54
98.60
52.87
53.55
89.58
91.82
Qwen3-VL-30B-A3B 4
67.75
84.47
15.62
66.93
65.03
73.65
95.43
98.95
Table 4: Multiple-choice evaluation results on OVIBench. MA denotes answer selection accuracy, and CA denotes interruption-type classification accuracy. Δ indicates the performance improvement of our model over the baseline. The best results in each column are highlighted in bold.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Score
Criterion
10
Immediately enters the correct strategy; no repetition, filler transition, or semantic jump; reads like natural human real-time dialogue.
9
Correct strategy executed immediately; may contain a very short transition word; no repetition or semantic discontinuity.
8
Strategy is correct; may contain one explicit transition sentence; no semantic confusion; no repetition exceeding one sentence.
7
Strategy is correct, with only minor repetition or slightly rigid structure.
6
Strategy is correct, but repeats original content for more than one sentence or briefly becomes illogical before recovering.
5
Eventually shifts to the correct strategy, but contains noticeable discontinuity or delayed revision.
Appendix
Table 5: Detailed scoring rubric for Textual Fluency Score (TFS).
Score
Criterion
10
All mentioned objects, actions, and attributes (e.g., color, position, state) are genuinely present; no hallucination; the description accurately reflects the visual state at the interruption moment; no temporal misalignment.
9
All visual elements genuinely exist; only extremely minor detail imprecision is present; timing is essentially synchronized.
8
Major visual elements are correct; one minor detail may be vague or slightly inaccurate; no obvious hallucination.
7
Core objects and actions are correct, with only one minor visual error or slight temporal lag.
6
Most visual content is correct, but there is one obvious but non-core visual error, or a noticeable temporal synchronization deviation.
5
Core visual elements are correct, but multiple minor visual errors are present, or there is partial over-generalization.
Appendix
Table 6: Detailed scoring rubric for Visual Evidence Score (VES).
Score
Criterion
10
Explicitly states that the old conclusion was incorrect; fully retracts the conflicting semantics; completely reconstructs the response under the new constraints; no residual old semantics remain.
9
Fully follows the new constraints; does not explicitly apologize or deny the old conclusion, but contains no actual residual semantic error.
8
Major semantic conflicts are corrected; only extremely minor and irrelevant old phrasing remains.
7
The core conflict is corrected, but some secondary old semantics remain uncleared.
6
The model attempts correction, but still retains noticeable fragments of the old conclusion.
5
The model performs partial rewriting, but the core constraints are not fully satisfied.
Appendix
Table 7: Detailed scoring rubric for Correction Following Score (CFS).
Score
Criterion
10
All semantic units that should be preserved are fully retained; only minimal necessary modifications are made; no irrelevant expansion or structural disruption occurs.
9
Almost all preservable context is retained; only a very small amount of non-core information is lost.
8
All core semantics are preserved, but some secondary details are missing.
7
The main semantics are preserved, but some core information is slightly weakened.
6
Core semantics are only partially preserved, and some unrelated content is clearly deleted or rewritten.
5
Only part of the core semantics is preserved, and the structure is noticeably reorganized.
Appendix
Table 8: Detailed scoring rubric for Contextual Consistency Score (CCS).
Condition
Maximum Score
One core visual error
≤4
Two or more core visual errors
≤3
Obvious fabricated scene
≤2
Content completely detached from the video
1
Appendix
Table 9: Score cap conditions for Visual Evidence Score (VES).
Condition
Maximum Score
Clearly continues the core old conclusion
≤4
Old and new conclusions coexist
≤3
Completely ignores the correction signal
1
Appendix
Table 10: Score cap rules for Correction Following Score (CFS).
Condition
Maximum Score
Unjustified deletion of core semantic units
≤4
Large-scale rewriting of original context
≤3
Complete topic shift
1
Appendix
Table 11: Score cap rules for Contextual Consistency Score (CCS).
Model/Metric
IAC
TFS
VES
IFA
CFS
CCS
CA
doubao
95.54%
9.35
7.18
64.53%
1.69
5.80
100%
gemini
79.94%
9.47
7.49
37.90%
2.20
5.13
100%
Qwen2.5-VL-7B
51.45%
7.55
4.83
27.76%
1.34
4.56
100%
Qwen3-VL-8B
90.40%
9.13
6.80
67.79%
1.85
6.55
100%
VideoChat-R1-7B
58.24%
7.81
5.03
36.79%
1.33
4.26
100%
VideoLLaMA3-7B
74.17%
7.33
4.99
40.38%
1.51
5.26
100%
Appendix
Table 12: Experimental results on the open-ended subset of OVIBench for Cancellation. The best results in each column are highlighted in bold.
Model/Metric
IAC
TFS
VES
IFA
CFS
CCS
CA
doubao
95.33%
9.81
6.57
78.48%
8.73
7.22
88%
gemini
69.62%
8.02
6.42
67.94%
7.14
7.33
87%
Qwen2.5-VL-7B
92.92%
9.02
6.63
77.14%
6.49
5.79
92%
Qwen3-VL-8B
98.53%
9.77
7.28
89.96%
8.25
6.30
87%
VideoChat-R1-7B
94.18%
9.16
6.65
74.59%
6.76
5.73
92%
VideoLLaMA3-7B
97.63%
9.06
6.14
66.25%
7.35
6.15
90%
Appendix
Table 13: Experimental results on the open-ended subset of OVIBench for Correction. The best results in each column are highlighted in bold.
Model/Metric
IAC
TFS
VES
IFA
CFS
CCS
CA
doubao
96.74%
9.49
4.55
96.83%
1.92
8.87
98%
gemini
86.51%
8.10
4.73
85.64%
1.30
8.15
99%
Qwen2.5-VL-7B
19.62%
4.60
3.27
19.16%
0.69
4.12
67%
Qwen3-VL-8B
87.19%
8.62
5.35
77.06%
1.41
7.88
85%
VideoChat-R1-7B
19.25%
4.61
3.09
18.60%
0.69
4.21
75%
VideoLLaMA3-7B
34.52%
6.35
5.28
48.27%
0.91
5.70
84%
Appendix
Table 14: Experimental results on the open-ended subset of OVIBench for False Trigger. The best results in each column are highlighted in bold.
Figure 5: Illustrative examples of the six evaluation metrics in OVIBench. Each case shows a short video stream with a user interruption and the model’s responses. Green boxes denote responses that achieve high scores, while red boxes indicate failure cases.
Model
All
False Trigger
Cancellation
Correction
MA
CA
MA
CA
MA
CA
MA
CA
Open-Source Models
Qwen2.5-VL-7B 42
62.78
79.44
48.33
91.67
42.78
50.00
85.56
94.67
Qwen2.5-VL-72B 42
69.44
90.56
86.67
96.67
38.33
91.67
83.33
83.33
VideoChat-R1-7B 16
68.33
81.67
52.78
91.67
49.44
58.33
88.89
95.00
Qwen3-VL-30B-A3B 4
70.00
86.11
28.33
73.33
68.89
81.67
96.67
98.33
Appendix
Table 15: Multiple-choice evaluation results on the human-written OVI-Human subset. MA denotes answer selection accuracy, and CA denotes interruption-type classification accuracy. Δ indicates the performance improvement of our model over the baseline. The best results in each column are highlighted in bold.
Recent advances in video large language models (VideoLLMs) call for new evaluation protocols and benchmarks for video-language understanding. Inspired by the needle in a haystack test widely used by LLMs, we introduce a novel task of Needle in a Montage (NeMo), which is designed to assess the temporal understanding capabilities of advanced VideoLLMs. Specifically, the proposed task focuses on two fundamental abilities critical for temporal understanding, i.e., retrieval-style long-context recall and temporal grounding. To generate video question answering data for our task, we develop a scalable automated data generation pipeline that facilitates high-quality data synthesis. Built upon the proposed pipeline, we present NeMoBench, a video-language benchmark centered on our task. Specifically, our full set of NeMoBench features 31,378 automatically generated question-answer (QA) pairs from 13,486 videos with various durations ranging from seconds to hours. Experiments demonstrate that our pipeline can reliably and automatically generate high-quality evaluation data, enabling NeMoBench to be continuously updated with the latest videos. We evaluate 20 state-of-the-art models on our benchmark, providing extensive results and key insights into their capabilities and limitations. Our project page is available at: https://lavi-lab.github.io/NeMoBench.
Zi-Yuan Hu, Shuo Liang, Duo Zheng +10
The Chinese University of Hong Kong · Phoenix TV · Stanford University +2
Video large language models (LLMs) are often constrained by computation and memory budgets, leading them to use reduced frame rates and spatial resolutions, which may cause them to miss critical information for question answering (QA). A practical and efficient solution is a two-stage paradigm: first perform coarse video understanding to localize relevant segments, and then re-watch these segments at higher temporal or spatial fidelity. In this paper, we present video-SALMONN-R3, the first end-to-end video-LLM that enables re-watch through reinforcement learning without relying on chain-of-thought (CoT) cold-start. This design removes the need for costly CoT data annotations and avoids CoT-based supervised fine-tuning (SFT), which can otherwise degrade the pretrained video understanding abilities. To address the mismatch between the reasoning-first behavior induced by re-watch and the answer-first tendency of pretrained video-LLMs, we propose a re-answer strategy, in which the model first produces a direct answer in the first watch and then refines it after re-watching. Finally, to improve question adherence during re-watching, we propose a re-ask mechanism that re-injects the query when revisiting localized segments. Experimental results show that video-SALMONN-R3 consistently outperforms both the base model and the QA-SFT baseline, while surpassing prior re-watch-based approaches with significantly lower computational cost. Code, models, and data will be publicly released upon acceptance.
Reliable online video question answering requires tracking state transitions while selectively abstaining when visual evidence is insufficient. Existing benchmarks focus on static recognition or long-range retrieval, rarely evaluating these coupled capabilities under evolving and incomplete evidence. We present STORM-Bench, comprising 5,736 questions across 630 compact, change-dense episodes spanning five egocentric domains (STORM-Real) and two controlled simulation subsets (STORM-Sim) at 1 FPS. Questions are stratified by a proxy for accumulated change intensity (Low, Medium, High) and query-time answerability (Known, Uncertain). To measure reliability, we introduce STORM-BR, a harmonic metric over joint answer-status correctness that exposes abstention failures masked by aggregate accuracy, alongside STORM-BR-ATTR for uncertainty attribution. Across 14 video LLMs, online accuracy peaks at 60.3% (mean 51.7%), whereas STORM-BR ranges from 5.7% to 35.6% (mean 18.8%), driven by pervasive overconfidence on uncertain queries. STORM-Bench shows that task accuracy masks these gaps in epistemic reliability and state tracking. Benchmark and code are available at https://github.com/siruzhong/STORM-Bench.
Siru Zhong, Shenghan Tan, Rihong Yan +6
HKUST (GZ) · Beta Infinity · Beihang University +2