OVIBench: Benchmarking Online Video Question Answering under Interruption

Naiming Liu; Zhiheng Wu; Shuning Wang; Tie Zhang; Bowen Liu; Tong Wang

cs.CVAug 23, 2026

OVIBench: Benchmarking Online Video Question Answering under Interruption

Authors: Naiming Liu, Zhiheng Wu, Shuning Wang, Tie Zhang, Bowen Liu, Tong Wang

Organizations: HIT · CASIA · ZJU · UESTC ·

Abstract

Recent vision language models (VLMs) have achieved strong progress in video understanding. However, most existing video QA research and benchmarks still follow an offline, single-round paradigm, overlooking realistic interactions where users may interrupt the model during answer generation. To address this gap, we formulate the task of Online Video Question Answering under Interruption and introduce OVIBench, the first standardized benchmark for evaluating VLMs in this setting. OVIBench categorizes interruptions into three types: Cancellation, False Trigger, Correction and supports both open-ended and multiple-choice evaluations. To enable large-scale and reproducible testing, we develop an offline simulation protocol that reproduces interruption during generation under a unified temporal setup, together with a multi-dimensional metric suite for assessing interruption understanding and response generation. Experiments demonstrate that OVIBench effectively distinguishes models' interruption-handling abilities, especially in following correction requests. Finally, we construct a train set OVI-Train for interruption-aware fine-tuning. Models fine-tuned on this dataset achieve significant gains on OVIBench, validating the effectiveness of our benchmark and data design. OVIBench, OVI-Train, and the evaluation code will be released.

Figures & tables

Figure 1: Illustration of three interruption types in online video question answering: Cancellation, False Trigger, and Correction. In Online interactions, users may terminate the response, provide irrelevant signals, or introduce semantic corrections during generation.

Figure 2: (a) Distribution of video sources across five public datasets during data collection. (b) Sample size distribution of the benchmark under open-ended and multiple-choice evaluation settings. For each dataset, three bars are shown from left to right, corresponding to the False Trigger, Cancellation, and Correction interruption types.

Figure 3: Overview of OVIBench. (a) Benchmark data construction pipeline, including scene segmentation, VLM-based question generation and interruption insertion for both open-ended and multiple-choice formats. (b) Offline simulation of online video interruption, where the model generates responses autoregressively and reacts to interruption signals during generation.

Model	MAE ${}_{\text{IAC}}$	MAE ${}_{\text{TFS}}$	MAE ${}_{\text{IFA}}$	MAE ${}_{\text{CCS}}$	MAE ${}_{\text{CFS}}$	MAE ${}_{\text{VES}}$
Qwen3-VL-8B 4	0.2070	1.1445	0.1641	3.1289	2.0703	1.6016
Qwen3-VL-30B-A3B 4	0.1406	0.8984	0.1250	3.1602	2.3359	0.8477
Qwen3-VL-235B-A22B 4	0.0508	0.3672	0.0625	0.9453	0.5547	1.0273
Doubao-Seed-1.6 12	0.0859	1.0859	0.1484	2.6016	1.9766	2.1445
Gemini-2.5-Flash 9	0.0938	1.2656	0.1406	2.8984	1.8711	-
GPT-4.1-mini 1	0.0938	0.7500	0.1445	3.0938	1.6484	-

Table 1: MAE between different Judge models and human annotations on the validation set, and lower is better. The best value in each column is highlighted in bold.

Model/Metric	IAC $\uparrow$	TFS $\uparrow$	VES $\uparrow$	IFA $\uparrow$	CFS $\uparrow$	CCS $\uparrow$	CA $\uparrow$
Doubao-Seed-1.6 12	95.73%	9.61	6.23	79.42%	5.24	7.26	92.85 %
Gemini-2.5-Flash 9	76.45%	8.41	6.28	64.58%	4.42	6.97	73.39%
Qwen2.5-VL-7B 42	64.14%	7.55	5.34	50.11%	3.73	5.06	83.64%
Qwen3-VL-8B 4	93.63%	9.32	6.68	81.07%	4.91	6.75	88.61%
VideoChat-R1-7B 16	66.42%	7.69	5.36	51.03%	3.86	4.98	87.27%
VideoLLaMA3-7B 43	72.33%	7.50	5.25	57.92%	4.06	5.63	89.61%

Table 2: Experimental results on the open-ended subset of OVIBench. CA represents the accuracy of interruption-type classification. IAC and IFA are binary classification accuracy. TFS, VES, CFS, and CCS are scored on a 1-10 scale. Higher values indicate better performance (

\uparrow

). The best results in each column are highlighted in bold.

Figure 4: Comparison of model performance across different interruption types on OVIBench. Each axis represents the recognition accuracy (%) for one interruption type, and larger areas indicate better overall interruption understanding capability.

	Cancellaton	Correction	False Trigger	Total
ActivityNet	840	1616	799	3255
MovieChat	475	913	477	1865
QVHighlights	2736	5365	2744	10845
UCF_Crime	173	297	140	610
YouCook2	1041	2050	1099	4190

Table 3: Statistics of OVI-Train instances by video source and interruption type.

Open-Source Models
Model	All		False Trigger		Cancellation		Correction
	MA	CA	MA	CA	MA	CA	MA	CA
Qwen2.5-VL-7B 42	65.54	81.39	43.46	98.69	45.69	42.74	86.99	92.79
Qwen2.5-VL-72B 42	71.46	93.36	95.37	99.16	30.35	98.49	65.93	77.24
VideoChat-R1-7B 16	71.92	83.67	56.54	98.60	52.87	53.55	89.58	91.82
Qwen3-VL-30B-A3B 4	67.75	84.47	15.62	66.93	65.03	73.65	95.43	98.95

Table 4: Multiple-choice evaluation results on OVIBench. MA denotes answer selection accuracy, and CA denotes interruption-type classification accuracy.

\Delta

indicates the performance improvement of our model over the baseline. The best results in each column are highlighted in bold.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Score	Criterion
10	Immediately enters the correct strategy; no repetition, filler transition, or semantic jump; reads like natural human real-time dialogue.
9	Correct strategy executed immediately; may contain a very short transition word; no repetition or semantic discontinuity.
8	Strategy is correct; may contain one explicit transition sentence; no semantic confusion; no repetition exceeding one sentence.
7	Strategy is correct, with only minor repetition or slightly rigid structure.
6	Strategy is correct, but repeats original content for more than one sentence or briefly becomes illogical before recovering.
5	Eventually shifts to the correct strategy, but contains noticeable discontinuity or delayed revision.

Appendix

Table 5: Detailed scoring rubric for Textual Fluency Score (TFS).

Score	Criterion
10	All mentioned objects, actions, and attributes (e.g., color, position, state) are genuinely present; no hallucination; the description accurately reflects the visual state at the interruption moment; no temporal misalignment.
9	All visual elements genuinely exist; only extremely minor detail imprecision is present; timing is essentially synchronized.
8	Major visual elements are correct; one minor detail may be vague or slightly inaccurate; no obvious hallucination.
7	Core objects and actions are correct, with only one minor visual error or slight temporal lag.
6	Most visual content is correct, but there is one obvious but non-core visual error, or a noticeable temporal synchronization deviation.
5	Core visual elements are correct, but multiple minor visual errors are present, or there is partial over-generalization.

Appendix

Table 6: Detailed scoring rubric for Visual Evidence Score (VES).

Score	Criterion
10	Explicitly states that the old conclusion was incorrect; fully retracts the conflicting semantics; completely reconstructs the response under the new constraints; no residual old semantics remain.
9	Fully follows the new constraints; does not explicitly apologize or deny the old conclusion, but contains no actual residual semantic error.
8	Major semantic conflicts are corrected; only extremely minor and irrelevant old phrasing remains.
7	The core conflict is corrected, but some secondary old semantics remain uncleared.
6	The model attempts correction, but still retains noticeable fragments of the old conclusion.
5	The model performs partial rewriting, but the core constraints are not fully satisfied.

Appendix

Table 7: Detailed scoring rubric for Correction Following Score (CFS).

Score	Criterion
10	All semantic units that should be preserved are fully retained; only minimal necessary modifications are made; no irrelevant expansion or structural disruption occurs.
9	Almost all preservable context is retained; only a very small amount of non-core information is lost.
8	All core semantics are preserved, but some secondary details are missing.
7	The main semantics are preserved, but some core information is slightly weakened.
6	Core semantics are only partially preserved, and some unrelated content is clearly deleted or rewritten.
5	Only part of the core semantics is preserved, and the structure is noticeably reorganized.

Appendix

Table 8: Detailed scoring rubric for Contextual Consistency Score (CCS).

Condition	Maximum Score
One core visual error	$\leq 4$
Two or more core visual errors	$\leq 3$
Obvious fabricated scene	$\leq 2$
Content completely detached from the video	$1$

Appendix

Table 9: Score cap conditions for Visual Evidence Score (VES).

Condition	Maximum Score
Clearly continues the core old conclusion	$\leq 4$
Old and new conclusions coexist	$\leq 3$
Completely ignores the correction signal	$1$

Appendix

Table 10: Score cap rules for Correction Following Score (CFS).

Condition	Maximum Score
Unjustified deletion of core semantic units	$\leq 4$
Large-scale rewriting of original context	$\leq 3$
Complete topic shift	$1$

Appendix

Table 11: Score cap rules for Contextual Consistency Score (CCS).

Model/Metric	IAC	TFS	VES	IFA	CFS	CCS	CA
doubao	95.54%	9.35	7.18	64.53%	1.69	5.80	100%
gemini	79.94%	9.47	7.49	37.90%	2.20	5.13	100%
Qwen2.5-VL-7B	51.45%	7.55	4.83	27.76%	1.34	4.56	100%
Qwen3-VL-8B	90.40%	9.13	6.80	67.79%	1.85	6.55	100%
VideoChat-R1-7B	58.24%	7.81	5.03	36.79%	1.33	4.26	100%
VideoLLaMA3-7B	74.17%	7.33	4.99	40.38%	1.51	5.26	100%

Appendix

Table 12: Experimental results on the open-ended subset of OVIBench for Cancellation. The best results in each column are highlighted in bold.

Model/Metric	IAC	TFS	VES	IFA	CFS	CCS	CA
doubao	95.33%	9.81	6.57	78.48%	8.73	7.22	88%
gemini	69.62%	8.02	6.42	67.94%	7.14	7.33	87%
Qwen2.5-VL-7B	92.92%	9.02	6.63	77.14%	6.49	5.79	92%
Qwen3-VL-8B	98.53%	9.77	7.28	89.96%	8.25	6.30	87%
VideoChat-R1-7B	94.18%	9.16	6.65	74.59%	6.76	5.73	92%
VideoLLaMA3-7B	97.63%	9.06	6.14	66.25%	7.35	6.15	90%

Appendix

Table 13: Experimental results on the open-ended subset of OVIBench for Correction. The best results in each column are highlighted in bold.

Model/Metric	IAC	TFS	VES	IFA	CFS	CCS	CA
doubao	96.74%	9.49	4.55	96.83%	1.92	8.87	98%
gemini	86.51%	8.10	4.73	85.64%	1.30	8.15	99%
Qwen2.5-VL-7B	19.62%	4.60	3.27	19.16%	0.69	4.12	67%
Qwen3-VL-8B	87.19%	8.62	5.35	77.06%	1.41	7.88	85%
VideoChat-R1-7B	19.25%	4.61	3.09	18.60%	0.69	4.21	75%
VideoLLaMA3-7B	34.52%	6.35	5.28	48.27%	0.91	5.70	84%

Appendix

Table 14: Experimental results on the open-ended subset of OVIBench for False Trigger. The best results in each column are highlighted in bold.

Figure 5: Illustrative examples of the six evaluation metrics in OVIBench. Each case shows a short video stream with a user interruption and the model’s responses. Green boxes denote responses that achieve high scores, while red boxes indicate failure cases.

Open-Source Models
Model	All		False Trigger		Cancellation		Correction
	MA	CA	MA	CA	MA	CA	MA	CA
Qwen2.5-VL-7B 42	62.78	79.44	48.33	91.67	42.78	50.00	85.56	94.67
Qwen2.5-VL-72B 42	69.44	90.56	86.67	96.67	38.33	91.67	83.33	83.33
VideoChat-R1-7B 16	68.33	81.67	52.78	91.67	49.44	58.33	88.89	95.00
Qwen3-VL-30B-A3B 4	70.00	86.11	28.33	73.33	68.89	81.67	96.67	98.33

Appendix

Table 15: Multiple-choice evaluation results on the human-written OVI-Human subset. MA denotes answer selection accuracy, and CA denotes interruption-type classification accuracy.

\Delta

indicates the performance improvement of our model over the baseline. The best results in each column are highlighted in bold.

Video QA

Video question answering studies how AI systems use visual and audio evidence over time to answer questions about video.

See all 184 papers

NeMo: Needle in a Montage for Video-Language Understanding
Sep 29, 2025Zi-Yuan Hu, Shuo Liang, Duo Zheng +10Temporal Video Grounding Video-Language Model Evaluation
The Chinese University of Hong Kong · Phoenix TV · Stanford University +2
video-SALMONN-R $^3$ : Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understanding
Jun 23, 2026Yixuan Li, Guangzhi Sun, Yudong Yang +1Efficient VLM Inference Video QA
Tsinghua University · University of Cambridge
STORM-Bench: Evaluating Online Video QA under Evolving and Incomplete Evidence
Sep 25, 2026Siru Zhong, Shenghan Tan, Rihong Yan +6Video QA Temporal Video Understanding
HKUST (GZ) · Beta Infinity · Beihang University +2

Sep 29, 2025J/K move · Enter open · S save

NeMo: Needle in a Montage for Video-Language Understanding

Zi-Yuan Hu, Shuo Liang, Duo Zheng, Yanyang Li+9

The Chinese University of Hong Kong · Phoenix TV · Stanford University+2

Temporal Video Grounding Video-Language Model Evaluation Video QA Long-Video Understanding Multimodal Model Evaluation Temporal Video Understanding

Open paper AskPDF arXiv

Recent advances in video large language models (VideoLLMs) call for new evaluation protocols and benchmarks for video-language understanding. Inspired by the needle in a haystack test widely used by LLMs, we introduce a novel task of Needle in a Montage (NeMo), which is designed to assess the temporal understanding capabilities of advanced VideoLLMs. Specifically, the proposed task focuses on two fundamental abilities critical for temporal understanding, i.e., retrieval-style long-context recall and temporal grounding. To generate video question answering data for our task, we develop a scalable automated data generation pipeline that facilitates high-quality data synthesis. Built upon the proposed pipeline, we present NeMoBench, a video-language benchmark centered on our task. Specifically, our full set of NeMoBench features 31,378 automatically generated question-answer (QA) pairs from 13,486 videos with various durations ranging from seconds to hours. Experiments demonstrate that our pipeline can reliably and automatically generate high-quality evaluation data, enabling NeMoBench to be continuously updated with the latest videos. We evaluate 20 state-of-the-art models on our benchmark, providing extensive results and key insights into their capabilities and limitations. Our project page is available at: https://lavi-lab.github.io/NeMoBench.

Abstract

Video QA

Explore similar work

NeMo: Needle in a Montage for Video-Language Understanding

video-SALMONN-R3^33: Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understanding

STORM-Bench: Evaluating Online Video QA under Evolving and Incomplete Evidence

video-SALMONN-R $^3$ : Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understanding