STORM-Bench: Evaluating Online Video QA under Evolving and Incomplete Evidence

Siru Zhong; Shenghan Tan; Rihong Yan; Xiaohui Lv; Yuzheng Zhuang; Shuai Tao; Wulong Liu; Haohuan Fu

cs.CVSep 25, 2026

STORM-Bench: Evaluating Online Video QA under Evolving and Incomplete Evidence

Authors: Siru Zhong, Shenghan Tan, Rihong Yan, Xiaohui Lv, Yuzheng Zhuang, Shuai Tao, Wulong Liu, Haohuan Fu,

Organizations: HKUST (GZ) · Beta Infinity · Beihang University · UESTC ·

Abstract

Reliable online video question answering requires tracking state transitions while selectively abstaining when visual evidence is insufficient. Existing benchmarks focus on static recognition or long-range retrieval, rarely evaluating these coupled capabilities under evolving and incomplete evidence. We present STORM-Bench, comprising 5,736 questions across 630 compact, change-dense episodes spanning five egocentric domains (STORM-Real) and two controlled simulation subsets (STORM-Sim) at 1 FPS. Questions are stratified by a proxy for accumulated change intensity (Low, Medium, High) and query-time answerability (Known, Uncertain). To measure reliability, we introduce STORM-BR, a harmonic metric over joint answer-status correctness that exposes abstention failures masked by aggregate accuracy, alongside STORM-BR-ATTR for uncertainty attribution. Across 14 video LLMs, online accuracy peaks at 60.3% (mean 51.7%), whereas STORM-BR ranges from 5.7% to 35.6% (mean 18.8%), driven by pervasive overconfidence on uncertain queries. STORM-Bench shows that task accuracy masks these gaps in epistemic reliability and state tracking. Benchmark and code are available at https://github.com/siruzhong/STORM-Bench.

Figures & tables

Figure 1: Overview of STORM -Bench. (a) Domain mix with Real/Sim and Known/Uncertain splits. (b) Storm-BR ranking of 14 video LLMs. (c) Nearly matched accuracy hides a 7.50-point Storm-BR gap. (d) By change intensity: Known remains stronger for most models; Uncertain often collapses under overconfidence (seven-subset $B_{1}$ – $B_{3}$ average; cf. Table 20 , Figure 10 ).

Figure 2: Word cloud of question text. Temporal expressions, state terms, and references to objects, actions, and spatial relations summarize the vocabulary of the questions.

Figure 3: Overview of STORM -Bench. (a) Five real egocentric domains and four controlled simulation room categories. (b) Compact multi-visit episodes requiring update, aggregation, and abstention. (c) Six question types stratified by intensity $\times$ answerability, with uncertainty causes.

Figure 4: Compared to existing Video-QA suites, STORM -Bench combines compact durations (a), higher sampled visual-change rates (b), and denser QA supervision (c). Dots/bars denote medians; error bars in (b) show IQRs. See Appendix A.5.1 .

Figure 5: STORM -Bench diagnostic framework. Questions posed at query time $t_{q}$ are answered strictly from the observed history $V_{\leq t_{q}}$ across six semantic types and two diagnostic axes. Balanced reliability and fine-grained root-cause attribution are scored via Storm-BR and Storm-BR-Attr .

Figure 6: Real-scene episode construction pipeline. Region-aware timelines record object interactions and state transitions. Temporally separated visits to the same functional area are compiled into compact revisit episodes, followed by query-time QA grounding and cross-annotator validation.

Figure 7: Simulated-scene construction pipeline. Scripted rollouts generate video and execution traces; event extraction joins state changes with visual masks; query selection enforces causal bounds; question construction emits balanced options and diagnostic labels. The illustrated THOR rollout uses disappearance/reappearance events; VHome instantiates the same stages with native agent actions and sampled visibility records (Appendix A.2.5 ).

Model	Storm-BR by domain							Avg. BR	Avg. Acc
Model	Cook	Bike	Health	Music	Sports	THOR	VHome	Avg. BR	Avg. Acc
Molmo2-8B ( 2026 )	34.45 [27.4, 39.3]	33.59 [22.7, 40.4]	31.80 [18.4, 38.8]	43.29 [24.2, 55.4]	31.34 [21.8, 36.8]	34.15 [21.6, 42.4]	40.35 [31.8, 44.9]	35.57	60.31
Qwen3.5-9B ( 2026 )	33.29 [27.7, 37.8]	22.22 [12.2, 25.7]	33.23 [25.0, 38.7]	42.70 [25.1, 52.4]	24.06 [18.5, 28.2]	25.89 [12.6, 33.2]	15.10 [9.5, 18.1]	28.07	60.33
Qwen3-VL-4B ( 2025 )	26.77 [21.9, 30.4]	31.49 [23.1, 35.3]	29.71 [21.3, 34.8]	36.92 [20.9, 44.6]	20.89 [13.4, 25.2]	19.27 [10.8, 27.7]	18.96 [11.0, 24.9]	26.29	55.16
Qwen3.5-4B ( 2026 )	34.04 [28.8, 38.5]	8.72 [6.3, 12.3]	33.95 [23.8, 39.9]	36.97 [22.9, 46.0]	18.09 [11.0, 22.2]	15.42 [8.3, 20.0]	16.65 [8.1, 20.7]	23.41	58.20
MiniCPM-V 4.5 ( 2025 )	22.17 [16.5, 26.0]	31.97 [22.8, 37.1]	18.71 [8.5, 25.9]	27.89 [14.9, 38.8]	17.16 [8.5, 22.3]	21.70 [11.5, 26.2]	20.33 [10.9, 27.5]	22.84	54.78

Table 1: Main evaluation on STORM -Bench (RQ1). Online balanced reliability ( BR , Laplace-smoothed Storm-BR ) per domain with 95% bootstrap intervals, overall average BR ( Avg. BR ), and accuracy ( Avg. Acc ). Both are unweighted means across five real domains, THOR, and VHome.

Figure 8: Whole-bench evaluation across five real domains and two simulation subsets. (a) Average Storm-BR across seven evaluation subsets. (b) Per-domain Storm-BR scores. (c) Parameter scaling across matched model families. (d) Domain difficulty distribution across all 14 models.

Figure 9: Diagnostic protocol and metric analysis (Cook domain except panel b). (a) Online accuracy vs. Storm-BR (marker size

\propto

scale). (b) Per-domain Kendall’s

\tau

(95% bootstrap CIs; highlighted bar pools domains). (c,d) Shift in Storm-BR and accuracy under full-video access.

Figure 10: Answerability diagnostics and error analysis on Cook. (a) Joint correctness across intensity $\times$ answerability cells. (b) Known vs. Uncertain accuracy with overconfidence ( $\mathrm{OC}$ ). (c) Joint correctness vs. cause F1 across sources. (d) Stratification by $\mathrm{OC}$ (dashed diagonal indicates parity).

Figure 11: Temporal diagnostics and input controls. (a) Macro accuracy across question types. (b) Cross-model median and IQR by evidence age. (c) Cook joint accuracy across intensity bins (

B_{1}

–

B_{3}

). (d) Accuracy and Storm-BR across input controls on Cook.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Kitchen	Living Room	Bedroom	Bathroom
Scene numbers	7, 12, 13, 19, 20, 27, 28	202, 209, 211, 221, 222, 223, 229	301, 306, 308, 318, 327, 329, 330	401, 404, 406, 408, 410, 422, 428
Camera–target distance (m)	0.5–1.2	0.7–2.2	0.6–2.0	0.5–1.2
Maximum segment speed (m/s)	0.85	2.20	1.60	1.30
Maximum station step (m)	3.5	8.0	5.0	4.0
Minimum target area (pixels)	1,400	600	600	500
Minimum box dimension (pixels)	28	18	18	16

Appendix

Table 2: Simulation rollout settings. THOR scene numbers denote the suffix of FloorPlan . Each THOR room category contributes seven videos and 75 QA items; VHome settings are summarized below. Pixel-area and bounding-box thresholds apply when the target is required to be visible.

Type	Known target (THOR; VHome)	Uncertain target	THOR K/U	VHome K/U
Current state	Room; current visibility or physical state	Hidden physical state or location	53/1	51/1
Factual retrieval	Room; earlier visibility or physical state	State during missing observation	52/2	50/2
History aggregation	Last object in a return pair; return/exit counts	Number of hidden manipulations	39/12	37/12
State change	Appearance change; visibility or reviewed state sequence	Cause of an unobserved change	48/3	46/3
Object tracking	Returning category; return or reviewed object state	Physical identity across a gap	26/24	26/23
Temporal reasoning	Return order; return or reviewed state order	Ordering of unobserved actions	27/13	27/12

Appendix

Table 3: Simulation question construction and release counts. K/U denote Known/Uncertain. Known-target descriptions give THOR followed by VHome; Uncertain builders query hidden physical facts in both subsets.

Type	Operational Definition	Example Question
Factual retrieval	Retrieve an object, attribute, or location from an earlier principal interval after the target leaves the view.	“What color was the towel previously seen on the radiator?”
Current state	Integrate visible history up to $t_{q}$ to determine an entity’s latest state or location after observed changes.	“Where is the spice bottle at $t_{q}$ after the observed relocations?”
State change	Compare visual evidence across a transition boundary to identify how physical state or placement changed.	“How did the milk frother’s state change between 4 s and 7 s?”
Object tracking	Determine whether observations across timestamps refer to the same instance; report uncertainty if ambiguous.	“Which earlier bowl, if any, is the bowl observed after the transition?”
Temporal reasoning	Determine event ordering, first/last, or $k$ -th occurrence of actions without relying on common-sense priors.	“Which occurred first: rinsing the knife, opening the drawer, or placing the bowl?”
History aggregation	Count, accumulate, or summarize discrete visible action segments or entities up to $t_{q}$ under a predefined unit.	“How many distinct visible drawer-opening segments occurred by $t_{q}$ ?”

Appendix

Table 4: Semantic question types in STORM -Real. Each type targets a temporal reasoning operation, annotated independently from accumulated change intensity and epistemic status. Table 3 specifies the corresponding Sim builders.

Domain	QA	Ep.	Hours	Dur.	QA/Ep.	Known	Unc.	$L/M/H$
STORM -Real / Cook	2,640	314	3.739	42.86 s	8.41	2,122	518	782 / 1,111 / 747
STORM -Real / Bike	786	61	0.661	39.00 s	12.89	638	148	371 / 370 / 45
STORM -Real / Health	800	83	0.948	41.13 s	9.64	642	158	289 / 370 / 141
STORM -Real / Music	120	16	0.186	41.74 s	7.50	96	24	68 / 52 / 0
STORM -Real / Sports	800	99	1.126	40.93 s	8.08	640	160	406 / 299 / 95
THOR	300	28	0.467	60.05 s	10.71	245	55	37 / 57 / 206

Appendix

Table 5: Dataset overview, scale, and diagnostic composition.

L/M/H

denote low (1–3), medium (4–6), and high (7–10) change-intensity bins. Dur. denotes the mean video duration per episode in seconds. QA/Ep. denotes the average number of QA pairs per episode. Domain scores are evaluated independently without pooling.

Question-Type Distribution
Domain	Current state	Factual retrieval	History aggregation	State change	Object tracking	Temporal reasoning	All
Cook	446	429	442	439	444	440	2,640
Bike	142	142	120	137	145	100	786
Health	158	167	103	155	133	84	800
Music	25	26	11	22	22	14	120
Sports	170	171	96	148	141	74	800

Appendix

Table 6: Benchmark structural composition: question types, uncertainty sources, and evidence protocols. Evidence ends denotes the fraction of items whose last evidence endpoint equals the query time; mean duration excludes instantaneous spans; the last column counts items with at least one span longer than 10 seconds.

Dataset	Duration $n$	Median (s)	V/W	QA/min	Change/min [IQR]
EgoSchema	1,157	180.00	15/43	0.33	13.67 [10.50, 16.97]
LongVideoBench	3,991	60.80	15/31	0.23	7.45 [4.46, 14.85]
LVBench	103	3664.41	15/45	0.22	14.35 [10.07, 20.58]
MLVU	2,106	480.02	15/45	0.12	18.00 [10.77, 23.35]
MVBench	5,061	11.97	13/15	3.13	9.23 [8.16, 12.41]
Video-MME	900	487.94	15/41	0.18	17.98 [13.04, 23.51]

Appendix

Table 7: Coverage and descriptive statistics for Figure 4 . Duration statistics use local media; change statistics use selected windows. QA density follows the supplied QA/video-time totals. STORM-Sim combines THOR and VHome (component rows use their portions of the joint sample); its 15 sampled episodes comprise seven THOR and eight VHome episodes. Duration and change values are medians, with the change interquartile range in brackets; V/W denotes source videos/windows.

Figure 12: Dataset patterns across domains. Inventory counts follow release manifests (Tables 5 and 6 ). (A) Six-cell Known / Uncertain occupancy for Storm-BR , with an empty high-change bin for Music. (B) Within-type Uncertain rates are near 20% in STORM -Real, while both simulation subsets concentrate uncertainty in tracking and temporal items. (C) Uncertainty-source mix (labels per Uncertain item); missing observation dominates except in Health and Music. (D) Most STORM -Real items have evidence ending within 2 s of query, whereas THOR places every latest endpoint at $t_{q}$ ; VHome includes delayed retrieval queries.

Figure 13: Cook: recognizing the direction of a state change. At $t_{q}=9\,\mathrm{s}$ , visual evidence indicates that the digital scale reading changes from 554 to zero. Qwen3.5-9B correctly tracks this state transition, whereas Qwen3-VL-8B predicts the inverted direction. Both state probes correctly predict Known , demonstrating that accurate epistemic-status classification does not guarantee correct interpretation of the underlying dynamic change.

Figure 14: Bike Repair: identifying a completed physical separation. At $t_{q}=31\,\mathrm{s}$ , the tire is visibly separated from the rim. Both models select the correct task option, but Qwen3.5-9B incorrectly predicts Uncertain in the decoupled probe, whereas Qwen3-VL-8B correctly identifies Known . This highlights the decoupling between task answer selection and epistemic confidence verification.

Figure 15: Health: grounding current object configuration. At $t_{q}=8\,\mathrm{s}$ , the COVID test box is tilted open with its white plastic tray partially extracted. Qwen3.5-9B correctly identifies this spatial configuration, whereas Qwen3-VL-8B selects a distractor asserting the box lies flat with its lid tucked under. While both models correctly report Known , they diverge on fine-grained spatial interpretation.

Figure 16: Music: avoiding unsupported event-completion claims. Visual history does not establish a completed page turn by $t_{q}=27\,\mathrm{s}$ , mandating Cannot be determined . Both models correctly predict Uncertain in the decoupled probe, yet overconfidently commit to an option claiming a completed turn, illustrating a dissociation between uncertainty recognition and option choice.

Figure 17: Sports: localizing a dynamic object at query time. At $t_{q}=16\,\mathrm{s}$ , the basketball is located inside the hoop. Qwen3-VL-8B correctly grounds its query-time position and predicts Known . In contrast, Qwen3.5-9B misattributes the position to the backboard and incorrectly predicts Uncertain in the status probe.

Figure 18: THOR: distinguishing visual reappearance from instance identity. A hand towel appears at 14 s, the ring becomes empty at 38 s, and a towel reappears at 40–41 s. At $t_{q}=41\,\mathrm{s}$ , the history does not uniquely identify whether the returning object is the identical physical instance. Qwen3.5-9B correctly identifies Uncertain in the decoupled probe, yet overconfidently selects the same-instance answer in the task interface.

Figure 19: VHome: an observation gap leaves the action unresolved. The displayed views bracket a gap in which the target and its operator are outside the observer’s view. At $t_{q}=17.05$ s, the visible prefix does not determine the action that caused the change. Molmo2-8B selects the abstention option and predicts Uncertain ; Qwen3.5-9B also predicts Uncertain but chooses an unsupported physical explanation. Both predictions use the complete causal prefix, including the explicit timing and observation contract (Appendix A.2.5 ).

Model	Family	Max	Unit	Model	Family	Max	Unit
Qwen2.5-VL-3B	Qwen	768	2	Molmo2-8B	Molmo2	384	1
Qwen2.5-VL-7B	Qwen	768	2	MiniCPM-V 4.5	MiniCPM	180	1 (pack 3)
Qwen3-VL-4B	Qwen	768	2	InternVideo2.5-8B	InternVideo	512	4
Qwen3-VL-8B	Qwen	768	2	Eagle2.5-8B	Eagle	256	2
Qwen3.5-4B	Qwen	768	2	VideoLLaMA3-7B	VideoLLaMA3	128	2
Qwen3.5-9B	Qwen	768	2	LLaVA-NeXT-Video-7B-DPO	LLaVA-NeXT-Video	32	1

Appendix

Table 8: Per-model frame budget under the 1 FPS online protocol (RQ1). Checkpoints share the 1 FPS clock and retain recent frames ( Max ) up to native caps, padding the final frame to temporal unit ( Unit ). A larger cap is not uniformly better: a bounded 16-frame window improves Storm-BR for 9 of 14 models on Cook (Table 26 ). MiniCPM-V 4.5 packs up to three frames per temporal group.

Benchmark	Temporal Protocol	Temporal Target	Answerability Target	Diagnostic Focus	Reported Metrics
EgoSchema ( 2023 )	Offline	Extended context	Forced choice	Egocentric QA	Accuracy
StreamingBench ( 2024 )	Online	Timestamped QA	Forced choice	Online understanding	Accuracy
SVCBench ( 2026 )	Online	Count trajectories	Numeric answers	Counts and updates	GPA/MoC/UDA
EGOSTREAM ( 2026 )	Online	Validity windows	Valid-answer recall	Memory retention	Recall-group Acc.
EgoSAT ( 2026 )	Online	Past/present/future	Future predictability	Confidence diagnostics	Task Acc./Conf.
StreamReady ( 2026 )	Readiness	Evidence windows	Response readiness	Correctness and timing	Acc./ARS

Appendix

Table 9: Taxonomic comparison with representative Video-QA benchmark protocols. Existing suites cover complementary temporal and answerability targets; STORM -Bench couples accumulated change intensity with query-time answerability under an online protocol.

Benchmark	Protocol	Video / sample units	QA	Avg. dur.	Hours	Primary scope
Offline and long-video benchmarks
EgoSchema ( 2023 )	Offline	5,031 clips	5,031	3.0 min	251.6	Long-form egocentric reasoning
MVBench ( 2024 )	Offline	4,000 samples a	4,000	5–35 s	–	Twenty temporal skills
LongVideoBench ( 2024 )	Offline	3,763 videos	6,678	7.9 min	494.5 b	Referring and long-context reasoning
MLVU ( 2025 )	Offline	1,730 videos	3,102	15.5 min	446.9 b	Multi-task long-video understanding
LVBench ( 2025a )	Offline	103 videos	1,549	68.4 min	117.0	Extreme-duration video QA

Appendix

Table 10: Scale of offline video-understanding benchmarks. Counts follow the canonical paper or official release. A video unit denotes a source video, extracted clip, or derived episode. Dashes mark values not reported in a directly comparable form.

Benchmark	Protocol	Video / sample units	QA	Avg. dur.	Hours	Primary scope
Dynamic-environment, online, and uncertainty-aware benchmarks
Env-QA ( 2021 )	Offline	23,261 videos	85,072	0.33 min	129.2 b	Multi-event environment-state QA
SVBench ( 2025 )	Online	1,353 videos	49,979	–	–	Temporal multi-turn streaming QA
StreamBench ( 2025 )	Online	306 videos	$\sim$ 1,800	4.5 min	24.8	Online multi-turn interaction
VStream-QA ( 2024 )	Online	32 source videos c	3,370	40 min c	21.3 c	Long-stream retrieval and QA
StreamingBench ( 2024 )	Online	900 videos	4,500	–	–	Timestamped streaming perception

Appendix

Table 11: Scale of dynamic-environment and uncertainty-aware video benchmarks. “Online” excludes frames after the query time; “Deferred” permits later evidence; and “Mixed” combines protocols. The final two rows are the proposed real and simulated tracks.

	Low Change		Medium Change		High Change		Summary
Model	K	U	K	U	K	U	Worst	OC	BR-Attr
Qwen2.5-VL-3B ( 2025 )	57.79	10.53	53.12	10.43	49.72	12.74	10.43	66.41	9.85
Qwen2.5-VL-7B ( 2025 )	57.08	21.05	52.67	23.91	48.79	21.70	21.05	38.42	16.28
Qwen3-VL-4B ( 2025 )	58.64	17.11	48.92	20.43	43.18	16.51	16.51	20.66	13.14
Qwen3-VL-8B ( 2025 )	70.25	18.42	65.83	5.22	63.93	8.02	5.22	41.12	8.37
Qwen3.5-4B ( 2026 )	75.78	25.00	70.49	25.22	63.93	17.92	17.92	41.70	21.56

Appendix

Table 12: Complete six-cell decomposition on Cook under the online protocol (RQ3). Each cell reports joint answer–status accuracy (%). K/U denote Known/Uncertain ground truth; L/M/H denote change-intensity bins (1–3, 4–6, and 7–10). Worst indicates the minimum cell; OC is the overconfidence rate on Uncertain items. BR-Attr is the unsmoothed harmonic score with item-level cause-F1 modulation on Uncertain items (Eq. ( 4 )).

	Low Change		Medium Change		High Change		Summary
Model	K	U	K	U	K	U	Worst	OC	BR
Qwen2.5-VL-3B ( 2025 )	9.62	24.14	8.45	22.09	10.20	16.28	8.45	10.76	13.48
Qwen2.5-VL-7B ( 2025 )	7.31	41.38	3.17	32.56	3.06	18.60	3.06	1.27	7.78
Qwen3-VL-4B ( 2025 )	54.62	31.03	35.92	20.93	34.69	18.60	18.60	23.42	29.71
Qwen3-VL-8B ( 2025 )	56.92	17.24	49.30	9.30	51.02	11.63	9.30	31.65	21.28
Qwen3.5-4B ( 2026 )	56.15	27.59	50.00	33.72	53.06	16.28	16.28	32.91	33.95

Appendix

Table 13: Transfer decomposition: complete six-cell reliability on Health under the online protocol (RQ3). Each cell reports joint answer–status accuracy (%). K/U denote Known/Uncertain ground truth and L/M/H denote change-intensity bins (1–3, 4–6, and 7–10). Occupancy is L

=

260 Known / 29 Uncertain, M

=

284 / 86, and H

=

98 / 43. Worst indicates the minimum cell; OC is the overconfidence rate on Uncertain items. BR is the Laplace-smoothed six-cell Storm-BR .

	Low Change		Medium Change		High Change		Summary
Model	K	U	K	U	K	U	Worst	OC	BR
Qwen2.5-VL-3B ( 2025 )	6.18	22.00	4.52	12.82	9.52	15.62	4.52	13.12	9.62
Qwen2.5-VL-7B ( 2025 )	1.97	24.00	2.26	19.23	1.59	28.12	1.59	3.75	4.73
Qwen3-VL-4B ( 2025 )	38.20	10.00	28.96	14.10	31.75	21.88	10.00	23.75	20.89
Qwen3-VL-8B ( 2025 )	39.33	8.00	37.10	8.97	30.16	9.38	8.00	36.25	16.05
Qwen3.5-4B ( 2026 )	34.55	10.00	34.84	8.97	31.75	15.62	8.97	34.38	18.09

Appendix

Table 14: Transfer decomposition: complete six-cell reliability on Sports under the online protocol (RQ3). Each cell reports joint answer–status accuracy (%). K/U denote Known/Uncertain ground truth and L/M/H denote change-intensity bins (1–3, 4–6, and 7–10). Occupancy is L

=

356 Known / 50 Uncertain, M

=

221 / 78, and H

=

63 / 32. Worst indicates the minimum cell; OC is the overconfidence rate on Uncertain items. BR is the Laplace-smoothed six-cell Storm-BR .

Model	Full	Online	$\Delta$	Model	Full	Online	$\Delta$
Qwen2.5-VL-3B	18.52	19.22	-0.71	MiniCPM-V 4.5	21.87	22.17	-0.30
Qwen2.5-VL-7B	31.32	31.60	-0.28	InternVideo2.5	13.33	13.39	-0.06
Qwen3-VL-4B	26.69	26.77	-0.08	Eagle2.5-8B	21.80	23.63	-1.83
Qwen3-VL-8B	15.99	15.21	+0.78	VideoLLaMA3	10.02	11.10	-1.08
Qwen3.5-4B	32.01	34.04	-2.03	LLaVA-NeXT	1.13	1.13	0.00
Qwen3.5-9B	29.33	33.29	-3.96	GLM-4.1V-Think	19.63	21.40	-1.77

Appendix

Table 15: Full-video versus online Storm-BR on Cook (RQ2). Both columns report the six-cell Storm-BR metric.

\Delta

denotes full-video minus online performance in percentage points.

Domain pair	Accuracy	Known joint	Uncertain joint	Storm-BR
Cook–Bike	0.71	0.54	0.80	0.34
Cook–Health	0.84	0.67	0.76	0.54
Cook–Music	0.54	0.64	0.79	0.54
Cook–Sports	0.85	0.69	0.71	0.52
Bike–Health	0.60	0.69	0.84	0.36
Bike–Music	0.42	0.64	0.76	0.54

Appendix

Table 16: What transfers across domains: model-rank agreement on STORM -Real (RQ1). Kendall’s

\tau_{b}

is computed across all 14 models for each domain pair. Known and Uncertain are status-conditioned joint accuracies. Mean

\tau_{b}

summarises the 10 pairs; Kendall’s

W

measures joint agreement across all five domains.

Model	Cook	Bike	Health	Music	Sports
Qwen2.5-VL-3B	+42.4	-9.2	-11.7	-33.3	-10.3
Qwen2.5-VL-7B	+30.6	-26.0	-25.6	-45.8	-20.5
Qwen3-VL-4B	+32.4	+25.2	+21.2	-3.1	+20.0
Qwen3-VL-8B	+58.5	+48.4	+41.3	+22.9	+28.9
Qwen3.5-4B	+48.4	-24.2	+25.1	+1.0	+23.8
Qwen3.5-9B	+55.4	+1.2	+29.9	+16.7	+16.4

Appendix

Table 17: Which side is the bottleneck: status-conditioned reliability balance across STORM -Real domains (RQ1, RQ3). Each value is Known minus Uncertain joint accuracy in percentage points. Positive values indicate that Uncertain performance is limiting; negative values indicate that Known performance is limiting.

Model	$\Delta$ Acc.	$\Delta$ Known	$\Delta$ Uncertain	$\Delta$ OC	$\Delta$ UC	$\Delta$ BR
THOR minus Cook
Qwen2.5-VL-3B	-8.67	-27.69	+4.97	-59.14	+48.15	-8.92
Qwen2.5-VL-7B	-13.39	-38.06	-4.41	-38.42	+58.32	-21.43
Qwen3-VL-4B	-14.36	-1.32	+8.93	+2.98	-14.78	-7.50
Qwen3-VL-8B	-20.66	-21.52	+4.43	-17.48	+4.78	-3.98
Qwen3.5-4B	-18.26	-54.27	+8.71	-41.70	+64.09	-18.62

Appendix

Table 18: Cook-to-Sim shifts in model behaviour (RQ1). Deltas are the indicated simulation subset minus Cook in percentage points. Known/Uncertain denote status-conditioned joint accuracy; OC and UC denote overconfidence and underconfidence rates.

Model	Full-video	Online	KCI	Epi.	Storm-BR
THOR
Qwen2.5-VL-3B ( 2025 )	40.33	45.00	45.35	21.24	10.30 [7.7–11.9]
Qwen2.5-VL-7B ( 2025 )	44.00	47.67	44.88	16.64	10.17 [7.4–11.6]
Qwen3-VL-4B ( 2025 )	48.00	51.67	53.67	38.33	19.27 [10.8–27.7]
Qwen3-VL-8B ( 2025 )	41.00	44.00	48.21	29.02	11.23 [8.0–12.9]
Qwen3.5-4B ( 2026 )	48.00	55.00	51.68	23.62	15.42 [8.3–20.0]

Appendix

Table 19: Controlled stress test: complete 14-model reliability on STORM -Sim (RQ1, RQ2). THOR contains 300 items over 28 AI2-THOR episodes; VHome adds 290 items over 29 VirtualHome episodes. Full-video and Online report task-choice accuracy with the complete sequence and the query-time prefix, respectively; both use the original query-time target labels. KCI, Epi., and Storm-BR use online predictions. KCI is Known Change-Intensity; Epi. is the Known/Uncertain joint-accuracy macro-average. Storm-BR is the Laplace-smoothed harmonic mean over the six cells, with 95% episode-clustered bootstrap intervals (1,000 resamples). In THOR, Low-change Uncertain has

n{=}8

; that cell is a transfer diagnostic. Best and second-best in each column are bold and underlined. All values are percentages.

Model	K joint	U joint	Hist.	Recent–Old
Q2.5-VL-3B	19.55	17.85	32.66	+0.94
Q2.5-VL-7B	13.80	28.52	31.39	-3.53
Q3-VL-4B	40.08	22.88	42.55	-3.69
Q3-VL-8B	46.30	11.40	34.82	+5.47
Q3.5-4B	33.30	28.02	34.10	+2.37
Q3.5-9B	40.05	28.80	35.61	+6.88

Appendix

Table 20: Domain-macro diagnostics from frozen online predictions. Known (K) and Uncertain (U) joint accuracy and history-aggregation task accuracy (Hist.) average seven subsets equally. Recent–Old is the joint-accuracy difference between the 0–2s and

>5

s slices, averaged over the five real domains and VHome; THOR has no old-evidence items. All values are percentages or percentage-point differences.

Model	Current	Fact	History	Change	Track	Temporal	Recent	Old
Qwen2.5-VL-3B	53.81	64.80	29.19	66.74	63.74	44.09	46.77	37.85
Qwen2.5-VL-7B	58.74	70.63	35.97	72.44	74.10	54.77	48.48	38.49
Qwen3-VL-4B	65.92	74.36	45.48	72.44	76.80	61.36	44.35	46.06
Qwen3-VL-8B	63.68	73.43	42.99	73.35	71.62	63.18	57.35	43.53
Qwen3.5-4B	75.11	84.62	47.74	82.69	81.98	67.73	62.24	54.89
Qwen3.5-9B	78.03	85.31	55.66	83.83	84.01	71.82	66.50	58.99

Appendix

Table 21: What temporal tracking requires: semantic-type and evidence-age results on Cook (RQ4). The six type columns report online accuracy (%). Recent and Old report joint answer–status accuracy for evidence ending 0–2 s and more than 5 s before the query. All values cover the same 14-model panel visualized in Figure 11 a,b.

Source	$n$	Joint	Attribution F1
Missing observation	405	11.23	54.14
Partial observation	187	13.37	1.04
Ambiguous evidence	142	16.55	1.40
Low visual quality	71	34.51	24.07
Multiple candidates	26	17.31	3.12
Ambiguous attribute	15	40.00	0.00

Appendix

Table 22: Where uncertainty attribution fails: source-conditioned results on Cook (RQ3). Values are medians over 14 models. Joint is answer–status correctness conditioned on each ground-truth source; Attribution is the source-specific F1. Sources are multi-label and may overlap.

Domain	$\tau$ rank	Raw top	Smooth top	$\|\Delta\|_{\max}$ (pp)	Zero / 14
Cook	1.00	Molmo2-8B	Molmo2-8B	1.13	1
Bike	0.95	Molmo2-8B	Molmo2-8B	8.72	3
Health	0.96	Qwen3.5-4B	Qwen3.5-4B	3.98	1
Sports	0.93	Molmo2-8B	Molmo2-8B	8.29	2
THOR	0.97	Molmo2-8B	Molmo2-8B	14.00	8
VHome	0.91	Molmo2-8B	Molmo2-8B	13.25	4

Appendix

Table 23: Metric stability: raw versus add-one-smoothed Storm-BR (RQ2). Both scores are recomputed from identical frozen prediction files. Kendall

\tau

compares the 14-model raw and smoothed rankings, with tied pairs dropped.

|\Delta|_{\max}

is the largest model-wise score change in percentage points; a raw zero occurs when at least one joint cell has no correct prediction. Smoothing does not change the leader on any listed domain, while making rare-cell scores finite and comparable.

Domain	$n$	$\tau$	$\|\Delta\|_{\max}$ (pp)	LOO span (pp)	Leader
Cook	2,640	0.98	1.17	6.0–10.5	Molmo2-8B
Bike	786	0.91	5.53	6.3–14.3	Molmo2-8B
Health	800	0.93	2.78	7.4–12.8	Qwen3.5-4B
Sports	800	0.98	2.22	5.1–8.5	Molmo2-8B
THOR	300	0.52	16.52	8.7–15.9	Molmo2-8B
Music	120	n/a	n/a	n/a	Molmo2-8B

Appendix

Table 24: Aggregation robustness of Storm-BR on sparse cells (RQ2). The collapsed variant pools the three change-intensity bins into a single Known/Uncertain pair; leave-one-cell-out (LOO) drops each of the six cells in turn.

\tau

is the Kendall rank correlation between the 14-model six-cell and collapsed scores,

|\Delta|_{\max}

is the largest model-wise score change, and the LOO column summarizes score sensitivity to single-cell deletion. Collapsing preserves the leader on every evaluated subset, while THOR changes scores most (up to 16.5 pp).

Domain	$\tau$ median [min]	Leader in top-3	Top-3 set	$\|\Delta\|_{\max}$ (pp)	Known–Unc. sign
Cook	0.96 [0.91]	15/15	14/15	4.89	100.0%
Bike	0.93 [0.85]	15/15	15/15	9.63	98.1%
Health	0.96 [0.78]	15/15	6/15	7.76	100.0%
Sports	0.96 [0.89]	15/15	15/15	7.21	100.0%
THOR	0.76 [0.69]	15/15	3/15	9.36	89.0%
VHome	0.78 [0.67]	12/12	0/12	4.57	95.2%

Appendix

Table 25: Sensitivity to alternative change-intensity partitions (RQ2). Each of the 15 alternative partitions reuses the same 14 frozen online prediction files and changes only the two cutpoints that define the Low/Medium/High bins (canonical cuts

(3,6)

). The alternatives are the nearest valid cutpoint pairs to

(3,6)

; validity requires six non-empty intensity–answerability cells. VHome uses the 12 of these 15 alternatives that satisfy this condition; the three cuts starting at 1 leave its Low bin empty.

\tau

is the Kendall rank correlation between the 14-model ranking and the canonical ranking. “Leader in top-3” counts partitions in which the canonical leader remains within the top three, “Top-3 set” counts partitions that preserve the canonical top-3 set, and

|\Delta|_{\max}

is the largest change among the canonical top-3 scores in percentage points. The last column is the fraction of model

\times

partition comparisons that preserve the sign of the canonical Known–Uncertain reliability gap. Music lacks high-intensity queries (

B_{3}

) and is excluded from six-cell partition audits.

Model	Control	Sampling / cap	Acc.	KCI	Epi.	BR
Qwen2.5-VL-3B	Question + choices	no video	35.08	34.88	19.75	15.70
Qwen2.5-VL-3B	Shuffled frames	1 FPS / all	39.89	39.58	12.68	8.34
Qwen2.5-VL-3B	Recent history	1 FPS / 16	42.77	42.64	15.35	11.51
Qwen2.5-VL-7B	Question + choices	no video	42.54	42.34	30.31	0.28
Qwen2.5-VL-7B	Shuffled frames	1 FPS / all	46.48	46.40	17.00	2.84
Qwen2.5-VL-7B	Recent history	1 FPS / 16	48.64	48.76	19.61	4.53

Appendix

Table 26: Input configurations on Cook (RQ4). All 14 checkpoints are evaluated under text-only, shuffled-frame, and recent-16-frame input. Acc. is task accuracy; KCI is accuracy averaged over intensity bins; Epi. is Known/Uncertain joint-accuracy macro-average; BR is add-one-smoothed six-cell Storm-BR .

Model	Input / control	Sampling / cap	Acc.	BR
Qwen3.5-9B ( 2026 )	Official online	1 FPS / all	76.40	33.29
	Full-video offline	1 FPS / all	75.04	29.33
	Question + choices	no video	39.17	0.34
	Bounded recent history	1 FPS / 16	61.89	35.49
	Shuffled visible frames	1 FPS / all	57.54	33.56
Molmo2-8B ( 2026 )	Official online	1 FPS / all	76.33	34.45

Appendix

Table 27: Cook protocol and input-configuration results. Performance of two representative models under official online, full-video offline, and input-control configurations. Acc. denotes task accuracy, and BR denotes balanced reliability ( Storm-BR ).

Video QA

Video question answering studies how AI systems use visual and audio evidence over time to answer questions about video.

See all 184 papers

EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence
Jun 23, 2026Linpeng Huang, Weixing Chen, Zexin Chen +2VLM Evaluation Temporal Video Grounding
Sun Yat-sen University · Shenzhen University · Peng Cheng Laboratory
Evidence-Backed Video Question Answering
Jul 13, 2026Shijie Wang, Honglu Zhou, Ziyang Wang +5Temporal Video Grounding Video QA
Salesforce, Palo Alto, CA, USA · Brown University, Providence, RI, USA
VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification
Apr 2, 2026Jiahao Meng, Yue Tan, Qi Xu +9Video-Language Model Evaluation Long-Video QA
PKU · WHU · CASIA +2

Jun 23, 2026J/K move · Enter open · S save

EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence

Linpeng Huang, Weixing Chen, Zexin Chen, Yang Liu+1

Sun Yat-sen University · Shenzhen University · Peng Cheng Laboratory

VLM Evaluation Temporal Video Grounding Video QA

Open paper AskPDF arXiv

Recent advances in Video Large Language Models (Video-LLMs) have yielded promising performance on Video Question Answering (VideoQA). Nevertheless, existing benchmarks are predominantly evaluated through answer correctness, while the faithfulness of predicted evidence supporting those answers remains insufficiently evaluated. This disconnect between answer generation and evidence verification motivates the construction of the Evidence-Grounded Video Question Answering Benchmark (EG-VQA), a large-scale open-ended benchmark in which each QA pair is annotated with temporally localized textual evidence, requiring models to jointly produce answers and verifiable evidence. EG-VQA comprises 2,067 videos and 11,838 QA pairs with fine-grained evidence annotations. To evaluate predicted evidence, Evidence-Grounded F1 (EG-F1) is introduced as a metric that jointly measures temporal alignment and semantic consistency between predicted and ground-truth evidence. Experiments reveal a substantial discrepancy between answer correctness and evidence faithfulness: even strong proprietary models can achieve high answer accuracy while exhibiting gaps in temporal-semantic grounding. To investigate evidence-aware learning under EG-VQA, we develop EG-Reasoner, an evidence-aware model trained with explicit evidence supervision. EG-Reasoner achieves strong evidence grounding performance among open-source models while maintaining competitive answer generation results compared with proprietary systems. These findings highlight the importance of explicitly modeling evidence grounding and suggest that evidence-aware supervision provides an effective direction toward more reliable and interpretable VideoQA systems.

Abstract

Video QA

Explore similar work

EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence

Evidence-Backed Video Question Answering

VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification