Reliable online video question answering requires tracking state transitions while selectively abstaining when visual evidence is insufficient. Existing benchmarks focus on static recognition or long-range retrieval, rarely evaluating these coupled capabilities under evolving and incomplete evidence. We present STORM-Bench, comprising 5,736 questions across 630 compact, change-dense episodes spanning five egocentric domains (STORM-Real) and two controlled simulation subsets (STORM-Sim) at 1 FPS. Questions are stratified by a proxy for accumulated change intensity (Low, Medium, High) and query-time answerability (Known, Uncertain). To measure reliability, we introduce STORM-BR, a harmonic metric over joint answer-status correctness that exposes abstention failures masked by aggregate accuracy, alongside STORM-BR-ATTR for uncertainty attribution. Across 14 video LLMs, online accuracy peaks at 60.3% (mean 51.7%), whereas STORM-BR ranges from 5.7% to 35.6% (mean 18.8%), driven by pervasive overconfidence on uncertain queries. STORM-Bench shows that task accuracy masks these gaps in epistemic reliability and state tracking. Benchmark and code are available at https://github.com/siruzhong/STORM-Bench.
Figures & tables
Figure 1: Overview of STORM -Bench. (a) Domain mix with Real/Sim and Known/Uncertain splits. (b) Storm-BR ranking of 14 video LLMs. (c) Nearly matched accuracy hides a 7.50-point Storm-BR gap. (d) By change intensity: Known remains stronger for most models; Uncertain often collapses under overconfidence (seven-subset B1 – B3 average; cf. Table 20 , Figure 10 ).
Figure 2: Word cloud of question text. Temporal expressions, state terms, and references to objects, actions, and spatial relations summarize the vocabulary of the questions.
Figure 3: Overview of STORM -Bench. (a) Five real egocentric domains and four controlled simulation room categories. (b) Compact multi-visit episodes requiring update, aggregation, and abstention. (c) Six question types stratified by intensity × answerability, with uncertainty causes.
Figure 4: Compared to existing Video-QA suites, STORM -Bench combines compact durations (a), higher sampled visual-change rates (b), and denser QA supervision (c). Dots/bars denote medians; error bars in (b) show IQRs. See Appendix A.5.1 .
Figure 5: STORM -Bench diagnostic framework. Questions posed at query time tq are answered strictly from the observed history V≤tq across six semantic types and two diagnostic axes. Balanced reliability and fine-grained root-cause attribution are scored via Storm-BR and Storm-BR-Attr .
Figure 6: Real-scene episode construction pipeline. Region-aware timelines record object interactions and state transitions. Temporally separated visits to the same functional area are compiled into compact revisit episodes, followed by query-time QA grounding and cross-annotator validation.
Figure 7: Simulated-scene construction pipeline. Scripted rollouts generate video and execution traces; event extraction joins state changes with visual masks; query selection enforces causal bounds; question construction emits balanced options and diagnostic labels. The illustrated THOR rollout uses disappearance/reappearance events; VHome instantiates the same stages with native agent actions and sampled visibility records (Appendix A.2.5 ).
Model
Storm-BR by domain
Avg. BR
Avg. Acc
Cook
Bike
Health
Music
Sports
THOR
VHome
Molmo2-8B ( 2026 )
34.45 [27.4, 39.3]
33.59 [22.7, 40.4]
31.80 [18.4, 38.8]
43.29 [24.2, 55.4]
31.34 [21.8, 36.8]
34.15 [21.6, 42.4]
40.35 [31.8, 44.9]
35.57
60.31
Qwen3.5-9B ( 2026 )
33.29 [27.7, 37.8]
22.22 [12.2, 25.7]
33.23 [25.0, 38.7]
42.70 [25.1, 52.4]
24.06 [18.5, 28.2]
25.89 [12.6, 33.2]
15.10 [9.5, 18.1]
28.07
60.33
Qwen3-VL-4B ( 2025 )
26.77 [21.9, 30.4]
31.49 [23.1, 35.3]
29.71 [21.3, 34.8]
36.92 [20.9, 44.6]
20.89 [13.4, 25.2]
19.27 [10.8, 27.7]
18.96 [11.0, 24.9]
26.29
55.16
Qwen3.5-4B ( 2026 )
34.04 [28.8, 38.5]
8.72 [6.3, 12.3]
33.95 [23.8, 39.9]
36.97 [22.9, 46.0]
18.09 [11.0, 22.2]
15.42 [8.3, 20.0]
16.65 [8.1, 20.7]
23.41
58.20
MiniCPM-V 4.5 ( 2025 )
22.17 [16.5, 26.0]
31.97 [22.8, 37.1]
18.71 [8.5, 25.9]
27.89 [14.9, 38.8]
17.16 [8.5, 22.3]
21.70 [11.5, 26.2]
20.33 [10.9, 27.5]
22.84
54.78
Table 1: Main evaluation on STORM -Bench (RQ1). Online balanced reliability ( BR , Laplace-smoothed Storm-BR ) per domain with 95% bootstrap intervals, overall average BR ( Avg. BR ), and accuracy ( Avg. Acc ). Both are unweighted means across five real domains, THOR, and VHome.
Figure 8: Whole-bench evaluation across five real domains and two simulation subsets. (a) Average Storm-BR across seven evaluation subsets. (b) Per-domain Storm-BR scores. (c) Parameter scaling across matched model families. (d) Domain difficulty distribution across all 14 models.
Figure 9: Diagnostic protocol and metric analysis (Cook domain except panel b). (a) Online accuracy vs. Storm-BR (marker size ∝ scale). (b) Per-domain Kendall’s τ (95% bootstrap CIs; highlighted bar pools domains). (c,d) Shift in Storm-BR and accuracy under full-video access.
Figure 10: Answerability diagnostics and error analysis on Cook. (a) Joint correctness across intensity × answerability cells. (b) Known vs. Uncertain accuracy with overconfidence ( OC ). (c) Joint correctness vs. cause F1 across sources. (d) Stratification by OC (dashed diagonal indicates parity).
Figure 11: Temporal diagnostics and input controls. (a) Macro accuracy across question types. (b) Cross-model median and IQR by evidence age. (c) Cook joint accuracy across intensity bins ( B1 – B3 ). (d) Accuracy and Storm-BR across input controls on Cook.
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Kitchen
Living Room
Bedroom
Bathroom
Scene numbers
7, 12, 13, 19, 20, 27, 28
202, 209, 211, 221, 222, 223, 229
301, 306, 308, 318, 327, 329, 330
401, 404, 406, 408, 410, 422, 428
Camera–target distance (m)
0.5–1.2
0.7–2.2
0.6–2.0
0.5–1.2
Maximum segment speed (m/s)
0.85
2.20
1.60
1.30
Maximum station step (m)
3.5
8.0
5.0
4.0
Minimum target area (pixels)
1,400
600
600
500
Minimum box dimension (pixels)
28
18
18
16
Appendix
Table 2: Simulation rollout settings. THOR scene numbers denote the suffix of FloorPlan . Each THOR room category contributes seven videos and 75 QA items; VHome settings are summarized below. Pixel-area and bounding-box thresholds apply when the target is required to be visible.
Type
Known target (THOR; VHome)
Uncertain target
THOR K/U
VHome K/U
Current state
Room; current visibility or physical state
Hidden physical state or location
53/1
51/1
Factual retrieval
Room; earlier visibility or physical state
State during missing observation
52/2
50/2
History aggregation
Last object in a return pair; return/exit counts
Number of hidden manipulations
39/12
37/12
State change
Appearance change; visibility or reviewed state sequence
Cause of an unobserved change
48/3
46/3
Object tracking
Returning category; return or reviewed object state
Physical identity across a gap
26/24
26/23
Temporal reasoning
Return order; return or reviewed state order
Ordering of unobserved actions
27/13
27/12
Appendix
Table 3: Simulation question construction and release counts. K/U denote Known/Uncertain. Known-target descriptions give THOR followed by VHome; Uncertain builders query hidden physical facts in both subsets.
Type
Operational Definition
Example Question
Factual retrieval
Retrieve an object, attribute, or location from an earlier principal interval after the target leaves the view.
“What color was the towel previously seen on the radiator?”
Current state
Integrate visible history up to tq to determine an entity’s latest state or location after observed changes.
“Where is the spice bottle at tq after the observed relocations?”
State change
Compare visual evidence across a transition boundary to identify how physical state or placement changed.
“How did the milk frother’s state change between 4 s and 7 s?”
Object tracking
Determine whether observations across timestamps refer to the same instance; report uncertainty if ambiguous.
“Which earlier bowl, if any, is the bowl observed after the transition?”
Temporal reasoning
Determine event ordering, first/last, or k -th occurrence of actions without relying on common-sense priors.
“Which occurred first: rinsing the knife, opening the drawer, or placing the bowl?”
History aggregation
Count, accumulate, or summarize discrete visible action segments or entities up to tq under a predefined unit.
“How many distinct visible drawer-opening segments occurred by tq ?”
Appendix
Table 4: Semantic question types in STORM -Real. Each type targets a temporal reasoning operation, annotated independently from accumulated change intensity and epistemic status. Table 3 specifies the corresponding Sim builders.
Domain
QA
Ep.
Hours
Dur.
QA/Ep.
Known
Unc.
L/M/H
STORM -Real / Cook
2,640
314
3.739
42.86 s
8.41
2,122
518
782 / 1,111 / 747
STORM -Real / Bike
786
61
0.661
39.00 s
12.89
638
148
371 / 370 / 45
STORM -Real / Health
800
83
0.948
41.13 s
9.64
642
158
289 / 370 / 141
STORM -Real / Music
120
16
0.186
41.74 s
7.50
96
24
68 / 52 / 0
STORM -Real / Sports
800
99
1.126
40.93 s
8.08
640
160
406 / 299 / 95
THOR
300
28
0.467
60.05 s
10.71
245
55
37 / 57 / 206
Appendix
Table 5: Dataset overview, scale, and diagnostic composition. L/M/H denote low (1–3), medium (4–6), and high (7–10) change-intensity bins. Dur. denotes the mean video duration per episode in seconds. QA/Ep. denotes the average number of QA pairs per episode. Domain scores are evaluated independently without pooling.
Question-Type Distribution
Domain
Current state
Factual retrieval
History aggregation
State change
Object tracking
Temporal reasoning
All
Cook
446
429
442
439
444
440
2,640
Bike
142
142
120
137
145
100
786
Health
158
167
103
155
133
84
800
Music
25
26
11
22
22
14
120
Sports
170
171
96
148
141
74
800
Appendix
Table 6: Benchmark structural composition: question types, uncertainty sources, and evidence protocols. Evidence ends denotes the fraction of items whose last evidence endpoint equals the query time; mean duration excludes instantaneous spans; the last column counts items with at least one span longer than 10 seconds.
Dataset
Duration n
Median (s)
V/W
QA/min
Change/min [IQR]
EgoSchema
1,157
180.00
15/43
0.33
13.67 [10.50, 16.97]
LongVideoBench
3,991
60.80
15/31
0.23
7.45 [4.46, 14.85]
LVBench
103
3664.41
15/45
0.22
14.35 [10.07, 20.58]
MLVU
2,106
480.02
15/45
0.12
18.00 [10.77, 23.35]
MVBench
5,061
11.97
13/15
3.13
9.23 [8.16, 12.41]
Video-MME
900
487.94
15/41
0.18
17.98 [13.04, 23.51]
Appendix
Table 7: Coverage and descriptive statistics for Figure 4 . Duration statistics use local media; change statistics use selected windows. QA density follows the supplied QA/video-time totals. STORM-Sim combines THOR and VHome (component rows use their portions of the joint sample); its 15 sampled episodes comprise seven THOR and eight VHome episodes. Duration and change values are medians, with the change interquartile range in brackets; V/W denotes source videos/windows.
Figure 12: Dataset patterns across domains. Inventory counts follow release manifests (Tables 5 and 6 ). (A) Six-cell Known / Uncertain occupancy for Storm-BR , with an empty high-change bin for Music. (B) Within-type Uncertain rates are near 20% in STORM -Real, while both simulation subsets concentrate uncertainty in tracking and temporal items. (C) Uncertainty-source mix (labels per Uncertain item); missing observation dominates except in Health and Music. (D) Most STORM -Real items have evidence ending within 2 s of query, whereas THOR places every latest endpoint at tq ; VHome includes delayed retrieval queries.
Figure 13: Cook: recognizing the direction of a state change. At tq=9s , visual evidence indicates that the digital scale reading changes from 554 to zero. Qwen3.5-9B correctly tracks this state transition, whereas Qwen3-VL-8B predicts the inverted direction. Both state probes correctly predict Known , demonstrating that accurate epistemic-status classification does not guarantee correct interpretation of the underlying dynamic change.
Figure 14: Bike Repair: identifying a completed physical separation. At tq=31s , the tire is visibly separated from the rim. Both models select the correct task option, but Qwen3.5-9B incorrectly predicts Uncertain in the decoupled probe, whereas Qwen3-VL-8B correctly identifies Known . This highlights the decoupling between task answer selection and epistemic confidence verification.
Figure 15: Health: grounding current object configuration. At tq=8s , the COVID test box is tilted open with its white plastic tray partially extracted. Qwen3.5-9B correctly identifies this spatial configuration, whereas Qwen3-VL-8B selects a distractor asserting the box lies flat with its lid tucked under. While both models correctly report Known , they diverge on fine-grained spatial interpretation.
Figure 16: Music: avoiding unsupported event-completion claims. Visual history does not establish a completed page turn by tq=27s , mandating Cannot be determined . Both models correctly predict Uncertain in the decoupled probe, yet overconfidently commit to an option claiming a completed turn, illustrating a dissociation between uncertainty recognition and option choice.
Figure 17: Sports: localizing a dynamic object at query time. At tq=16s , the basketball is located inside the hoop. Qwen3-VL-8B correctly grounds its query-time position and predicts Known . In contrast, Qwen3.5-9B misattributes the position to the backboard and incorrectly predicts Uncertain in the status probe.
Figure 18: THOR: distinguishing visual reappearance from instance identity. A hand towel appears at 14 s, the ring becomes empty at 38 s, and a towel reappears at 40–41 s. At tq=41s , the history does not uniquely identify whether the returning object is the identical physical instance. Qwen3.5-9B correctly identifies Uncertain in the decoupled probe, yet overconfidently selects the same-instance answer in the task interface.
Figure 19: VHome: an observation gap leaves the action unresolved. The displayed views bracket a gap in which the target and its operator are outside the observer’s view. At tq=17.05 s, the visible prefix does not determine the action that caused the change. Molmo2-8B selects the abstention option and predicts Uncertain ; Qwen3.5-9B also predicts Uncertain but chooses an unsupported physical explanation. Both predictions use the complete causal prefix, including the explicit timing and observation contract (Appendix A.2.5 ).
Model
Family
Max
Unit
Model
Family
Max
Unit
Qwen2.5-VL-3B
Qwen
768
2
Molmo2-8B
Molmo2
384
1
Qwen2.5-VL-7B
Qwen
768
2
MiniCPM-V 4.5
MiniCPM
180
1 (pack 3)
Qwen3-VL-4B
Qwen
768
2
InternVideo2.5-8B
InternVideo
512
4
Qwen3-VL-8B
Qwen
768
2
Eagle2.5-8B
Eagle
256
2
Qwen3.5-4B
Qwen
768
2
VideoLLaMA3-7B
VideoLLaMA3
128
2
Qwen3.5-9B
Qwen
768
2
LLaVA-NeXT-Video-7B-DPO
LLaVA-NeXT-Video
32
1
Appendix
Table 8: Per-model frame budget under the 1 FPS online protocol (RQ1). Checkpoints share the 1 FPS clock and retain recent frames ( Max ) up to native caps, padding the final frame to temporal unit ( Unit ). A larger cap is not uniformly better: a bounded 16-frame window improves Storm-BR for 9 of 14 models on Cook (Table 26 ). MiniCPM-V 4.5 packs up to three frames per temporal group.
Benchmark
Temporal Protocol
Temporal Target
Answerability Target
Diagnostic Focus
Reported Metrics
EgoSchema ( 2023 )
Offline
Extended context
Forced choice
Egocentric QA
Accuracy
StreamingBench ( 2024 )
Online
Timestamped QA
Forced choice
Online understanding
Accuracy
SVCBench ( 2026 )
Online
Count trajectories
Numeric answers
Counts and updates
GPA/MoC/UDA
EGOSTREAM ( 2026 )
Online
Validity windows
Valid-answer recall
Memory retention
Recall-group Acc.
EgoSAT ( 2026 )
Online
Past/present/future
Future predictability
Confidence diagnostics
Task Acc./Conf.
StreamReady ( 2026 )
Readiness
Evidence windows
Response readiness
Correctness and timing
Acc./ARS
Appendix
Table 9: Taxonomic comparison with representative Video-QA benchmark protocols. Existing suites cover complementary temporal and answerability targets; STORM -Bench couples accumulated change intensity with query-time answerability under an online protocol.
Benchmark
Protocol
Video / sample units
QA
Avg. dur.
Hours
Primary scope
Offline and long-video benchmarks
EgoSchema ( 2023 )
Offline
5,031 clips
5,031
3.0 min
251.6
Long-form egocentric reasoning
MVBench ( 2024 )
Offline
4,000 samples a
4,000
5–35 s
–
Twenty temporal skills
LongVideoBench ( 2024 )
Offline
3,763 videos
6,678
7.9 min
494.5 b
Referring and long-context reasoning
MLVU ( 2025 )
Offline
1,730 videos
3,102
15.5 min
446.9 b
Multi-task long-video understanding
LVBench ( 2025a )
Offline
103 videos
1,549
68.4 min
117.0
Extreme-duration video QA
Appendix
Table 10: Scale of offline video-understanding benchmarks. Counts follow the canonical paper or official release. A video unit denotes a source video, extracted clip, or derived episode. Dashes mark values not reported in a directly comparable form.
Benchmark
Protocol
Video / sample units
QA
Avg. dur.
Hours
Primary scope
Dynamic-environment, online, and uncertainty-aware benchmarks
Env-QA ( 2021 )
Offline
23,261 videos
85,072
0.33 min
129.2 b
Multi-event environment-state QA
SVBench ( 2025 )
Online
1,353 videos
49,979
–
–
Temporal multi-turn streaming QA
StreamBench ( 2025 )
Online
306 videos
∼ 1,800
4.5 min
24.8
Online multi-turn interaction
VStream-QA ( 2024 )
Online
32 source videos c
3,370
40 min c
21.3 c
Long-stream retrieval and QA
StreamingBench ( 2024 )
Online
900 videos
4,500
–
–
Timestamped streaming perception
Appendix
Table 11: Scale of dynamic-environment and uncertainty-aware video benchmarks. “Online” excludes frames after the query time; “Deferred” permits later evidence; and “Mixed” combines protocols. The final two rows are the proposed real and simulated tracks.
Low Change
Medium Change
High Change
Summary
Model
K
U
K
U
K
U
Worst
OC
BR-Attr
Qwen2.5-VL-3B ( 2025 )
57.79
10.53
53.12
10.43
49.72
12.74
10.43
66.41
9.85
Qwen2.5-VL-7B ( 2025 )
57.08
21.05
52.67
23.91
48.79
21.70
21.05
38.42
16.28
Qwen3-VL-4B ( 2025 )
58.64
17.11
48.92
20.43
43.18
16.51
16.51
20.66
13.14
Qwen3-VL-8B ( 2025 )
70.25
18.42
65.83
5.22
63.93
8.02
5.22
41.12
8.37
Qwen3.5-4B ( 2026 )
75.78
25.00
70.49
25.22
63.93
17.92
17.92
41.70
21.56
Appendix
Table 12: Complete six-cell decomposition on Cook under the online protocol (RQ3). Each cell reports joint answer–status accuracy (%). K/U denote Known/Uncertain ground truth; L/M/H denote change-intensity bins (1–3, 4–6, and 7–10). Worst indicates the minimum cell; OC is the overconfidence rate on Uncertain items. BR-Attr is the unsmoothed harmonic score with item-level cause-F1 modulation on Uncertain items (Eq. ( 4 )).
Low Change
Medium Change
High Change
Summary
Model
K
U
K
U
K
U
Worst
OC
BR
Qwen2.5-VL-3B ( 2025 )
9.62
24.14
8.45
22.09
10.20
16.28
8.45
10.76
13.48
Qwen2.5-VL-7B ( 2025 )
7.31
41.38
3.17
32.56
3.06
18.60
3.06
1.27
7.78
Qwen3-VL-4B ( 2025 )
54.62
31.03
35.92
20.93
34.69
18.60
18.60
23.42
29.71
Qwen3-VL-8B ( 2025 )
56.92
17.24
49.30
9.30
51.02
11.63
9.30
31.65
21.28
Qwen3.5-4B ( 2026 )
56.15
27.59
50.00
33.72
53.06
16.28
16.28
32.91
33.95
Appendix
Table 13: Transfer decomposition: complete six-cell reliability on Health under the online protocol (RQ3). Each cell reports joint answer–status accuracy (%). K/U denote Known/Uncertain ground truth and L/M/H denote change-intensity bins (1–3, 4–6, and 7–10). Occupancy is L = 260 Known / 29 Uncertain, M = 284 / 86, and H = 98 / 43. Worst indicates the minimum cell; OC is the overconfidence rate on Uncertain items. BR is the Laplace-smoothed six-cell Storm-BR .
Low Change
Medium Change
High Change
Summary
Model
K
U
K
U
K
U
Worst
OC
BR
Qwen2.5-VL-3B ( 2025 )
6.18
22.00
4.52
12.82
9.52
15.62
4.52
13.12
9.62
Qwen2.5-VL-7B ( 2025 )
1.97
24.00
2.26
19.23
1.59
28.12
1.59
3.75
4.73
Qwen3-VL-4B ( 2025 )
38.20
10.00
28.96
14.10
31.75
21.88
10.00
23.75
20.89
Qwen3-VL-8B ( 2025 )
39.33
8.00
37.10
8.97
30.16
9.38
8.00
36.25
16.05
Qwen3.5-4B ( 2026 )
34.55
10.00
34.84
8.97
31.75
15.62
8.97
34.38
18.09
Appendix
Table 14: Transfer decomposition: complete six-cell reliability on Sports under the online protocol (RQ3). Each cell reports joint answer–status accuracy (%). K/U denote Known/Uncertain ground truth and L/M/H denote change-intensity bins (1–3, 4–6, and 7–10). Occupancy is L = 356 Known / 50 Uncertain, M = 221 / 78, and H = 63 / 32. Worst indicates the minimum cell; OC is the overconfidence rate on Uncertain items. BR is the Laplace-smoothed six-cell Storm-BR .
Model
Full
Online
Δ
Model
Full
Online
Δ
Qwen2.5-VL-3B
18.52
19.22
-0.71
MiniCPM-V 4.5
21.87
22.17
-0.30
Qwen2.5-VL-7B
31.32
31.60
-0.28
InternVideo2.5
13.33
13.39
-0.06
Qwen3-VL-4B
26.69
26.77
-0.08
Eagle2.5-8B
21.80
23.63
-1.83
Qwen3-VL-8B
15.99
15.21
+0.78
VideoLLaMA3
10.02
11.10
-1.08
Qwen3.5-4B
32.01
34.04
-2.03
LLaVA-NeXT
1.13
1.13
0.00
Qwen3.5-9B
29.33
33.29
-3.96
GLM-4.1V-Think
19.63
21.40
-1.77
Appendix
Table 15: Full-video versus online Storm-BR on Cook (RQ2). Both columns report the six-cell Storm-BR metric. Δ denotes full-video minus online performance in percentage points.
Domain pair
Accuracy
Known joint
Uncertain joint
Storm-BR
Cook–Bike
0.71
0.54
0.80
0.34
Cook–Health
0.84
0.67
0.76
0.54
Cook–Music
0.54
0.64
0.79
0.54
Cook–Sports
0.85
0.69
0.71
0.52
Bike–Health
0.60
0.69
0.84
0.36
Bike–Music
0.42
0.64
0.76
0.54
Appendix
Table 16: What transfers across domains: model-rank agreement on STORM -Real (RQ1). Kendall’s τb is computed across all 14 models for each domain pair. Known and Uncertain are status-conditioned joint accuracies. Mean τb summarises the 10 pairs; Kendall’s W measures joint agreement across all five domains.
Model
Cook
Bike
Health
Music
Sports
Qwen2.5-VL-3B
+42.4
-9.2
-11.7
-33.3
-10.3
Qwen2.5-VL-7B
+30.6
-26.0
-25.6
-45.8
-20.5
Qwen3-VL-4B
+32.4
+25.2
+21.2
-3.1
+20.0
Qwen3-VL-8B
+58.5
+48.4
+41.3
+22.9
+28.9
Qwen3.5-4B
+48.4
-24.2
+25.1
+1.0
+23.8
Qwen3.5-9B
+55.4
+1.2
+29.9
+16.7
+16.4
Appendix
Table 17: Which side is the bottleneck: status-conditioned reliability balance across STORM -Real domains (RQ1, RQ3). Each value is Known minus Uncertain joint accuracy in percentage points. Positive values indicate that Uncertain performance is limiting; negative values indicate that Known performance is limiting.
Model
Δ Acc.
Δ Known
Δ Uncertain
Δ OC
Δ UC
Δ BR
THOR minus Cook
Qwen2.5-VL-3B
-8.67
-27.69
+4.97
-59.14
+48.15
-8.92
Qwen2.5-VL-7B
-13.39
-38.06
-4.41
-38.42
+58.32
-21.43
Qwen3-VL-4B
-14.36
-1.32
+8.93
+2.98
-14.78
-7.50
Qwen3-VL-8B
-20.66
-21.52
+4.43
-17.48
+4.78
-3.98
Qwen3.5-4B
-18.26
-54.27
+8.71
-41.70
+64.09
-18.62
Appendix
Table 18: Cook-to-Sim shifts in model behaviour (RQ1). Deltas are the indicated simulation subset minus Cook in percentage points. Known/Uncertain denote status-conditioned joint accuracy; OC and UC denote overconfidence and underconfidence rates.
Model
Full-video
Online
KCI
Epi.
Storm-BR
THOR
Qwen2.5-VL-3B ( 2025 )
40.33
45.00
45.35
21.24
10.30 [7.7–11.9]
Qwen2.5-VL-7B ( 2025 )
44.00
47.67
44.88
16.64
10.17 [7.4–11.6]
Qwen3-VL-4B ( 2025 )
48.00
51.67
53.67
38.33
19.27 [10.8–27.7]
Qwen3-VL-8B ( 2025 )
41.00
44.00
48.21
29.02
11.23 [8.0–12.9]
Qwen3.5-4B ( 2026 )
48.00
55.00
51.68
23.62
15.42 [8.3–20.0]
Appendix
Table 19: Controlled stress test: complete 14-model reliability on STORM -Sim (RQ1, RQ2). THOR contains 300 items over 28 AI2-THOR episodes; VHome adds 290 items over 29 VirtualHome episodes. Full-video and Online report task-choice accuracy with the complete sequence and the query-time prefix, respectively; both use the original query-time target labels. KCI, Epi., and Storm-BR use online predictions. KCI is Known Change-Intensity; Epi. is the Known/Uncertain joint-accuracy macro-average. Storm-BR is the Laplace-smoothed harmonic mean over the six cells, with 95% episode-clustered bootstrap intervals (1,000 resamples). In THOR, Low-change Uncertain has n=8 ; that cell is a transfer diagnostic. Best and second-best in each column are bold and underlined. All values are percentages.
Model
K joint
U joint
Hist.
Recent–Old
Q2.5-VL-3B
19.55
17.85
32.66
+0.94
Q2.5-VL-7B
13.80
28.52
31.39
-3.53
Q3-VL-4B
40.08
22.88
42.55
-3.69
Q3-VL-8B
46.30
11.40
34.82
+5.47
Q3.5-4B
33.30
28.02
34.10
+2.37
Q3.5-9B
40.05
28.80
35.61
+6.88
Appendix
Table 20: Domain-macro diagnostics from frozen online predictions. Known (K) and Uncertain (U) joint accuracy and history-aggregation task accuracy (Hist.) average seven subsets equally. Recent–Old is the joint-accuracy difference between the 0–2s and >5 s slices, averaged over the five real domains and VHome; THOR has no old-evidence items. All values are percentages or percentage-point differences.
Model
Current
Fact
History
Change
Track
Temporal
Recent
Old
Qwen2.5-VL-3B
53.81
64.80
29.19
66.74
63.74
44.09
46.77
37.85
Qwen2.5-VL-7B
58.74
70.63
35.97
72.44
74.10
54.77
48.48
38.49
Qwen3-VL-4B
65.92
74.36
45.48
72.44
76.80
61.36
44.35
46.06
Qwen3-VL-8B
63.68
73.43
42.99
73.35
71.62
63.18
57.35
43.53
Qwen3.5-4B
75.11
84.62
47.74
82.69
81.98
67.73
62.24
54.89
Qwen3.5-9B
78.03
85.31
55.66
83.83
84.01
71.82
66.50
58.99
Appendix
Table 21: What temporal tracking requires: semantic-type and evidence-age results on Cook (RQ4). The six type columns report online accuracy (%). Recent and Old report joint answer–status accuracy for evidence ending 0–2 s and more than 5 s before the query. All values cover the same 14-model panel visualized in Figure 11 a,b.
Source
n
Joint
Attribution F1
Missing observation
405
11.23
54.14
Partial observation
187
13.37
1.04
Ambiguous evidence
142
16.55
1.40
Low visual quality
71
34.51
24.07
Multiple candidates
26
17.31
3.12
Ambiguous attribute
15
40.00
0.00
Appendix
Table 22: Where uncertainty attribution fails: source-conditioned results on Cook (RQ3). Values are medians over 14 models. Joint is answer–status correctness conditioned on each ground-truth source; Attribution is the source-specific F1. Sources are multi-label and may overlap.
Domain
τ rank
Raw top
Smooth top
∣Δ∣max (pp)
Zero / 14
Cook
1.00
Molmo2-8B
Molmo2-8B
1.13
1
Bike
0.95
Molmo2-8B
Molmo2-8B
8.72
3
Health
0.96
Qwen3.5-4B
Qwen3.5-4B
3.98
1
Sports
0.93
Molmo2-8B
Molmo2-8B
8.29
2
THOR
0.97
Molmo2-8B
Molmo2-8B
14.00
8
VHome
0.91
Molmo2-8B
Molmo2-8B
13.25
4
Appendix
Table 23: Metric stability: raw versus add-one-smoothed Storm-BR (RQ2). Both scores are recomputed from identical frozen prediction files. Kendall τ compares the 14-model raw and smoothed rankings, with tied pairs dropped. ∣Δ∣max is the largest model-wise score change in percentage points; a raw zero occurs when at least one joint cell has no correct prediction. Smoothing does not change the leader on any listed domain, while making rare-cell scores finite and comparable.
Domain
n
τ
∣Δ∣max (pp)
LOO span (pp)
Leader
Cook
2,640
0.98
1.17
6.0–10.5
Molmo2-8B
Bike
786
0.91
5.53
6.3–14.3
Molmo2-8B
Health
800
0.93
2.78
7.4–12.8
Qwen3.5-4B
Sports
800
0.98
2.22
5.1–8.5
Molmo2-8B
THOR
300
0.52
16.52
8.7–15.9
Molmo2-8B
Music
120
n/a
n/a
n/a
Molmo2-8B
Appendix
Table 24: Aggregation robustness of Storm-BR on sparse cells (RQ2). The collapsed variant pools the three change-intensity bins into a single Known/Uncertain pair; leave-one-cell-out (LOO) drops each of the six cells in turn. τ is the Kendall rank correlation between the 14-model six-cell and collapsed scores, ∣Δ∣max is the largest model-wise score change, and the LOO column summarizes score sensitivity to single-cell deletion. Collapsing preserves the leader on every evaluated subset, while THOR changes scores most (up to 16.5 pp).
Domain
τ median [min]
Leader in top-3
Top-3 set
∣Δ∣max (pp)
Known–Unc. sign
Cook
0.96 [0.91]
15/15
14/15
4.89
100.0%
Bike
0.93 [0.85]
15/15
15/15
9.63
98.1%
Health
0.96 [0.78]
15/15
6/15
7.76
100.0%
Sports
0.96 [0.89]
15/15
15/15
7.21
100.0%
THOR
0.76 [0.69]
15/15
3/15
9.36
89.0%
VHome
0.78 [0.67]
12/12
0/12
4.57
95.2%
Appendix
Table 25: Sensitivity to alternative change-intensity partitions (RQ2). Each of the 15 alternative partitions reuses the same 14 frozen online prediction files and changes only the two cutpoints that define the Low/Medium/High bins (canonical cuts (3,6) ). The alternatives are the nearest valid cutpoint pairs to (3,6) ; validity requires six non-empty intensity–answerability cells. VHome uses the 12 of these 15 alternatives that satisfy this condition; the three cuts starting at 1 leave its Low bin empty. τ is the Kendall rank correlation between the 14-model ranking and the canonical ranking. “Leader in top-3” counts partitions in which the canonical leader remains within the top three, “Top-3 set” counts partitions that preserve the canonical top-3 set, and ∣Δ∣max is the largest change among the canonical top-3 scores in percentage points. The last column is the fraction of model × partition comparisons that preserve the sign of the canonical Known–Uncertain reliability gap. Music lacks high-intensity queries ( B3 ) and is excluded from six-cell partition audits.
Model
Control
Sampling / cap
Acc.
KCI
Epi.
BR
Qwen2.5-VL-3B
Question + choices
no video
35.08
34.88
19.75
15.70
Qwen2.5-VL-3B
Shuffled frames
1 FPS / all
39.89
39.58
12.68
8.34
Qwen2.5-VL-3B
Recent history
1 FPS / 16
42.77
42.64
15.35
11.51
Qwen2.5-VL-7B
Question + choices
no video
42.54
42.34
30.31
0.28
Qwen2.5-VL-7B
Shuffled frames
1 FPS / all
46.48
46.40
17.00
2.84
Qwen2.5-VL-7B
Recent history
1 FPS / 16
48.64
48.76
19.61
4.53
Appendix
Table 26: Input configurations on Cook (RQ4). All 14 checkpoints are evaluated under text-only, shuffled-frame, and recent-16-frame input. Acc. is task accuracy; KCI is accuracy averaged over intensity bins; Epi. is Known/Uncertain joint-accuracy macro-average; BR is add-one-smoothed six-cell Storm-BR .
Model
Input / control
Sampling / cap
Acc.
BR
Qwen3.5-9B ( 2026 )
Official online
1 FPS / all
76.40
33.29
Full-video offline
1 FPS / all
75.04
29.33
Question + choices
no video
39.17
0.34
Bounded recent history
1 FPS / 16
61.89
35.49
Shuffled visible frames
1 FPS / all
57.54
33.56
Molmo2-8B ( 2026 )
Official online
1 FPS / all
76.33
34.45
Appendix
Table 27: Cook protocol and input-configuration results. Performance of two representative models under official online, full-video offline, and input-control configurations. Acc. denotes task accuracy, and BR denotes balanced reliability ( Storm-BR ).
Recent advances in Video Large Language Models (Video-LLMs) have yielded promising performance on Video Question Answering (VideoQA). Nevertheless, existing benchmarks are predominantly evaluated through answer correctness, while the faithfulness of predicted evidence supporting those answers remains insufficiently evaluated. This disconnect between answer generation and evidence verification motivates the construction of the Evidence-Grounded Video Question Answering Benchmark (EG-VQA), a large-scale open-ended benchmark in which each QA pair is annotated with temporally localized textual evidence, requiring models to jointly produce answers and verifiable evidence. EG-VQA comprises 2,067 videos and 11,838 QA pairs with fine-grained evidence annotations. To evaluate predicted evidence, Evidence-Grounded F1 (EG-F1) is introduced as a metric that jointly measures temporal alignment and semantic consistency between predicted and ground-truth evidence. Experiments reveal a substantial discrepancy between answer correctness and evidence faithfulness: even strong proprietary models can achieve high answer accuracy while exhibiting gaps in temporal-semantic grounding. To investigate evidence-aware learning under EG-VQA, we develop EG-Reasoner, an evidence-aware model trained with explicit evidence supervision. EG-Reasoner achieves strong evidence grounding performance among open-source models while maintaining competitive answer generation results compared with proprietary systems. These findings highlight the importance of explicitly modeling evidence grounding and suggest that evidence-aware supervision provides an effective direction toward more reliable and interpretable VideoQA systems.
Linpeng Huang, Weixing Chen, Zexin Chen +2
Sun Yat-sen University · Shenzhen University · Peng Cheng Laboratory
Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding. Existing explainability efforts rely on textual rationales or sparse bounding boxes, which struggle to capture complex video dynamics such as occlusions and non-rigid deformations. We propose Evidence-Backed Video Question Answering (E-VQA), a novel task requiring models to jointly output a semantic answer and precise spatio-temporal evidence: temporal segments and dense, tracked object segmentation masklets. To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and true visual perception that scaling alone fails to bridge. To address this, we develop scalable, automated generation pipelines to create ST-Evidence-Instruct, a 160k-scale dataset bridging high-level reasoning with fine-grained grounding. Fine-tuning grounded Video LLMs on this data yields substantial gains over the corresponding size-matched UniPixel baselines (e.g., +27.2 t-mean and +13.8 J&F on a 7B model), establishing a robust baseline for explainable, evidence-backed video understanding. Code and data are available at https://github.com/SalesforceAIResearch/EVQA.
Shijie Wang, Honglu Zhou, Ziyang Wang +5
Salesforce, Palo Alto, CA, USA · Brown University, Providence, RI, USA
Video multimodal large language models achieve strong results on existing benchmarks, but answer accuracy alone does not establish whether they can locate the evidence needed to answer a question. We introduce VideoZeroBench, a challenging long-video benchmark with manually annotated question-answer pairs spanning 13 video domains. Questions target fine-grained cues, fleeting events, and evidence distributed across multiple segments. Temporal intervals and key-frame boxes are annotated where applicable. All questions undergo two rounds of cross-verification for answer validity and evidence quality. Our five-level diagnostic protocol compares answering with and without evidence hints, then combines answer correctness with independently evaluated temporal and spatial grounding. Across 19 evaluated models, the best standard QA accuracy is 24.8% (Level-3), achieved by Gemini-3.7-Flash. No model exceeds 1.8% when correct answers and accurate spatio-temporal localization are jointly required (Level-5). Analyses of atomic abilities, evidence spans, input modalities, and thinking-with-videos inference further characterize where the evaluated systems struggle. These findings motivate more precise evidence search and localization for long-video question answering. Our code and data are publicly released.