Organizations: Nanyang Technological University, Singapore · ByteDance Inc. · State Key Laboratory of Complex and Critical Software Environment, School of Computer Science and Engineering, Beihang University · Westlake University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
Long instructional videos are increasingly used in medical education, industrial training, and procedural knowledge transfer, yet their evidence is often incomplete or unreliable: subtitles may be missing, ASR noisy, OCR sparse, and decisive actions briefly shown without narration. Existing VideoQA systems typically return labels or free-form text, making answer support difficult to inspect. We study Long Video Moment Question Answering (LVMQA), where a system returns timestamped answer-supporting evidence. We propose EP-LVMQA, an evidence-preserving coarse-to-fine framework comprising an Uncertainty-Aware Multimodal Evidence Timeline, a Coarse Evidence Planner, and an Evidence-Conditioned Fine Localizer. An evidence card transfers planner scores, matched cues, source-confidence proxies, modality-specific evidence, neighboring context, and a temporal prior between the latter two modules. On MedVidQA, EP-LVMQA achieves 26.1 [email protected] and 31.8 mIoU, improving the strongest non-ours results by 6.3 and 5.1 points. It raises Suff.@2 and Ver.@2 to 25.4 and 22.6, gains of 9.5 and 8.8 points. Under matched low-quality perturbations, average relative degradation is reduced from the strongest non-ours value of 14.6% to 11.1%. On Ego4D-NLQ and QVHighlights, EP-LVMQA remains broadly comparable to established task-specific localization methods.
Figures & tables
Figure 1: (a) Text-driven methods pass ASR/OCR-retrieved clips or scores to a clip-conditioned localizer, yielding a loose interval without provenance. (b) EP-LVMQA transfers multimodal evidence and source-aware metadata in an evidence card, enabling routed, prior-guided refinement and an inspectable primary evidence moment.
Figure 2: EP-LVMQA comprises three modules: (a) an Uncertainty-Aware Multimodal Evidence Timeline, (b) a Coarse Evidence Planner, and (c) an Evidence-Conditioned Fine Localizer. Question-Conditioned Evidence Cues condition the planner and localizer, which communicate through an Evidence-Preserving Candidate Interface implemented as an evidence card.
Table 1: Main comparison with explicit result provenance. MedVidQA is the target setting; Ego4D-NLQ and QVHighlights are compatibility checks. Prot.: Off.=reported/official reference, Rep.=our reproduction, M=matched feature-based evaluation, MP=matched-prompt Video-LLM evaluation. Off./Rep. means that localization is official while audit values come from reproduced saved predictions; these values are not combined into a matched superiority claim. All audit columns are measured strict-majority rates on the common blinded 300-query set. Bold and underlining denote the best and second-best listed value per metric, irrespective of protocol class.
Figure 3: Cross-benchmark performance on (a) Ego4D-NLQ and (b) QVHighlights, showing the top three methods for each metric. Ours denotes EP-LVMQA.
Method
Clean
ASR-50
OCR-50
HN-Cap
Conflict
Avg. drop ↓
Conflict Suff.@2
VPTSL/TAGV-style
18.2
11.4
16.6
13.2
12.0
26.9
9.1
Moment-DETR ( Lei et al. 2021 )
14.1
11.5
13.1
10.9
10.2
19.0
6.8
QD-DETR ( Moon et al. 2023b )
16.7
13.4
15.4
12.5
11.7
20.7
7.9
CG-DETR ( Moon et al. 2023a )
18.4
15.2
16.9
13.7
13.0
20.1
9.0
UniVTG ( Lin et al. 2023 )
19.8
16.7
18.2
15.0
14.4
18.8
10.6
CausalVTG ( Wang et al. 2025b )
19.2
16.5
17.9
15.8
15.4
14.6
11.9
Table 2: MedVidQA robustness in [email protected]. ASR/OCR-50 removes half the corresponding evidence; HN-Cap uses semantically similar non-overlapping caption negatives; Conflict retains ASR but replaces captions. Avg. drop is the mean relative loss across the four perturbations. All methods receive the same degraded input views.
Setting
Cand. R
Sel. dur.
Final R1
Cached time
@0.3 ↑
(%) ↓
@0.5 ↑
(s/query) ↓
A. Planner comparison under Top-5
Uniform sampling
31.8
11.6
11.9
0.54
Text-only planner
49.2
10.8
15.4
0.49
Visual-only planner
55.6
11.0
16.6
0.48
MomentSeeker Top-5 ( Yuan et al. 2025 )
61.9
12.3
18.2
0.69
Table 3: Planner and candidate-budget analysis on MedVidQA videos over 15 min. Block A fixes Top-5; Block B varies K . Cand. [email protected] is candidate recall; Sel. dur. is candidate-union/video duration. Cached time excludes offline extraction. † Oracle changes selection only; the fixed localizer receives no ground-truth boundaries.
Configuration
TL
Card
Src
Rtr
Cue
Ctx
Prior
R1
mIoU
Conflict
Suff.@2
Nec.@2
Ver.@2
@0.5
@0.7
Stage-level references
Standard MR reference
–
–
–
–
–
–
–
19.8
10.1
26.0
14.4
14.8
13.2
12.6
Coarse planner only
–
–
–
–
–
–
–
20.5
8.7
27.0
15.8
17.6
15.4
14.7
Component ablations
− TL
×
✓
×
×
✓
✓
✓
22.3
11.5
28.5
15.9
18.2
16.3
15.6
Table 4: Stage-level references and component ablations on MedVidQA. Checks/crosses mark enabled/removed components or downstream unavailability. TL is the evidence timeline; Src contains modality evidence, masks, and confidence; Rtr is the source router; Conflict is [email protected] under ASR–caption conflict. Suff.@2 and Ver.@2 are primary outcomes; Nec.@2 is auxiliary. Audit values use the common blinded set and strict-majority rule; all variants share the split, encoders, Top-5 budget, backbone, optimizer, and evaluation.
Figure 4: Frame-level comparisons under complete, missing, and conflicting evidence. EP-LVMQA is tighter or more resilient than TimeZero, retaining target overlap with incomplete or inconsistent evidence.
While LVLMs rapidly improve, long-video question answering still remains challenging: relevant evidence is sparse, and question-relevant context often fails to provide cues that discriminate the correct answer from plausible alternatives. Diagnostic analysis on a manually annotated subset of MMR-V shows that prior agentic systems substantially improve cue retrieval over direct VLM inference yet fail to achieve a corresponding gain in answer accuracy, indicating that the bottleneck lies in option-discriminative evidence rather than topical relevance alone. We propose PACE (Progressive Acquisition of Critical Evidence), a factor-guided framework for long-video evidence acquisition. PACE proceeds in two stages: it first indexes clip-level descriptions guided by question-derived factors without observing the candidate answers; it then uses the candidate answers to derive contrastive cues and queries the index for verification. On MMR-V with the open-source Qwen3-VL backbone, PACE achieves 42.6% accuracy, outperforming direct inference and prior agentic baselines including Deep Video Discovery (DVD). On the same diagnostic subset, PACE recovers 66.9% of the annotated cues, providing empirical evidence that its gains are associated with improved evidence recovery rather than stronger answer-side priors alone. Consistent gains over DVD on LVBench, Video-MME, EgoSchema, and LongVideoBench suggest that option-aware evidence acquisition transfers beyond MMR-V. Code is available at https://github.com/HKUST-KnowComp/PACE.
Baixuan Xu, Yinyui Xu, Tianshi Zheng +9
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China
Video multimodal large language models achieve strong results on existing benchmarks, but answer accuracy alone does not establish whether they can locate the evidence needed to answer a question. We introduce VideoZeroBench, a challenging long-video benchmark with manually annotated question-answer pairs spanning 13 video domains. Questions target fine-grained cues, fleeting events, and evidence distributed across multiple segments. Temporal intervals and key-frame boxes are annotated where applicable. All questions undergo two rounds of cross-verification for answer validity and evidence quality. Our five-level diagnostic protocol compares answering with and without evidence hints, then combines answer correctness with independently evaluated temporal and spatial grounding. Across 19 evaluated models, the best standard QA accuracy is 24.8% (Level-3), achieved by Gemini-3.7-Flash. No model exceeds 1.8% when correct answers and accurate spatio-temporal localization are jointly required (Level-5). Analyses of atomic abilities, evidence spans, input modalities, and thinking-with-videos inference further characterize where the evaluated systems struggle. These findings motivate more precise evidence search and localization for long-video question answering. Our code and data are publicly released.
Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding. Existing explainability efforts rely on textual rationales or sparse bounding boxes, which struggle to capture complex video dynamics such as occlusions and non-rigid deformations. We propose Evidence-Backed Video Question Answering (E-VQA), a novel task requiring models to jointly output a semantic answer and precise spatio-temporal evidence: temporal segments and dense, tracked object segmentation masklets. To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and true visual perception that scaling alone fails to bridge. To address this, we develop scalable, automated generation pipelines to create ST-Evidence-Instruct, a 160k-scale dataset bridging high-level reasoning with fine-grained grounding. Fine-tuning grounded Video LLMs on this data yields substantial gains over the corresponding size-matched UniPixel baselines (e.g., +27.2 t-mean and +13.8 J&F on a 7B model), establishing a robust baseline for explainable, evidence-backed video understanding. Code and data are available at https://github.com/SalesforceAIResearch/EVQA.
Shijie Wang, Honglu Zhou, Ziyang Wang +5
Salesforce, Palo Alto, CA, USA · Brown University, Providence, RI, USA