Video Evidence Indexing: Learning Where to Look from Video Previews for Token-Budgeted Long-Video Question Answering

Haowen Guan; Shengzhi Li; Shichao Pei

cs.CVSep 30, 2026

Video Evidence Indexing: Learning Where to Look from Video Previews for Token-Budgeted Long-Video Question Answering

Authors: Haowen Guan, Shengzhi Li, Shichao Pei

Abstract

Long-video question answering is limited by the high cost of visual tokens and by the fixed context width of current VLMs. A long-video question may require broad temporal coverage, but the answer is often supported by only a compact set of moments. To locate these moments efficiently, we propose token-budgeted Video Evidence Indexing (VEI): given a dense low-resolution Video Preview, the model constructs a compact high-resolution Evidence Set for final reasoning. We treat VEI as a policy that must jointly solve \textit{evidence localization}, which finds question-relevant moments, and \textit{budget planning}, which decides where to spend the limited high-resolution frame budget. We implement this idea with an inference pipeline: the Video Preview provides cheap global coverage, Video Evidence Indexing constructs the Evidence Set, and Answer Generation combines both inputs for final VQA. To address missing frame-level supervision, we adopt privileged self-distillation, where an answer-aware teacher guides the normal test-time policy on student-generated indexing traces. We explore previews at 1, 6, 12, and 24 visual tokens per frame, training a single policy that supports all four resolutions. Experiments show that Video Evidence Indexing improves accuracy under limited visual budgets, and self-distillation further improves both QA accuracy and temporal evidence localization.

Figures & tables

Figure 1: Overview of Video Evidence Indexing. A densely sampled low-resolution Video Preview (VPre) provides cheap global coverage under a strict context limit. We explore how VEI can use VPre to locate complementary evidence and reserve high-resolution detail for the moments that matter.

Figure 2: Accuracy–context trade-off. Mean accuracy across three datasets (Section 4 ). All methods use Qwen3-VL-8B. Comparators cover predictive compression (AdaCodec), multi-agent aggregation (MACF), encoder-based selection (AKS, AdaQ), and RL-trained video-tool acquisition (ParaVT).

Figure 3: Privileged self-distillation for VEI. (a) The student generates observation targets and timestamp lists; the complete VEI prompt $P_{\text{VEI}}$ appears in fig. 6 . (b) Privileged information enables the teacher to guide the student’s search toward evidence supporting the correct answer. Distilling this guidance teaches the student what to look for and how to allocate its frame budget from compressed previews, without frame-level annotations or privileged inputs at inference.

Method	Context Limit	Preview Size $(\#\mathrm{Frame}/\mathrm{Tok})$	VideoMME w/o sub.				LVBench	MLVUMCQ	Average
Method	Context Limit	Preview Size $(\#\mathrm{Frame}/\mathrm{Tok})$	Short	Medium	Long	Overall	Train	Test	Score	%
Use 100% Context Width Across All Benchmarks
Qwen3-VL-8B	$256\mathrm{K}$	$2048/640_{\mathrm{Tok}}$	$77.8$	$70.7$	$64.7$	$71.0$	$57.6$	$67.7$	$65.4$	$100.0$
Use 5.9% Context Width Across All Benchmarks
Uniform FS	$15\mathrm{K}$	$1\mathrm{fps}/1_{\mathrm{Tok}}$	$73.1$	$60.0$	$58.4$	$63.8$	$40.2$	$52.2$	$52.1$	$79.6$
Training-Free VEI	$15\mathrm{K}$	$1\mathrm{fps}/1_{\mathrm{Tok}}$	$70.4$	$53.9$	$51.4$	$58.6$	$36.7$	$46.2$	$47.2$	$72.1\;{\color[rgb]{0,0,1}(-7.5)}$

Table 1: Main results across preview budgets. One self-distilled policy supports all four preview resolutions, outperforming uniform sampling even at one token per frame under the same 32-frame EvSet budget. The

1

fps preview rate is reduced when the input exceeds the context limit.

Model	Size	Token Budget	VideoMME		LVBench	MLVU	AVG
Model	Size	Token Budget	Long	Overall	LVBench	MLVU	AVG
(a) Proprietary Models
GPT-4o	–	$128\mathrm{K}$	65.3	71.9	64.4	64.6	67.0
Gemini-2.5-Pro	–	$1\mathrm{M}$	–	87.0	69.2	81.2	79.1
(b) Open-Source / Public Pretrained Models
LLaVA-OneVision ( Li et al., 2024 )	7B	$32\mathrm{K}$	46.7	58.2	26.9	50.5	45.2

Table 2: Comparison with existing VLMs and evidence-acquisition methods. Prior results in (a–c) use published evaluations and budgets; methods in (c) use Qwen3-VL-8B. CLIP † is our retrieval baseline (Appendix H ). Bold marks the best across (b–d).

Figure 4: Evidence localization on LVBench. Training-Free and Self-Distilled VEI are compared across four preview budgets. Evidence Recall is the key metric, which measures how much of the annotated evidence the VEI policy recovers in its indexing.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Selector	Comparable questions	Base FS	+ EL	+ EL + BP (VEI)
Training-Free VEI	220 / 502	70.0	66.4	74.6
Self-Distilled VEI	218 / 502	78.9	78.0	80.7

Appendix

Table 6: VEI component ablation with comparable selection counts. Accuracy (%) on shared questions within each row; every configuration uses a final 32-frame Evidence Set.

Panel of Figure 6 — Figure 6: Video Evidence Indexing Prompt. The normal indexer receives the Video Preview, question, and answer choices. During self-distillation, the teacher additionally receives privileged information, including the question type and correct answer.

Method	Video Processed $(\#\mathrm{Frames}/\mathrm{Token})$	Average Accuracy All Benchmarks	Estimated Cost All Benchmarks
Qwen3-VL-8B	$2048_{\mathrm{Max}}/640_{\mathrm{Tok}}$	65.4	$\sim\$ 87.8$
Self-Distilled VEI	$1\mathrm{fps}/24_{\mathrm{Tok}}+32/128_{\mathrm{Tok}}$	66.9	$\sim\$ 19.1$

Appendix

Table 7: Long-video accuracy and evaluation cost. The estimate covers 4,751 test questions across the three datasets.

Figure 7: EvSet Case Analysis. Uniform sampling misses the close-up needed to compare decorations. CLIP retrieves repetitive tree views, while VEI includes detailed views of the completed decoration and answers correctly.

Question	Outcome	Diagnosis
61	Hit, wrong	The selected evidence reaches the annotated event but misses its later outcome.
65	Hit, wrong	The interaction is localized, but the answer misses the semantic detail that no bribe occurred.
73	Miss, wrong	Selection concentrates on unrelated moments and misses the annotated event.
78	Miss, correct	The answer is correct despite missing the annotated window; the preview or alternative evidence may provide support.

Appendix

Table 8: Representative LVBench cases. These cases distinguish localization, incomplete event coverage, and downstream reasoning, and show that a single annotated window need not exhaust all useful evidence.

Method	Context Limit	VideoMME w/o sub.				LVBench	MLVUMCQ	Average Score ( $\%$ )
Method	Context Limit	Short	Medium	Long	Overall	Train	Test	Average Score ( $\%$ )
Use 5.9% Context Width Across All Benchmarks
Training-Free VEI	$15\mathrm{K}$	$70.4$	$53.9$	$51.4$	$58.6$	$36.7$	$46.2$	$47.2$
w/o Evidence Set	$15\mathrm{K}$	$45.1$	$43.1$	$45.0$	$44.4$	$30.9$	$32.9$	$36.1$ $(-11.1)$
Use 7.8% Context Width Across All Benchmarks
Training-Free VEI	$20\mathrm{K}$	$73.2$	$64.1$	$55.9$	$64.4$	$46.1$	$61.2$	$57.2$

Appendix

Table 9: Effect of Evidence Set access under matched configured context limits. For each low-resolution preview budget, we compare training-free VEI with the same model after removing access to the selected high-resolution Evidence Set. Blue values report the absolute average-score drop caused by removing Evidence Set frames.

Parameter	SD
Learning Rate	$1\times 10^{-4}$
Effective Batch Size	$8$
LoRA Rank ( $r$ )	$64$
LoRA Alpha ( $\alpha$ )	$128$
LoRA Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Max Completion Length	$1024$

Appendix

Table 10: Training Configuration for Privileged Self-Distillation

Explore similar work

Ground, Cover, and Refine: Evidence-Centric Frame Selection for Long-Video Question Answering
Aug 3, 2026Fan Wei, Siru Zhong, Runmin Dong +3Long Video Question Answering Question
Department of Earth System Science, Tsinghua University · 5The Hong Kong University of Science and Technology (Guangzhou) · School of Artificial Intelligence, Sun Yat-Sen University +2
MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering
Jun 4, 2026Qing Yang, Pengcheng Huang, Xinze Li +6Long Video Question Answering Peak Memory
School of Computer Science and Engineering, Northeastern University, Shenyang, China · Department of Computer Science and Technology, Tsinghua University, Beijing, China · Digital China Group, Beijing, China
PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding
Sep 20, 2026Siru Zhong, Qiongyan Wang, Xiaohui Lv +5Frozen Vision-Language Models Long-Video Benchmarks
The Hong Kong University of Science and Technology (Guangzhou) · Beta Infinity · Tsinghua University

Aug 3, 2026, cs.CVJ/K move · Enter open · S save

Ground, Cover, and Refine: Evidence-Centric Frame Selection for Long-Video Question Answering

Fan Wei, Siru Zhong, Runmin Dong, Miao Yang+2

Department of Earth System Science, Tsinghua University · 5The Hong Kong University of Science and Technology (Guangzhou) · School of Artificial Intelligence, Sun Yat-Sen University+2

Long Video Question Answering Question Relevance Grounding Data-Curation

Open paper AskPDF arXiv

Long-video question answering requires identifying sparse yet critical evidence from videos containing thousands of frames under a constrained visual-token budget. Existing methods either select query-aware frames in a single pass or rely on timestamped text solely as retrieval guidance, leading to two key limitations. First, selected frames tend to cluster around local relevance peaks, and once the budget is exhausted, omitted evidence cannot be recovered. Second, textual and visual evidence remain weakly aligned. We propose GCR, a training-free framework that casts fixed-budget frame selection as a joint evidence curation problem. Ground converts timestamped text into temporal events, selects query-relevant real frame anchors, and renders each event text onto its temporally aligned frame. Cover supplements grounded events with direct visual anchors for complementary visual evidence and applies global maximal marginal relevance to preserve diverse context. Refine revisits omitted temporal regions and replaces the weakest revisable context frame with a real-frame medoid---but only when the medoid offers greater evidence value. GCR maintains a fixed number of chronologically ordered frames and requires no VLM training or architectural modification. Experiments on LongVideoBench and Video-MME, across three 7B backbones and frame budgets of 8, 32, and 64, demonstrate consistent improvements in long-video QA. With the 7B LLaVA-OV backbone and 32 frames, GCR achieves 64.25% and 62.15% on the two benchmarks, outperforming the strongest reproduced baselines by 2.54 and 1.93 percentage points, respectively.

Abstract

Explore similar work

Ground, Cover, and Refine: Evidence-Centric Frame Selection for Long-Video Question Answering

MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering

PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding