Video Evidence Indexing: Learning Where to Look from Video Previews for Token-Budgeted Long-Video Question Answering

Haowen Guan; Shengzhi Li; Shichao Pei

cs.CVSep 30, 2026

Video Evidence Indexing: Learning Where to Look from Video Previews for Token-Budgeted Long-Video Question Answering

Authors: Haowen Guan, Shengzhi Li, Shichao Pei

Abstract

Long-video question answering is limited by the high cost of visual tokens and by the fixed context width of current VLMs. A long-video question may require broad temporal coverage, but the answer is often supported by only a compact set of moments. To locate these moments efficiently, we propose token-budgeted Video Evidence Indexing (VEI): given a dense low-resolution Video Preview, the model constructs a compact high-resolution Evidence Set for final reasoning. We treat VEI as a policy that must jointly solve \textit{evidence localization}, which finds question-relevant moments, and \textit{budget planning}, which decides where to spend the limited high-resolution frame budget. We implement this idea with an inference pipeline: the Video Preview provides cheap global coverage, Video Evidence Indexing constructs the Evidence Set, and Answer Generation combines both inputs for final VQA. To address missing frame-level supervision, we adopt privileged self-distillation, where an answer-aware teacher guides the normal test-time policy on student-generated indexing traces. We explore previews at 1, 6, 12, and 24 visual tokens per frame, training a single policy that supports all four resolutions. Experiments show that Video Evidence Indexing improves accuracy under limited visual budgets, and self-distillation further improves both QA accuracy and temporal evidence localization.

Figures & tables

Figure 1: Overview of Video Evidence Indexing. A densely sampled low-resolution Video Preview (VPre) provides cheap global coverage under a strict context limit. We explore how VEI can use VPre to locate complementary evidence and reserve high-resolution detail for the moments that matter.

Figure 2: Accuracy–context trade-off. Mean accuracy across three datasets (Section 4 ). All methods use Qwen3-VL-8B. Comparators cover predictive compression (AdaCodec), multi-agent aggregation (MACF), encoder-based selection (AKS, AdaQ), and RL-trained video-tool acquisition (ParaVT).

Figure 3: Privileged self-distillation for VEI. (a) The student generates observation targets and timestamp lists; the complete VEI prompt $P_{\text{VEI}}$ appears in fig. 6 . (b) Privileged information enables the teacher to guide the student’s search toward evidence supporting the correct answer. Distilling this guidance teaches the student what to look for and how to allocate its frame budget from compressed previews, without frame-level annotations or privileged inputs at inference.

Method	Context Limit	Preview Size $(\#\mathrm{Frame}/\mathrm{Tok})$	VideoMME w/o sub.				LVBench	MLVUMCQ	Average
Method	Context Limit	Preview Size $(\#\mathrm{Frame}/\mathrm{Tok})$	Short	Medium	Long	Overall	Train	Test	Score	%
Use 100% Context Width Across All Benchmarks
Qwen3-VL-8B	$256\mathrm{K}$	$2048/640_{\mathrm{Tok}}$	$77.8$	$70.7$	$64.7$	$71.0$	$57.6$	$67.7$	$65.4$	$100.0$
Use 5.9% Context Width Across All Benchmarks
Uniform FS	$15\mathrm{K}$	$1\mathrm{fps}/1_{\mathrm{Tok}}$	$73.1$	$60.0$	$58.4$	$63.8$	$40.2$	$52.2$	$52.1$	$79.6$
Training-Free VEI	$15\mathrm{K}$	$1\mathrm{fps}/1_{\mathrm{Tok}}$	$70.4$	$53.9$	$51.4$	$58.6$	$36.7$	$46.2$	$47.2$	$72.1\;{\color[rgb]{0,0,1}(-7.5)}$

Table 1: Main results across preview budgets. One self-distilled policy supports all four preview resolutions, outperforming uniform sampling even at one token per frame under the same 32-frame EvSet budget. The

1

fps preview rate is reduced when the input exceeds the context limit.

Model	Size	Token Budget	VideoMME		LVBench	MLVU	AVG
Model	Size	Token Budget	Long	Overall	LVBench	MLVU	AVG
(a) Proprietary Models
GPT-4o	–	$128\mathrm{K}$	65.3	71.9	64.4	64.6	67.0
Gemini-2.5-Pro	–	$1\mathrm{M}$	–	87.0	69.2	81.2	79.1
(b) Open-Source / Public Pretrained Models
LLaVA-OneVision ( Li et al., 2024 )	7B	$32\mathrm{K}$	46.7	58.2	26.9	50.5	45.2

Table 2: Comparison with existing VLMs and evidence-acquisition methods. Prior results in (a–c) use published evaluations and budgets; methods in (c) use Qwen3-VL-8B. CLIP † is our retrieval baseline (Appendix H ). Bold marks the best across (b–d).

Figure 4: Evidence localization on LVBench. Training-Free and Self-Distilled VEI are compared across four preview budgets. Evidence Recall is the key metric, which measures how much of the annotated evidence the VEI policy recovers in its indexing.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Selector	Comparable questions	Base FS	+ EL	+ EL + BP (VEI)
Training-Free VEI	220 / 502	70.0	66.4	74.6
Self-Distilled VEI	218 / 502	78.9	78.0	80.7

Appendix

Table 6: VEI component ablation with comparable selection counts. Accuracy (%) on shared questions within each row; every configuration uses a final 32-frame Evidence Set.

Panel of Figure 6 — Figure 6: Video Evidence Indexing Prompt. The normal indexer receives the Video Preview, question, and answer choices. During self-distillation, the teacher additionally receives privileged information, including the question type and correct answer.

Method	Video Processed $(\#\mathrm{Frames}/\mathrm{Token})$	Average Accuracy All Benchmarks	Estimated Cost All Benchmarks
Qwen3-VL-8B	$2048_{\mathrm{Max}}/640_{\mathrm{Tok}}$	65.4	$\sim\$ 87.8$
Self-Distilled VEI	$1\mathrm{fps}/24_{\mathrm{Tok}}+32/128_{\mathrm{Tok}}$	66.9	$\sim\$ 19.1$

Appendix

Table 7: Long-video accuracy and evaluation cost. The estimate covers 4,751 test questions across the three datasets.

Figure 7: EvSet Case Analysis. Uniform sampling misses the close-up needed to compare decorations. CLIP retrieves repetitive tree views, while VEI includes detailed views of the completed decoration and answers correctly.

Question	Outcome	Diagnosis
61	Hit, wrong	The selected evidence reaches the annotated event but misses its later outcome.
65	Hit, wrong	The interaction is localized, but the answer misses the semantic detail that no bribe occurred.
73	Miss, wrong	Selection concentrates on unrelated moments and misses the annotated event.
78	Miss, correct	The answer is correct despite missing the annotated window; the preview or alternative evidence may provide support.

Appendix

Table 8: Representative LVBench cases. These cases distinguish localization, incomplete event coverage, and downstream reasoning, and show that a single annotated window need not exhaust all useful evidence.

Method	Context Limit	VideoMME w/o sub.				LVBench	MLVUMCQ	Average Score ( $\%$ )
Method	Context Limit	Short	Medium	Long	Overall	Train	Test	Average Score ( $\%$ )
Use 5.9% Context Width Across All Benchmarks
Training-Free VEI	$15\mathrm{K}$	$70.4$	$53.9$	$51.4$	$58.6$	$36.7$	$46.2$	$47.2$
w/o Evidence Set	$15\mathrm{K}$	$45.1$	$43.1$	$45.0$	$44.4$	$30.9$	$32.9$	$36.1$ $(-11.1)$
Use 7.8% Context Width Across All Benchmarks
Training-Free VEI	$20\mathrm{K}$	$73.2$	$64.1$	$55.9$	$64.4$	$46.1$	$61.2$	$57.2$

Appendix

Table 9: Effect of Evidence Set access under matched configured context limits. For each low-resolution preview budget, we compare training-free VEI with the same model after removing access to the selected high-resolution Evidence Set. Blue values report the absolute average-score drop caused by removing Evidence Set frames.

Parameter	SD
Learning Rate	$1\times 10^{-4}$
Effective Batch Size	$8$
LoRA Rank ( $r$ )	$64$
LoRA Alpha ( $\alpha$ )	$128$
LoRA Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Max Completion Length	$1024$

Appendix

Table 10: Training Configuration for Privileged Self-Distillation

Explore similar work

Aug 3, 2026cs.CV

Ground, Cover, and Refine: Evidence-Centric Frame Selection for Long-Video Question Answering

Long-video question answering requires identifying sparse yet critical evidence from videos containing thousands of frames under a constrained visual-token budget. Existing methods either select query-aware frames in a single pass or rely on timestamped text solely as retrieval guidance, leading to two key limitations. First, selected frames tend to cluster around local relevance peaks, and once the budget is exhausted, omitted evidence cannot be recovered. Second, textual and visual evidence remain weakly aligned. We propose GCR, a training-free framework that casts fixed-budget frame selection as a joint evidence curation problem. Ground converts timestamped text into temporal events, selects query-relevant real frame anchors, and renders each event text onto its temporally aligned frame. Cover supplements grounded events with direct visual anchors for complementary visual evidence and applies global maximal marginal relevance to preserve diverse context. Refine revisits omitted temporal regions and replaces the weakest revisable context frame with a real-frame medoid---but only when the medoid offers greater evidence value. GCR maintains a fixed number of chronologically ordered frames and requires no VLM training or architectural modification. Experiments on LongVideoBench and Video-MME, across three 7B backbones and frame budgets of 8, 32, and 64, demonstrate consistent improvements in long-video QA. With the 7B LLaVA-OV backbone and 32 frames, GCR achieves 64.25% and 62.15% on the two benchmarks, outperforming the strongest reproduced baselines by 2.54 and 1.93 percentage points, respectively.

Fan Wei, Siru Zhong, Runmin Dong +3

Department of Earth System Science, Tsinghua University · 5The Hong Kong University of Science and Technology (Guangzhou) · School of Artificial Intelligence, Sun Yat-Sen University +2

Long Video Question Answering Question Relevance Grounding Data-Curation

Ask PDF

Jun 4, 2026cs.CV

MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering

Long-video question answering remains challenging for Vision-Language Models (VLMs), as answer-relevant evidence is often sparse, transient, and temporally dispersed across lengthy video contexts. Existing frame-centric approaches improve efficiency through uniform sampling, query-aware frame selection, visual-token compression, and adaptive resolution strategies. However, they still rely on isolated and fragmented frames as the fundamental evidence units, limiting VLMs' ability to effectively capture coherent event-level semantics. To address this limitation, we propose MemoryCard, a video-memory-based augmentation framework that organizes long videos into self-contained Memory Cards. Specifically, MemoryCard first performs a self-reading process over videos and aligned utterances to segment the video into semantically coherent units, each corresponding to a distinct topic or event. For each unit, it generates an event-level video gist and selects representative visual moments, which are then rendered into unified Memory Cards for retrieval and question answering. Experimental results demonstrate that MemoryCard consistently improves long-video QA performance under comparable visual-token budgets, achieving up to a 21.8% relative improvement in accuracy. All code is available at https://github.com/NEUIR/MemoryCard.

Qing Yang, Pengcheng Huang, Xinze Li +6

School of Computer Science and Engineering, Northeastern University, Shenyang, China · Department of Computer Science and Technology, Tsinghua University, Beijing, China · Digital China Group, Beijing, China

Long Video Question Answering Peak Memory Data Compression Methods

Ask PDF

Sep 20, 2026cs.CV

PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding

Long-video understanding must capture transient visual evidence under strict token budgets, yet existing methods compress frames, append memory tokens, or alter internal key-value (KV) caches. We introduce Prefix-Steered Recurrent Memory (PREM), a memory-token-free framework for frozen vision-language models (VLMs). PREM separates video ingestion from query answering: a recurrent writer distills visual streams into a compact 256 KiB multi-slot associative state, while a question-conditioned readout adds memory-derived key/value (K/V) steering modulations to existing non-visual prompt prefixes during prefill. This enables write-once, query-many inference without extra prompt tokens or decoding recurrence. Across six long-video benchmarks in offline and streaming end-of-stream settings, PREM consistently outperforms frozen baselines at every evaluated visual budget. Under a constrained budget of 16 frames, PREM improves macro-average accuracy by 3.06% on Qwen2.5-VL-3B, with gains of 11.0% on action antonym identification and 9.9% on localized needle retrieval. These gains require tuning 0.24% of backbone parameters at 0.03 GiB of peak GPU memory overhead.

Siru Zhong, Qiongyan Wang, Xiaohui Lv +5

The Hong Kong University of Science and Technology (Guangzhou) · Beta Infinity · Tsinghua University

Frozen Vision-Language Models Long-Video Benchmarks Visual Evidence Recurrent Model

Ask PDF