Organizations: Institute of Cyberspace Security, Zhejiang University of Technology. · Binjiang Institute of Artificial Intelligence, Zhejiang University of Technology. · Hong Kong University of Science and Technology (Guangzhou).
Long-video reasoning requires a system to localize sparse evidence over extended temporal sequences, preserve temporal structure, and generate answers despite substantial multimodal distractors. Existing video retrieval-augmented generation methods typically build indexes from uniformly sampled video segments. In long-video settings, this design readily introduces irrelevant context and is unreliable for questions with explicit temporal constraints. We propose VTG-RAG, a framework for long-video question answering that combines query-adaptive coarse-to-fine sampling with temporal graph retrieval. First, a two-stage sampling strategy selects core segments with high mean query relevance or high intra-segment relevance variance. The framework then integrates subtitles, speech transcripts, visual descriptions, entities, and relations from these segments into a temporally ordered multimodal graph. Finally, it performs temporally filtered, two-pass retrieval and reasoning across frame, text, and graph evidence spaces. A lot of experiments have proven the effectiveness of the VTG-RAG. Specifically, we evaluate the framework on Video-MME and LongVideoBench. On LongVideoBench, the reported Qwen2.5-VL-32B and 72B results improve from 48.16% to 50.56% and from 52.13% to 53.32%, respectively, with 24 answer-input frames. These correspond to gains of 2.40 and 1.19 percentage points. The ablation experiments of VTG-RAG further support that the contributions of each component are very important.
Figures & tables
Figure 1: Motivation for VTG-RAG. Uniform segmentation treats all temporal intervals equally, introducing irrelevant or temporally invalid evidence into long-video question answering. In contrast, VTG-RAG combines query-adaptive coarse-to-fine sampling with temporal graph retrieval to construct a compact and ordered evidence context, applying timestamp-based filtering when a query time is available
Figure 2: Overall architecture of VTG-RAG for long-video question answering. The framework comprises three modules. Query-Adaptive Coarse-to-Fine sampling combines variance-aware coarse scoring with candidate refinement to identify a compact set of core segments. Temporal graph construction integrates visual descriptions, transcripts, entities, relations, and temporal links into a multimodal graph. Time-constrained retrieval and reasoning combines query rewriting, reasoning planning, multi-space retrieval, and working memory to retrieve supporting evidence and generate the final answer
Model
Size
Video-MME
Avg
ΔOurs
w/o official subtitles
w/ official subtitles
Proprietary LVLMs
Gemini 1.5 Pro
–
42.69
48.46
45.58
–
GPT-4o
–
53.62
59.17
56.40
–
Open-source LVLMs
LLaVA-OV
7B
35.08
35.52
35.30
–
Table 1: Comparison of the main results on Video-MME. ΔOurs denotes the absolute improvement of VTG-RAG in Avg over the corresponding method. The subtitle conditions refer to the availability of official subtitles. VTG-RAG uses ASR-derived transcripts in both conditions; the condition without official subtitles is therefore not visual-only.
Model
Text
Size
Frames
Acc. (%)
Gain (pp)
Tokens
Acc./1M tokens
LVLMs
LongVA
–
7B
8
24.2
–
16.7M
1.45
LongVA + VTG-RAG
1.8K
7B
8
40.1
+15.9
10.4M
3.86
LLaVA
–
7B
16
18.8
–
11.3M
1.65
LLaVA + VTG-RAG
2.0K
7B
16
35.2
+16.4
4.8M
7.33
InternVL3
–
8B
16
23.2
–
12.7M
1.83
Table 2: Performance and token efficiency of different vision–language models on LongVideoBench. Text denotes the retrieved-text budget of VTG-RAG, and Gain denotes the absolute accuracy improvement over the direct baseline using the same model. Direct baselines use official subtitles, whereas VTG-RAG uses ASR-derived transcripts and generated visual descriptions without official subtitles. The Text column reports the retrieved-text budget of VTG-RAG; “–” does not indicate the absence of subtitle input for the direct baselines
Model
Retrieved text
Size
Frames
Accuracy (%)
Gain (pp)
Qwen2.5-VL
–
32B
24
48.16
–
Qwen2.5-VL + VTG-RAG
2.1K
32B
24
50.56
+2.40
Qwen2.5-VL
–
72B
24
52.13
–
Qwen2.5-VL + VTG-RAG
2.1K
72B
24
53.32
+1.19
Table 3: Scaling performance of VTG-RAG with larger Qwen2.5-VL models on the official labeled validation set of LongVideoBench. All configurations used 24 visual frames and the same evaluation protocol. Retrieved text denotes the textual-evidence budget. Gain is the absolute accuracy improvement, in percentage points, over the direct baseline with the same model size and visual-frame budget
Configuration
Long w/ subtitles (%)
Full model
43.8
– w/o query-adaptive sampling
40.7
– w/o graph retrieval
42.1
– w/o two-pass reasoning
42.9
Table 4: Core ablation experiments for VTG-RAG. All results are reported under the Video-MME Long setting with subtitles
Figure 3: Effect of the retrieval frame count on Qwen2.5-VL models at different scales
Graph-based retrieval-augmented generation (RAG) provides a scalable paradigm for long-video understanding, but existing systems typically inherit a fixed temporal granularity from video segmentation when constructing their retrieval index. We argue that this design unnecessarily couples indexing granularity with evidence granularity: coarse representations can often suffice for locating relevant temporal regions, while fine-grained evidence remains important for downstream reasoning. We propose Density-Aware Graph Construction (DAGC), a training-free approach that decouples a query-independent coarse retrieval index from the original fine-grained evidence space. DAGC constructs a compact, density-adaptive graph index by merging visually redundant neighboring chunks, while preserving mappings to the original temporal units. Retrieved coarse regions are subsequently expanded back to the original chunk granularity for fine-grained evidence refinement and answer generation. Experiments on MLVU, VideoMME, and LongVideoBench show that DAGC retains only about 40--50% of the original graph nodes and achieves 1.3--1.7× end-to-end wall-clock acceleration while preserving approximately 99% of the original QA performance. The gains transfer across different LVLM backbones and video RAG pipelines, suggesting that long-video RAG need not maintain the same temporal granularity for indexing and evidence reasoning.
Zhe Jin, Zhimin Lin, Bin Zheng +2
Beijing University of Posts and Telecommunications · Soochow University
Current large video-language models (LVLMs) still face challenges when dealing with long videos, mainly because frames are often processed independently, making it difficult to capture temporal dependencies across events. Although retrieval-augmented approaches have been introduced to provide additional context, most of them operate at the frame or snippet level, which limits their ability to model how events evolve over time and relate to each other. In this paper, we propose Event Chain Retrieval-Augmented Generation (EC-RAG), a training-free framework that organizes video content into an explicit event chain before question answering. Instead of retrieving isolated frames or text segments, EC-RAG first partitions the video into semantically coherent segments, represents each segment using multi-modal signals, and then links them into a structured chain that preserves temporal order and captures inter-event relationships. Given a query, the system identifies relevant events within this chain and gathers supporting evidence from the associated modalities. Our approach offers several practical advantages: (i) event-level abstraction that better reflects how video content is naturally structured, enabling more reliable localization compared to frame-level retrieval; (ii) structured multi-modal fusion that aggregates speech, text, and visual cues at the event level, allowing complementary information to be more effectively utilized during reasoning; and (iii) plug-and-play compatibility with existing LVLM backbones, requiring no additional training or reliance on proprietary models. Experiments on Video-MME, MLVU, and LongVideoBench show that this event-centric design consistently outperforms frame-level retrieval baselines, highlighting the importance of modeling temporal structure for long-video understanding.
Yuhao Qin, Junbo Wang, Yuke Li +1
Northwestern Polytechnical University Xi’an, China
Retrieval-augmented generation is moving beyond text into long, egocentric video, where systems must select query-relevant chunks across multiple modalities and temporal granularities. Yet progress in VideoRAG is limited by two gaps: existing benchmarks allow queries to be answered without the video, obscuring retrieval errors, and prior methods apply a single modality-granularity configuration per query, ignoring chunk-level variability. We address both by introducing V-RAGBench, a benchmark of ⟨query, evidence chunk, answer⟩ triplets that enables faithful, decoupled evaluation of retrieval and generation, and CARVE, a simple method that runs parallel retrievers across configurations and employs chunk-adaptive reranking to identify the winning configuration for each chunk. Each chunk then enters the generator under its winning configuration selected during retrieval, yielding an interleaved evidence form where the chunk-level decision propagates across both stages. CARVE outperforms eight recent VideoRAG baselines, with the chunks supplied to the generator interleaving multiple configurations rather than sharing a single one, a behavior unattainable by query-level methods.
Yuho Lee, Jisu Shin, Nicole Hee-Yeon Kim +5
KAIST, YH, Seoul, Republic of Korea · Qualcomm AI Research, Qualcomm Korea, YH, Seoul, Republic of Korea