Organizations: Institute of Cyberspace Security, Zhejiang University of Technology. · Binjiang Institute of Artificial Intelligence, Zhejiang University of Technology. · Hong Kong University of Science and Technology (Guangzhou).
Long-video reasoning requires a system to localize sparse evidence over extended temporal sequences, preserve temporal structure, and generate answers despite substantial multimodal distractors. Existing video retrieval-augmented generation methods typically build indexes from uniformly sampled video segments. In long-video settings, this design readily introduces irrelevant context and is unreliable for questions with explicit temporal constraints. We propose VTG-RAG, a framework for long-video question answering that combines query-adaptive coarse-to-fine sampling with temporal graph retrieval. First, a two-stage sampling strategy selects core segments with high mean query relevance or high intra-segment relevance variance. The framework then integrates subtitles, speech transcripts, visual descriptions, entities, and relations from these segments into a temporally ordered multimodal graph. Finally, it performs temporally filtered, two-pass retrieval and reasoning across frame, text, and graph evidence spaces. A lot of experiments have proven the effectiveness of the VTG-RAG. Specifically, we evaluate the framework on Video-MME and LongVideoBench. On LongVideoBench, the reported Qwen2.5-VL-32B and 72B results improve from 48.16% to 50.56% and from 52.13% to 53.32%, respectively, with 24 answer-input frames. These correspond to gains of 2.40 and 1.19 percentage points. The ablation experiments of VTG-RAG further support that the contributions of each component are very important.
Figures & tables
Figure 1: Motivation for VTG-RAG. Uniform segmentation treats all temporal intervals equally, introducing irrelevant or temporally invalid evidence into long-video question answering. In contrast, VTG-RAG combines query-adaptive coarse-to-fine sampling with temporal graph retrieval to construct a compact and ordered evidence context, applying timestamp-based filtering when a query time is available
Figure 2: Overall architecture of VTG-RAG for long-video question answering. The framework comprises three modules. Query-Adaptive Coarse-to-Fine sampling combines variance-aware coarse scoring with candidate refinement to identify a compact set of core segments. Temporal graph construction integrates visual descriptions, transcripts, entities, relations, and temporal links into a multimodal graph. Time-constrained retrieval and reasoning combines query rewriting, reasoning planning, multi-space retrieval, and working memory to retrieve supporting evidence and generate the final answer
Model
Size
Video-MME
Avg
ΔOurs
w/o official subtitles
w/ official subtitles
Proprietary LVLMs
Gemini 1.5 Pro
–
42.69
48.46
45.58
–
GPT-4o
–
53.62
59.17
56.40
–
Open-source LVLMs
LLaVA-OV
7B
35.08
35.52
35.30
–
Table 1: Comparison of the main results on Video-MME. ΔOurs denotes the absolute improvement of VTG-RAG in Avg over the corresponding method. The subtitle conditions refer to the availability of official subtitles. VTG-RAG uses ASR-derived transcripts in both conditions; the condition without official subtitles is therefore not visual-only.
Model
Text
Size
Frames
Acc. (%)
Gain (pp)
Tokens
Acc./1M tokens
LVLMs
LongVA
–
7B
8
24.2
–
16.7M
1.45
LongVA + VTG-RAG
1.8K
7B
8
40.1
+15.9
10.4M
3.86
LLaVA
–
7B
16
18.8
–
11.3M
1.65
LLaVA + VTG-RAG
2.0K
7B
16
35.2
+16.4
4.8M
7.33
InternVL3
–
8B
16
23.2
–
12.7M
1.83
Table 2: Performance and token efficiency of different vision–language models on LongVideoBench. Text denotes the retrieved-text budget of VTG-RAG, and Gain denotes the absolute accuracy improvement over the direct baseline using the same model. Direct baselines use official subtitles, whereas VTG-RAG uses ASR-derived transcripts and generated visual descriptions without official subtitles. The Text column reports the retrieved-text budget of VTG-RAG; “–” does not indicate the absence of subtitle input for the direct baselines
Model
Retrieved text
Size
Frames
Accuracy (%)
Gain (pp)
Qwen2.5-VL
–
32B
24
48.16
–
Qwen2.5-VL + VTG-RAG
2.1K
32B
24
50.56
+2.40
Qwen2.5-VL
–
72B
24
52.13
–
Qwen2.5-VL + VTG-RAG
2.1K
72B
24
53.32
+1.19
Table 3: Scaling performance of VTG-RAG with larger Qwen2.5-VL models on the official labeled validation set of LongVideoBench. All configurations used 24 visual frames and the same evaluation protocol. Retrieved text denotes the textual-evidence budget. Gain is the absolute accuracy improvement, in percentage points, over the direct baseline with the same model size and visual-frame budget
Configuration
Long w/ subtitles (%)
Full model
43.8
– w/o query-adaptive sampling
40.7
– w/o graph retrieval
42.1
– w/o two-pass reasoning
42.9
Table 4: Core ablation experiments for VTG-RAG. All results are reported under the Video-MME Long setting with subtitles
Figure 3: Effect of the retrieval frame count on Qwen2.5-VL models at different scales