Long video understanding (LVU) remains a core challenge in multimodal learning. Although recent vision-language models (VLMs) have made notable progress, existing benchmarks mainly focus on either fine-grained perception or coarse summarization, offering limited insight into temporal understanding over long contexts. In this work, we define a scene as a coherent segment of a video in which both visual and semantic contexts remain consistent, aligning with human perception. This leads us to a key question: can current VLMs reason effectively over long, scene-level contexts? To answer this, we introduce a new benchmark, SceneBench, designed to provide scene-level challenges. Our evaluation reveals a sharp drop in accuracy when VLMs attempt to answer scene-level questions, indicating significant forgetting of long-range context. To further validate these findings, we propose Scene Retrieval-Augmented Generation (Scene-RAG), which constructs a dynamic scene memory by retrieving and integrating relevant context across scenes. This Scene-RAG improves VLM performance by +2.50%, confirming that current models still struggle with long-context retention. We hope SceneBench will encourage future research toward VLMs with more robust, human-like video comprehension.
Figures & tables
Figure 1: Longer SceneQA distances lead to lower accuracy. Video-RAG [ 31 ] improves the baseline in short- and long-range settings but struggles in the mid-range, while our Scene-RAG achieves consistent gains, especially for mid- and long-term reasoning. Curves are smoothed using a rolling mean. Results use Qwen2.5-VL [ 1 ] . Solid markers denote actual measurements; curves are cubic-spline interpolations for visual clarity.
Figure 2: We divide video information into frame, clip, scene, and video levels. For frame-level information, it focuses mainly on describing the details of the subject within the frame. Clip-level information includes temporal information but can only describe the objective behavior of objects. Scene-level information contains a lot of clip-level information and forms a complete scenario event, while video consists of a lot of scene-level content, with related scenes forming a logical storyline based on cause and effect. For convenience of quantification, we use two minutes to distinguish between clip and scene levels.
Benchmark
Year
#Videos
Len.(s)
#Tasks
#Genres
#QA Pairs
QA Len. (s)
QA Density
Multi- level
Scene Reliance
Audio Reliance
LVBench [ 47 ]
2025
103
4,101
6
21
1,549
–
0.38
✗
✗
✗
LongVideoBench [ 52 ]
2024
3,763
473
17
10
6,678
–
14.1
✗
✗
✗
HourVideo [ 3 ]
2024
500
2,742
4
16
12,976 †
–
4.7
✗
✗
✗
ALLVB [ 43 ]
2025
1,376
7,200
9
30
252,000 †
–
35.0
✗
✗
✗
Video-MME [ 14 ]
2025
900
1,024
12
30
2,700
–
2.6
✓
✗
✗
MoVQA [ 66 ]
2023
930
992
6
1
21,953
–
22.1
✓
✗
✗
Table 1: Comparison of SceneBench with existing long-video understanding benchmarks. SceneBench offers a balanced number of videos, diverse genres, and high–time-range, scene-based QA annotations. The average temporal distance between each question and its corresponding answer in SceneQA is 262s, with a standard deviation of 310s, illustrating the wide temporal diversity of the dataset. “ † ” denotes automatically or partially automatically generated QA pairs.
Figure 3: Statistical overview of our SceneBench benchmark. (A) Distribution of scene lengths. (B) Distribution of task counts. (C) Distribution of SceneQA and SceneQA-Audio length duration proportion over the full video.
Figure 4: Comparison of Scene-RAG with traditional RAGs working on video understanding tasks. Scene-RAG first aggregates long-range visual scenes via scene tiling, stores aligned visual–audio evidence, and retrieves task-relevant segments conditioned on user queries.
Method
Release
Frame Count
Title Pred.
Comm. Pred.
Clip QA
Scene QA
SceneQA -Audio
I-VQA
Avg.
Proprietary MLLMs
Gemini 2.5 Pro [ 9 ]
2025
1fps
99.4
92.9
80.5
60.7
65.8
82.4
80.3
Gemini 2.5 Flash-Lite [ 9 ]
2025
1fps
99.1
78.2
48.4
25.6
30.1
41.6
53.8
Kimi 2.5v [ 44 ]
2025
128
99.4
90.9
73.1
54.0
59.6
60.7
73.0
Open-source MLLMs
MovieChat [ 41 ]
2023
2048
25.2
28.6
52.2
21.9
21.4
21.6
28.5
Table 2: Benchmark results on the SceneBench. “Frame Count” indicates the number of frames used as input. Comm.: Comment.
Method
RAG Type
LLM Params
Frame Count
Title Pred.
Comm. Pred.
Clip QA
Scene QA
SceneQA -Audio
I-VQA
Avg.
Gain
LongVA [ 67 ]
-
7B
128
96.8
71.1
54.3
27.7
29.5
40.1
53.3
-
LongVA [ 67 ]
+ Video-RAG [ 31 ]
7B
128
97.7
70.4
53.7
28.3
30.2
39.2
53.4
+0.1
LongVA [ 67 ]
+ Scene-RAG
7B
128
97.7
71.5
54.6
28.9
32.4
41.4
54.4
+1.1
Long-LLaVA [ 58 ]
-
7B
64
96.2
79.7
57.1
26.1
25.4
33.0
52.9
-
Long-LLaVA [ 58 ]
+ Video-RAG [ 31 ]
7B
64
96.2
80.8
56.5
24.3
24.3
35.4
52.9
+0.0
Long-LLaVA [ 58 ]
+ Scene-RAG
7B
64
96.2
81.8
57.6
26.8
28.0
37.0
54.6
+1.7
Table 3: Performance of different MLLMs on the SceneBench benchmark. The average improvement of Scene-RAG is ∼ 1.40% in average.
Method
RAG Type
LLM Params
Frame
Short
Medium
Long
Overall
Avg.
Gain
LongVA [ 67 ]
-
7B
128
60.9
49.3
44.0
51.4
51.4
-
LongVA [ 67 ]
+ Video-RAG [ 31 ]
7B
128
66.1
60.4
59.4
62.0
62.0
+10.6
LongVA [ 67 ]
+ Scene-RAG
7B
128
66.9
60.6
59.7
63.4
62.4
+11.0
Long-LLaVA [ 58 ]
-
7B
64
60.3
51.4
44.1
52.0
52.0
-
Long-LLaVA [ 58 ]
+ Video-RAG [ 31 ]
7B
64
66.4
60.2
59.8
62.1
62.1
+10.1
Long-LLaVA [ 58 ]
+ Scene-RAG
7B
64
68.0
62.1
61.4
64.8
63.8
+11.8
Table 4: Performance of different MLLMs on the Video-MME [ 14 ] .
Table 5: Scene-RAG ablation study on SceneBench and MLVU [ 71 ] . Each column indicates whether the corresponding modality or component is used. Results are reported with an additional Gain column showing improvements over the no-RAG baseline.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: QA Accuracy Across Frame Lengths . Performance of SceneQA and SceneQA-Audio across different input frame lengths (16, 32, 64, and 128 frames). While SceneQA shows slight improvement with longer inputs, SceneQA-Audio performance peaks at moderate frame lengths and slightly declines for longer sequences.
Temporal
SceneQA
SceneQA-Audio
Range (s)
#Samples
Avg. Span
#Samples
Avg.Span
[0,250)
2110
163.13
2953
158.95
[250,500)
700
342.91
769
337.94
[500,750)
256
608.29
187
588.53
[750,1000)
57
853.72
34
861.50
[1000,1250)
68
1139.16
22
1109.36
Appendix
Table 6: Temporal span distribution of SceneQA and SceneQA-Audio. We report the number of samples and average temporal span (seconds) for each temporal range.
The evaluation of long-term video quality understanding remains an open challenge for large vision-language models (LVLMs). Existing video quality benchmarks predominantly focus on short clips and isolated distortions, overlooking the temporal continuity, cumulative degradation, and reasoning complexity inherent in long-duration content. To address these limitations, we present LongVQUBench, a comprehensive benchmark for long-term video quality understanding. LongVQUBench contains over 1200 diverse videos spanning movies, documentaries, surveillance footage, egocentric recordings, and animated content, accompanied by 1500 multiple-choice and open-ended questions for validation and testing. To assess perceptual reasoning across different temporal scopes, we introduce three progressively complex evaluation levels: (i) local event quality understanding (LQU) for analyzing localized distortions; (ii) cross-event quality reasoning (CQR) for integrating multiple degraded events; and (iii) global quality understanding (GQU) for holistic perceptual evaluation over extended durations. Furthermore, a needle distortion question-answering (NDQA) paradigm is embedded across all three levels, where spatial or temporal artifacts are sparsely inserted to probe fine-grained detection and reasoning capabilities. Extensive experiments on 14 state-of-the-art LVLMs reveal significant performance degradation with increasing video length and reasoning depth, highlighting their limited capacity for long-range temporal integration and perceptual attribution. We envision LongVQUBench as a foundational step toward the systematic, hierarchical, and explainable evaluation of LVLMs' long-term video quality understanding.
Real-world long video understanding requires models to perform continuous tracking, information integration and memory retention over massive temporal spans within extreme video durations. Mastering this intense cognitive load constitutes the fundamental bottleneck in long video understanding. While existing benchmarks have driven progress by scaling up video duration, their evaluation tasks often require comprehending only short and isolated video segments, falling short of capturing the challenge of ultra-long-context reasoning. To measure this cognitive load, we emphasize continuous certificate length, defined as the video length a human must continuously watch to definitively answer a given question. Driven by this metric, we introduce VideoOdyssey, a benchmark specifically designed for ultra-long-context and omni-modal video understanding. VideoOdyssey is characterized by three key features: 1) Extreme video duration and diversity: spanning 11 domains and 54 subcategories with an average video duration of 109 minutes; 2) Comprehensive evaluation scenarios: offering two subsets to address different research focuses, i.e., VideoOdyssey-V for probing the limits of visual understanding in MLLMs, and VideoOdyssey-AV for evaluating synchronized audio-visual understanding for omni-modal models; 3) Ultra-long and multi-level continuous certificates: extending the average continuous certificate to 16 minutes for VideoOdyssey-V and 12.8 minutes for VideoOdyssey-AV. Crucially, we design 5 granular levels from seconds to hours, providing a comprehensive diagnostic tool to evaluate models across varying context lengths and cognitive loads. Extensive evaluations show that bottlenecks of current MLLMs extend beyond simple retrieval to include struggles with continuous reasoning across varying context lengths, fine-grained perception, and non-verbal omni-modal understanding.
Haichen He, Jiayi Zhou, Sifeng Shang +3
Hong Kong Baptist University · GVC Lab, Great Bay University · S-Lab, Nanyang Technological University
Multimodal Large Language Models (MLLMs) have advanced image and video understanding and can increasingly handle longer visual inputs. Long-horizon tasks such as autonomous driving and robotic navigation require more than recognizing the current view, as models must remember and retrieve previously observed spatial layouts, routes, viewpoint changes, and object states. To evaluate this capability, we introduce LongSpace-Bench, a room-tour video benchmark for long-horizon spatial memory, covering scene perception, spatial relations, and spatial memory. In this work, we further propose LongSpace, a memory framework for long-video spatial reasoning. LongSpace models long videos as sequential chunks, incorporates 3D structural cues into early decoder layers, and constructs layer-aware memory for question-guided retrieval. Experiments on multiple spatial reasoning benchmarks show that LongSpace improves long-video spatial understanding, further demonstrating explicit spatial memory as a key capability for long-horizon video MLLMs.
Shiqiang Lang, Jing Liu, Haoyang He +6
Beijing University of Posts and Telecommunications · Zhongguancun Academy · Institute of Automation, Chinese Academy of Sciences +2