Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark

Seng Nam Chen; Hao Chen; Chenglam Ho; Xinyu Mao; Jinping Wang; Yu Zhang; Chao Li

cs.CVMar 28, 2026

Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark

Authors: Seng Nam Chen, Hao Chen, Chenglam Ho, Xinyu Mao, Jinping Wang, Yu Zhang, Chao Li

Organizations: CUHK (SZ) · University of Cambridge · UESTC · CUHK · GPNU · Shanghai Jiao Tong University

Abstract

Long video understanding (LVU) remains a core challenge in multimodal learning. Although recent vision-language models (VLMs) have made notable progress, existing benchmarks mainly focus on either fine-grained perception or coarse summarization, offering limited insight into temporal understanding over long contexts. In this work, we define a scene as a coherent segment of a video in which both visual and semantic contexts remain consistent, aligning with human perception. This leads us to a key question: can current VLMs reason effectively over long, scene-level contexts? To answer this, we introduce a new benchmark, SceneBench, designed to provide scene-level challenges. Our evaluation reveals a sharp drop in accuracy when VLMs attempt to answer scene-level questions, indicating significant forgetting of long-range context. To further validate these findings, we propose Scene Retrieval-Augmented Generation (Scene-RAG), which constructs a dynamic scene memory by retrieving and integrating relevant context across scenes. This Scene-RAG improves VLM performance by +2.50%, confirming that current models still struggle with long-context retention. We hope SceneBench will encourage future research toward VLMs with more robust, human-like video comprehension.

Figures & tables

Figure 1: Longer SceneQA distances lead to lower accuracy. Video-RAG [ 31 ] improves the baseline in short- and long-range settings but struggles in the mid-range, while our Scene-RAG achieves consistent gains, especially for mid- and long-term reasoning. Curves are smoothed using a rolling mean. Results use Qwen2.5-VL [ 1 ] . Solid markers denote actual measurements; curves are cubic-spline interpolations for visual clarity.

Figure 2: We divide video information into frame, clip, scene, and video levels. For frame-level information, it focuses mainly on describing the details of the subject within the frame. Clip-level information includes temporal information but can only describe the objective behavior of objects. Scene-level information contains a lot of clip-level information and forms a complete scenario event, while video consists of a lot of scene-level content, with related scenes forming a logical storyline based on cause and effect. For convenience of quantification, we use two minutes to distinguish between clip and scene levels.

Benchmark	Year	#Videos	Len.(s)	#Tasks	#Genres	#QA Pairs	QA Len. (s)	QA Density	Multi- level	Scene Reliance	Audio Reliance
LVBench [ 47 ]	2025	103	4,101	6	21	1,549	–	0.38	✗	✗	✗
LongVideoBench [ 52 ]	2024	3,763	473	17	10	6,678	–	14.1	✗	✗	✗
HourVideo [ 3 ]	2024	500	2,742	4	16	12,976 †	–	4.7	✗	✗	✗
ALLVB [ 43 ]	2025	1,376	7,200	9	30	252,000 †	–	35.0	✗	✗	✗
Video-MME [ 14 ]	2025	900	1,024	12	30	2,700	–	2.6	✓	✗	✗
MoVQA [ 66 ]	2023	930	992	6	1	21,953	–	22.1	✓	✗	✗

Table 1: Comparison of SceneBench with existing long-video understanding benchmarks. SceneBench offers a balanced number of videos, diverse genres, and high–time-range, scene-based QA annotations. The average temporal distance between each question and its corresponding answer in SceneQA is 262s, with a standard deviation of 310s, illustrating the wide temporal diversity of the dataset. “

\dagger

” denotes automatically or partially automatically generated QA pairs.

Figure 3: Statistical overview of our SceneBench benchmark. (A) Distribution of scene lengths. (B) Distribution of task counts. (C) Distribution of SceneQA and SceneQA-Audio length duration proportion over the full video.

Figure 4: Comparison of Scene-RAG with traditional RAGs working on video understanding tasks. Scene-RAG first aggregates long-range visual scenes via scene tiling, stores aligned visual–audio evidence, and retrieves task-relevant segments conditioned on user queries.

Method	Release	Frame Count	Title Pred.	Comm. Pred.	Clip QA	Scene QA	SceneQA -Audio	I-VQA	Avg.
Proprietary MLLMs
Gemini 2.5 Pro [ 9 ]	2025	1fps	99.4	92.9	80.5	60.7	65.8	82.4	80.3
Gemini 2.5 Flash-Lite [ 9 ]	2025	1fps	99.1	78.2	48.4	25.6	30.1	41.6	53.8
Kimi 2.5v [ 44 ]	2025	128	99.4	90.9	73.1	54.0	59.6	60.7	73.0
Open-source MLLMs
MovieChat [ 41 ]	2023	2048	25.2	28.6	52.2	21.9	21.4	21.6	28.5

Table 2: Benchmark results on the SceneBench. “Frame Count” indicates the number of frames used as input. Comm.: Comment.

Method	RAG Type	LLM Params	Frame Count	Title Pred.	Comm. Pred.	Clip QA	Scene QA	SceneQA -Audio	I-VQA	Avg.	Gain
LongVA [ 67 ]	-	7B	128	96.8	71.1	54.3	27.7	29.5	40.1	53.3	-
LongVA [ 67 ]	+ Video-RAG [ 31 ]	7B	128	97.7	70.4	53.7	28.3	30.2	39.2	53.4	+0.1
LongVA [ 67 ]	+ Scene-RAG	7B	128	97.7	71.5	54.6	28.9	32.4	41.4	54.4	+1.1
Long-LLaVA [ 58 ]	-	7B	64	96.2	79.7	57.1	26.1	25.4	33.0	52.9	-
Long-LLaVA [ 58 ]	+ Video-RAG [ 31 ]	7B	64	96.2	80.8	56.5	24.3	24.3	35.4	52.9	+0.0
Long-LLaVA [ 58 ]	+ Scene-RAG	7B	64	96.2	81.8	57.6	26.8	28.0	37.0	54.6	+1.7

Table 3: Performance of different MLLMs on the SceneBench benchmark. The average improvement of Scene-RAG is

\sim

1.40% in average.

Method	RAG Type	LLM Params	Frame	Short	Medium	Long	Overall	Avg.	Gain
LongVA [ 67 ]	-	7B	128	60.9	49.3	44.0	51.4	51.4	-
LongVA [ 67 ]	+ Video-RAG [ 31 ]	7B	128	66.1	60.4	59.4	62.0	62.0	+10.6
LongVA [ 67 ]	+ Scene-RAG	7B	128	66.9	60.6	59.7	63.4	62.4	+11.0
Long-LLaVA [ 58 ]	-	7B	64	60.3	51.4	44.1	52.0	52.0	-
Long-LLaVA [ 58 ]	+ Video-RAG [ 31 ]	7B	64	66.4	60.2	59.8	62.1	62.1	+10.1
Long-LLaVA [ 58 ]	+ Scene-RAG	7B	64	68.0	62.1	61.4	64.8	63.8	+11.8

Table 4: Performance of different MLLMs on the Video-MME [ 14 ] .

Table 5: Scene-RAG ablation study on SceneBench and MLVU [ 71 ] . Each column indicates whether the corresponding modality or component is used. Results are reported with an additional Gain column showing improvements over the no-RAG baseline.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: QA Accuracy Across Frame Lengths . Performance of SceneQA and SceneQA-Audio across different input frame lengths (16, 32, 64, and 128 frames). While SceneQA shows slight improvement with longer inputs, SceneQA-Audio performance peaks at moderate frame lengths and slightly declines for longer sequences.

Temporal	SceneQA		SceneQA-Audio
Range (s)	#Samples	Avg. Span	#Samples	Avg.Span
$[0,250)$	2110	163.13	2953	158.95
$[250,500)$	700	342.91	769	337.94
$[500,750)$	256	608.29	187	588.53
$[750,1000)$	57	853.72	34	861.50
$[1000,1250)$	68	1139.16	22	1109.36

Appendix

Table 6: Temporal span distribution of SceneQA and SceneQA-Audio. We report the number of samples and average temporal span (seconds) for each temporal range.

Stage	Visual Enc.	Audio	LLM	Total
	(InternVideo2)	(QwenAudio2)	(Qwen3)	(Seconds)
Offline Preprocess	273.52	3.20	-	276.72
Online Inference	1.04	0.19	61.06	62.29

Appendix

Table 7: Runtime latency breakdown (video Length: 2,767s).

Figure 6: Overview of the Annotation pipeline. Zoom in for better visibility.

Setting	TV-L1 $\alpha$	TV-L1 $L_{\min}$ (s)	Scene-RAG $K$	Avg. Result	Gain
Full Model (Ours)	1.5	3.0	10	62.4	-
$\alpha$ ↓ (0.5)	0.5	3.0	10	61.2	-1.2
$\alpha$ ↑ (2.0)	2.0	3.0	10	61.7	-0.7
$L_{\min}$ ↓ (2s)	1.5	2.0	10	61.9	-0.5
$K$ ↓ (5)	1.5	3.0	5	62.1	-0.3
$K$ ↑ (15)	1.5	3.0	15	62.0	-0.4

Appendix

Table 8: Ablation over SceneTiling

(\alpha,L_{\min})

and Scene-RAG retrieval size

K

on VideoMME [ 12 ] . Model: Longva [ 67 ] .

Figure 7: Examples QAs of SceneBench. Overview of the problem set.

Figure 8: Examples QAs of SceneBench. Overview of the problem set.

Explore similar work

LongVQUBench: Benchmarking Long-Term Video Quality Understanding of Vision-Language Models
Jul 1, 2026Arpita Nema, Hanwei Zhu, Xi Zhang +1VLM Evaluation Video Quality Assessment
Nanyang Technological University, Singapore
VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding
May 21, 2026Haichen He, Jiayi Zhou, Sifeng Shang +3VLM Evaluation Video Understanding
Hong Kong Baptist University · GVC Lab, Great Bay University · S-Lab, Nanyang Technological University
LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video
Jun 4, 2026Shiqiang Lang, Jing Liu, Haoyang He +6Visual Spatial Reasoning Multimodal Large Language Models
Beijing University of Posts and Telecommunications · Zhongguancun Academy · Institute of Automation, Chinese Academy of Sciences +2

Jul 1, 2026J/K move · Enter open · S save

LongVQUBench: Benchmarking Long-Term Video Quality Understanding of Vision-Language Models

Arpita Nema, Hanwei Zhu, Xi Zhang, Weisi Lin

Nanyang Technological University, Singapore

VLM Evaluation Video Quality Assessment Long-Video Understanding

Open paper AskPDF arXiv

The evaluation of long-term video quality understanding remains an open challenge for large vision-language models (LVLMs). Existing video quality benchmarks predominantly focus on short clips and isolated distortions, overlooking the temporal continuity, cumulative degradation, and reasoning complexity inherent in long-duration content. To address these limitations, we present LongVQUBench, a comprehensive benchmark for long-term video quality understanding. LongVQUBench contains over 1200 diverse videos spanning movies, documentaries, surveillance footage, egocentric recordings, and animated content, accompanied by 1500 multiple-choice and open-ended questions for validation and testing. To assess perceptual reasoning across different temporal scopes, we introduce three progressively complex evaluation levels: (i) local event quality understanding (LQU) for analyzing localized distortions; (ii) cross-event quality reasoning (CQR) for integrating multiple degraded events; and (iii) global quality understanding (GQU) for holistic perceptual evaluation over extended durations. Furthermore, a needle distortion question-answering (NDQA) paradigm is embedded across all three levels, where spatial or temporal artifacts are sparsely inserted to probe fine-grained detection and reasoning capabilities. Extensive experiments on 14 state-of-the-art LVLMs reveal significant performance degradation with increasing video length and reasoning depth, highlighting their limited capacity for long-range temporal integration and perceptual attribution. We envision LongVQUBench as a foundational step toward the systematic, hierarchical, and explainable evaluation of LVLMs' long-term video quality understanding.

Abstract

Explore similar work

LongVQUBench: Benchmarking Long-Term Video Quality Understanding of Vision-Language Models

VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding

LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video