VideoMemBench: How Well Do MLLMs Remember in Video Conversations?

Zhengdong Hu; Chao Wang; Kaituo Feng; Huanjin Yao; Mingfei Lu; Hehe Fan; Yi Yang

cs.CVOct 29, 2026

VideoMemBench: How Well Do MLLMs Remember in Video Conversations?

Authors: Zhengdong Hu, Chao Wang, Kaituo Feng, Huanjin Yao, Mingfei Lu, Hehe Fan, Yi Yang

Organizations: University of Technology Sydney, Sydney, Australia · The Chinese University of Hong Kong, Hong Kong, China · The Hong Kong University of Science and Technology, Hong Kong, China · Zhejiang University, Hangzhou, China

Abstract

Multimodal large language models (MLLMs) can now process hour-long videos, making extended multi-turn dialogue over a single video a natural mode of use. Sustaining such a dialogue requires more than answering each question in isolation: the model must retrieve and reuse information established earlier in the conversation, both what it has seen and what it has said, an ability we term in-context video memory. Existing video QA benchmarks leave this ability unmeasured: their questions are mutually independent, never reference one another, and each is answerable from the video alone, so nothing ever requires carrying information across turns. We introduce VideoMemBench, in which each sample pairs a single video with a long multi-turn dialogue, whose turns are connected by typed cross-turn dependencies: an Episodic Dependency refers back to a visual episode described in an earlier turn, probing memory of what the model has seen, while an Answer Dependency builds on the model's own earlier answer, probing memory of what the model has said. Every dependency is verified considering two properties: 1) necessity, meaning the question cannot be answered without its earlier turns, and 2) non-leakage, meaning the question does not restate the information from its earlier turns. To enforce these properties, we design a construction pipeline with three LLM roles: a planner, a generator, and a judger, with each dialogue reviewed by human experts. VideoMemBench comprises 326 videos with 2,476 turns, of which 1,387 depend on earlier turns, including multi-parent dependencies and long-range call-back. Among over 20 evaluated open- and closed-source MLLMs, Claude Opus 5 performs the best with an overall score of only 45.7%, highlighting the challenge of in-context video memory. Our dataset and evaluation are released.

Figures & tables

Figure 1: Example of VideoMemBench with cross-turn dependencies: Independent turns ( e.g ., $T_{1}$ , $T_{8}$ ) establish visually grounded facts; dependent turns refer back to earlier turns through expressions such as “the scene from $T_{1}$ ” or “the target from $T_{1}$ ”, without restating earlier content. An Episodic Dependency ( e.g ., $T_{2}$ ) requires recalling what the model has seen : the model must recover which scene $T_{1}$ described before answering the question. An Answer Dependency ( e.g ., $T_{3}$ , $T_{4}$ ) requires recalling what the model has said : each builds on the answer produced at an earlier turn.

Benchmark	Input	#Videos	#Dialogs	#QAs	Max Dur.	Max Turns	Multi-turn	Dep. Graph	Typed	M-Parent
MVBench ( Li et al., 2024 )	Video	3,641	–	4,000	3 min	1	✗	✗	✗	✗
Video-MME ( Fu et al., 2025 )	Video	900	–	2,700	1 h	1	✗	✗	✗	✗
LVBench ( Wang et al., 2025 )	Video	103	–	1,549	2 h	1	✗	✗	✗	✗
MMDU ( Liu et al., 2024b )	Images	–	110	1,645	–	27	✓	✗	✗	✗
MT-Video-Bench ( Pan et al., 2026 )	Video	135	1,000	5,887	20 min	8	✓	✗	✗	✗
VideoMemBench (ours)	Video	326	326	2,476	23.47 h	21	✓	✓	✓	✓

Table 1: Comparison with existing benchmarks. Max Dur. is the maximum video duration in the benchmark. Multi-turn indicates multi-turn dialogue evaluation. Max Turns is the largest number of QA turns in one dialogue. Dep. Graph denotes structured dependency graphs for individual turns; Typed distinguishes answer and episodic dependencies; M-Parent indicates multiple parents.

Figure 2: The VideoMemBench construction pipeline. Starting from a curated video pool, a planner specifies turn intents and a typed dependency graph. A generator constructs the dialogue turn by turn, while a judger checks necessity, non-leakage, and visual grounding, returning failed turns for regeneration. Completed dialogues are checked against the plan and reviewed by human experts.

Figure 3: Dataset statistics of VideoMemBench. (a) Category distribution across four major domains and their fine-grained subcategories. (b) Distribution of videos by duration. (c) Distribution of the number of dialogue turns per video. (d) Distribution of question dependencies, including independent questions, episodic dependencies, and answer dependencies. (e) Text-length statistics for questions, answers, and answer choices, reported as mean $\pm$ standard deviation.

Model	Frames	TurnAcc (%)	EpiDepAcc (%)	AnsDepAcc (%)	Overall (%)
Human	–	88.1	86.0	84.2	86.1
Closed-source models
Claude Opus 5	128	52.3	49.1	35.7	45.7
Gemini 3.8 Flash	128	50.4	46.6	30.3	42.4
GPT-5.6-sol	128	50.0	47.8	27.1	41.7
Gemini 3.5 Flash	128	49.6	45.5	29.7	41.6

Table 2: Main results on VideoMemBench. Models are evaluated under the self-rollout protocol. TurnAcc: accuracy over all turns, scored individually. EpiDepAcc: accuracy on turns with at least one Episodic dependency. AnsDepAcc: accuracy on turns with at least one Answer dependency under chain crediting. Overall: the arithmetic mean of the three metrics. The best and second-best results in each metric within each model group are bold and underlined , respectively.

Figure 4: Analysis of factors affecting in-context video memory. (a) Accuracy on dependent turns by dependency distance, measured from the current turn to its earliest parent; the dashed line shows the mean across all evaluated models. (b) Accuracy on dependent turns by the number of parents. (c) Independent- and dependent-turn accuracy by dialogue position for two models: the light segment shows dependent-turn accuracy, the full bar height shows independent-turn accuracy, and the dark segment represents their difference. (d) Accuracy on answer-dependent turns under self-rollout and oracle histories. Oracle histories replace previous model responses with ground-truth answers.

Model	Indep. (%)	EpiDepAcc (%)	AnsDepAcc (%)	Dep. (%)	$\Delta$ (pp)
Kimi K3	51.4	49.0	27.1	38.1	13.4
GPT-5.6-sol	50.6	47.8	27.1	37.5	13.1
Claude Sonnet 5	50.0	45.3	27.3	36.3	13.7
Qwen3.5-27B	46.0	39.8	20.9	30.3	15.7
Qwen3-VL-235B-A22B	47.4	41.6	18.6	30.1	17.3

Table 3: Independent versus dependent turns. Indep.: accuracy over the independent turns. Dep.: mean of EpiDepAcc and AnsDepAcc.

\Delta=\mathrm{Indep.}-\mathrm{Dep.}

, measured in percentage points (pp).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The expert review interface. For each dialogue, the expert replays the video in the top panel, where the evidence window of the current turn is marked on the timeline, and reads the dialogue turn by turn, where the parent turns and the type of each dependency are highlighted. The expert answers the current question before the reference answer is revealed, then marks the turn as correct , answer wrong , dependency not required , wording leaks a parent , or duplicate , and edits the question, the options, or the answer in place. A dialogue is retained only after every turn has been marked correct or revised, and the interface records each decision together with the time spent on the turn. Referent here means episodic dependency. Please zoom in to see the details.

Model	Frames	TurnAcc (%)	EpiDepAcc (%)	AnsDepAcc (%)	Overall (%)
Qwen3-VL-235B-A22B	16	33.7	32.3	11.7	25.9
Qwen3.5-122B-A10B	16	33.2	32.4	11.9	25.8
Qwen3.5-27B	16	33.9	33.7	8.2	25.3
Qwen3-VL-32B	16	30.6	28.9	10.5	23.3
Qwen3-VL-30B-A3B	16	29.6	30.6	7.6	22.6
Qwen3.5-35B-A3B	16	30.7	28.1	9.0	22.6

Appendix

Table 4: Frame-count ablation on VideoMemBench under self-rollout. Each model is evaluated with 16, 32, and 64 uniformly sampled frames per video, using the same metrics as the 128-frame evaluation in Table 2 . Models are ranked by the overall score within each frame budget, with the best result for each metric shown in bold .

Figure 6: A soccer dialogue example from VideoMemBench. Seven dialogue turns cover player identification, spatial tracking, and match events, with later questions referring to previously discussed players. Reference answers are shown in red, and model responses marked correct at the turn level are shown in green.

Figure 7: A basketball dialogue example from VideoMemBench. Selected video frames are shown above, followed by eight dialogue turns with reference answers and responses from five models. Reference answers are shown in red, and responses marked correct are shown in green.

MLLM Evaluation

See all 40 papers

Jun 5, 2026

Watch, Remember, Reason: Human-View Video Understanding with MLLMs

Video understanding is being rapidly transformed by multimodal large language models (MLLMs), as research moves from short clips to long, multimodal, and knowledge-intensive video scenarios. These scenarios require models to handle sparse evidence, long-range dependencies, multimodal alignment, and reliable inference under limited computational budgets. This work presents a human-view perspective on LLM-based video understanding, organized around three functional abilities: watching, remembering, and reasoning. Rather than treating video tasks as isolated benchmarks, this view provides a unified structure for analyzing how video MLLMs acquire evidence, preserve context, and produce grounded outputs. We introduce a formulation that characterizes video understanding systems by their perceptual representations, memory states, reasoning traces, and final predictions. Based on this formulation, we identify challenges in spatio-temporal perception, efficient long-video processing, memory modeling, streaming understanding, and faithful reasoning. Representative methods are organized by their roles in video MLLM systems. Watching covers fine-grained, comprehensive, audio-visual, and efficient perception. Remembering includes offline and streaming memory, while reasoning covers text-only reasoning and thinking with videos. We further examine application domains such as egocentric, sports, instructional, medical, and narrative videos, and cover training datasets and evaluation benchmarks across task types, supervision formats, modalities, and capability dimensions. Finally, we outline open problems and future directions for scalable, memory-aware, and evidence-grounded video intelligence. Related works will be continuously traced at https://github.com/marinero4972/Awesome-HumanView-VideoUnderstanding.

Jiahao Meng, Yue Tan, Qi Xu +12

School of Intelligence Science and Technology, Peking University · Wuhan University · Shanghai Jiao Tong University +7

Streaming Video Understanding Video-Language Models Long-Video Understanding Video Reasoning Multimodal Understanding

Ask PDF

Oct 17, 2026

Write Once, Query Many: Evolving Compact Reusable Memory for Long Videos

Long videos often exceed the context capacity of multimodal large language models (MLLMs), while agentic search repeats costly video processing and grounding for each query. Reusable memory can amortize this cost, but must preserve information for questions unknown during construction. We propose Video-EMem, an evolvable video memory harness built on time-anchored perceptual observations. This representation combines compact free-form text with sparse images to preserve semantic content and complementary visual details across temporal scales, supporting temporal navigation without a fixed semantic schema. A Writer encodes each video once by consolidating overlapping multiscale observations, and a Reader retrieves and organizes evidence solely from the resulting memory. With the underlying models frozen, we evolve their workflows and operators using distinct feedback: query-independent source fidelity and coverage for the Writer, and QA failures in evidence localization and utilization for the Reader. Experiments support this separation for generalization beyond development queries, with our evolution strategy improving average accuracy by 7.2% over the unevolved harness. While reducing storage by about 99.86% on average relative to raw videos, Video-EMem improves Qwen3-VL-30B by an average of 2.6% across three long-video QA benchmarks over direct input of 2,048 frames. Using the same o3 answer model as the state-of-the-art agentic method, it improves average accuracy by 2.5% with lower per-query latency.

Kerui Chen, Jinglu Wang, Yun Wang +2

College of Artificial Intelligence, Zhejiang University. · Microsoft Research Asia

Memory-Augmented Video Understanding Memory-Augmented VLMs Long-Video Understanding Long-Video QA

Ask PDF

Mar 31, 2026

Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism

Long video understanding is a key challenge that plagues the advancement of Multimodal Large language Models (MLLMs). In this paper, we study this problem from the perspective of visual memory mechanism, and proposed a novel and training-free approach, termed Flexible Memory (FlexMem). In principle, FlexMem aims to mimic human behavior of video watching, i.e., continually watching video content and recalling the most relevant memory fragments to answer the question. In this way, FlexMem can help MLLMs achieve video understanding of infinite lengths, unlike previous methods that process all video information at once and have input upper-limit. Concretely, FlexMem first consider the visual KV caches as the memory sources, and realize the effective memory transfer and writing via a dual-pathway compression design. Afterwards, FlexMem also explores different memory reading strategies for the diverse video understanding tasks, including the popular streaming one. To validate FlexMem, we apply it to two popular video-MLLMs, and conduct extensive experiments on five long video and one streaming video task. The experimental results show that on a single 3090 GPU, our FlexMem can achieve obvious improvements than existing efficient video understanding methods and process more than 1k frames, which also helps the base MLLMs achieve comparable or even better performance than SOTA MLLMs on some benchmarks, e.g. , GPT-4o and Gemini-1.5 Pro.

Tao Chen, Kun Zhang, Qiong Wu +5

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · National University of Defense Technology, 230000, P.R. China.

Long-Video Understanding Memory-Augmented Video Understanding Multimodal Large Language Models Streaming Video Understanding

Ask PDF

Abstract

MLLM Evaluation

Explore similar work

Watch, Remember, Reason: Human-View Video Understanding with MLLMs

Write Once, Query Many: Evolving Compact Reusable Memory for Long Videos

Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism