Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline

Guo Chen; Lidong Lu; Yicheng Liu; Liangrui Dong; Lidong Zou; Jixin Lv; Zhenquan Li; Xinyi Mao

cs.CVMar 5, 2026

Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline

Authors: Guo Chen, Lidong Lu, Yicheng Liu, Liangrui Dong, Lidong Zou, Jixin Lv, Zhenquan Li, Xinyi Mao, +12 more

Organizations: 1Nanjing University · 2NVIDIA · 3Zhejiang University · 4Shanghai Jiao Tong University · 5The University of Tokyo

Abstract

While datasets for video understanding have scaled to hour-long durations, they typically consist of densely concatenated clips that differ from natural, unscripted daily life. To bridge this gap, we introduce MM-Lifelong, a dataset designed for Multimodal Lifelong Understanding. Comprising 181.1 hours of footage, it is structured across Day, Week, and Month scales to capture varying temporal densities. Extensive evaluations reveal two critical failure modes in current paradigms: end-to-end MLLMs suffer from a Working Memory Bottleneck due to context saturation, while representative agentic baselines experience Global Localization Collapse when navigating sparse, month-long timelines. To address this, we propose the Recursive Multimodal Agent (ReMA), which employs dynamic memory management to iteratively update a recursive belief state, significantly outperforming existing methods. Finally, we establish dataset splits designed to isolate temporal and domain biases, providing a rigorous foundation for future research in supervised learning and out-of-distribution generalization.

Figures & tables

Figure 1: Physical Temporal Span vs. Scale. The x-axis represents the Physical Temporal Span (

T_{span}

), while bubble size indicates Observational Duration (

T_{dur}

). Unlike existing datasets clustered in the bottom-left (short clips,

T_{span}\approx T_{dur}

), MM-Lifelong occupies the unique Lifelong Regime (top-right). This regime is characterized by high temporal sparsity (

T_{span}\gg T_{dur}

), requiring models to bridge unobserved gaps across days to months.

Figure 2: Performance Scaling Analysis. As the number of input frames increases, end-to-end MLLMs initially improve but soon exhibit performance oscillation and even sharp degradation due to context saturation and noise accumulation. In contrast, ReMA consistently scales with more recursion rounds, effectively mitigating this bottleneck via dynamic memory management and demonstrating superior scaling potential and stability.

Figure 3: 1) Live stream subset of MM-Lifelong comprises 105.6 hours of broadcast footage spanning 51 days. 2) An example of a multi-clue (hop) reasoning question with an ultra-long temporal certificate: The task requires identifying all occurrences where the streamer sings a specific song on subways across multiple cities. Successfully answering this requires persistent memory and the ability to perform multi-event inference over more than 10 hours of continuous livestream data.

Scale	Domain	Subject	$T_{dur}$	$T_{span}$	Data Source Description
Day	Gamer’s Journey	The Protagonist	23.6h	$\sim$ 24h	Complete narrative walkthrough tracking the avatar’s inventory and skill progression.
Week	Egocentric Life	The Wearer	51.9h	$\sim$ 7d	Continuous first-person recording of daily routines and household interactions from EgoLife ( Yang et al., 2025b ) .
Month	Live Stream	The Streamer	105.6h	$\sim$ 51d	Unscripted IRL stream tracking the influencer’s travel across cities and social events.

Table 1: The Multi-Scale Split of MM-Lifelong Dataset. We structure the dataset around the continuous experience of a Cognitive Subject . Distinct from surveillance, each domain tracks the state accumulation of a specific agent.

Statistics	Number
Total Duration	181.1 hours
Total Questions	1289
* Avg. Question Length	26.79 words
* Avg. Answer Length	4.80 words
Total Clue Intervals	1810 (100%)
* Short ( $<$ 90s)	1039 (57.40%)

Table 2: Statistics of the MM-Lifelong dataset.

I. Short-Context Multimodal Dataset
Dataset	Modalities	#Samples	Max. Dur	Max. Span	Anno.	QA	Clue
MMMU ( Yue et al., 2024 )	Image	11.5k	0	0	M	11.5k	✗
AIR-Bench ( Yang et al., 2024 )	Audio	19k	19.4s	19.4s	A&M	19k	✗
OmniBench ( Li et al., 2024b )	Audio+Image	1.1k	30s	30s	A&M	1.1k	✗
MVBench ( Li et al., 2024a )	Video	4.0k	2.95m	2.95m	A	4.0k	✗
II. Long-Context Multimodal Dataset

Table 3: Comparison with representative multimodal datasets with increasing context lengths. We categorize existing datasets into Short-Context, Long-Context, and Lifelong horizon. Max. Dur (

T_{dur}

) denotes the maximum playback duration of processed clips, while Max. Span (

T_{span}

) represents the actual physical timeline covered by the event. Unlike prior datasets where

T_{span}\approx T_{dur}

, MM-Lifelong introduces the Lifelong Horizon (

T_{span}\gg T_{dur}

), requiring reasoning over unobserved temporal gaps spanning up to 2 months. Notably, it is one of the few datasets providing manual, clue-grounded annotations ( Clue ) for continuous audio-visual streams.

Figure 6: The Architecture of the Recursive Multimodal Agent (ReMA). ReMA follows an offline two-phase architecture for long-form multimodal reasoning. The agent maintains a global Memory Bank for belief state aggregation and leverages a Multimodal Toolkit (e.g., MMInspect and MemManage ), backed by foundation models (e.g., Whisper, Qwen3-VL), to perform global perception and iterative control for query answering.

End-to-End MLLMs
		Val@Month		Test@Week		Test@Day
Methods	Frames	Acc	Ref@300	Acc	Ref@300	Acc	Ref@300
Human	Full	80.4	33.5	95.6	42.4	99.2	49.8
GPT-5 ( Singh and OpenAI, 2026 )	50	14.87	0.44	15.00	0.92	15.25	0.53
Qwen3-VL-235B-A22B ( Yang et al., 2025a )	1536	14.33	0.06	15.63	0.80	12.44	0.79
Qwen3-VL-30B-A3B ( Yang et al., 2025a )	1536	11.92	0.64	11.07	0.77	11.48	0.42

Table 4: Performance comparison on val@month, test@week, and test@day set of MM-Lifelong.

Figure 7: Ablation on Recursive Depth. Impact of the maximum allowed tool-call rounds on Answer Accuracy. Performance saturates around 4-5 rounds.

$\Delta t$	Acc	Ref@60	Ref@300	Ref@600	Avg. Rounds
2min	12.83	7.82	11.23	13.12	4.92
5min	9.40	6.28	9.28	11.28	4.91
15min	8.07	4.08	5.79	7.39	4.92
1hour	6.27	1.93	2.37	3.34	5.24
Full	3.72	0.18	0.24	0.31	6.81

Table 5: Impact of Perception Granularity (

\Delta t

). We compare accuracy, grounding performance, and reasoning cost (Average Rounds) across different memory update intervals.

Backbone for MLLM Tool
Model	Acc	Ref@60	Ref@300	Ref@600	Avg. Rounds
Qwen3-VL-A3B	9.40	6.29	9.28	11.28	4.91
GPT-5	10.57	8.14	11.48	14.51	5.31
Backbone for Central Controller
GPT-5	9.40	6.29	9.28	11.28	4.91
Qwen3-VL-A3B	7.12	1.17	1.82	2.28	3.79

Table 6: Component Analysis. Performance comparison using different backbones for the MLLM Inspection Tool (top) and the Central Controller (bottom).

Model Judge	Acc Score	F1 Score
GPT-5 ( Singh and OpenAI, 2026 )	9.40	99.39
GPT-o4-mini ( OpenAI, 2024a )	9.22	98.78
GPT-4.1 ( OpenAI, 2025 )	9.56	98.20
Human	9.22	100.00

Table 7: Judge Consistency (vs. Human).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Table 8: Temporal metadata of video clips across subsets. Each clip is annotated with its begin and end time, using relative timestamps or absolute UTC timestamps.

Category	Definition	Day	Week	Month	Total
Counting	For the object(s) or event(s) mentioned, ask how many times they appear or repeat.	40 (20.00%)	59 (29.50%)	213 (23.96%)	312 (24.20%)
Causal Reasoning	For an event mentioned, ask about its cause or the result it leads to.	20 (10.00%)	17 (8.50%)	151 (16.99%)	188 (14.58%)
Entity Recognition	Identify a specific entity (object, person, or place) referenced in the question.	31 (15.50%)	23 (11.50%)	132 (14.85%)	186 (14.43%)
Temporal Reasoning	Ask about temporal order, chronological sequencing, or duration of events.	31 (15.50%)	36 (18.00%)	85 (9.56%)	152 (11.79%)
Hallucination Det.	Given a set of statements, ask which statements are correct or which are wrong.	31 (15.50%)	10 (5.00%)	71 (7.99%)	112 (8.69%)
Event Recognition	Perform a recognition or identification of an event mentioned in the question.	11 (5.50%)	17 (8.50%)	80 (9.00%)	108 (8.38%)

Appendix

Table 12: Statistics and definitions of question categories. The categories are sorted by the total number of samples.

Question sub-category	QA pair example
Counting	Q: In the first live stream after returning from both the China trip and the Mongolia trip, how many times did [The Streamer] hear the song ’Sunshine Rainbow Little White Horse’ while browsing videos on Discord?
	A: 8 times
Event Recognition	Q: In the game FRAGPUNK played by [The Streamer], in the sixth round of the new match after the final score was 4:2, what event occurred that shocked [The Streamer]?
	A: [The Streamer] and [The Streamer]’s teammates were all killed by an invisible enemy.
Language Content Recall	In Day 5, when the first-person protagonist arrives at the supermarket, what is written on the front of their tablet’s case?
	A: Scholar.

Appendix

Table 13: QA example of each category.

Module	Component	Configuration
Vector Store	FAISS	Euclidean distance
Embedder	OpenAI	text-embedding-3-large
LLM	OpenAI	GPT-4.1-mini (T=0.1)
Reranker	LLM-based	GPT-4.1-mini (T=0, top- $k$ =5)

Appendix

Table 14: Implementation details of the memory system based on Mem0.

End-to-End MLLMs
		Train@Month		Val@Month		Test@Day		Test@Week
Methods	Frames	Acc	Ref@300	Acc	Ref@300	Acc	Ref@300	Acc	Ref@300
Human	Full	82.5	31.2	80.4	33.5	99.2	49.8	95.6	42.4
GPT-5 ( Singh and OpenAI, 2026 )	50	10.15	1.39	14.87	0.44	15.25	0.53	15.00	0.92
Qwen3-VL-235B-A22B ( Yang et al., 2025a )	1536	9.09	0.39	14.33	0.06	12.44	0.79	15.63	0.80
Qwen3-VL-30B-A3B ( Yang et al., 2025a )	1536	8.33	0.48	11.92	0.64	11.48	0.42	11.07	0.77

Appendix

Table 15: Performance comparison on train@month, val@month, test@week, and test@day set of MM-Lifelong.

Panel of Figure 8 — Figure 8: Ref@N performance under different N settings across four data splits: (a) Day-scale subset, (b) Week-scale subset, (c) Month-scale subset, and (d) the Full dataset.

Explore similar work

Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoning
May 8, 2026Jiazheng Li, Chi-Hao Wu, Yunze Liu +3Video Understanding Multimodal Memory
University of Connecticut · Northwestern University · University of Virginia
Agentic Very Long Video Understanding
Jan 26, 2026Aniket Rege, Arka Sadhu, Yuliang Li +5Question Answering Egocentric Video Understanding
Reality Labs Research at Meta · University of Wisconsin-Madison · Work done during internship at Meta
InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning
Jun 10, 2026Ziang Yan, Sheng Xia, Jiashuo Yu +10Efficient Multimodal Inference Multimodal Reasoning
Shanghai AI Laboratory · Shanghai Innovation Institute · Nanjing University

May 8, 2026J/K move · Enter open · S save

Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoning

Jiazheng Li, Chi-Hao Wu, Yunze Liu, Kaize Ding+2

University of Connecticut · Northwestern University · University of Virginia

Video Understanding Multimodal Memory Long-Video Understanding Agentic RAG Memory-Augmented Video Understanding Long-Video QA

Open paper AskPDF arXiv

Understanding ultra-long videos such as egocentric recordings, live streams, or surveillance footage spanning days to weeks, remains a challenge. For current multimodal LLMs: even with million-token context windows, frame budgets cover only tens of minutes of densely sampled video, and most evidence is discarded before inference begins. Memory-augmented and agentic approaches help with scale, but their retrieval remains fragmented across modalities and lacks long-range narrative summaries that span days or weeks. We propose MAGIC-Video, a training-free framework built around a multimodal memory graph with interleaved narrative chain: the graph unifies episodic, semantic, and visual content through six typed edges and supports cross-modal retrieval, while the chain distils long-horizon entity biographies and recurring activity events. At inference time, an agentic loop interleaves graph retrieval with narrative fact injection, covering both the modality and time dimensions of ultra-long video in a single retrieval pipeline. On EgoLifeQA, Ego-R1 and MM-Lifelong, MAGIC-Video consistently outperforms strong general-purpose, long-video, and agentic baselines, with gains of 10.1, 7.4, and 5.9 points over the prior best agentic system on each benchmark. Code is available at https://github.com/lijiazheng0917/MAGIC-video.

Abstract

Explore similar work

Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoning

Agentic Very Long Video Understanding

InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning