MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories

Guangzhi Xiong; Xinyuan Zhang; Xiao Yang; Hyokun Yun; Kai Zhang; Shiun-Zu Kuo; Hyeonjeong Ha; Xilun Chen

cs.CVSep 30, 2026

MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories

Authors: Guangzhi Xiong, Xinyuan Zhang, Xiao Yang, Hyokun Yun, Kai Zhang, Shiun-Zu Kuo, Hyeonjeong Ha, Xilun Chen, +9 more

Organizations: Meta Reality Labs · University of Virginia · Work done at Meta · University of Illinois Urbana-Champaign

Abstract

Long-term egocentric video enables personalized AI assistants to reason about daily life. However, as video histories grow to hundreds of hours spanning months or years, reprocessing raw clips for every query becomes computationally prohibitive. Memory systems offer a scalable alternative by compacting videos into text representations, but often fail on practical benchmarks: either the memory does not preserve key evidence, or the retriever fails to locate relevant entries due to retrieval competition in growing search spaces. To address these challenges, we introduce MemLife, a multimodal memory system that constructs entity-grounded, first-person text episodes and retrieves them via a time-indexed agentic reader. Without training or query-time video access, MemLife improves over the strongest training-free baseline by 4.6--12.0% across four long-horizon benchmarks. To further improve memory quality, we propose MemOpt, a reinforcement learning framework that optimizes the memory writer to produce faithful, informative, and retrievable memories. MemOpt consistently improves MemLife by 2.7--5.0% across different video and question distributions, with gains that generalize across writer and reader backbones and memory systems.

Figures & tables

Figure 1: Failure modes of long-term video memory systems. Dotted objects denote lost information (e.g., Week-1 video is deleted). Systems fail when relevant evidence is omitted during memory writing or missed during retrieval. Our proposed solutions outperform prior state-of-the-art systems.

Figure 2: Overview of MemLife ’s episodic writer and time-indexed agentic reader, together with MemOpt ’s decomposed writer supervision. Direct source-video access is optional for the reader.

Method	SuperMemory-VQA		EgoLifeQA		SuperMemory-LVQA		EgoLife-EQA
Method	Accuracy	Recall	Accuracy	Recall	Accuracy	Recall	Accuracy	Recall
Reference
Oracle Context	67.58	100.00	66.20	100.00	67.58	100.00	61.00	100.00
Without Memory-Writer Training
Video ReCap	36.28	54.01	35.20	27.40	39.17	16.79	38.00	27.00
EgoRAG	49.28	66.79	48.20	29.60	47.51	47.90	38.00	13.00

Table 1: Comparison with existing video memory systems. Oracle Context bypasses retrieval by supplying all annotated source-video evidence directly to the reader. MemLife -V permits source-video access. Bold and underlined values mark the best and second-best results within each group.

Figure 3: Memory quality across EgoLifeQA question categories. Oracle accuracy is measured by providing memory entries aligned with annotated evidence directly to the reader.

Frames
Question	Q: […] Did I add the milk before or after the eggs when making the batter?	Q: […] Did I set a reminder for a backup dinner plan?
	A: You did not add milk or eggs to the batter; you only added lentils, salt, and spices.	A: No […] However, you did mention earlier that you have baked chicken available.
MemLife	I am in a kitchen […] transfer yellow corn kernels from a small food processor bowl […] move toward the sink area […]	[…] preparing food […] they respond to my question about being hungry by saying they can wait for the food […]
MemLife + MemOpt	I am in a kitchen […] yellowish-orange granular material, which appears to be cooked lentils […] to the sink area […]	[…] I am preparing a meal, specifically baked chicken, and offering it to B , who indicates they are not hungry and can wait.

Table 2: Examples of memory corrections learned through MemOpt . Red highlights errors and yellow highlights corrected content.

Writer	Training	Qwen3.5-9B Reader		Qwen3.6-27B Reader
Writer	Training	SuperMemory-VQA	EgoLifeQA	SuperMemory-VQA	EgoLifeQA
Qwen3.5-9B	Zero-shot	56.50	52.80	66.93	59.60
Qwen3.5-9B	MemOpt	60.35	56.60	67.90	60.20
Qwen3.6-27B	Zero-shot	57.14	55.20	66.77	59.80
Qwen3.6-27B	MemOpt	60.67	56.60	68.06	60.60

Table 3: Answer accuracy (%) across writer and reader backbones.

Figure 4: Generalizability of MemOpt writer training across memory systems.

Writer Ablation
Multimodal Fusion	Entity Grounding	First-person Narration	SuperMemory-VQA		EgoLifeQA		EgoLifeQA Recall
Multimodal Fusion	Entity Grounding	First-person Narration	Zero-shot	MemOpt	Zero-shot	MemOpt	Zero-shot	MemOpt
✔	✗	✗	52.33	56.98	52.80	52.60	48.60	47.00
✔	✔	✗	54.09	59.87	52.00	54.40	45.60	47.60
✔	✔	✔	56.50	60.35	52.80	56.60	48.40	50.60
Reader Ablation

Table 4: Ablation studies on the writer and reader components in MemLife . Accuracy is reported for SuperMemory-VQA and EgoLifeQA. Recall on the long-term EgoLifeQA task is also reported.

Writer Supervision					SuperMemory-VQA		EgoLifeQA
Teacher	Accuracy	Retrievability	Informativeness	Faithfulness	Accuracy	Recall	Accuracy	Recall
✗	✗	✗	✗	✗	56.50	80.53	52.80	48.40
✔	✗	✗	✗	✗	57.78	83.59	53.00	49.00
✗	✔	✗	✗	✗	57.46	79.01	52.60	47.80
✗	✗	✔	✗	✗	53.45	82.63	54.40	53.20
✗	✗	✔	✔	✗	60.03	84.35	51.40	47.60

Table 5: Comparison of supervision signals used during memory writer training.

Training Design		SuperMemory-VQA		EgoLifeQA
Faithfulness	Aggregation	Accuracy	Recall	Accuracy	Recall
Sequence-level	Additive	58.75	83.78	55.00	46.40
Token-level	Additive	58.91	81.68	56.00	49.40
Token-level	Multiplicative	60.35	81.49	56.60	50.60

Table 6: Ablation of faithfulness granularity and reward aggregation.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Type	# Questions	# Options	Avg. Correct Options	Avg. Evidence Segments	Avg. Distinct Days
Frequency counting	45	8	2.33	2.31	2.31
Routine	31	2–4	1.00	2.97	2.97
Comparison	24	3	1.00	8.04	5.54
All	100	–	1.60	3.89	3.29

Appendix

Table 7: Composition of EgoLife-EQA by question type. Correct options, evidence segments, and distinct evidence days are averaged per question.

Dataset	# Questions	Avg. Video Hours	Avg. Time Span (days)	Avg. Evidence Segments	Avg. Evidence Duration (s)
SuperMemory-VQA	4771	3.97	12.50	1.34	57.1
EgoLifeQA	500	22.65	2.80	1.10	32.3
SuperMemory-LVQA	623	47.90	127.36	1.24	56.3
EgoLife-EQA	100	43.06	6.33	3.89	116.7

Appendix

Table 8: Dataset statistics. Video hours and time spans are averaged over the history available to each question. Evidence segments and their total duration are averaged per question.

Group	Parameter	Value
Sampling	Group size $G$ (rollouts per segment)	5
	Rollout temperature / top- $p$ / top- $k$	1.0 / 1.0 / disabled
	Maximum prompt length (tokens)	12,288
	Maximum response length (tokens)	512
Optimization	Learning rate	$1\times 10^{-6}$
	Weight decay	0.01

Appendix

Table 9: MemOpt training hyperparameters.

Method	SuperMemory-VQA		EgoLifeQA
Method	Accuracy	Recall	Accuracy	Recall
VST	56.50	65.08	29.40	7.60
TaskMem	48.96	51.15	48.20	35.00
M3-Agent	53.93	54.20	35.40	9.60
MemLife + MemOpt	60.35	81.49	56.60	50.60

Appendix

Table 10: Controlled comparison with trained baselines. All methods use Qwen3.5-9B and task-specific training data only from SuperMemory-VQA. EgoLifeQA is evaluated out of distribution, and bold marks the best result.

Method	SuperMemory-VQA				EgoLifeQA
	Accuracy		Recall		Accuracy		Recall
	$T=0.0$	$T=1.0$	$T=0.0$	$T=1.0$	$T=0.0$	$T=1.0$	$T=0.0$	$T=1.0$
EgoRAG	49.28	49.18 $\pm$ 0.95	66.79	66.79 $\pm$ 0.00	48.20	46.48 $\pm$ 0.95	29.60	28.80 $\pm$ 0.00
MemLife	56.50	55.31 $\pm$ 1.53	80.53	80.69 $\pm$ 0.32	52.80	50.16 $\pm$ 1.83	48.40	47.48 $\pm$ 1.36
MemLife -V	57.78	58.78 $\pm$ 0.41	84.35	84.58 $\pm$ 0.64	53.20	52.44 $\pm$ 1.72	50.20	49.72 $\pm$ 0.30
Method	SuperMemory-LVQA				EgoLife-EQA

Appendix

Table 11: Stability across reader decoding settings.

T=0.0

reports greedy decoding, while

T=1.0

reports mean

\pm

standard deviation over five runs.

Method	Memory Storage (MB/h)		Reading Speed (s/question)
Method	SuperMemory-VQA	EgoLifeQA	SuperMemory-VQA	EgoLifeQA
Video ReCap	3.86	3.66	20.4	54.2
EgoRAG	4.98	5.49	33.2	43.5
Video-RAG	1.03	3.44	60.9	76.9
VideoARM	–	–	47.5	110.4
EGAgent	21.50	22.20	200.3	230.7

Appendix

Table 12: Storage and reading efficiency of training-free video memory systems. Storage is normalized by source-video duration, and reading time is averaged per question. Dashes indicate methods without a persistent memory store.

Training-time proxy	SuperMemory-VQA	EgoLifeQA
None (zero-shot)	56.50	52.80
Original question	57.62	55.40
Agent actions	60.35	56.60

Appendix

Table 13: Effect of the training-time retrievability approximation on answer accuracy (%).

Figure 5: Dynamics of the three reward signals during MemOpt training.

Writer trained	Reader trained	SuperMemory-VQA		EgoLifeQA
Writer trained	Reader trained	Accuracy	Recall	Accuracy	Recall
✗	✗	56.50	80.53	52.80	48.40
✔	✗	60.35	81.49	56.60	50.60
✗	✔	61.00	89.50	51.40	59.00
✔	✔	63.08	87.98	54.20	56.80

Appendix

Table 14: Effects of writer and reader training. All values are percentages.

Predecessor Context	Conversational Memory	Intent Recall	In-Context Retrieval	Timeline Reconstruction	Object-Location Memory	Visual Recall	Overall
None	63.41	77.42	43.18	57.41	52.29	44.12	56.50
Text	62.60	86.02	37.50	48.15	45.87	45.10	54.25
Multimodal	69.92	82.80	37.50	47.22	46.79	43.14	54.90

Appendix

Table 15: Answer accuracy across SuperMemory-VQA question types under different forms of predecessor context for memory writing. Bold values indicate the best result in each column.

Panel of Figure 6 — Figure 6: Complete system instruction and user template for the MemLife memory writer.

Panel of Figure 7 — Figure 7: System prompt for the default MemLife reader.

Panel of Figure 8 — Figure 8: Additional prompt block for MemLife -V. The tool count is updated to three, and all other text follows Figure 7 .

Panel of Figure 9 — Figure 9: Prompt template for grounded key-fact extraction.

Panel of Figure 10 — Figure 10: Prompt template for informativeness evaluation.

Panel of Figure 11 — Figure 11: Prompt template for faithfulness evaluation.

Explore similar work

EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding
May 11, 2026Ziyang Wang, Yue Zhang, Shoubin Yu +6Multimodal Memory Egocentric Video
UNC Chapel Hill · NTU Singapore
Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoning
May 8, 2026Jiazheng Li, Chi-Hao Wu, Yunze Liu +3Video Agent Multimodal Memory
University of Connecticut · Northwestern University · University of Virginia
Visual Agentic Memory: Enabling Online Long Video Understanding via Online Indexing, Hierarchical Memory, and Agentic Retrieval
May 15, 2026Aiden Yiliu Li, Nels Numan, Anthony SteedVideo Understanding Agentic Memory
University College London

May 11, 2026, cs.CVJ/K move · Enter open · S save

EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding

Ziyang Wang, Yue Zhang, Shoubin Yu, Ce Zhang+5

UNC Chapel Hill · NTU Singapore

Multimodal Memory Egocentric Video Reasoning Benchmark

Open paper AskPDF arXiv

Next-generation visual assistants, such as smart glasses, embodied agents, and always-on life-logging systems, must reason over an entire day or more of continuous visual experience. In ultra-long video settings, relevant information is sparsely distributed across hours or days, making memory a fundamental challenge: models must accumulate information over time, recall prior states, track temporal order, and abstract recurring patterns. However, existing week-long video benchmarks are primarily designed for perception and recognition, such as moment localization or global summarization, rather than reasoning that requires integrating evidence across multiple days. To address this gap, we introduce EgoMemReason, a comprehensive benchmark that systematically evaluates week-long egocentric video understanding through memory-driven reasoning. EgoMemReason evaluates three complementary memory types: entity memory, tracking how object states evolve and change across days; event memory, recalling and ordering activities separated by hours or days; and behavior memory, abstracting recurring patterns from sparse, repeated observations over the whole week period. EgoMemReason comprises 500 questions across three memory types and six core challenges, with an average of 5.1 video segments of evidence per question and 25.9 hours of memory backtracking. We evaluate EgoMemReason on 17 methods across MLLMs and agentic frameworks, revealing that even the best model achieves only 39.6% overall accuracy. Further analysis shows that the three memory types fail for distinct reasons and that performance degrades as evidence spans longer temporal horizons, revealing that long-horizon memory remains far from solved. We believe EgoMemReason establishes a strong foundation for evaluating and advancing long-context, memory-aware multimodal systems.

Abstract

Explore similar work

EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding

Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoning

Visual Agentic Memory: Enabling Online Long Video Understanding via Online Indexing, Hierarchical Memory, and Agentic Retrieval