Evaluating Memory Capability in Continuous Lifelog Scenario

Jianjie Zheng; Zhichen Liu; Zhanyu Shen; Jingxiang Qu; Guanhua Chen; Yile Wang; Yang Xu; Yang Liu

cs.CLApr 13, 2026

Evaluating Memory Capability in Continuous Lifelog Scenario

Authors: Jianjie Zheng, Zhichen Liu, Zhanyu Shen, Jingxiang Qu, Guanhua Chen, Yile Wang, Yang Xu, Yang Liu,

Organizations: Southern University of Science and Technology · Shenzhen University · Shanghai Jiao Tong University · Tsinghua University

Abstract

Nowadays, wearable devices can continuously lifelog ambient conversations, creating substantial opportunities for memory systems. However, existing benchmarks primarily focus on online one-on-one chatting or human-AI interactions, thus neglecting the unique demands of real-world scenarios. Given the scarcity of public lifelogging audio datasets, we propose a hierarchical synthesis framework to curate LifeDialBench, a novel benchmark comprising two complementary subsets: EgoMem, built on real-world egocentric videos, and LifeMem, constructed using simulated virtual community. Crucially, to address the issue of temporal leakage in traditional offline settings, we propose an Online Evaluation protocol that strictly adheres to temporal causality, ensuring systems are evaluated in a realistic streaming fashion. Our experimental results reveal a counterintuitive finding: current sophisticated memory systems fail to outperform a simple RAG-based baseline. This highlights the detrimental impact of over-designed structures and lossy compression in current approaches, emphasizing the necessity of high-fidelity context preservation for lifelog scenarios.

Figures & tables

Figure 1: Comparison between (1) The microphone-always-on scenario, which continuously recording dialogue with others in daily life, and (2) Chatting with AI scenario, which on-demand logging to form the chat history.

Figure 2: We introduce LifeDialBench , which consists of two subsets: EgoMem (top left), constructed from real-world egocentric videos (EgoLife), and LifeMem (right), a more comprehensive dataset built upon a Human-in-the-loop Hierarchical Life-Simulation Framework. Additionally, we propose a novel online evaluation method that assesses model performance incrementally during data storage, in contrast to conventional evaluation conducted only at the end of storage.

Benchmark	Scenario	Time Span	#Sessions	Cont. Rec.	TS	Online
LoCoMo ( Maharana et al., 2024 )	Person-Person	Few months	1k	✗	✗	✗
MemoryBank ( Zhong et al., 2024 )	Person-AI	10 days	300	✗	✓	✗
LongMemEval ( Wu et al., 2024 )	Person-AI	N/A	50k	✗	✓	✗
MemBench ( Tan et al., 2025 )	Person-AI	N/A	65k	✗	✗	✗
EgoMem (Ours)	Multi-Person	7 days	1.7k	✓	✓	✓
LifeMem (Ours)	Multi-Person	1 year	3.8k	✓	✓	✓

Table 1: Comparison of memory benchmarks. The key properties are summarized including the type of scenario ( Scenario ), the temporal coverage ( Time Span ), number of sessions ( #Sessions ), whether continuous recording is supported ( Cont. Rec. ), whether the queries contain explicit timestamp for simulation ( TS ), whether support for online evaluation ( Online ).

LifeMem	Total	Filtered	Keep Rate	Model Acc.
Daily	1464	1430	97.68%	99.23%
Weekly	248	241	97.18%	98.76%
Monthly	48	46	95.83%	100%
All	1760	1717	97.56%	99.18%

Table 2: The number of generated questions, the number of questions remained after filtering and human annotation, and the model accuracy in the final answerable verification.

Figure 3: Distributional statistics of the LifeMem dataset. The plots summarize event types, social roles, locations, and monthly dialogue counts, showing that the dataset is balanced and closely aligned with real-world lifelog patterns.

		Open-Ended					Multiple-Choices
Models	Method	QT1	QT2	QT3	QT4	Overall	QT1	QT2	QT3	QT4	Overall
EgoMem
gpt-4o-mini	RAG	39.11	57.91	10.91	26.57	34.07	73.79	86.25	60.69	57.20	69.86
	A-Mem	36.29	53.33	8.73	30.18	32.48	70.56	86.66	52.83	55.40	66.77
	Mem0	15.72	17.91	3.49	16.21	13.41	56.85	53.75	44.54	37.83	48.56
	MemOS	26.61	33.75	6.55	11.71	20.02	69.35	72.91	66.37	38.73	62.30

Table 3: Main results of memory systems’ performance on LifeDialBench in the online evaluation setting. The method with the best overall performance are bold , the second are underlined .

Figure 4: Comparison of accuracy decay rates across memory systems. RAG exhibits the steepest decline, whereas abstraction-based methods (A-Mem, MemOS) maintain better stability over long-term timelines despite lower absolute accuracy.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Comparison on the performances of memory systems using different backbone LLMs.

Method	Task	Top- $k$
Method	Task	10	20	40
A-Mem	Open-Ended	29.06	35.10	34.89
A-Mem	MCQ	61.90	65.15	68.46
MemOS	Open-Ended	27.07	32.89	33.43
MemOS	MCQ	59.10	64.26	67.79

Appendix

Table 4: Performance sensitivity analysis across varying top-

k

retrieval configurations on LifeMem.

Time	Speaker	Utterance
[08:10:15]	Jane	All done eating. I’ll go clear the bowls first, and then shall we tidy up the cabinet in the living room?
[08:10:22]	Jeremy	Okay, I’ll help you clear up. No point letting them pile up.
[08:10:30]	Jane	Yeah, and give the tablecloth a good shake while you’re at it, there are some breadcrumbs.
[08:10:38]	Jeremy	Alright, you go change into some clothes you don’t mind getting dirty. I’ll be over as soon as I finish here.
[08:11:05]	Jane	Hey, the bottom drawer of the cabinet is stuck. Can you give it a pull?
[08:11:10]	Jeremy	Let me see… Push it in a bit, then give it a sharp tug – There, it’s open.

Appendix

Table 5: Jeremy and Jane at Home Organizing Old Items (2024-01-01)

Time	Speaker	Utterance
[16:00:12]	Jeremy	Mom, Jane, the Spring Festival Gala replay has started. The tea is freshly brewed, have it while it’s hot.
[16:00:18]	Mother	Oh, this tea aroma is so comforting. Hangzhou Longjing really is something else.
[16:00:25]	Jane	Mmm, so light and refreshing. One sip and I feel completely relaxed.
[16:03:40]	Mother	These hosts look the same as always, wearing red dresses every year, smiling like flowers.
[16:05:10]	Jeremy	Mom, don’t just look at what they’re wearing. There’s a cross-talk performance later, you love those.
[16:08:33]	Jane	I recognize this skit actor. He was hilarious last time playing that delivery guy.

Appendix

Table 6: Jeremy with Family Watching Spring Festival Gala (2024-02-10)

Time	Speaker	Utterance
[10:15:00]	Jeremy	Is everyone here? Let’s get started. As you all saw, last night’s incident had a significant impact. We need to quickly piece together the timeline and identify the root causes.
[10:15:45]	Mike	Yes, we came straight from the morning stand-up. Wei and the Ops reps are here too.
[10:15:55]	Jeremy	Good. Let me briefly recap the timeline. Last night at 21:47, our monitoring platform started receiving a flood of 503 errors, concentrated on the user login and permission verification APIs. Frontend service response times spiked from an average of 80 milliseconds to over two seconds, lasting roughly twenty minutes.
[10:17:10]	Alex	On the backend side, we didn’t receive alerts until 21:49, two minutes after the problem started. Furthermore, the initial alerts were scattered; no one realized it was a systemic issue initially.
[10:17:45]	Wei	The test environment monitoring didn’t trigger because we hadn’t simulated failure states for that authentication component. It appears a vulnerability in the third-party SDK was triggered by a scanning tool, causing it to crash outright, which then cascaded to our authorization service.
[10:18:35]	Other	Correct. Checking the logs confirms it’s the CVE-2024-3187 mentioned in their urgent patch bulletin – a high-severity privilege escalation vulnerability. When their service restarted, our persistent connections were all severed, and we lacked reconnection safeguards.

Appendix

Table 7: Jeremy in Emergency Project Post-Mortem Meeting (2024-06-03)

		Open-Ended					Multiple-Choices
Models	Method	QT1	QT2	QT3	QT4	Overall	QT1	QT2	QT3	QT4	Overall
EgoMem
gpt-4o-mini	RAG	37.09	51.25	9.60	23.87	30.88	68.14	83.75	56.33	49.09	64.74
	A-Mem	33.06	51.25	9.17	27.47	30.56	68.54	81.25	55.02	54.05	64.74
	Mem0	12.09	16.25	2.62	14.41	11.39	54.03	51.35	46.28	32.88	46.11
	MemOS	26.20	32.50	9.60	9.90	19.91	66.93	72.08	68.55	32.88	60.59

Appendix

Table 8: Results on Offline Evaluation Setting

Figure 6: The Distribution of QA types.

Figure 7: Definition of the structure of a memory system, and a comparison table of current memory system approaches under this structure.

LLM Memory

Large language model memory studies how models store, retrieve, update, and forget information across prompts, tasks, and interactions.

See all 95 papers

LifeBench: A Benchmark for Long-Horizon Multi-Source Memory
Mar 4, 2026Zihao Cheng, Weixin Wang, Yu Zhao +15Agent Memory Memory Agent Benchmarks
State Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Huawei Technologies Co., Ltd.
VitaBench: Memory Lifecycles Beyond Retrieval
Oct 10, 2026Muquan Yu, Dihan HuMemory Agent Benchmarks Agent Memory Management
University of Hong Kong · Chinese University of Hong Kong · Cornell University
DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings
Jun 22, 2026Wenya Xie, Shengming Zhou, Zelin Li +9Agent Memory LLM Agent Memory
University of Minnesota · University of North Carolina at Charlotte · Johns Hopkins University +2

Mar 4, 2026J/K move · Enter open · S save

LifeBench: A Benchmark for Long-Horizon Multi-Source Memory

Zihao Cheng, Weixin Wang, Yu Zhao, Ziyang Ren+14

State Key Laboratory for Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Huawei Technologies Co., Ltd.

Agent Memory Memory Agent Benchmarks Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

Long-term memory is fundamental for personalized agents capable of accumulating knowledge, reasoning over user experiences, and adapting across time. However, existing memory benchmarks primarily target declarative memory, specifically semantic and episodic types, where all information is explicitly presented in dialogues. In contrast, real-world actions are also governed by non-declarative memory, including habitual and procedural types, and need to be inferred from diverse digital traces. To bridge this gap, we introduce Lifebench, which features densely connected, long-horizon event simulation. It pushes AI agents beyond simple recall, requiring the integration of declarative and non-declarative memory reasoning across diverse and temporally extended contexts. Building such a benchmark presents two key challenges: ensuring data quality and scalability. We maintain data quality by employing real-world priors, including anonymized social surveys, map APIs, and holiday-integrated calendars, thus enforcing fidelity, diversity and behavioral rationality within the dataset. Towards scalability, we draw inspiration from cognitive science and structure events according to their partonomic hierarchy; enabling efficient parallel generation while maintaining global coherence. Performance results show that top-tier, state-of-the-art memory systems reach just 55.2% accuracy, highlighting the inherent difficulty of long-horizon retrieval and multi-source integration within our proposed benchmark. The dataset and data synthesis code are available at https://github.com/1754955896/LifeBench.

Abstract

LLM Memory

Explore similar work

LifeBench: A Benchmark for Long-Horizon Multi-Source Memory

VitaBench: Memory Lifecycles Beyond Retrieval

DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings