Organizations: Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan · Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Institute of Information Science, Academia Sinica, Taipei, Taiwan · Department of Computer Science and Information Engineering, AI Research Center (AINTU), National Taiwan University, Taipei, Taiwan
Since individuals may struggle to recall all life details and often confuse events, establishing a system to assist users in recalling forgotten experiences is essential. While numerous studies have proposed memory recall systems, these primarily rely on deep learning techniques that require extensive training and often face data scarcity due to the limited availability of personal lifelogs. As lifelogs grow over time, systems must also adapt quickly to newly accumulated data. Recently, large language models (LLMs) have demonstrated remarkable capabilities across various tasks, making them promising for personalized applications. In this work, we present a framework that leverages LLMs for proactive information access, integrating personal knowledge graphs to enhance the detection of access needs through a refined decision-making process. Our framework offers high flexibility, enabling the replacement of base models and the modification of fact retrieval methods for continuous improvement. Experimental results demonstrate that our approach effectively identifies forgotten events, supporting users in recalling past experiences more efficiently.
Figures & tables
Fig. 1: Overview of our graph-empowered refinement (GER) framework for personal information access support.
Model
CST
INC
ADD
FGT
UFG
GPT-3.5
0.6834
0.0000
0.7772
0.7733
0.6961
Llama3 70B
0.7063
0.0000
0.7846
0.7934
0.7234
SEEN (Base)
0.7792
0.0833
0.7905
0.8113
0.7500
SEEN (Large)
0.7910
0.1667
0.8207
0.8371
0.7442
GER (ours)
0.7650
0.0417
0.8338
0.8635
0.7248
TABLE I: Recall score of each event type.
Model
CST
INC
ADD
FGT
UFG
GPT-3.5
0.6834
0.0000
0.7772
0.7733
0.6961
Llama3 70B
0.7063
0.0000
0.7846
0.7934
0.7234
SEEN (Base)
0.7395
0.0952
0.8173
0.8136
0.7470
SEEN (Large)
0.7625
0.2500
0.8374
0.8271
0.7568
GER
0.7543
0.0741
0.8370
0.8364
0.7584
TABLE II: F-score of each event type.
Base Model
Support Model
Correction Model
Consistent
Inconsistent
Additional
Forgotten
Unforgotten
SEEN (Base)
N/A
N/A
0.7395
0.0952
0.8173
0.8136
0.7470
SEEN (Base)
Llama3 70B
GPT-3.5
0.7489
0.0572
0.8261
0.8173
0.7470
SEEN (Base)
GPT-3.5
GPT-3.5
0.7401
0.0000
0.8258
0.8212
0.7481
SEEN (Base)
Llama3 70B
GPT-4o
0.7516
0.0000
0.8321
0.8243
0.7516
SEEN (Base)
GPT-3.5
GPT-4o
0.7465
0.0000
0.8323
0.8376
0.7514
TABLE III: Results with SEEN (Base) as the base module.
Model
CST
INC
ADD
FGT
UFG
Baseline
0.7625
0.2500
0.8374
0.8271
0.7568
Llama3 70B
0.7576
0.0715
0.8364
0.8237
0.7486
GPT-3.5
0.7457
0.1480
0.8287
0.8284
0.7495
Ground Truth
0.8546
0.2759
0.9028
0.9014
0.8684
TABLE IV: Results of each event type with different support modules. The base module is SEEN (Large), and the correction module is GPT-3.5.
Model
CST
INC
ADD
FGT
UFG
GPT-3.5
0.7163
0.0000
0.7971
0.7914
0.7137
Llama3 70B
0.7204
0.0000
0.7988
0.8004
0.7240
SEEN (Base)
0.7401
0.0000
0.8258
0.8212
0.7481
SEEN (Large)
0.7457
0.1480
0.8287
0.8284
0.7495
TABLE V: The importance of base module. Both the support module and correction module use GPT-3.5. The model’s name indicates the base module used.
SEEN (Base)
CST
INC
ADD
FGT
UFG
GPT-3.5
0.7401
0.0000
0.8258
0.8212
0.7481
GPT-4o
0.7465
0.0000
0.8323
0.8376
0.7514
SEEN (Large)
CST
INC
ADD
FGT
UFG
GPT-3.5
0.7457
0.1480
0.8287
0.8284
0.7495
GPT-4o
0.7543
0.0741
0.8370
0.8364
0.7584
TABLE VI: The importance of the correction module. The first row represents the model of the base module. All the support module uses GPT-3.5.
Fail
Success
Alternative Insight
Count
Consistent
16.17%
58.36%
25.47%
1,268
Inconsistent
45.83%
20.83%
33.34%
24
Additional
7.50%
74.87%
17.63%
1,986
Forgotten
6.59%
77.81%
15.60%
1,897
Unforgotten
19.24%
52.16%
28.60%
1,388
TABLE VII: Prediction analysis of support module with the setting of SEEN (Large), GPT-3.5, GPT-4o.
Fail
Success
Count
Consistent
28.48%
71.52%
323
Inconsistent
25.00%
75.00%
8
Additional
51.71%
48.29%
350
Forgotten
45.27%
54.73%
296
Unforgotten
28.97%
71.03%
397
TABLE VIII: Prediction analysis of correction module with the setting of SEEN (Large), GPT-3.5, GPT-4o.
School of Computer Science, Peking University · Kuaishou Technology · Center for Data Science, Academy for Advanced Interdisciplinary Studies, Peking University