ProMemAgent: Long-Horizon Proactive Assistance with Context-Aware Prospective Memory from Sensory Perceptions

Lilin Xu; Linyang He; Kaiyuan Hou; Yuang Fan; Minghui Zhao; Xuhai Xu; Xiaofan Jiang

cs.AIOct 19, 2026

ProMemAgent: Long-Horizon Proactive Assistance with Context-Aware Prospective Memory from Sensory Perceptions

Authors: Lilin Xu, Linyang He, Kaiyuan Hou, Yuang Fan, Minghui Zhao, Xuhai Xu, Xiaofan Jiang

Organizations: Columbia University

Abstract

Large language model (LLM) agents are moving beyond reactive responses toward proactive assistance that anticipates user needs without explicit requests. However, existing proactive agents primarily address immediate needs, while many everyday situations involve delayed intentions that can only be acted on hours or days later. Humans handle such intentions through prospective memory, the ability to remember to act on an intention when the right moment arises. In this paper, we introduce ProMemAgent, the first proactive agent to incorporate prospective memory over everyday sensory contexts for long-horizon proactive assistance. ProMemAgent organizes past sensory perceptions (e.g., egocentric video and audio) into context-aware memory, generating prospective cues and linking earlier events with subsequent developments for future retrieval. Given the current context, it retrieves relevant past events and performs memory-augmented proactive reasoning to determine whether assistance is needed. We further introduce CPMBench, the first benchmark for evaluating long-horizon proactive assistance over sensory contexts, comprising 1,000 instances across eight everyday categories. Extensive experiments show that ProMemAgent achieves the best memory retrieval performance and consistently outperforms all baselines on long-horizon proactive assistance.

Figures & tables

Figure 1: Comparison of prior work (top) and ours (bottom). Reactive agents require explicit requests, and prior proactive agents address immediate needs. ProMemAgent incorporates context-aware prospective memory to connect current and historical contexts and assist with implicit delayed intentions at the right moment.

Figure 2: Distribution of timeline instances.

Figure 3: Overview of ProMemAgent. It extracts sensory contexts and constructs context-aware memory through prospective cue generation and cross-event linking. During inference, it retrieves past events via cue-triggered retrieval and performs memory-augmented reasoning to provide long-horizon proactive assistance when needed.

	Method	Memory Retrieval		Proactive Decision		End-to-End Assistance
		DIR $\uparrow$	SCR $\uparrow$	FD $\downarrow$	Acc-P $\uparrow$	Acc-PA $\uparrow$	Set-P $\uparrow$	Set-R $\uparrow$	Set-F1 $\uparrow$
Proactive Agents	Proactive Agent (8B)	-	-	7.0	77.8	74.7	63.8	30.1	40.9
	ContextAgent (8B)	-	-	43.3	59.8	46.8	17.7	22.8	19.9
Full-Context LLMs (Open- and Closed-Source )	Qwen2.5-32B	-	-	15.7	73.2	68.3	43.4	29.3	35.0
	Qwen3-4B	-	-	19.0	69.8	63.8	32.1	22.0	26.1
	Qwen3-8B	-	-	27.0	69.8	63.4	37.7	39.8	38.7

Table 1: Main results on CPMBench. All metrics are in percentages (%). The best and second-best results are in bold and underlined , respectively. “-” denotes inapplicable retrieval metrics.

Method	DIR $\uparrow$	SCR $\uparrow$	Set-P $\uparrow$	Set-R $\uparrow$	Set-F1 $\uparrow$
Proactive Agent (8B)	-	-	63.6	21.5	32.2
ContextAgent (8B)	-	-	11.0	12.3	11.6
Qwen3-32B	-	-	35.2	49.2	41.0
GPT-5.4	-	-	50.0	61.5	55.2
text-embedding-3-small	20.0	10.7	55.6	15.4	24.1
T-Mem (Horizon-based)	26.2	14.3	44.8	20.0	27.7

Table 2: Main results on CPMBench-Omni.

Figure 4: Qualitative results of ProMemAgent in proactive and non-proactive cases.

Method	CPMBench					Direct QA
	DIR $\uparrow$	SCR $\uparrow$	FD $\downarrow$	Set-R $\uparrow$	Set-F1 $\uparrow$	Recall@10 $\uparrow$
ProMemAgent	81.3	72.0	4.3	66.7	75.2	98.1
w/o $\mathcal{L}_{f}$	81.3	38.0	14.7	66.7	65.9	98.1
w/o $\mathcal{K}_{f}$	47.2	42.0	2.7	32.5	46.8	99.5
w/o $\mathcal{W}_{f}$	66.7	61.0	3.0	54.5	67.3	98.1
w/o $\mathcal{M}^{CT}$	26.0	15.0	4.0	21.1	32.3	99.5

Table 3: Memory components ablation on CPMBench.

Panel of Figure 5 — Figure 5: Impact of reasoner variants on CPMBench. $K$ denotes training context sizes and “Qw” denotes Qwen.

Figure 6: Sensitivity to the content and cue-triggered retrieval budgets (

k^{C}

k^{CT}

) on CPMBench.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: An example scenario from CPMBench and its four variants, showing representative events from the Positive timeline and its annotation, followed by the variant-specific events of the other three variants.

Panel of Figure 8 — Figure 8: Summarized prompt for sensory context extraction in ProMemAgent.

Panel of Figure 9 — Figure 9: Summarized prompt for prospective cue generation in ProMemAgent’s memory construction.

Panel of Figure 10 — Figure 10: Summarized prompt for cross-event linking in ProMemAgent’s memory construction.

Panel of Figure 11 — Figure 11: Summarized prompt for proactive reasoning in ProMemAgent.

Panel of Figure 12 — Figure 12: Summarized prompt for data generation.

Panel of Figure 13 — Figure 13: Summarized judge prompt in the evaluation.

	Annotator 1	Annotator 2	GPT-5-mini	GPT-5.6 Luna
Annotator 1	-	93.9	95.7	89.6
Annotator 2	93.9	-	94.8	95.7
GPT-5-mini	95.7	94.8	-	92.2
GPT-5.6 Luna	89.6	95.7	92.2	-

Appendix

Table 4: Pairwise agreement (%) between two human annotators and two LLM judges.

	Method	Memory Retrieval		Proactive Decision		End-to-End Assistance
		DIR $\uparrow$	SCR $\uparrow$	FD $\downarrow$	Acc-P $\uparrow$	Acc-PA $\uparrow$	Set-P $\uparrow$	Set-R $\uparrow$	Set-F1 $\uparrow$
Proactive Agents	Proactive Agent (8B)	-	-	4.8	77.7	74.7	63.6	21.5	32.2
	ContextAgent (8B)	-	-	38.7	59.8	47.6	11.0	12.3	11.6
Full-Context LLMs (Open- and Closed-Source )	Qwen2.5-32B	-	-	3.0	75.4	72.1	50.0	7.7	13.3
	Qwen3-4B	-	-	19.6	67.9	62.2	23.3	15.4	18.5
	Qwen3-8B	-	-	28.6	62.9	57.9	23.8	23.1	23.4

Appendix

Table 5: Main results on CPMBench-Omni. All metrics are in percentages (%). The best and second-best results are in bold and underlined , respectively. “-” denotes inapplicable retrieval metrics.

	Method	Memory Retrieval		Proactive Decision		End-to-End Assistance
		DIR $\uparrow$	SCR $\uparrow$	FD $\downarrow$	Acc-P $\uparrow$	Acc-PA $\uparrow$	Set-P $\uparrow$	Set-R $\uparrow$	Set-F1 $\uparrow$
Proactive Agents	Proactive Agent (8B)	-	-	6.9	77.9	74.4	65.6	30.5	41.7
	ContextAgent (8B)	-	-	37.9	60.5	48.3	15.3	16.0	15.7
Full-Context LLMs (Open- and Closed-Source )	Qwen2.5-32B	-	-	12.4	73.0	67.5	41.5	20.6	27.6
	Qwen3-4B	-	-	13.4	70.8	64.8	30.5	13.7	18.9
	Qwen3-8B	-	-	24.5	70.1	64.1	39.5	37.4	38.4

Appendix

Table 6: Out-of-domain results on CPMBench, with three categories excluded from reasoner fine-tuning.

Proactive Agents

See all 35 papers

MemoProAgent: Toward Memory-Driven Proactive Assistance
Oct 15, 2026Shenglai Zeng, Xiao Yang, Tiffany Ma +15Proactive Agents Proactive Assistance
Michigan State University · Meta Reality Labs · Meta Superintelligence Labs +3
PM-Bench: Evaluating Prospective Memory in LLM Agents
Jul 14, 2026Genglin Liu, Saadia GabrielLLM Agent Memory LLM Agent Evaluation
University of California, Los Angeles
PASK: Toward Intent-Aware Proactive Agents with Long-Term Memory
Apr 9, 2026Zhifei Xie, Zongzheng Hu, Fangda Ye +10Proactive Agents Agentic AI
NTU · Pask-Core · NUS

Oct 15, 2026J/K move · Enter open · S save

MemoProAgent: Toward Memory-Driven Proactive Assistance

Shenglai Zeng, Xiao Yang, Tiffany Ma, Xinyuan Zhang+14

Michigan State University · Meta Reality Labs · Meta Superintelligence Labs+3

Proactive Agents Proactive Assistance Agent Memory Personal AI Agents Memory Agent Benchmarks

Open paper AskPDF arXiv

As assistants move from reactive answering toward proactive intervention, memory changes role: it no longer only supplies the answer to a query, it determines whether and when to speak at all. Prior work in this setting hands the system the relevant fact in context; we study the regime that makes the problem hard, where the evidence justifying an intervention must be mined from a long-horizon personal memory. We introduce MemoProBench, a benchmark for memory-driven proactive assistance (MPA), pairing egocentric video with year-long, distractor-rich user memories (~618K tokens on average), human-verified trigger windows, and a protocol that scores content and timing jointly. Every baseline we evaluate performs poorly (at most 15%), including frontier models given summarized or retrieval-augmented memory. A controlled study explains why: with oracle evidence delivered inside the trigger window GPT-5.3-Chat reaches 96%; letting it choose the timing drops it to 38%, and replacing oracle evidence with retrieval drops it to 15%. Content and timing are separately necessary, and missing either is catastrophic. Guided by this, we build MemoProAgent with a plug-and-play harness that surfaces memory from a graph-organized store and gates it down to what is worth interrupting for, plus a lightweight RL stage that curbs early and repeated firing. Without any training, MemoProAgent lifts strict accuracy from below 10% to 37.5% on Qwen3.8-27B and 38% on GPT-5.3-Chat; RL post-training reaches 42.4%.

Abstract

Proactive Agents

Explore similar work

MemoProAgent: Toward Memory-Driven Proactive Assistance

PM-Bench: Evaluating Prospective Memory in LLM Agents

PASK: Toward Intent-Aware Proactive Agents with Long-Term Memory