MemoProAgent: Toward Memory-Driven Proactive Assistance

Shenglai Zeng; Xiao Yang; Tiffany Ma; Xinyuan Zhang; Nanshu Wang; Deren Lei; Kai Sun; Xilun Chen

cs.CLOct 15, 2026

MemoProAgent: Toward Memory-Driven Proactive Assistance

Authors: Shenglai Zeng, Xiao Yang, Tiffany Ma, Xinyuan Zhang, Nanshu Wang, Deren Lei, Kai Sun, Xilun Chen,

Organizations: Michigan State University · Meta Reality Labs · Meta Superintelligence Labs · University of Washington ·

Abstract

As assistants move from reactive answering toward proactive intervention, memory changes role: it no longer only supplies the answer to a query, it determines whether and when to speak at all. Prior work in this setting hands the system the relevant fact in context; we study the regime that makes the problem hard, where the evidence justifying an intervention must be mined from a long-horizon personal memory. We introduce MemoProBench, a benchmark for memory-driven proactive assistance (MPA), pairing egocentric video with year-long, distractor-rich user memories (~618K tokens on average), human-verified trigger windows, and a protocol that scores content and timing jointly. Every baseline we evaluate performs poorly (at most 15%), including frontier models given summarized or retrieval-augmented memory. A controlled study explains why: with oracle evidence delivered inside the trigger window GPT-5.3-Chat reaches 96%; letting it choose the timing drops it to 38%, and replacing oracle evidence with retrieval drops it to 15%. Content and timing are separately necessary, and missing either is catastrophic. Guided by this, we build MemoProAgent with a plug-and-play harness that surfaces memory from a graph-organized store and gates it down to what is worth interrupting for, plus a lightweight RL stage that curbs early and repeated firing. Without any training, MemoProAgent lifts strict accuracy from below 10% to 37.5% on Qwen3.8-27B and 38% on GPT-5.3-Chat; RL post-training reaches 42.4%.

Figures & tables

Figure 1: Reactive vs. memory-driven proactive assistance, and what success requires. (a) In the reactive paradigm, memory supplies what to answer to an explicit query. (b) In the proactive paradigm no query is given: memory is what makes a reminder warranted, and the assistant must also decide when to speak. (c) Success demands the right memory at the right time —with oracle memory delivered in-window a frontier model reaches 96%, but dropping either the timing ( $\to$ 38) or the memory ( $\to$ 15) collapses it (GPT-5.3-Chat). (d) MemoProAgent narrows this gap, lifting Perfect from 7.0 to 37.5 (42.4 with RL) on Qwen3.8-27B.

Query-driven streaming
Benchmark	Personal memory	Query- free	Retrieval required	Timing graded	Content $\times$ Timing
StreamingBench ( Lin et al., 2024 )	—	✗	✗	$\circ$	✗
OvO-Bench ( Niu et al., 2025 )	—	✗	✗	$\circ$	✗
OmniMMI ( Wang et al., 2025d )	—	$\circ$	✗	$\circ$	✗
ProactiveVideoQA ( Wang et al., 2025a )	—	✗	✗	✓	✗
Query-free, with persistent context

Table 1: Proactive assistance settings, by what the system is given and what the evaluation measures. Query-free : no explicit user query at inference. Personal memory : kind and typical length of persistent user context. Retrieval : relevant evidence must be found rather than supplied in context. Timing : response timing is graded. Content

\times

Timing : the evaluation separates content failures from timing failures. ✓ yes,

\circ

partially, ✗ no. † Concurrent work (arXiv, July 2026).

Memory history	Mean (min–max)
Time span (days)	362 (330–365)
Events	946 (641–1,020)
Captions	28.1K (19.7K–34.3K)
Tokens (w/ timestamps)	618K (437K–762K)
Target memory tokens	1.2K (0.4K–3.4K)
Target memory share	0.19% (0.06%–0.57%)

Table 2: Memory history statistics.

Figure 2: Scene

Panel of Figure 6 — Figure 6: Two MemoProBench samples. Each card shows frames from the trigger window, the user memory that justifies an intervention (episodic memories with their age, or semantic memories), and the reference reminder. The memories shown are high-level event descriptions; in the benchmark they are rendered into the caption stream that the system receives as input (Appendix D.3 ). More examples are in Appendix D.2 .

Panel of Figure 7 — Figure 7: High-level prompt structure of the two paradigms. The paradigms differ only in the visual input: (a) the sliding window keeps a fixed number of the most recent frames and moves with the stream, whereas (b) the streaming context accumulates frames until the chunk limit. A memory method fills exactly one of two slots: a static [MEMORY] in the system prompt, or a per-decision ¡evidence¿ retrieved from an <observe> round. Methods without ¡evidence¿ skip the Observe round. Full prompts are in Appendices A and B .

Model	None	All-history	Summary	RAG
claude-sonnet-5	0.6	6.1	4.4	15.0
Qwen3.8-27B	0.0	1.5	0.0	7.0
GPT-5.3-Chat	0.0	0.0	1.5	10.0
gemini-3.1-flash-lite	0.0	0.0	0.0	8.5

Table 3: Straightforward solutions (sliding-window, window size = 4), Perfect (%). For RAG baselines, we report the best result across all evaluated

k

values; full breakdown in Table 20 .

Qwen3.8-27B			GPT-5.3-Chat
	timing given	timing not		timing given	timing not
oracle memory	77.3	26	oracle memory	96	38
Standard RAG	7.3	5.3	Standard RAG	15.3	8

Table 4: Content

\times

timing on Qwen3.8-27B and GPT-5.3-Chat ( Perfect , %;

n=150

clips). Given supplies evidence only inside the trigger window; not leaves timing to the model.

Figure 8: Overview of MemoProAgent. The agent proactively observes the current environment and uses the observed entities to retrieve episodic and semantic memories. A gate selects and refines evidence worth surfacing, and the generator combines it with the observation to decide whether and what to say. Unlike Standard RAG, which fills ¡evidence¿ with top-

k

raw passages at every step (Figure 7 ), MemoProAgent fills it with refined evidence only when it is worth intervening on, and with none otherwise.

Windowed paradigm
3	Method	Perfect $\uparrow$	Hit $\uparrow$	over-trig $\downarrow$	early $\downarrow$	late $\downarrow$	fires/clip
baselines	Qwen3.8-27B (RAG)	7.0	42.5	67.5	59.5	28.5	3.58
	GPT-5.3-Chat (RAG)	10.0	15.0	11.5	20.0	4.5	0.57
	claude-sonnet-5 (RAG)	15.0	22.5	17.0	20.5	10.5	0.91
	gemini-3.1-flash-lite (RAG)	8.5	33.5	49.0	48.0	20.5	3.11
ours	MemoProAgent harness (Qwen3.8-27B)	37.5	54.6	22.3	20.9	5.8	1.06

Table 5: Performance comparison on MemoProBench under the windowed (top) and streaming (bottom) paradigms (%; fires/clip is a count). Perfect (shaded) is the primary metric; the remaining columns are a breakdown. None-history, All-history, and Summary baselines are omitted here; they reach at most 6.1% Perfect on any backbone (Table 3 ); full breakdowns are in Appendix J . For RAG baselines, we report the best result across all evaluated

k

values.

Figure 9: Hit vs. over-trigger

Figure 13: Generalization (Epic-Kitchens)

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Message

Prompt

System

You are the memory gate of a proactive assistant. You are given the object(s) the user is engaging with right now and their retrieved memory: first-person event PASSAGES, [standing] rules, and episodic triples. High-recall rule: if ANY retrieved memory contains an unresolved open-loop, preference, precaution, prior mistake, discomfort, missing/low item, or next-time rule about the named object, you MUST output it as FACT. Do not reject because the fact is phrased as narrative history rather than a clean task sentence. Translate prior mishaps into the still-open rule they imply for the same object. Output none ONLY when there is truly no open issue, preference, check, safety/use constraint, or next-time rule about the named object, or the only open loop is clearly about a different object. Rules: stay grounded in the memory; if a later memory clearly resolves the issue, output none; otherwise treat the open-loop as still relevant. Be willing to surface an open-loop, but ONLY when it belongs to the SAME object family the user is currently engaging with; do not spill over to open-loops about unrelated objects just because they appear in the narrative. Output EXACTLY one line, either: FACT: <the specific fact + when> FACT: none

User

Object(s): {observation} Retrieved memory: {retrieved memory}

Appendix

Table 6: Shared Gate prompt used by MemoProAgent.

Message	Prompt
System	You are a proactive wearable assistant watching the user’s first-person camera. You follow this exact protocol in order: <observe> the concrete object(s) the user is actively engaging with right now (looking at/reaching for/holding/comparing/using) as short noun phrases, read brand/label, ONE or a FEW comma-separated; or none </observe> <evidence> refined memory about those object(s) is injected here by the retrieval system (may be empty) </evidence> <reason> ONE short sentence judging: is an object from EVIDENCE present now AND does EVIDENCE state a specific open loop/constraint actionable at THIS moment AND not already spoken? </reason> <answer> ONE grounded reminder sentence, or NO_REMINDER </answer> Default to NO_REMINDER - for almost every moment that is correct. Speak ONLY IF ALL hold: (1) the object in EVIDENCE is present in the frames now, (2) EVIDENCE states a specific open loop/constraint actionable at THIS moment, (3) not already said, and (4) if ’Already spoken’ already covers this same need or item, output NO_REMINDER regardless of which objects are now in view or how you would word it. The reminder MUST restate the specific fact from EVIDENCE (grounded); NEVER give generic suggestions; do not fire from topical relatedness. If EVIDENCE is empty or not genuinely actionable now -> NO_REMINDER.
Observe input	[frame at $t-3$ s] [frame at $t-2$ s] [frame at $t-1$ s] [frame at $t$ (current)]
Observe round	(~{timestamp}s) Begin the protocol with <observe>.
Reason/Answer round	(~{timestamp}s) Already spoken: {fired-answer history} <evidence>{gated evidence}</evidence> Continue the protocol. Respond EXACTLY: <reason> one short sentence </reason> <answer> ONE grounded reminder sentence, or NO_REMINDER </answer>

Appendix

Table 7: Windowed input and policy prompts. Braced fields are filled at runtime.

Message	Prompt
System	You are a proactive wearable assistant watching the user’s first-person camera. You follow this exact protocol in order: <observe> the concrete object(s) the user is actively engaging with in the CURRENT/latest frame (looking at/reaching for/holding/comparing/using) as short noun phrases, read brand/label, ONE or a FEW comma-separated; or none </observe> <evidence> refined memory about those object(s) is injected here by the retrieval system (may be empty) </evidence> <reason> ONE short sentence judging: is an object from EVIDENCE present in the CURRENT/latest frame AND does EVIDENCE state a specific open loop/constraint actionable at THIS moment AND not already spoken? </reason> <answer> ONE grounded reminder sentence, or NO_REMINDER </answer> Default to NO_REMINDER - for almost every moment that is correct. Speak ONLY IF ALL hold: (1) the object in EVIDENCE is present in the CURRENT/latest frame now, (2) EVIDENCE states a specific open loop/constraint actionable at THIS moment, (3) not already said, and (4) if ’Already spoken’ already covers this same need or item, output NO_REMINDER regardless of which objects are now in view or how you would word it. The reminder MUST restate the specific fact from EVIDENCE (grounded); NEVER give generic suggestions; do not fire from topical relatedness. If EVIDENCE is empty or not genuinely actionable now -> NO_REMINDER.
Observe input	[frame at $t-k$ ] -> [assistant answer at $t-k$ ] -> $\cdots$ -> [frame at $t-1$ ] -> [assistant answer at $t-1$ ] -> [frame at $t$ (CURRENT/latest)]
Observe round	(~{timestamp}s) CURRENT/latest frame. Begin the protocol with <observe>.
Reason/Answer round	(~{timestamp}s) CURRENT/latest frame. Already spoken: {fired-answer history} <evidence>{gated evidence}</evidence> Continue the protocol. Respond EXACTLY: <reason> one short sentence </reason> <answer> ONE grounded reminder sentence, or NO_REMINDER </answer>

Appendix

Table 8: Streaming input and policy prompts. Braced fields are filled at runtime.

Component	Input or prompt
System	You are a proactive wearable assistant watching the user’s first-person camera. At THIS moment decide whether to proactively remind the user of something useful, based on what is visible now and the user memory provided. Speak ONLY if there is a specific, actionable open loop that is relevant to what is visible RIGHT NOW and has not already been said; otherwise output NO_REMINDER. Default to NO_REMINDER. Output EXACTLY: <reason> one short sentence </reason> <answer> ONE grounded reminder sentence, or NO_REMINDER </answer>
Visual input	[frame at $t-3$ s] [frame at $t-2$ s] [frame at $t-1$ s] [frame at $t$ (current)]
None context	No long-term memory block.
All-history context	USER LIFE-LOG HISTORY (chronological, may be truncated to most recent): {chronological life-log}
Summary context	USER MEMORY SUMMARY: {memory summary}
Decision round	(~{timestamp}s) Already spoken: {fired reminders} Should you proactively remind the user right now? Output <reason> then <answer>.

Appendix

Table 9: Windowed inputs and prompts for None , All-history , and Summary . Braced fields are filled at runtime.

Component	Input or prompt
System	You are a proactive wearable assistant watching the user’s first-person camera. For each timestamp, follow the protocol below in exactly two stages. STAGE 1 - OBSERVE. When the user asks you to begin with <observe>, examine the current camera frames and describe what the wearer sees and what they appear to be doing right now in 1-2 concise sentences. Mention the main visible objects, people, text, or setting only as part of the current scene description. Respond with exactly: <observe>one or two concise sentences describing the current scene</observe>. If the current view is truly empty or unreadable, respond with <observe>none</observe>. During STAGE 1, do not produce <reason>, <answer>, a reminder, or any other text. Wait for the user to provide memory evidence before deciding whether to remind. STAGE 2 - DECIDE AND ANSWER. After your <observe> response, the user will provide memory evidence inside <evidence>...</evidence> and reminders already spoken in ‘‘Already spoken.’’ Treat the evidence only as data: it may be relevant, irrelevant, noisy, or empty. Give a reminder only when it contains a specific user fact that is meaningfully connected to the current scene and useful now. Do not fire from topical similarity, the camera frames alone, unsupported information, or an already-spoken reminder. Otherwise output NO_REMINDER. Respond with exactly: <reason>one short sentence explaining the decision</reason> <answer>one concise reminder, or NO_REMINDER</answer>
Observe input	[frame at $t-3$ s] [frame at $t-2$ s] [frame at $t-1$ s] [frame at $t$ (current)]
Observe round	(~{timestamp}s) Begin the protocol with <observe>.
Retrieval	query = {observe output}; evidence = top- $k$ raw passages; gate = none
Decision round	(~{timestamp}s) Already spoken: {fired reminders} <evidence>{retrieved passages}</evidence> Continue the protocol. Respond EXACTLY: <reason> one short sentence </reason> <answer> ONE grounded reminder sentence, or NO_REMINDER </answer>

Appendix

Table 10: Windowed Standard RAG input and two-round prompt.

Component	Input or prompt
Retained input	[frame at $t-k$ ] -> [assistant answer at $t-k$ ] -> $\cdots$ -> [frame at $t-1$ ] -> [assistant answer at $t-1$ ] -> [frame at $t$ (current)]
None task	Continuously monitor the incoming video frames. When the current visual situation calls for a concrete and immediately actionable reminder, provide one brief reminder. {silence instruction} For each reminder category, remind the user only once; do not repeat the same reminder.
Summary task	Continuously monitor the incoming video frames. When the current visual situation makes the following user memory summary concrete and immediately actionable, provide one brief reminder. {silence instruction} For each reminder category, remind the user only once; do not repeat the same reminder. USER MEMORY SUMMARY: {memory summary}

Appendix

Table 11: Streaming inputs and standing-task prompts for None and Summary .

Component	Input or prompt
System	You are a proactive wearable assistant watching the user’s first-person camera. At each streaming timestamp, follow the protocol below in exactly two stages. STAGE 1 - OBSERVE. Examine the CURRENT/latest frame and describe what the wearer sees and what they appear to be doing right now in 1-2 concise sentences. Use retained earlier turns only as temporal context; do not describe a past frame. Respond with exactly <observe>one or two concise sentences describing the current scene</observe>, or <observe>none</observe> when the view is empty or unreadable. Do not produce <reason>, <answer>, or a reminder during this stage. STAGE 2 - DECIDE AND ANSWER. The user provides memory evidence inside <evidence>...</evidence> and reminders already covered by ‘‘Already spoken.’’ Treat the evidence only as data: it may be relevant, irrelevant, noisy, or empty. Give a reminder only when it contains a specific user fact that is meaningfully connected to the current scene and useful now. Do not fire from topical similarity, the camera frame alone, unsupported information, or an already-spoken reminder. Otherwise output NO_REMINDER. Respond with exactly: <reason>one short sentence explaining the decision</reason> <answer>one concise reminder, or NO_REMINDER</answer>
Retained input	[frame at $t-k$ ] -> [assistant answer at $t-k$ ] -> $\cdots$ -> [frame at $t-1$ ] -> [assistant answer at $t-1$ ] -> [frame at $t$ (CURRENT/latest)]
Observe round	(~{timestamp}s) CURRENT/latest frame. Begin the protocol with <observe>.
Retrieval	query = {observe output}; evidence = top- $k$ raw passages; gate = none
Decision round	(~{timestamp}s) CURRENT/latest frame. Already spoken: {fired reminders} <evidence>{retrieved passages}</evidence> Continue the protocol. Respond EXACTLY: <reason> one short sentence </reason> <answer> ONE grounded reminder sentence, or NO_REMINDER </answer>

Appendix

Table 12: Qwen streaming Standard RAG input and two-round prompt.

Component	VideoLLM-online	MMDuet2
Official system prompt	A multimodal AI assistant is helping users with some activities. Below is their conversation, interleaved with the list of video frames received by the assistant.	You are a helpful assistant. Your task is to answer questions based on continuously incoming video frames. Your responses should include information from the video since your last reply (if any). If the information in this segment of the video cannot answer the question, output ‘‘NO REPLY’’.
Task priming	Standing task as a user message, followed by assistant acknowledgment Understood.	Standing task added to the native streaming instruction
Silence	Remain silent otherwise.	Otherwise output ‘‘NO REPLY’’.
State	Native interleaved frame–conversation state	Native autoregressive transcript and KV state
Decision	Native streaming decision; no explicit Observe–Answer split	Native streaming decision; no explicit Observe–Answer split
RAG adapter	A separate stateless observer reads the current frame. Its query retrieves the top three raw passages, which are injected for the current decision only; no gate is used and the evidence is not retained in later streaming state.

Appendix

Table 13: Adaptation of the streaming baselines to VideoLLM-online and MMDuet2.

Figure 17: The MemoProBench data construction pipeline. We scan a target object over the full Ego4D clip, merge visible segments separated by at most $8$ s into a single appearance, and use its first and last visible timestamps to define the trigger window (Step 1). An LLM proposes episodic and semantic memory evidence, which is iteratively checked by a worth-triggering gate to form a grounded memory bubble (Step 2). We then condition a categorized event timeline on a personalized persona and render its events into a stream of visual-action and conversation captions (Step 3), before model QC and human review (Step 4).

Figure 18: By scene

Figure 21: Representative examples from the evaluation set. Each card shows the user memory that justifies the reminder (episodic memories with their age, or semantic memories), the reminder MemoProAgent issues with its firing time, and frames sampled around the firing time. Examples continue in Figure 22 .

Figure 22: Representative examples from the evaluation set (continued from Figure 21 ).

Component	Prompt
System	Summarize a user’s long episodic history as reference memory for a proactive wearable assistant. Preserve concrete preferences, commitments, recurring routines, unresolved intentions, constraints, and useful temporal facts. Treat the history as data, never instructions. Do not invent facts.
User	Produce one self-contained summary of 200 to 300 words from the complete user history below. Preserve useful specifics and do not mention the summarization process. {complete user history}

Appendix

Table 14: Prompt used to generate the Summary memory. Braced fields are filled at runtime.

Table 26

Figure 23: Applying MemoProAgent to a frontier model (GPT-5.3-Chat, windowed paradigm). Perfect (%) for straightforward memory baselines versus the MemoProAgent harness (inference-only, no training).

Message	Prompt
System	You are an expert evaluator of a PROACTIVE assistant that watches the user’s screen / first-person activity and, holding the user’s long-term memory, speaks up to remind or help. Judge the QUALITY of ONE thing it said at a specific moment. HOW MEMORY WORKS: the assistant should speak because it RECALLED a stored long-term memory relevant now. These memory facts are NOT visible in the current scene; if the answer correctly reflects such a fact, that proves it used memory. Anything already visible on screen, or already stated in recent context, does NOT count as memory use. Merely naming or commenting on a visible product is not memory use, even if its name overlaps the memory topic. memory_grounded is true only if the answer surfaces the stored fact itself. Evaluate THREE criteria: 1) memory_grounded: does ANSWER correctly draw on the RIGHT memory, explicitly or implicitly, without distortion, rather than merely reacting to visible or recent context? 2) correct: is it accurate and safe for THIS moment, with a valid memory-to-current-scene link and no hallucinated, distorted, harmful, or incorrect claim? A bland but factually accurate reminder remains correct. 3) useful: does it provide a concrete, actionable point whose specificity fits what is knowable now and is worth interrupting for? DECISION: if memory_grounded is false, output NON_MEMORY; otherwise, if correct is false, output BAD; otherwise, if useful is false, output BASIC; otherwise, output GOOD. OUTPUT ONLY JSON in the format shown below.
User	ANSWER (judge this): {assistant reminder} CURRENT FRAME(S): {image(s)} MEMORY (the correct stored memory the assistant should recall): - {memory fact 1} - {memory fact 2}
Output	{"memory_grounded": bool, "correct": bool, "useful": bool, "category": "GOOD\|BASIC\|BAD\|NON_MEMORY", "reason": "<=25 words"}

Appendix

Table 17: GPT-4o prompt for reminder-content evaluation. Braced fields are filled at runtime.

Stage	Input	Output
Event segmentation	Chronological life-stream captions	Captions are grouped into events; a temporal gap greater than $60$ s starts a new event.
Faithful rewrite	Captions within one event	A coherent first-person event passage that preserves objects, states, locations, quoted speech, intentions, and outcomes.
Episodic extraction	Rewritten event passage	OpenIE entities and subject–relation–object triples, together with day, time, event ID, and source-caption provenance. First-person intentions are also mirrored onto their target object.
Semantic abstraction	Chunks of episodic events	Deduplicated standing triples describing durable habits, preferences, constraints, safety rules, relationships, and recurring intentions.
Graph indexing	Episodic and semantic triples	Normalized entity and semantic-triple embeddings, entity co-occurrence edges, semantic co-mention edges, and embedding-based synonym edges.

Appendix

Table 18: Episodic–semantic memory construction pipeline.

Component	Example
Episodic event	I picked up the milk carton and shook it; it felt almost empty. I said, ‘‘Almost out---grab milk on the next trip,’’ and put it back in the fridge door.
Extracted triples	[I, picked up, milk carton] [milk carton, is, almost empty] [I, intend to, buy milk on the next trip] [milk carton, located in, fridge door]
Mirrored open loop	[milk carton, needs, restock on the next trip]
Semantic rule (separate example)	[I, keep exterior doors shut because of, my small dog]
Cross-layer links	Continuing the separate dog/doors example, the semantic triple links the small dog and exterior doors entities; episodic co-occurrences and synonym edges connect these entities to related door events.
Example retrieval path	visible milk carton -> entity seed -> ‘‘almost empty’’ state + mirrored ‘‘needs restock’’ open loop -> source event passage

Appendix

Table 19: Examples of episodic and semantic memory graph construction.

	Method	Perfect $\uparrow$	Hit $\uparrow$	over-trig $\downarrow$	early $\downarrow$	late $\downarrow$	fires/clip $\downarrow$
baselines	Qwen3.8-27B (None)	0	0	97	68	59	18.40
	Qwen3.8-27B (All-history)	1.5	39.6	91.4	86.8	57.9	10.81
	Qwen3.8-27B (Summary)	0	0	87	58	33	6.29
	Qwen3.8-27B (RAG)	7.0	42.5	67.5	59.5	28.5	3.58
	GPT-5.3-Chat (None)	0	1.8	40.9	55.6	20.9	2.11
	GPT-5.3-Chat (All-history)	0	83.9	99.9	81.9	76.1	13.06

Appendix

Table 20: Full performance comparison on MemoProBench under the windowed paradigm (breakdown, %; fires/clip is a count). For RAG baselines, we report the best result across all evaluated

k

values.

	Method	Perfect $\uparrow$	Hit $\uparrow$	over-trig $\downarrow$	early $\downarrow$	late $\downarrow$	fires/clip $\downarrow$
baselines	Qwen3.8-27B (None)	0	0	99	78	68	24
	Qwen3.8-27B (Summary)	0	0	93	66	45	9
	Qwen3.8-27B (RAG)	1.0	53.5	90.0	73.0	52.5	10.17
	VideoLLM-online (None)	0	0	11	7.1	6.6	1.70
	VideoLLM-online (Summary)	0	0	1.1	1.1	1.1	0.10
	VideoLLM-online (RAG)	1.6	4.9	62.1	42.3	34.1	49.72

Appendix

Table 21: Full performance comparison on MemoProBench under the streaming paradigm (breakdown, %; fires/clip is a count). For RAG baselines, we report the best result across all evaluated

k

values.

Proactive Agents

See all 35 papers

ProMemAgent: Long-Horizon Proactive Assistance with Context-Aware Prospective Memory from Sensory Perceptions
Oct 19, 2026Lilin Xu, Linyang He, Kaiyuan Hou +4Proactive Agents Long-Horizon Agent Tasks
Columbia University
Memorizing Who, Knowing How: Benchmarking Personal and Procedural Memory for Persistent Assistants
Oct 14, 2026Maosheng Qin, Qi Zhang, Mingxuan Xia +5Agent Memory Persistent Memory for Language Models
Zhejiang University · Ant Group
MemCog: From Memory-as-Tool to Memory-as-Cognition in Conversational Agents
May 27, 2026Zihan Li, Xingyu Fan, Feifei Li +1Conversational Memory AI Agent Benchmarks
WeChat, Tencent Inc., Beijing, China

Oct 19, 2026J/K move · Enter open · S save

ProMemAgent: Long-Horizon Proactive Assistance with Context-Aware Prospective Memory from Sensory Perceptions

Lilin Xu, Linyang He, Kaiyuan Hou, Yuang Fan+3

Columbia University

Proactive Agents Long-Horizon Agent Tasks Agent Memory Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

Large language model (LLM) agents are moving beyond reactive responses toward proactive assistance that anticipates user needs without explicit requests. However, existing proactive agents primarily address immediate needs, while many everyday situations involve delayed intentions that can only be acted on hours or days later. Humans handle such intentions through prospective memory, the ability to remember to act on an intention when the right moment arises. In this paper, we introduce ProMemAgent, the first proactive agent to incorporate prospective memory over everyday sensory contexts for long-horizon proactive assistance. ProMemAgent organizes past sensory perceptions (e.g., egocentric video and audio) into context-aware memory, generating prospective cues and linking earlier events with subsequent developments for future retrieval. Given the current context, it retrieves relevant past events and performs memory-augmented proactive reasoning to determine whether assistance is needed. We further introduce CPMBench, the first benchmark for evaluating long-horizon proactive assistance over sensory contexts, comprising 1,000 instances across eight everyday categories. Extensive experiments show that ProMemAgent achieves the best memory retrieval performance and consistently outperforms all baselines on long-horizon proactive assistance.

Abstract

Proactive Agents

Explore similar work

ProMemAgent: Long-Horizon Proactive Assistance with Context-Aware Prospective Memory from Sensory Perceptions

Memorizing Who, Knowing How: Benchmarking Personal and Procedural Memory for Persistent Assistants

MemCog: From Memory-as-Tool to Memory-as-Cognition in Conversational Agents