An Agentic Framework for Proactive Robots with Procedural Assistance Memory

Dohyun Lee; Minho Park; Jeonghoon Park; Dongyoon Hwang; Junha Hyung; Sungwon Hwang; Byungkun Lee; Jaegul Choo

cs.ROOct 15, 2026

An Agentic Framework for Proactive Robots with Procedural Assistance Memory

Authors: Dohyun Lee, Minho Park, Jeonghoon Park, Dongyoon Hwang, Junha Hyung, Sungwon Hwang, Byungkun Lee, Jaegul Choo

Organizations: KAIST AI

Abstract

Proactive robots should anticipate user needs and provide assistance before being explicitly asked. A straightforward approach is to infer human intentions from commonsense knowledge, but the same observed behavior can imply different needs across users, potentially leading to unwanted assistance. We observe that everyday routines in the real world exhibit recurring patterns, so learning these patterns can reduce ambiguity in proactive decisions. Based on this, we propose a memory-based agentic framework in which a VLM task predictor reads a Procedural Assistance Memory (PAM) alongside the current observation to decide when to assist and what task to perform, and a VLA executor performs the predicted task. PAM is structured as a finite-state machine over activity phases, per-state assistance rules, and event-driven transitions; a reflection VLM revises PAM from each day's interaction trace without weight updates or retraining. In our 5-day desk-assistance evaluation, memory-free baselines remain conservative, with limited recall on proactive tasks. Our system improves consistently with accumulated experience, achieving the highest precision, recall, and F1 and reducing first-fire delay from approximately 5 seconds on Day 1 to 1 second by Day 5. Our approach also adapts to changes in the user's routine through continued memory reflection, recovering from memory--routine misalignment after a single memory update. The project page is available at https://davian-robotics.github.io/PAM/

Figures & tables

Figure 1: Motivation for memory-based proactive assistance. Visually similar observations map to different correct actions depending on user-specific habits, and our PAM memory resolves this ambiguity by conditioning the task predictor on accumulated behavioral context.

Figure 2: Overview of our framework. (a) A VLM task predictor reads PAM and recent frames to select assistive tasks for the VLA executor. (b) At the end of each day, a VLM memory reflector updates PAM from the day’s video by adding or revising states, trigger-action pairs, and transitions.

Figure 3: An example of PAM , comprising states, trigger–action pairs, and transitions.

Figure 4: Tasks and dataset. The six proactive tasks form a continuous desk-activity sequence: (T1) Cereal $\rightarrow$ Person, (T2) Milk $\rightarrow$ Person, (T3) Cereal $\rightarrow$ Storage, (T4) Milk $\rightarrow$ Storage, (T5) Bowl $\rightarrow$ Storage, (T6) Used wet wipe $\rightarrow$ Trash.

Figure 5: Multi-day proactive task prediction (Days 1–5). Memory-free baselines show limited T1–T6 recall overall, while Ours (FSM Memory) consistently improves across days, achieving the highest F1 and reducing first-fire delay from ${\sim}5$ s on Day 1 to ${\sim}1$ s by Day 5.

Figure 6: Memory adaptation (Scenario A

\rightarrow

B). Scenario A refers to the scenario shown in Figure 4 , and Scenario B refers to the scenario shown in Figure 20 . A-Frozen fixes A’s Day-5 memory, while A-Evolving updates it at the end of each day in Scenario B. Scratch-Evolving builds memory from scratch on B; Zero-shot VLM uses no memory.

Figure 7: Qualitative analysis of FSM-based Procedural Assistance Memory evolution. From Day 3–5, missing trigger-action pairs are recovered and per-state rules are completed. When the user’s routine changes, end-of-day reflection revises the affected transitions and rules accordingly.

Method	Cereal $\rightarrow$ Person	Milk $\rightarrow$ Person	Cereal $\rightarrow$ Storage	Milk $\rightarrow$ Storage	Bowl $\rightarrow$ Storage	Wet Wipe $\rightarrow$ Trash
$\pi_{0.5}$ [ 2 ]	7 / 10	1 / 10	6 / 10	0 / 10	5 / 10	4 / 10
$\pi_{0.5}$ (Reasoning)	6 / 10	0 / 10	0 / 10	0 / 10	5 / 10	5 / 10
Ours	9 / 10	3 / 10	8 / 10	5 / 10	8 / 10	8 / 10

Table 1: Task-wise success rate comparison with monolithic VLA [ 2 ] with and without subtask reasoning. Bold = best per task.

Figure 8: Qualitative comparison of closed-loop execution. Our System 1–2 framework correctly selects and executes proactive tasks, while $\pi_{0.5}$ and $\pi_{0.5}$ (Reasoning) produce incorrect actions due to joint optimization of reasoning and low-level control under limited data.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Snapshots of the five evaluation days. Each day captures the same canonical desk routine performed by a single user: bringing out a bowl, pouring cereal and milk, eating, using the laptop with the bowl set aside, wiping the mouth with a wet wipe, and leaving the desk.

Day	Frames	NO_ACT	T1	T2	T3	T4	T5	T6
1	270	133	12	21	5	34	35	30
2	248	143	11	27	6	18	24	19
3	224	154	9	23	6	8	14	10
4	207	125	8	23	5	16	16	14
5	197	118	7	19	4	13	21	15
Total	1,146	673	47	113	26	89	110	88

Appendix

Table 2: Per-day annotation statistics on the five evaluation days. Frames are extracted at

1

fps; deleted frames are excluded from the totals. T1: Cereal

\rightarrow

Person, T2: Milk

\rightarrow

Person, T3: Cereal

\rightarrow

Storage, T4: Milk

\rightarrow

Storage, T5: Bowl

\rightarrow

Storage, T6: Wet wipe

\rightarrow

Trash. NO_ACT marks frames where no intervention is appropriate.

Figure 10: Camera setup. Two back-facing cameras (left and right) observe the user and the desk from behind the Franka arm, while a wrist camera mounted on the end-effector provides close-up grasp context. The VLM-based task predictor and reflector consume only the left and right back views; the VLAs additionally receive the wrist view.

Figure 11: Per-task precision, recall, and F1. Memory-based approaches outperform the memory-free baselines on Milk $\rightarrow$ Person and Cereal $\rightarrow$ Storage while maintaining comparable NO_ACT F1. FSM Memory provides further gains over List Memory on Cereal $\rightarrow$ Storage and Milk $\rightarrow$ Storage.

Task	Method	D1	D2	D3	D4	D5
T1	LIT	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$
	Event-Driven	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.40\!\pm\!0.49$	$0.10\!\pm\!0.30$	$0.30\!\pm\!0.46$
	Zero-shot VLM	$0.50\!\pm\!0.50$	$0.20\!\pm\!0.40$	$0.88\!\pm\!0.15$	$0.96\!\pm\!0.09$	$0.85\!\pm\!0.14$
	Ours (List Memory)	$0.60\!\pm\!0.49$	$0.30\!\pm\!0.40$	$0.46\!\pm\!0.42$	$0.85\!\pm\!0.24$	$0.77\!\pm\!0.24$
	Ours (FSM Memory)	$0.40\!\pm\!0.49$	$0.40\!\pm\!0.49$	$0.89\!\pm\!0.14$	$0.98\!\pm\!0.04$	$0.93\!\pm\!0.13$
T2	LIT	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.08\!\pm\!0.00$	$0.00\!\pm\!0.00$

Appendix

Table 3: Per-task Precision on D1–D5 (mean

\pm

standard deviation). T1: Cereal

\rightarrow

Person; T2: Milk

\rightarrow

Person; T3: Cereal

\rightarrow

Storage; T4: Milk

\rightarrow

Storage; T5: Bowl

\rightarrow

Storage; T6: Wet wipe

\rightarrow

Trash. NO_ACT denotes no intervention.

Task	Method	D1	D2	D3	D4	D5
T1	LIT	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$
	Event-Driven	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.04\!\pm\!0.05$	$0.01\!\pm\!0.04$	$0.04\!\pm\!0.07$
	Zero-shot VLM	$0.05\!\pm\!0.06$	$0.03\!\pm\!0.06$	$0.50\!\pm\!0.15$	$0.54\!\pm\!0.11$	$0.66\!\pm\!0.09$
	Ours (List Memory)	$0.07\!\pm\!0.06$	$0.13\!\pm\!0.18$	$0.36\!\pm\!0.35$	$0.58\!\pm\!0.36$	$0.66\!\pm\!0.29$
	Ours (FSM Memory)	$0.03\!\pm\!0.04$	$0.05\!\pm\!0.07$	$0.36\!\pm\!0.16$	$0.68\!\pm\!0.22$	$0.69\!\pm\!0.25$
T2	LIT	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.26\!\pm\!0.00$	$0.00\!\pm\!0.00$

Appendix

Table 4: Per-task Recall on D1–D5 (mean

\pm

standard deviation). T1: Cereal

\rightarrow

Person; T2: Milk

\rightarrow

Person; T3: Cereal

\rightarrow

Storage; T4: Milk

\rightarrow

Storage; T5: Bowl

\rightarrow

Storage; T6: Wet wipe

\rightarrow

Trash. NO_ACT denotes no intervention.

Task	Method	D1	D2	D3	D4	D5
T1	LIT	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$
	Event-Driven	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.08\!\pm\!0.10$	$0.02\!\pm\!0.07$	$0.08\!\pm\!0.11$
	Zero-shot VLM	$0.09\!\pm\!0.10$	$0.05\!\pm\!0.10$	$0.63\!\pm\!0.17$	$0.68\!\pm\!0.11$	$0.73\!\pm\!0.08$
	Ours (List Memory)	$0.12\!\pm\!0.11$	$0.18\!\pm\!0.25$	$0.35\!\pm\!0.32$	$0.56\!\pm\!0.23$	$0.62\!\pm\!0.16$
	Ours (FSM Memory)	$0.06\!\pm\!0.08$	$0.09\!\pm\!0.12$	$0.49\!\pm\!0.19$	$0.77\!\pm\!0.15$	$0.78\!\pm\!0.21$
T2	LIT	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.00\!\pm\!0.00$	$0.12\!\pm\!0.00$	$0.00\!\pm\!0.00$

Appendix

Table 5: Per-task F1 on D1–D5 (mean

\pm

standard deviation). T1: Cereal

\rightarrow

Person; T2: Milk

\rightarrow

Person; T3: Cereal

\rightarrow

Storage; T4: Milk

\rightarrow

Storage; T5: Bowl

\rightarrow

Storage; T6: Wet wipe

\rightarrow

Trash. NO_ACT denotes no intervention.

Figure 12: Effect of memory self-evolution. Per-day precision, recall, and F1 averaged over the six proactive tasks (top) and on NO_ACT (bottom), with first-fire delay (top right), across the five days. We compare our self-evolving memory against a memory frozen after Day 1 and a memory-free zero-shot VLM; continued reflection yields greater overall improvement across days.

Panel of Figure 13 — Figure 13: FSM memory after each of the four end-of-day reflections (one chain run). Blue outlines mark new states or states with updated rules; blue text marks new (trigger, action) pairs and transitions, or only the newly added or -clauses within an existing trigger. Unchanged text remains gray or black.

Panel of Figure 14 — Figure 14: Our final FSM-structured memory instance after the five evaluation days. States are activity phases of the user’s routine; per-state rules describe assistance opportunities associated with each activity phase; edges are event-driven transitions between states.

Figure 20: A different routine (vitamin scenario). The user takes vitamins with water and drinks Coke while working on the laptop. The six assistance tasks involve delivering needed items and clearing them after use.

Figure 21: During eating, milk and cereal are already in storage. FSM Memory yields NO_ACT , while List Memory triggers T2 (Milk $\to$ Person).

Figure 22: Results on the vitamin scenario across five days. Per-day precision, recall, and F1 averaged over the six proactive tasks (top) and on NO_ACT (bottom), with first-fire delay (top right).

Figure 23: Memory adaptation from Scenario B to Scenario A. The evolving source-memory condition adapts toward the from-scratch baseline, while the frozen source-memory underperforms. The A

\rightarrow

B results are shown in Figure 6 in the main paper.

Proactive Assistance

Proactive assistance studies how AI systems can anticipate users’ needs and decide what help to offer, when to offer it, or whether to stay silent.

See all 46 papers

ProMemAgent: Long-Horizon Proactive Assistance with Context-Aware Prospective Memory from Sensory Perceptions
Oct 19, 2026Lilin Xu, Linyang He, Kaiyuan Hou +4Proactive Agents Long-Horizon Agent Tasks
Columbia University
MemoProAgent: Toward Memory-Driven Proactive Assistance
Oct 15, 2026Shenglai Zeng, Xiao Yang, Tiffany Ma +15Proactive Agents Proactive Assistance
Michigan State University · Meta Reality Labs · Meta Superintelligence Labs +3
Robots That Take Initiative: A Framework for Building and Evaluating Proactive Robots
Sep 24, 2026Maithili Patel, Sonia ChernovaHuman-in-the-Loop Evaluation Robotic Policy Evaluation
School of Interactive Computing, Georgia Institute of Technology, USA

Oct 19, 2026J/K move · Enter open · S save

ProMemAgent: Long-Horizon Proactive Assistance with Context-Aware Prospective Memory from Sensory Perceptions

Lilin Xu, Linyang He, Kaiyuan Hou, Yuang Fan+3

Columbia University

Proactive Agents Long-Horizon Agent Tasks Agent Memory Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

Large language model (LLM) agents are moving beyond reactive responses toward proactive assistance that anticipates user needs without explicit requests. However, existing proactive agents primarily address immediate needs, while many everyday situations involve delayed intentions that can only be acted on hours or days later. Humans handle such intentions through prospective memory, the ability to remember to act on an intention when the right moment arises. In this paper, we introduce ProMemAgent, the first proactive agent to incorporate prospective memory over everyday sensory contexts for long-horizon proactive assistance. ProMemAgent organizes past sensory perceptions (e.g., egocentric video and audio) into context-aware memory, generating prospective cues and linking earlier events with subsequent developments for future retrieval. Given the current context, it retrieves relevant past events and performs memory-augmented proactive reasoning to determine whether assistance is needed. We further introduce CPMBench, the first benchmark for evaluating long-horizon proactive assistance over sensory contexts, comprising 1,000 instances across eight everyday categories. Extensive experiments show that ProMemAgent achieves the best memory retrieval performance and consistently outperforms all baselines on long-horizon proactive assistance.

Abstract

Proactive Assistance

Explore similar work

ProMemAgent: Long-Horizon Proactive Assistance with Context-Aware Prospective Memory from Sensory Perceptions

MemoProAgent: Toward Memory-Driven Proactive Assistance

Robots That Take Initiative: A Framework for Building and Evaluating Proactive Robots