Memorizing Who, Knowing How: Benchmarking Personal and Procedural Memory for Persistent Assistants

Maosheng Qin; Qi Zhang; Mingxuan Xia; Jinglei Chen; Jiexiang Wang; Sheng Guo; Cheng Peng; Haobo Wang

cs.AIOct 14, 2026

Memorizing Who, Knowing How: Benchmarking Personal and Procedural Memory for Persistent Assistants

Authors: Maosheng Qin, Qi Zhang, Mingxuan Xia, Jinglei Chen, Jiexiang Wang, Sheng Guo, Cheng Peng, Haobo Wang

Organizations: Zhejiang University · Ant Group

Abstract

Memory systems for persistent personal agents must move beyond isolated memories, maintaining both personal memory about whom they serve and procedural memory about how to act. Crucially, in harness-based personal assistance, procedures or skills are inherently user-conditioned: personal context determines which procedures apply, how they are instantiated, and when they should change. Existing benchmarks typically emphasize either personal or procedural memory, leaving their interaction underexplored. We introduce PraxisMemBench, a harness-based benchmark for persistent personal assistance across daily-life and work trajectories, where personal and procedural memory evolve together and later tasks depend on both. Experiments show that maintaining two memory types separately is insufficient, with completion dropping from single-type to interdependent dependencies (84.6% to 77.9%). More broadly, across existing methods, relation-related failures increase from early to late sessions as interaction history grows (59.8% to 72.9%), revealing a continual-maintenance challenge: writing correct memories is not enough if their evolving content and organization cannot be preserved for later retrieval. To address this, we propose PraxisMem, which uses privileged joint-state supervision to internalize the ability to maintain memory content and organization into the memory manager, with location-aware GRPO and dynamic residual-routed OPSD. PraxisMem reaches 40.78% overall Session Success Rate, compared with 20.60% for Claude Code's native memory. Our code is publicly available at https://github.com/QMMMS/PraxisMemBench.

Figures & tables

Benchmark	Setting	Horizon	Eval.	Domains	Mixed	Personal	Procedural	State dep.
SkillsBench	Exec. agent	1 task	84 tasks	11	$\times$	$\times$	$\checkmark$	$\times$
ClawArena	Harness	24–30 rounds	12 scenarios	$\geq$ 7	$\times$	$\checkmark$	$\times$	$\times$
WorkBuddy Bench	Harness	1 task	260 tasks	4	$\times$	$\times$	$\times$	$\times$
LongMemEval	QA	50–500 sessions	500 Q	1	$\times$	$\checkmark$	$\times$	$\times$
Memoria-Bench	QA	Hundreds of turns	58,422 Q	7	$\times$	$\times$	$\checkmark$	$\times$
VitaBench 2.0	Exec. agent	10–20 tasks	56 users	3	$\checkmark$	$\checkmark$	$\times$	$\times$

Table 1: Comparison of representative long-term memory (unshaded) and general agent or skill (shaded) benchmarks.

\checkmark

\circ

, and

\times

denote full, partial, and absent coverage. Mixed denotes trajectories spanning multiple application domains; State dep. denotes cross-session dependence on agent-created state or artifacts.

Figure 1: Representative personal, procedural, and joint memory failures in persistent personal assistance.

Figure 2: PraxisMemBench trajectory construction. Each session is instantiated from the evolving user, personal and procedural memory, and application context, then independently validated before updating the canonical state for subsequent sessions.

Figure 3: PraxisMemBench evaluation lifecycle. Across sessions, the memory method continuously updates and retrieves personal and procedural memory from user interactions. The maintained memory persists across the trajectory and shapes later memory-dependent tasks, whose outcomes are evaluated by LLM judges and programmatic checks.

Panel of Figure 4 — Figure 4: Memory-dependency structure in the test split. From left to right: (a) Distribution of personal and procedural memory points required by executable tasks; (b) Temporal distance from source evidence for dependencies in later sessions; (c) Joint user-conditioning taxonomy.

Figure 5: Overview of PraxisMem . (1) Criteria missed by all rollouts are activated. (2) They are routed to chunks where the corresponding memory defects persist. (3) A privileged teacher receives the dynamic residual between current and target memory states to provide token-level guidance. (4) GRPO trains the full rollout, while OPSD supervises only routed chunks.

Diagnostic
Method	Tool				Memory QA				Artifact
	Pers.	Proc.	Step SR	SR	Recall	Precision	F1	SR	Pers.	Proc.	Assert.	SR
Oracle	90.42	93.98	93.59	83.90	98.45	99.29	98.82	99.29	85.18	90.32	93.16	77.42
Long Context	39.68	44.91	72.71	21.46	43.37	57.06	46.88	30.71	52.69	51.80	80.56	30.26
DualMem	27.73	51.30	79.23	15.90	16.21	35.61	20.19	0.71	39.77	47.41	86.89	13.73
Basic baselines

Table 2: Overall results on the PraxisMemBench test set. Tool tasks report personal-memory (Pers.), procedural-memory (Proc.), step success rate (Step SR), and session success rate (Session SR). Memory QA reports recall, precision, F1, and SR. Artifact tasks report Pers., Proc., artifact-assertion score (Assert.), and SR.

Figure 6: Memory use across dependency types and persistent interaction. From left to right: (a) DualMem Combined Completion by dependency type; (b) Long-range degradation by evidence age; (c) Counts of memory failures for MemOS, Hermes, and PraxisMem across trajectory stages.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Location	Maintained content
USER.md	Stable user attributes and current preferences or constraints.
MEMORY.md	Topic index linking current personal state to dated events.
events/*.md	Time-indexed changes with optional task and skill links.
skills/<domain>/SKILL.md	Domain rules and reusable procedural workflows.
Raw interaction archive	Source evidence retained for targeted search and inspection.

Appendix

Table 4: Persistent Markdown state used by PraxisMem .

Task	Input	Allowed behavior	Supervised outcome
Update	Store $M_{n}$ and ordered interaction chunks $\chi_{n,1:K}$	Search and read the store, archive evidence, and create or edit structured Markdown	The final state must contain the required changes in their designated locations while avoiding redundant units
Retrieval	Store $M_{n}$ and current request $q_{n}$	Use read-only list, search, and file-reading operations; return concise Markdown bullets	The returned context must cover the required personal and procedural points while excluding irrelevant content

Appendix

Table 5: Update and retrieval training tasks.

Function	Purpose	Update	Retrieval
save_raw	Archive the interaction evidence visible in the current chunk.	$\checkmark$	$\times$
list_memory_files	Inspect the store structure without reading file contents.	$\checkmark$	$\checkmark$
grep_memory	Search memory and return matches with file locations.	$\checkmark$	$\checkmark$
read_memory	Read a selected file or line range for focused inspection.	$\checkmark$	$\checkmark$
edit_memory	Create, replace, or insert structured Markdown content.	$\checkmark$	$\times$

Appendix

Table 6: Memory operations and task-specific availability.

Domain	Description
Training split
Availability Planner	Finds time windows and records schedule blocks.
Care Coordination	Selects services and records plans and check-ins.
Case Note	Maintains case notes, decisions, tags, and actions.
Client Correspondence	Prepares client replies and handoff messages.
Community Posting	Creates board posts, notices, and comments.

Appendix

Table 7: Application domains used in the training and test splits. Descriptions summarize the executable functionality used to instantiate SOPs.

Basic Identity

Age, gender, location, languages, education, occupation, income band, and cultural background.

Professional and Daily Role

Job title, sector, responsibilities, common tasks, work schedule, digital-tool proficiency, and frequent collaborators.

Personality and Values

Core values, social orientation, work and planning styles, privacy awareness, sense of responsibility, and positive, neutral, and negative traits.

Relationship Network

Appendix

Table 8: Persona attribute ontology used to initialize PraxisMemBench trajectories. Each gray row denotes a top-level field group; the following row lists the information represented within that group.

Session	Date	Family / domain	Task
s001	Jan. 8	Tool / entertainment	Create separate watchlists of relaxing movies or shows and games.
s002	Jan. 25	Memory QA	Recall changes in sense of responsibility.
s003	Feb. 6	Artifact / travel	Set a travel budget limit and save context grounded in the user’s travel preferences.
s004	Feb. 22	Tool / education	Find beginner Python resources and save a teaching-oriented plan for a pilot.
s005	Mar. 9	Memory QA	Recall changes in social tendency.
s006	Mar. 25	Artifact / travel	Create an SFO–JFK trip and save its booking receipt.

Appendix

Table 9: A complete test trajectory spanning nearly one year. QA rows name the personal fields queried; executable rows summarize the task presented to the assistant.

Benchmark	Setting	Sessions / turns	History tokens
LongMemEval	S	$\sim$ 40–50 sessions	115K
LongMemEval	M	500 sessions	1,500K
MemBench	Participation normal	Varies by trajectory	10K
MemBench	Participation long	Varies by trajectory	100K
PersonaMem-v2	Effective history	Not fixed	32K
PersonaMem-v2	Extended history	Not fixed	128K

Appendix

Table 10: Long-horizon scale of representative memory benchmarks and PraxisMemBench.

Panel of Figure 8 — Figure 8: Memory QA task example.

Panel of Figure 10 — Figure 10: Artifact task example.

Evaluator	Overall	Tool Argument	Artifact Slot	QA Recall	QA Precision
Human–Human	0.934	0.960	0.979	1.000	0.783
GLM-5	0.793	0.696	0.800	0.844	0.837
Qwen3.7-Plus	0.796	0.698	0.863	0.812	0.833
DeepSeek-V4-Flash	0.670	0.408	0.632	0.902	0.823
Kimi-K2.5	0.776	0.704	0.710	0.899	0.799

Appendix

Table 11: Agreement of independent expert annotations and LLM judges on semantic grading. Human–Human reports QWK before adjudication; LLM rows are measured against the adjudicated human reference. Each rubric contains 30 sampled scoring points.

Parameter	Value
Sampling
Rollouts per example	8
Temperature / top- $p$ / top- $k$	1.0 / 1.0 / $-1$
Max. prompt / response tokens	24,576 / 4,096
Optimization
Strategy	full-parameter FSDP2

Appendix

Table 12: Training configuration for PraxisMem .

Figure 11: Training dynamics of four memory-training variants, measured by Correct-location Recall and location-aware F-state.

Figure 12: Routing and stability diagnostics for the two OPSD variants over training steps.

Method	Tool				Memory QA				Artifact
	Pers.	Proc.	Step SR	SR	Recall	Precision	F1	SR	Pers.	Proc.	Assert.	SR
Oracle	90.42	93.98	93.59	83.90	98.45	99.29	98.82	99.29	85.18	90.32	93.16	77.42
Oracle-Action	95.42	97.30	96.67	94.10	100.00	100.00	100.00	100.00	100.00	95.24	97.67	95.65

Appendix

Table 13: Comparison of Oracle and Oracle-Action on the PraxisMemBench test set.

Diagnostic
Method	Tool				Memory QA				Artifact
	Pers.	Proc.	Step SR	SR	Recall	Precision	F1	SR	Pers.	Proc.	Assert.	SR
Oracle	84.05	93.55	89.96	82.93	97.02	97.77	97.03	100.00	85.67	84.62	92.36	74.19
Long Context	35.42	34.35	59.46	16.67	36.10	44.21	36.79	14.29	30.47	35.94	64.71	10.26
DualMem	25.47	19.11	61.26	12.50	24.07	40.53	27.53	5.71	29.69	19.27	73.82	15.38
Basic baselines

Appendix

Table 14: Overall results in the Autonomous setting on the PraxisMemBench test set. Tool tasks report personal-memory (Pers.), procedural-memory (Proc.), step success rate (Step SR), and session success rate (Session SR). Memory QA reports recall, precision, F1, and SR. Artifact tasks report Pers., Proc., artifact-assertion score (Assert.), and SR.

Figure 13: Task Completion by memory dependency type.

Task group	Sessions	Personal retrieved	Procedural retrieved
Personal-only	22	15/22 (68.2%)	–
Procedural-only	24	–	18/24 (75.0%)
Coordinated	181	112/181 (61.9%)	140/181 (77.3%)
Interdependent	122	96/122 (78.7%)	98/122 (80.3%)

Appendix

Table 15: DualMem required-evidence retrieval by dependency type. Cells report retrieved sessions over eligible sessions, followed by percentage.

	Access-related			Relation-related
Joint type	Personal missing	Procedural missing	Both missing	Fragmented relation	Competing memory
Coordinated ( $n=172$ )	54 (31.4%)	26 (15.1%)	15 (8.7%)	71 (41.3%)	6 (3.5%)
Interdependent ( $n=103$ )	17 (16.5%)	15 (14.6%)	9 (8.7%)	59 (57.3%)	3 (2.9%)

Appendix

Table 16: Primary failure attribution for failed DualMem joint sessions. Cells report counts followed by row percentages.

Memory method	Qwen3.7-Plus	GLM-5.2	DeepSeek-V4-Pro
OpenClaw native memory	10.0	26.4	31.0
MemOS	18.1	22.6	24.2
PraxisMem	43.1	46.7	49.8

Appendix

Table 17: Session SR under different task-agent backbones on eight fixed test trajectories in the Scheduled setting.

Panel of Figure 19 — Figure 19: Effectiveness and runtime trade-offs. (a) Session SR versus inference cost for completing the 500-session test split, including Task-LLM API and memory-GPU costs. (b) Session SR versus per-session memory update/build latency; harness-native memory latency is estimated from excess wall-clock time over stable task-LLM baselines.

Diagnostic
Method	Calls	Input tok.	Output tok.	Time (s)	Task API ($)	Mem. GPU ($)	Session SR
Oracle	7.66	268.51K	5.27K	87.55	0.116	–	86.20
Long Context	7.61	473.03K	5.56K	99.79	0.198	–	27.40
DualMem	7.83	264.86K	5.09K	75.24	0.114	0.072	10.86
Basic baselines
BM25	7.32	243.94K	4.95K	84.46	0.106	–	14.40

Appendix

Table 18: Per-session task-agent resources across memory settings. Input and output tokens include all Qwen3.7-Plus calls made by the task agent, with API cost computed at USD 0.40 per million input tokens and USD 1.60 per million output tokens. Memory-GPU cost uses a single A100 GPU at USD 1.59 per GPU-hour.

Diagnostic
Method	Ops.	Calls	Input tok.	Output tok.	Time (s)	Context tok.	Session SR
Oracle	1.00	–	–	–	0.00	1.14K	86.20
Long Context	1.00	–	–	–	0.00	81.44K	27.40
DualMem	14.07	13.99	125.97K	8.07K	163.58	3.39K	10.86
Retrieval baselines
BM25	1.00	–	–	–	0.02	3.88K	14.40

Appendix

Table 19: Per-session resources of separately observable memory methods, averaged over executed sessions. Input and output tokens refer to the generative Memory LLM, while context tokens measure memory text returned to the task agent. For ReMe, we report the resources and latency of its memory construction stage.

Agent Memory

Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.

See all 189 papers

Oct 15, 2026

MemoProAgent: Toward Memory-Driven Proactive Assistance

As assistants move from reactive answering toward proactive intervention, memory changes role: it no longer only supplies the answer to a query, it determines whether and when to speak at all. Prior work in this setting hands the system the relevant fact in context; we study the regime that makes the problem hard, where the evidence justifying an intervention must be mined from a long-horizon personal memory. We introduce MemoProBench, a benchmark for memory-driven proactive assistance (MPA), pairing egocentric video with year-long, distractor-rich user memories (~618K tokens on average), human-verified trigger windows, and a protocol that scores content and timing jointly. Every baseline we evaluate performs poorly (at most 15%), including frontier models given summarized or retrieval-augmented memory. A controlled study explains why: with oracle evidence delivered inside the trigger window GPT-5.3-Chat reaches 96%; letting it choose the timing drops it to 38%, and replacing oracle evidence with retrieval drops it to 15%. Content and timing are separately necessary, and missing either is catastrophic. Guided by this, we build MemoProAgent with a plug-and-play harness that surfaces memory from a graph-organized store and gates it down to what is worth interrupting for, plus a lightweight RL stage that curbs early and repeated firing. Without any training, MemoProAgent lifts strict accuracy from below 10% to 37.5% on Qwen3.8-27B and 38% on GPT-5.3-Chat; RL post-training reaches 42.4%.

Shenglai Zeng, Xiao Yang, Tiffany Ma +15

Michigan State University · Meta Reality Labs · Meta Superintelligence Labs +3

Proactive Agents Proactive Assistance Agent Memory Personal AI Agents Memory Agent Benchmarks

Ask PDF

Oct 21, 2026

Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences

Personal intelligence systems are taking on increasingly complex everyday tasks, yet repeated questions, unnecessary confirmations, and unwanted outreach can turn capable assistance into interaction burden. Reducing this burden requires agents to remember how users want to be assisted. However, existing memory benchmarks focus on factual recall and personalized answers and rarely test how agents learn and act on interaction preferences. We introduce MemPABench, an interactive environment for on-policy evaluation of memory in personal agents. MemPABench poses long-horizon tasks requiring agents to decide when to clarify, confirm, or act autonomously. Four grounded simulated users hold 14 preferences across work and personal contexts. Their feedback spans 99 sessions over four months, with five event-driven changes per user testing memory updates. We evaluate preference selection, delivered behavior, and sensitivity to context and preference change. Evaluating four memory systems across four LLM backbones, we find that current memory brings limited gains in preference selection. Retrieval-based systems (i.e. Mem0, MemOS, simple RAG) do not outperform No Memory, while File-based memory improves accuracy but applies preferences beyond relevant contexts. Our analysis shows that current memory struggles to retrieve interaction preferences and adapt to preference changes. Even with the correct preference available, overriding the agent's default interaction habits remains challenging. The environment and dataset are available at https://mempabench.com.

Jialin Li, Zhenhao Chen, Hanjun Luo +1

New York University Abu Dhabi, Abu Dhabi, UAE

Memory Agent Benchmarks Personal AI Agents User Preference Modeling Long-Horizon Agent Evaluation

Ask PDF

May 25, 2026

Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-horizon Agents

Existing large language model (LLM) based memory systems apply universal, static policies that overlook a fundamental reality: the contexts that are worth storing in memory are different across users. This misalignment wastes limited memory budget on transient interactions while failing to preserve critical context for long horizon tasks. To address this gap, we investigate an underexplored question: can LLM based memory systems learn personalized memory policies? We introduce PerMemBench, the first benchmark for evaluating personalized memory systems, featuring multi year, multi domain interaction histories across diverse user personas. We further present the first empirical study of memory personalization, proposing session level storage gating, a lightweight framework that selectively bypasses memory operations for transient sessions. Our study confirms that personalization yields substantial retention gains under perfect gating, yet reveals that accurate gating remains an open and critical challenge.

Yeonjun In, Wonjoong Kim, Sangwu Park +2

KAIST

Memorization in Language Models Persistent Memory for Language Models Memory Agent Benchmarks Long-Horizon LLM Agents Agent Memory Management

Ask PDF

Abstract

Agent Memory

Explore similar work

MemoProAgent: Toward Memory-Driven Proactive Assistance

Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences

Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-horizon Agents