Organizations: Jiangnan University, Wuxi, China · School of Advanced Interdisciplinary Sciences, UCAS, Beijing, China · State Key Lab of AI Safety, Institute of Computing Technology, CAS, Beijing, China · DataArc Tech Ltd., China · IDEA Research, International Digital Economy Academy, Shenzhen, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
Personal agents need to draw on long-term interaction history to retrieve and apply external knowledge suited to a user's current request. Even an answer grounded in relevant documents can be unsuitable when it overlooks the user's circumstances. This creates two connected challenges: using historical conditions to identify the knowledge needed for a new request, and applying that knowledge to the user's situation. Existing benchmarks advance memory recall, personalized retrieval, and memory-guided action, while the joint evaluation of these two challenges remains underexplored. We introduce ColaBench, a synthetic benchmark in health and daily living, with 349 tasks across 10 Profiles and a knowledge base of 13,033 documents. Its Evidence Graph connects personal conditions to their disclosures across public interaction sessions, records prior document exposure, and links answer requirements to personal and newly acquired document evidence. We also introduce MemKB, which uses state summaries to inform queries and rule application, and memory-document links to retrieve knowledge associated with the user's history. MemKB achieves 86.25% Accuracy with an average of 4.07 interaction rounds per task. Task-level analysis finds larger gains on tasks integrating information across multiple historical Facts; error analysis further identifies unsupported personal assumptions and rule misapplication as obstacles to using retrieved evidence. ColaBench provides traceable tasks and an evaluation framework for studying how memory guides knowledge acquisition and rule application.
Figures & tables
Figure 1. Plan–Create–Review construction: (a) document collection; (b) grounded Profiles and Topics; (c) public interaction; (d) history-dependent Item construction; and (e) Item enhancement and independent validation. The growing Evidence Graph guides knowledge exploration and constrains generation and review through recorded Fact disclosures, document exposure, and supporting evidence. Panels a to c show document collection, grounded Profile and Topic construction, and public interaction with separate private and public inputs. Panel d shows initial Item construction: a question writer cannot see the reference answer, a blind solver cannot see history, and a history gate plus final audit select Items. Beside it, panel e analyzes each initial Item, creates an enhanced draft using either the original evidence or an alternative supporting document, and independently validates history dependence and evidence support. Passing drafts become enhanced Items; failed drafts are discarded. The bottom Evidence Graph guides knowledge exploration, constrains disclosed state and document novelty, and supports evidence validation across stages.
Component
Description
Request
Am I eligible for an HFC caregiver respite care grant?
History
In a meal-preparation conversation, the user states that they live alone . In another exchange, they mention caring for a relative with Alzheimer’s disease.
Rule
The care recipient must live at home with the applicant .
Judgment
Living alone does not satisfy the co-residence requirement; other undisclosed conditions remain unknown.
Table 1. A ColaBench task (paraphrased).
Statistic
Value
Profiles / Sessions / Messages
10 / 21 / 1,198
Facts / Topics / Items
52 / 600 / 349
Base / Added web documents
11,069 / 1,964
Items with multiple Facts
107
Items citing multiple sessions
153
Table 2. Structural statistics of ColaBench.
Figure 2. MemKB construction and answering. (a) Public history yields state summaries and memory cards linked to historical Read documents and unexposed KB neighbors. (b) States inform the first query; round-robin link expansion and equal-weight RRF combine memory-linked, semantic, and keyword candidates. Search returns memories, states, and document snippets. Public interaction history produces a compact state bank and Mem0-derived memory cards. Card vectors encode memory text, and explicit document links connect cards to historical Read documents and offline KB neighbors. The state bank is supplied before the first query. A query ranks cards, whose linked documents are allocated round-robin, alongside semantic and keyword KB matches. Matched memory text with source identifiers, the complete state bank, and RRF-fused document snippets return to the agent, which can search again, read documents, or answer.
Method
Acc. ↑
Read Recall ↑
Avg. Rounds ↓
No Memory+Hybrid ( Cormack et al., 2009 )
6.59
37.25
4.85
Graphiti+Hybrid ( Rasmussen et al., 2025 )
80.80
41.83
4.58
A-MEM+Hybrid ( Xu et al., 2025 )
81.38
39.26
4.42
Mem0+Hybrid ( Chhikara et al., 2025 )
79.94
44.70
4.44
MemKB (ours)
86.25
45.56
4.07
Table 3. Main comparison on 349 tasks. Acc. and Read Recall are percentages; rounds include the final answer.
Dimension
Mem0+Hybrid
MemKB
Gain
State Fidelity
4.523
4.595
+0.072
Condition Coverage
4.532
4.739
+0.207
Evidence Grounding
4.252
4.405
+0.153
Rule Application
4.396
4.604
+0.207
Answer Completeness
4.586
4.766
+0.180
Table 4. Model-assessed answer-quality scores on 111 tasks from three Profiles. Each dimension is scored from 1 to 5.
Figure 3. Task structure and evidence access on 349 tasks. (a,b) Accuracy by Fact and history-pair counts. (c) Both exposed: both methods receive the reference document through search or Read; both read: through Read. (d) Paired gains under common composition weights with 95% Profile-cluster bootstrap intervals. pp: percentage points; req.: answer requirements. Four panels show gains by task structure, shared reference access, and task-composition adjustments with confidence intervals.
Task and decisive premise
Mem0+Hybrid
MemKB
Breastfeeding suitability. The user has a history of childhood cancer and works in maternal and child health. Applying the guidance requires distinguishing past from current treatment.
Retrieved memories emphasize teaching experience. The answer claims no record of cancer treatment, omitting personal history relevant to rule applicability. Verdict: incorrect.
The initial state summary preserves the cancer history. The answer distinguishes past from current treatment and keeps unknown treatment details conditional. Verdict: correct.
Keeping naloxone at home. The user takes a prescription pain medicine of unspecified class. A personal recommendation depends on opioid use and overdose risk.
Reads the relevant guidance, then asserts in the answer that the user takes an opioid. The risk discussion proceeds from this unconfirmed premise. Verdict: incorrect.
The state summary records a prescription pain medicine. The answer still asserts opioid use, leaving dose and concurrent medicines as the unresolved details. Verdict: incorrect.
Table 5. Paired cases of personal context and rule application. Responses are paraphrased.
Method
Acc. ↑
Read Recall ↑
Avg. Rounds ↓
MemKB
91.89
42.34
4.27
w/o States
85.59 ( −6.31 )
40.54 ( −1.80 )
4.49 ( +0.22 )
w/o Links
81.08 ( −10.81 )
36.94 ( −5.41 )
4.41 ( +0.14 )
Table 6. Ablations on 111 tasks from three Profiles; changes from MemKB appear in parentheses (pp or rounds).
Figure 4. Primary error categories and model-assessed evidence visibility for 48 MemKB failures. Cells show counts; row totals include percentages. Other Issues has zero cases. A four-by-three count heatmap. Personal Assumptions has 1 not-visible, 10 partly-visible and 6 visible cases; Answer Omissions has 7, 5 and 1; Rule Misapplication has 2, 3 and 7; Context Underuse has 0, 3 and 3.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Documents
MedlinePlus
4,849
openFDA drug/label
2,803
National Cancer Institute (NCI)
2,371
NICHD Health Topics
818
MyHealthfinder
103
Open RN nursing textbooks
90
Appendix
Table 7. Sources of the frozen knowledge base. Web additions include selected supporting material and related candidate documents.
Count per task
1
2
3
4
5
6
Facts
242
90
15
2
0
0
Cited turn pairs
22
222
79
22
2
2
Requirements
3
110
171
47
17
1
Appendix
Table 8. Distribution of annotation counts per task. Each cell gives the number of tasks, and each row sums to 349.
Component
Configuration
Answer model
DeepSeek v4.1 Flash; temperature 0; output limit 16,384 tokens; Agent budget 128,000 tokens. Mem0 and MemKB use OpenCode; No Memory, Graphiti, and A-MEM use SiliconFlow with OpenCode fallback.
Online interaction
Up to four tool rounds and one final-answer round; early completion allowed. A round is one model response and may contain multiple tool calls. Read: at most 100 lines and 3,000 tokens per call.
Accuracy evaluator
GLM-5.3 Flash; low reasoning effort; temperature 0; output limit 32,768 tokens. SiliconFlow and OpenCode, with identical model inputs and parameters.
Memory construction
Mem0 SDK extraction and update with GLM-5.3 Flash; Mem0 and MemKB share the resulting memories.
Embeddings
Qwen3-Embedding-0.6B for knowledge retrieval, Mem0, MemKB and Graphiti; all-MiniLM-L6-v2 for A-MEM memory.
MemKB state bank
GLM-5.3 Flash; output limit 16,384 tokens (32,768 for one Profile). Up to 30 states with subjects, time, unknown conditions and source-message IDs; provided before the first query and in search results.
Appendix
Table 9. Configuration for the main comparison. Output limits apply per call, and the agent budget applies per task.
Current question
Knowledge to retrieve
History-informed judgment
Would my relative qualify for and be suited to a dementia-specific adult day program?
Target population and enrollment requirements.
Alzheimer’s disease matches the target population; formal diagnostic documentation required for enrollment remains to be confirmed.
Does respite care apply, at home or during a short inpatient stay?
Hospice eligibility and respite-care arrangements.
The user is a family caregiver. If hospice enrollment is established, explain home and short-term inpatient options.
How should my relative prepare for a winter storm or power outage?
Dementia-specific supplies, communication and local assistance.
Adapt preparations to cognitive impairment and long-distance caregiving; identify local helpers and agree on responsibilities.
Appendix
Table 10. Three tasks sharing the same personal background: the user cares for a relative with Alzheimer’s disease from a distance. Questions and answer points are paraphrased.
Agents today often take real-world actions that depend on long-term memory and context recall over time. However, most current memory benchmarks are built for a conversational question-answer format, where the question itself signals that some fact must be retrieved, and often which one. Moreover, benchmarks rarely require anything beyond accuracy from submissions, allowing memory systems to make unreasonable cost/time tradeoffs to achieve higher scores. We present DolphinBench, a benchmark that evaluates memory directly through an agent's task completion. DolphinBench includes three knowledge-work personas with roughly 500k tokens of user messages per persona and evaluates agents on tasks that depend on information from that history. We verify all 200 tasks per persona by running an agent with and without the relevant history, requiring success with it and failure without it. Finally, we require all evaluations to report total cost and latency alongside accuracy, which enables us to evaluate agent memory systems holistically. No existing memory benchmark combines all three. The dataset and evaluation code are available at https://dolphinbench.ai.
Personal intelligence systems are taking on increasingly complex everyday tasks, yet repeated questions, unnecessary confirmations, and unwanted outreach can turn capable assistance into interaction burden. Reducing this burden requires agents to remember how users want to be assisted. However, existing memory benchmarks focus on factual recall and personalized answers and rarely test how agents learn and act on interaction preferences. We introduce MemPABench, an interactive environment for on-policy evaluation of memory in personal agents. MemPABench poses long-horizon tasks requiring agents to decide when to clarify, confirm, or act autonomously. Four grounded simulated users hold 14 preferences across work and personal contexts. Their feedback spans 99 sessions over four months, with five event-driven changes per user testing memory updates. We evaluate preference selection, delivered behavior, and sensitivity to context and preference change. Evaluating four memory systems across four LLM backbones, we find that current memory brings limited gains in preference selection. Retrieval-based systems (i.e. Mem0, MemOS, simple RAG) do not outperform No Memory, while File-based memory improves accuracy but applies preferences beyond relevant contexts. Our analysis shows that current memory struggles to retrieve interaction preferences and adapt to preference changes. Even with the correct preference available, overriding the agent's default interaction habits remains challenging. The environment and dataset are available at https://mempabench.com.
Memory systems for persistent personal agents must move beyond isolated memories, maintaining both personal memory about whom they serve and procedural memory about how to act. Crucially, in harness-based personal assistance, procedures or skills are inherently user-conditioned: personal context determines which procedures apply, how they are instantiated, and when they should change. Existing benchmarks typically emphasize either personal or procedural memory, leaving their interaction underexplored. We introduce PraxisMemBench, a harness-based benchmark for persistent personal assistance across daily-life and work trajectories, where personal and procedural memory evolve together and later tasks depend on both. Experiments show that maintaining two memory types separately is insufficient, with completion dropping from single-type to interdependent dependencies (84.6% to 77.9%). More broadly, across existing methods, relation-related failures increase from early to late sessions as interaction history grows (59.8% to 72.9%), revealing a continual-maintenance challenge: writing correct memories is not enough if their evolving content and organization cannot be preserved for later retrieval. To address this, we propose PraxisMem, which uses privileged joint-state supervision to internalize the ability to maintain memory content and organization into the memory manager, with location-aware GRPO and dynamic residual-routed OPSD. PraxisMem reaches 40.78% overall Session Success Rate, compared with 20.60% for Claude Code's native memory. Our code is publicly available at https://github.com/QMMMS/PraxisMemBench.