cs.AIOct 22, 2026
SaveColaBench: Evaluating Memory-Guided Knowledge Acquisition in Personal Agents
Organizations: Jiangnan University, Wuxi, China · School of Advanced Interdisciplinary Sciences, UCAS, Beijing, China · State Key Lab of AI Safety, Institute of Computing Technology, CAS, Beijing, China · DataArc Tech Ltd., China ·
Abstract
Personal agents need to draw on long-term interaction history to retrieve and apply external knowledge suited to a user's current request. Even an answer grounded in relevant documents can be unsuitable when it overlooks the user's circumstances. This creates two connected challenges: using historical conditions to identify the knowledge needed for a new request, and applying that knowledge to the user's situation. Existing benchmarks advance memory recall, personalized retrieval, and memory-guided action, while the joint evaluation of these two challenges remains underexplored. We introduce ColaBench, a synthetic benchmark in health and daily living, with 349 tasks across 10 Profiles and a knowledge base of 13,033 documents. Its Evidence Graph connects personal conditions to their disclosures across public interaction sessions, records prior document exposure, and links answer requirements to personal and newly acquired document evidence. We also introduce MemKB, which uses state summaries to inform queries and rule application, and memory-document links to retrieve knowledge associated with the user's history. MemKB achieves 86.25% Accuracy with an average of 4.07 interaction rounds per task. Task-level analysis finds larger gains on tasks integrating information across multiple historical Facts; error analysis further identifies unsupported personal assumptions and rule misapplication as obstacles to using retrieved evidence. ColaBench provides traceable tasks and an evaluation framework for studying how memory guides knowledge acquisition and rule application.
Figures & tables
Figure 1. Plan–Create–Review construction: (a) document collection; (b) grounded Profiles and Topics; (c) public interaction; (d) history-dependent Item construction; and (e) Item enhancement and independent validation. The growing Evidence Graph guides knowledge exploration and constrains generation and review through recorded Fact disclosures, document exposure, and supporting evidence. Panels a to c show document collection, grounded Profile and Topic construction, and public interaction with separate private and public inputs. Panel d shows initial Item construction: a question writer cannot see the reference answer, a blind solver cannot see history, and a history gate plus final audit select Items. Beside it, panel e analyzes each initial Item, creates an enhanced draft using either the original evidence or an alternative supporting document, and independently validates history dependence and evidence support. Passing drafts become enhanced Items; failed drafts are discarded. The bottom Evidence Graph guides knowledge exploration, constrains disclosed state and document novelty, and supports evidence validation across stages.
| Component | Description |
|---|---|
| Request | Am I eligible for an HFC caregiver respite care grant? |
| History | In a meal-preparation conversation, the user states that they live alone . In another exchange, they mention caring for a relative with Alzheimer’s disease. |
| Rule | The care recipient must live at home with the applicant . |
| Judgment | Living alone does not satisfy the co-residence requirement; other undisclosed conditions remain unknown. |
Table 1. A ColaBench task (paraphrased).
| Statistic | Value |
|---|---|
| Profiles / Sessions / Messages | 10 / 21 / 1,198 |
| Facts / Topics / Items | 52 / 600 / 349 |
| Base / Added web documents | 11,069 / 1,964 |
| Items with multiple Facts | 107 |
| Items citing multiple sessions | 153 |
Table 2. Structural statistics of ColaBench.
Figure 2. MemKB construction and answering. (a) Public history yields state summaries and memory cards linked to historical Read documents and unexposed KB neighbors. (b) States inform the first query; round-robin link expansion and equal-weight RRF combine memory-linked, semantic, and keyword candidates. Search returns memories, states, and document snippets. Public interaction history produces a compact state bank and Mem0-derived memory cards. Card vectors encode memory text, and explicit document links connect cards to historical Read documents and offline KB neighbors. The state bank is supplied before the first query. A query ranks cards, whose linked documents are allocated round-robin, alongside semantic and keyword KB matches. Matched memory text with source identifiers, the complete state bank, and RRF-fused document snippets return to the agent, which can search again, read documents, or answer.
| Method | Acc. | Read Recall | Avg. Rounds |
|---|---|---|---|
| No Memory+Hybrid ( Cormack et al., 2009 ) | 6.59 | 37.25 | 4.85 |
| Graphiti+Hybrid ( Rasmussen et al., 2025 ) | 80.80 | 41.83 | 4.58 |
| A-MEM+Hybrid ( Xu et al., 2025 ) | 81.38 | 39.26 | 4.42 |
| Mem0+Hybrid ( Chhikara et al., 2025 ) | 79.94 | 44.70 | 4.44 |
| MemKB (ours) | 86.25 | 45.56 | 4.07 |
Table 3. Main comparison on 349 tasks. Acc. and Read Recall are percentages; rounds include the final answer.
| Dimension | Mem0+Hybrid | MemKB | Gain |
|---|---|---|---|
| State Fidelity | 4.523 | 4.595 | +0.072 |
| Condition Coverage | 4.532 | 4.739 | +0.207 |
| Evidence Grounding | 4.252 | 4.405 | +0.153 |
| Rule Application | 4.396 | 4.604 | +0.207 |
| Answer Completeness | 4.586 | 4.766 | +0.180 |
Table 4. Model-assessed answer-quality scores on 111 tasks from three Profiles. Each dimension is scored from 1 to 5.
Figure 3. Task structure and evidence access on 349 tasks. (a,b) Accuracy by Fact and history-pair counts. (c) Both exposed: both methods receive the reference document through search or Read; both read: through Read. (d) Paired gains under common composition weights with 95% Profile-cluster bootstrap intervals. pp: percentage points; req.: answer requirements. Four panels show gains by task structure, shared reference access, and task-composition adjustments with confidence intervals.
| Task and decisive premise | Mem0+Hybrid | MemKB |
|---|---|---|
| Breastfeeding suitability. The user has a history of childhood cancer and works in maternal and child health. Applying the guidance requires distinguishing past from current treatment. | Retrieved memories emphasize teaching experience. The answer claims no record of cancer treatment, omitting personal history relevant to rule applicability. Verdict: incorrect. | The initial state summary preserves the cancer history. The answer distinguishes past from current treatment and keeps unknown treatment details conditional. Verdict: correct. |
| Keeping naloxone at home. The user takes a prescription pain medicine of unspecified class. A personal recommendation depends on opioid use and overdose risk. | Reads the relevant guidance, then asserts in the answer that the user takes an opioid. The risk discussion proceeds from this unconfirmed premise. Verdict: incorrect. | The state summary records a prescription pain medicine. The answer still asserts opioid use, leaving dose and concurrent medicines as the unresolved details. Verdict: incorrect. |
Table 5. Paired cases of personal context and rule application. Responses are paraphrased.
| Method | Acc. | Read Recall | Avg. Rounds |
|---|---|---|---|
| MemKB | 91.89 | 42.34 | 4.27 |
| w/o States | 85.59 ( ) | 40.54 ( ) | 4.49 ( ) |
| w/o Links | 81.08 ( ) | 36.94 ( ) | 4.41 ( ) |
Table 6. Ablations on 111 tasks from three Profiles; changes from MemKB appear in parentheses (pp or rounds).
Figure 4. Primary error categories and model-assessed evidence visibility for 48 MemKB failures. Cells show counts; row totals include percentages. Other Issues has zero cases. A four-by-three count heatmap. Personal Assumptions has 1 not-visible, 10 partly-visible and 6 visible cases; Answer Omissions has 7, 5 and 1; Rule Misapplication has 2, 3 and 7; Context Underuse has 0, 3 and 3.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Source | Documents |
|---|---|
| MedlinePlus | 4,849 |
| openFDA drug/label | 2,803 |
| National Cancer Institute (NCI) | 2,371 |
| NICHD Health Topics | 818 |
| MyHealthfinder | 103 |
| Open RN nursing textbooks | 90 |
Table 7. Sources of the frozen knowledge base. Web additions include selected supporting material and related candidate documents.
| Count per task | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| Facts | 242 | 90 | 15 | 2 | 0 | 0 |
| Cited turn pairs | 22 | 222 | 79 | 22 | 2 | 2 |
| Requirements | 3 | 110 | 171 | 47 | 17 | 1 |
Table 8. Distribution of annotation counts per task. Each cell gives the number of tasks, and each row sums to 349.
| Component | Configuration |
|---|---|
| Answer model | DeepSeek v4.1 Flash; temperature 0; output limit 16,384 tokens; Agent budget 128,000 tokens. Mem0 and MemKB use OpenCode; No Memory, Graphiti, and A-MEM use SiliconFlow with OpenCode fallback. |
| Online interaction | Up to four tool rounds and one final-answer round; early completion allowed. A round is one model response and may contain multiple tool calls. Read: at most 100 lines and 3,000 tokens per call. |
| Accuracy evaluator | GLM-5.3 Flash; low reasoning effort; temperature 0; output limit 32,768 tokens. SiliconFlow and OpenCode, with identical model inputs and parameters. |
| Memory construction | Mem0 SDK extraction and update with GLM-5.3 Flash; Mem0 and MemKB share the resulting memories. |
| Embeddings | Qwen3-Embedding-0.6B for knowledge retrieval, Mem0, MemKB and Graphiti; all-MiniLM-L6-v2 for A-MEM memory. |
| MemKB state bank | GLM-5.3 Flash; output limit 16,384 tokens (32,768 for one Profile). Up to 30 states with subjects, time, unknown conditions and source-message IDs; provided before the first query and in search results. |
Table 9. Configuration for the main comparison. Output limits apply per call, and the agent budget applies per task.
| Current question | Knowledge to retrieve | History-informed judgment |
|---|---|---|
| Would my relative qualify for and be suited to a dementia-specific adult day program? | Target population and enrollment requirements. | Alzheimer’s disease matches the target population; formal diagnostic documentation required for enrollment remains to be confirmed. |
| Does respite care apply, at home or during a short inpatient stay? | Hospice eligibility and respite-care arrangements. | The user is a family caregiver. If hospice enrollment is established, explain home and short-term inpatient options. |
| How should my relative prepare for a winter storm or power outage? | Dementia-specific supplies, communication and local assistance. | Adapt preparations to cognitive impairment and long-distance caregiving; identify local helpers and agree on responsibilities. |
Table 10. Three tasks sharing the same personal background: the user cares for a relative with Alzheimer’s disease from a distance. Questions and answer points are paraphrased.
More in this topic
Memory-Augmented Agents
Memory-augmented agents use stored experience and knowledge to act coherently across long tasks, changing environments, and repeated interactions.
See all 348 papers