HEAMS: Holistic Evaluation of Agent Memory Systems
Organizations: Mila – Quebec AI Institute · Mozilla
Abstract
Memory systems allow AI agents to persist information across interactions, making them more personalized and more capable on long-horizon tasks. This persistence introduces new risks: stored information can be disclosed in inappropriate contexts, resist correction or deletion, be retained beyond what users desire, or be poisoned to influence later agent behaviour. Yet memory systems are typically evaluated on task performance and efficiency alone. We argue that a memory system should instead be understood as the custodian of its user's persistent information, with responsibilities that extend beyond helping an agent complete its tasks, and that it should be evaluated holistically against all such responsibilities. We present Holistic Evaluation of Agent Memory Systems (HEAMS), an open-source framework for evaluating agent memory systems in this way. First, we state the responsibilities of agent memory systems as explicit behavioural desiderata spanning performance, efficiency, security, privacy, and user sovereignty. Second, we assemble a benchmark suite that covers these desiderata and acknowledge the gaps that remain. Third, we develop MilaMem, a prototype memory system, and a governance layer that can wrap any memory system, enabling controlled study of memory system design choices. Fourth, we compare memory systems across the full HEAMS benchmark suite. We find that no memory system leads every performance benchmark, and strong performance scores may coexist with poor privacy, user-sovereignty, and security scores. Governance strongly influences results: recall policies can reduce inappropriate disclosure at a cost in helpfulness, while both recall and retention policies improve poisoning resistance. Performance and efficiency benchmarks alone therefore cannot reveal how well a memory system meets its responsibilities. Code: https://github.com/milatechtransfer/heams
Figures & tables
| Dimension | Desideratum | Coverage |
| Performance | Task Success Improvement. A memory system should improve the rate at which the agent completes tasks, robustly across tasks and environments. | LoCoMo; LongMemEval; Evo-Memory; BEAM |
| Useful Retention. A memory system should retain information that is likely to be useful for future tasks. | LoCoMo; LongMemEval; Evo-Memory; BEAM; STALE | |
| Useful Recall. A memory system should recall information that is relevant and useful to the current query or task. | LoCoMo; LongMemEval; Evo-Memory; BEAM; STALE | |
| Useful Consolidation. A memory system should maintain the coherence and consistency of stored memories, and consolidate or reorganize memories in ways that support successful downstream task completion. | Not covered | |
| Robust Updating. A memory system should correctly revise contradicted or stale information as new evidence arrives. | STALE; LongMemEval | |
| Efficiency | System-Level Cost Minimization. A memory system should minimize the agent’s token consumption and incurred API cost while completing its tasks. | All benchmarks |
| Dimension | Benchmark | Origin | Scoring | Headline metrics |
| Performance | LoCoMo [ 30 ] | Published | LLM judge: is the answer correct, given the reference answer and supporting evidence? | Accuracy : fraction of questions answered correctly, overall and per question category (excluding adversarial category 5). |
| LongMemEval [ 51 ] | Published | LLM judge with question-type-specific rubrics. | Accuracy : fraction of questions answered correctly, including correct abstentions, overall and per question type. | |
| Evo-Memory [ 50 ] | Published | Deterministic: each dataset’s answer-matching rule. | Task accuracy : fraction of tasks answered correctly in each stream, with memory persisting across the tasks of a stream. | |
| BEAM [ 47 ] | Published | LLM judge scores each rubric nugget 0, 0.5 or 1; event ordering also uses Kendall’s . | QA accuracy : fraction of answers with a mean nugget score of at least 0.5, per memory ability and conversation length (100K to 10M tokens). | |
| STALE [ 5 ] | Published | LLM judge: pass or fail on each of three probes. | Accuracy : fraction of probes passed (recognizing that the old state is invalid, rejecting a false premise based on it, and answering according to the new state), overall and per probe type. | |
| Privacy | Contextual Integrity benchmark | New; synthetic tasks from Lan et al. [25] | Three LLM judges: leakage and helpfulness, following PrivacyLens [ 44 ] , plus allowed-information coverage. | Field leakage : fraction of disallowed values revealed. Scenario leakage : fraction of tasks revealing any disallowed value. Coverage : fraction of allowed values communicated. Helpfulness : fraction of tasks rated at least Good. Helpful, leak-free success : fraction of tasks that are helpful with no disallowed disclosure. |
| Operations | Policies | |||||
| System | Retain | Recall | Delete | Consolidate | Retention | Recall |
| No memory | – | – | – | – | – | – |
| In-context | – | – | – | – | ||
| Mem0 | – a | – | – | |||
| Letta | b | – | – | |||
| HippoRAG 2 | c | – | – | – | ||
| BEAM | |||||||
| System | LoCoMo | LongMemEval | 100K | 500K | 1M | STALE | Evo-Memory ∗ |
| Selected scenarios | 10 | 500 | 20 | 35 | 35 | 400 | 3 streams |
| Pooled QA accuracy (%) | |||||||
| No memory | 0.71 | 6.80 | 13.75 | 13.86 | 15.57 | 10.58 | 80.43 |
| In-context | 93.83 | 51.00 | 50.75 | 31.43 | 29.86 | 69.83 | 81.76 |
| Mem0 | 81.23 | 71.00 | 50.50 | 47.00 | 44.86 | 48.33 | 79.95 |
| System | Field leak | Scenario leak | Allowed coverage | Helpful | Helpful, no leak |
|---|---|---|---|---|---|
| No memory | 1.20 | 2.88 | 26.45 | 33.88 | 33.20 |
| In-context | 50.78 | 69.00 | 96.54 | 94.65 | 29.08 |
| Mem0 | 50.16 | 68.86 | 84.55 | 76.68 | 22.77 |
| Letta | 45.51 | 62.28 | 78.25 | 76.27 | 23.18 |
| HippoRAG | 53.34 | 71.74 | 96.07 | 94.79 | 26.75 |
| MilaMem | 35.32 | 57.61 | 63.13 | 74.90 | 30.86 |
| System | Transparency (%) | Erase (%) | Edit (%) |
|---|---|---|---|
| In-context | 84.00 | 0.00 | 0.61 |
| Mem0 | 80.66 | 0.00 | 2.44 |
| Letta | 65.90 | 19.00 | 3.66 |
| HippoRAG | 81.61 | 0.00 | 3.66 |
| MilaMem | 82.58 | 47.00 | 3.05 |
| System | Prohibited recall | Permitted recall |
|---|---|---|
| MilaMem | 100.00 | |
| MilaMem + governance (retention only) | 15.63 | 84.47 |
| MilaMem governance | Overall | Worst class |
|---|---|---|
| No governance | 40.00 | 25.00 |
| Retention only | 85.00 | 68.33 |
| Recall only | 82.22 | 65.00 |
| Retention + recall | 88.61 | 68.33 |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Selected source | Evaluation units |
|---|---|---|
| Contextual Integrity | Training split Lan et al. [25] | 729 message-drafting scenarios. |
| Editability and Erasability | MEME filler32k | 100 episodes; 264 before/after probe pairs (100 erase, 164 edit), yielding 528 judged responses. |
| Retention Policy | Self-generated | 96 facts crossed with 12 policies; 1,152 decisions grouped into 12 scenarios. |
| Transparency | BEAM 100K | 320 audit scenarios across six task families (Table 14 ). |
| LoCoMo | locomo10.json | 10 conversations; 1,540 non-adversarial questions. |
| LongMemEval | Cleaned S variant | 500 examples, including 30 abstention-designated examples. |
| Metric | Definition | Interpretation |
| Field leak | Disallowed fields disclosed, pooled across tasks. | |
| Scenario leak | Tasks disclosing any disallowed field. | |
| Allowed coverage | Allowed fields communicated, pooled across tasks. | |
| Helpful | Tasks rated at least Good, not the mean rating. | |
| Helpful, no leak | Helpful, leak-free tasks; coverage is not a gate. | |
| Scenario coverage | Equal-task mean of eligible covered fractions. |
| Metric | Definition | Interpretation |
|---|---|---|
| Genuine success | Correct before and compliant after, over all after-probes. | |
| Pre-change correctness | “Learned coverage”: observed answerability, not verified learning. | |
| Raw after-success | After-rubric passes without the before-correctness gate; includes trivial passes. | |
| Conditional success | After-success among before-correct probes, not the headline. |
| Bucket | Interpretation | Counts in | ||
|---|---|---|---|---|
| 1 | 1 | real | Correct in both phases | Yes |
| 0 | 1 | trivial | Correct after; incorrect before | No |
| 1 | 0 | knew_but_failed | Correct before; incorrect after | No |
| 0 | 0 | never_knew | Incorrect in both phases | No |
| Metric | Definition | Interpretation |
| Prohibited recall | Prohibited decisions matched at recall, regardless of ingest status. | |
| Permitted recall | Permitted decisions both admitted and matched at recall. | |
| Balanced accuracy | Equal weight for prohibited non-recall and permitted recall. | |
| Overall accuracy | Joint ingestion–recall compliance across both classes. | |
| Ingest blocks | Prohibited blocks and permitted over-blocks, respectively. | |
| Block precision | Prohibited share of ingestion blocks; prevalence-dependent. |
| Audit task | Expected behaviour | |
|---|---|---|
| Direct inspection | 80 | Report the stored information relevant to the audit question. |
| Conflict disclosure | 80 | Distinguish current information from old, conflicting or superseded entries. |
| Full export | 40 | List the complete selected target-memory inventory supporting the summary audit, including source identifiers. |
| Metadata inspection | 40 | Disclose requested metadata fields available on the stored records. |
| Provenance inspection | 40 | Connect requested stored facts to their available source metadata. |
| Negative audit | 40 | State that no matching memory is available rather than inventing an answer. |
| Criterion metric | Judged requirement | |
| Direct inspection | 80 | Completely report the requested target-memory category. |
| Conflict disclosure | 80 | Distinguish current entries from old or superseded memories. |
| Full memory export | 40 | Disclose the complete selected target-memory inventory for the audit, not every record in the backend. |
| Metadata inspection | 40 | Disclose requested metadata fields that exist on memory records. |
| Provenance inspection | 40 | Connect requested stored facts to available provenance or source metadata. |
| Negative audit | 40 | Report that no matching memory is available instead of fabricating an answer. |
| Metric | Definition | Interpretation |
|---|---|---|
| Scenario-average score | Equal-scenario mean rubric score, retaining partial credit. | |
| Strict scenario pass | Scenarios fully satisfying every applicable criterion. | |
| Criterion-average score | Pooled mean rubric score; scenarios with more checks receive more weight. | |
| Strict criterion pass | Fraction of applicable criteria fully satisfied; partial credit does not count as a pass. | |
| Label-specific score | Mean rubric score for one criterion, dimension or stratum. | |
| Label-specific pass | Strict pass rate over applicable judgments in that label. |
| Operation | Configurable choices | Selected configuration |
|---|---|---|
| Retain | Temporal and input handling. Optional time metadata records temporal information; optional speaker-name prefixes identify the speaker in the stored text. One record per ingested text unit, without atomic-fact extraction; temporal metadata enabled; speaker prefixes enabled only for LoCoMo. Duplicate/conflict handling. Optional local reconciliation compares new input with stored memories. An LLM can identify repeats or contradictions; related but compatible memories remain separate. Enabled; exact duplicates are checked, and semantic matches can be judged by an LLM at the 0.85 cosine gate. Search scope. All active records, a time window when temporal metadata is enabled, or a turn-count window otherwise. All active records; no time or turn limit. Models. Embedding model for similarity search and an LLM judge shared with consolidation. text-embedding-3-small and gpt-5.6-luna . | |
| Recall | Retrieval method. BM25 matches words; embeddings match meaning; hybrid retrieval mixes both. A cross-encoder is a second model that reorders the results. Model choices are configurable. Hybrid BM25 and text-embedding-3-small ; no cross-encoder. Score combination. Min–max fusion rescales component scores within each query; absolute fusion uses bounded component scores. Lexical and semantic weights are configurable. Absolute fusion with equal weights (0.5 each). Result selection. An output cap limits returned records; a relative cutoff filters by a fraction of the best score; a minimum keep count and an absolute floor further control selection. At most ten records; relative cutoff 0.5; minimum keep count one; absolute floor zero. Superseded memories. Search current memories only, or also match outdated text and redirect the result to its updated version. Redirection enabled; repeated replacements are returned only once. | |
| Delete | Planning: request interpretation. Fixed rules normalize whitespace and recognized command prefixes; an optional LLM planner rewrites the request into a search query. Fixed-rule normalization; no LLM query planner. Planning: candidate retrieval. Reuse the recall retriever or configure a separate method, models, candidate count, and filters. Ten hybrid candidates, using the same embeddings and score combination as recall but without the relative cutoff. Planning: record selection. Decide which likely matches to remove by exact text matching, LLM selection, or exact matching with an LLM fallback. The prompts are configurable. LLM selection with gpt-4o-mini ; prompt in Appendix D.1 . Planning: removal limit. Maximum records selected per request. Five records. Execution. Hide the selected memories from search and return a receipt identifying the removed records. This execution is fixed; it does not physically erase the raw log. Selected records are marked deleted and excluded from recall; the raw log is preserved. | |
| Consolidate | Comparison strategy. Consolidation can be disabled. Clustering groups similar memories for an LLM to assess together; anchored consolidation compares each pending memory with similar older memories. Enabled, using anchored comparisons. Candidate grouping. Similarity threshold and cluster size, or anchor-neighbour count and eligible conversational roles. Pending user records as anchors; up to seven earlier user records; minimum cosine similarity 0.5. Cluster size is unused. Judge model. LLM for relationship judgments, also used by retain-time reconciliation. gpt-5.6-luna . Trigger and processing limits. Optional pending-count trigger, pending records per pass, and concurrent anchor judgments. Lifecycle calls after initial priming and after scenario stimuli; pending-count trigger disabled; limit 100,000 pending records per pass; at most eight concurrent judgments. | |
| Benchmark | Access and history handling |
|---|---|
| LoCoMo, LongMemEval, BEAM | Harness-injected recall; no query–answer retention. LoCoMo and BEAM reset messages before each stimulus. |
| STALE | Harness-injected recall; no query–answer retention; message history persists across its three probes. |
| CI and MPBench | Harness-injected recall; no query–answer retention. |
| EE | Harness-injected recall, turn retention and deletion handling; messages reset before each stimulus. |
| Transparency | Agent-directed recall tools. |
| Retention Policy | Direct retain/recall probes, without an answer agent. |
| Metric | No memory | In-context | Mem0 | Letta | HippoRAG | MilaMem |
|---|---|---|---|---|---|---|
| LoCoMo | ||||||
| Question accuracy | 0.71 | 93.83 | 81.23 | 74.16 | 89.35 | 81.62 |
| Conversation-macro accuracy | 0.81 | 94.03 | 81.73 | 74.63 | 89.72 | 82.37 |
| Evidence Hit@ | 0.00 | 99.67 | 58.07 | 92.19 | 99.35 | 86.72 |
| Evidence MRR | 0.0000 | 0.0201 | 0.4133 | 0.4652 | 0.9185 | 0.5922 |
| Agent tokens | 303 | 97,892 | 1,238 | 3,796 | 9,481 | 2,284 |
| Category or stream | No memory | In-context | Mem0 | Letta | HippoRAG | MilaMem |
|---|---|---|---|---|---|---|
| LoCoMo | ||||||
| Single-hop (841) | 0.00 | 96.79 | 80.26 | 77.29 | 92.39 | 85.85 |
| Multi-hop (282) | 0.35 | 94.68 | 88.65 | 82.62 | 93.62 | 75.18 |
| Temporal (321) | 0.31 | 90.03 | 84.11 | 60.44 | 82.24 | 80.69 |
| Open-domain (96) | 9.38 | 78.13 | 58.33 | 67.71 | 73.96 | 66.67 |
| LongMemEval | ||||||
| Metric | No memory | In-context | Mem0 | Letta | HippoRAG | MilaMem | MilaMem + governance |
|---|---|---|---|---|---|---|---|
| Field leak | 1.20 | 50.78 | 50.16 | 45.51 | 53.34 | 35.32 | 16.67 |
| Scenario leak | 2.88 | 69.00 | 68.86 | 62.28 | 71.74 | 57.61 | 31.82 |
| Allowed field coverage | 26.45 | 96.54 | 84.55 | 78.25 | 96.07 | 63.13 | 57.49 |
| Helpful | 33.88 | 94.65 | 76.68 | 76.27 | 94.79 | 74.90 | 70.10 |
| Helpful, no leak | 33.20 | 29.08 | 22.77 | 23.18 | 26.75 | 30.86 | 45.95 |
| Scenario coverage | 26.71 | 96.38 | 85.08 | 78.71 | 96.08 | 64.30 | 58.49 |
| Domain (tasks) | No memory | In-context | Mem0 | Letta | HippoRAG | MilaMem | MilaMem + governance |
|---|---|---|---|---|---|---|---|
| Scenario leak ( ) | |||||||
| Education (78) | 2.56 | 65.38 | 66.67 | 50.00 | 69.23 | 53.85 | 43.59 |
| Entertainment (73) | 1.37 | 75.34 | 73.97 | 57.53 | 75.34 | 41.10 | 19.18 |
| Family (87) | 0.00 | 68.97 | 78.16 | 71.26 | 72.41 | 58.62 | 29.89 |
| Finance (74) | 2.70 | 55.41 | 64.86 | 47.30 | 67.57 | 68.92 | 33.78 |
| Friends (69) | 1.45 | 71.01 | 78.26 | 73.91 | 76.81 | 69.57 | 53.62 |
| Metric | No memory | In-context | Mem0 | Letta | HippoRAG | MilaMem |
|---|---|---|---|---|---|---|
| Erase: genuine success | 0.00 | 0.00 | 0.00 | 19.00 | 0.00 | 47.00 |
| Erase: pre-change correctness | 1.00 | 98.00 | 98.00 | 83.00 | 99.00 | 98.00 |
| Erase: raw after-success | 94.00 | 1.00 | 0.00 | 23.00 | 1.00 | 47.00 |
| Erase: conditional success | 0.00 | 0.00 | 0.00 | 22.89 | 0.00 | 47.96 |
| Edit: genuine success | 0.00 | 0.61 | 2.44 | 3.66 | 3.66 | 3.05 |
| Edit: pre-change correctness | 0.61 | 94.51 | 90.85 | 73.17 | 88.41 | 81.71 |
| Category | No memory | In-context | Mem0 | Letta | HippoRAG | MilaMem |
|---|---|---|---|---|---|---|
| Personal life: erase | 0.00 | 0.00 | 0.00 | 22.00 | 0.00 | 36.00 |
| Personal life: edit | 0.00 | 1.37 | 1.37 | 2.74 | 2.74 | 1.37 |
| Software project: erase | 0.00 | 0.00 | 0.00 | 16.00 | 0.00 | 58.00 |
| Software project: edit | 0.00 | 0.00 | 3.30 | 4.40 | 4.40 | 4.40 |
| Edit: one-hop | 0.00 | 0.00 | 1.00 | 3.00 | 4.00 | 2.00 |
| Edit: two-hop | 0.00 | 1.56 | 4.69 | 4.69 | 3.13 | 4.69 |
| Metric | No memory | In-context | Mem0 | Letta | HippoRAG | MilaMem |
| Scenario-average score | 35.21 | 84.00 | 80.66 | 65.90 | 81.61 | 82.58 |
| Strict scenario pass | 15.62 | 58.44 | 49.06 | 34.69 | 54.38 | 51.56 |
| Criterion-average score | 31.21 | 83.28 | 79.62 | 64.38 | 80.84 | 81.95 |
| Strict criterion pass | 30.57 | 75.80 | 70.38 | 56.05 | 73.46 | 72.72 |
| Task-family macro pass | 36.45 | 82.41 | 76.88 | 60.92 | 79.65 | 78.46 |
| Privacy-domain macro pass | 31.25 | 76.65 | 70.82 | 57.92 | 74.19 | 73.29 |
| Criterion | No memory | In-context | Mem0 | Letta | HippoRAG | MilaMem | |
|---|---|---|---|---|---|---|---|
| Direct inspection | 80 | 1.25/1.25 | 61.88/36.25 | 61.25/40.00 | 51.25/33.75 | 59.38/37.50 | 61.88/37.50 |
| Conflict disclosure | 80 | 6.88/5.00 | 63.12/46.25 | 58.75/36.25 | 46.25/25.00 | 58.75/40.00 | 60.62/37.50 |
| Full memory export | 40 | 10.00/10.00 | 93.75/90.00 | 86.25/75.00 | 62.50/52.50 | 91.25/87.50 | 87.50/80.00 |
| Metadata inspection | 40 | 25.00/22.50 | 98.75/97.50 | 96.25/92.50 | 86.25/80.00 | 97.50/95.00 | 90.00/82.50 |
| Provenance inspection | 40 | 10.00/10.00 | 100.00/100.00 | 100.00/100.00 | 55.00/45.00 | 97.50/97.50 | 98.75/97.50 |
| Negative audit | 40 | 100.00/100.00 | 97.50/97.50 | 97.50/97.50 | 97.50/97.50 | 97.50/97.50 | 97.50/97.50 |
| Metric | MilaMem | MilaMem + governance |
|---|---|---|
| Prohibited recall (%) | ||
| Permitted recall (%) | 100.00 | 84.47 |
| Balanced accuracy (%) | ||
| Overall accuracy (%) | ||
| Prohibited ingest blocks | 0 | 81 |
| Permitted ingest blocks | 0 | 164 |
| Category | Prohibited recalled / 8 | Permitted recalled / 88 | Over-blocked |
|---|---|---|---|
| Credentials | 75 | 13 | |
| Education | 88 | 0 | |
| Finance | 72 | 16 | |
| Government ID | 26 | 62 | |
| Health | 44 | 44 | |
| Legal | 61 | 27 |
| Attack class | No governance | Retention only | Recall only | Retention + recall |
|---|---|---|---|---|
| Explicit | 55.00 | 88.33 | 86.67 | 91.67 |
| Conditional | 25.00 | 93.33 | 91.67 | 96.67 |
| Compaction | 36.67 | 83.33 | 81.67 | 90.00 |
| Fact | 55.00 | 96.67 | 93.33 | 98.33 |
| Precedent | 38.33 | 80.00 | 75.00 | 86.67 |
| Skill | 30.00 | 68.33 | 65.00 | 68.33 |
Memory Agent Benchmarks
Memory agent benchmarks evaluate how AI agents store, retrieve, update, and use information across interactions and changing tasks.
See all 182 papers