LEDGER: Auditable Agent Memory Under Change
Organizations: Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland
Abstract
Agent memory systems are mostly evaluated on fixed histories, and only their final answers are scored. In real use, facts change, sources differ in reliability, and users ask the agent to forget things. We present LEDGER, a memory layer in which a language model proposes facts and explicit rules decide which value is current, which source to trust, and what to delete. Each fact keeps its source and the period in which it held. LEDGER+EP adds a search over raw conversation turns for questions the ledger does not cover, and scrubs these turns on deletion. We compare both against no memory, full-history replay, retrieval over raw transcripts, and Mem0, with the same answering model and prompt, on a synthetic multi-session coding-agent workload and a 210-question subset of LongMemEval, and we score both the evidence each memory sends to the model and the answer. On the coding workload, LEDGER+EP in fallback mode answers 94.3% of questions correctly on five development seeds and 91.1% on five held-out seeds, against 82.1% and 83.3% for full replay, the strongest baseline. When a deletion request only describes the fact, no deleted value reached our systems' evidence in any evaluated probe. On LongMemEval, LEDGER+EP in augment mode has the same observed accuracy as full replay (74.8%) with 1% of its context. The main limitation is that the coding workload is synthetic and built by us; on LongMemEval, the differences between the strongest systems are within the confidence intervals.
Figures & tables
| Property | Error | Ledger | |
|---|---|---|---|
| I1 | Deletion : a fact recorded before the user asked to forget it is never returned again, also for questions about the past. The user may state it again later | resurrected | A1, A2 |
| I2 | Supersession : once an accepted update changes a fact, the old value is no longer the current answer. An update is accepted if its source is at least as trusted as the current value | stale | A1, A3 |
| I3 | Temporal : a question as of returns the value valid at , not a later one | future | A1, A4 |
| I4 | Deduplication : restating a fact or re-ingesting an event creates no new value entry; a restatement may add provenance | – | A1 |
| I5 | Contamination : a less trusted source (a stale README, the agent’s guess) does not override the user | contaminated | A1, A3 |
| I6 | Abstention : when nothing relevant is known, the memory says so | no abstention | A2 |
| Day | Source (trust) | Claim | Rule and effect |
|---|---|---|---|
| 1 | user (3) | PostgreSQL 14 | introduce: entry |
| 3 | README (2) | Postgres 12 | dispute: kept for audit only |
| 5 | user (3) | PostgreSQL 14 | restate: event added to provenance |
| 9 | migrate.log (3) | PostgreSQL 16 | supersede: closed; opened |
| 12 | user (3) | “forget the database” | delete: both entries, the dispute and the events erased; tombstone retained |
| Asked on day 7: PostgreSQL 14; on day 10 as of day 6: PostgreSQL 14; on day 10: PostgreSQL 16; on day 13: abstain, “deleted at the user’s request”. | |||
| Source | Trust | Introduce | Update | Extracted ( Ledger / Ledger+Ep ) |
|---|---|---|---|---|
| User message, tool result, code change | 3 | yes | any value | yes / yes |
| Document | 2 | yes | values of trust | yes / yes |
| Assistant message, tool call | 1 | no (held) | never (disputed) | yes / no |
| Small | Held-out | Long | LongMem- | |
| s0–4 | s5–9 | s0–2 | Eval | |
| Users (histories) | 3 | 3 | 6 | 210 |
| Sessions | 48 | 48 | 360 | 48 |
| Events | 184–218 | 184–217 | 1,151–1,203 | 493 |
| Tokens | 1.8k–2.3k | 1.9k–2.4k | 11.3k–12.0k | 95k–104k |
| Facts | 22–34 | 20–33 | 60–81 | – |
| System | Answer | Evidence | Wrong ans. | Forbidden in evidence: probes (s/f/c) | Deleted value | Tokens | $ |
|---|---|---|---|---|---|---|---|
| No memory | 29.5 1.5 | 0.0 0.0 | 0 | 0 (0/0/0) | 0 / 0 | 0 | 0.05 |
| Full replay | 82.1 3.0 | 21.9 1.8 | 65 | 249 (129/93/93) | 46 / 4 | 1,069 | 0.28 |
| Transcript RAG | 74.8 3.6 | 27.0 5.0 | 88 | 195 (101/69/66) | 39 / 0 | 128 | 0.08 |
| Mem0 | 66.8 1.8 | 34.9 2.7 | 66 | 107 (45/56/13) | 18 / 17 | 141 | 0.37 |
| Mem0 (coding prompt) | 70.6 6.4 | 34.7 2.9 | 82 | 143 (44/76/36) | 16 / 13 | 158 | 0.43 |
| Ledger (rules) | 39.5 6.9 | 38.7 6.3 | 9 | 9 (9/0/0) | 0 / 0 | 5 | 0.05 |
| System | Answer | Evidence | Wrong ans. |
|---|---|---|---|
| No memory | 29.4 2.9 | 0.0 0.0 | 0 |
| Full replay | 83.3 2.8 | 21.1 0.7 | 58 |
| Transcript RAG | 78.5 3.1 | 23.8 3.6 | 81 |
| Mem0 | 66.2 7.1 | 34.8 6.2 | 59 |
| Mem0 (coding prompt) | 68.4 4.3 | 36.0 3.1 | 94 |
| Ledger (rules) | 37.0 2.2 | 37.0 2.2 | 13 |
| System | Accuracy | SS-U | SS-A | Pref | Multi | KU | Temp | Abs | Tokens | Write | Read | Ans. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No memory | 16.2 (11–21) | 0 | 0 | 13 | 0 | 0 | 0 | 100 | 0 | 0.0 | 0.0 | 0.1 |
| Full replay | 74.8 (69–80) | 100 | 97 | 63 | 53 | 90 | 60 | 60 | 108,447 | 0.0 | 0.0 | 43.5 |
| Transcript RAG | 71.4 (65–78) | 100 | 97 | 33 | 43 | 73 | 67 | 87 | 1,072 | 0.0 | 0.0 | 0.6 |
| Mem0 | 71.0 (65–77) | 83 | 40 | 70 | 53 | 83 | 80 | 87 | 221 | 160.9 | 0.0 | 0.2 |
| Ledger (LLM) | 41.0 (34–48) | 63 | 3 | 13 | 40 | 60 | 13 | 93 | 15 | 49.5 | 1.5 | 0.1 |
| Ledger+Ep fallback | 70.5 (65–76) | 97 | 93 | 30 | 50 | 67 | 67 | 90 | 593 | 49.5 | 1.5 | 0.4 |
| System | LLM calls/session | Ingest ms | Retrieve ms | Store KiB | Tokens | Write m$ | Read m$ | Ans m$ |
|---|---|---|---|---|---|---|---|---|
| No memory | 0.00 | 0.00 | 0.00 | 0.0 | 0 | 0.00 | 0.00 | 0.10 |
| Full replay | 0.00 | 0.00 | 0.16 | 8.1 | 1,069 | 0.00 | 0.00 | 0.55 |
| Transcript RAG | 0.00 | 0.00 | 0.35 | 8.1 | 128 | 0.00 | 0.00 | 0.16 |
| Mem0 | 1.65 | 2989 | 174 | 1.5 | 141 | 1.06 | 0.00 | 0.16 |
| Mem0 (coding prompt) | 1.83 | 3272 | 202 | 1.9 | 158 | 1.24 | 0.00 | 0.17 |
| Ledger (rules) | 0.00 | 0.03 | 0.02 | 0.1 | 5 | 0.00 | 0.00 | 0.11 |
| Change | Metric | Before | After |
|---|---|---|---|
| (a) Single-factor trust-rule changes (cached) | |||
| Hold assistant-first values (trust 2) ∗ | wrong answers | 6.0 | 4.0 |
| Hold document-first values too (trust 3) ∗ | evidence | 91.5 | 89.7 |
| (a ′ ) Complete-configuration comparisons | |||
| Rule-based LLM extractor | evidence | 38.7 | 91.7 |
| Ledger Ledger+Ep fallback | answer | 90.9 | 94.3 |
LLM Agent Memory
Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.
See all 398 papers