Knowing the Store: What a Memory Backend Must Write Down Before an Agent Can Read It
Organizations: Bilkent University, Ankara, Türkiye
Abstract
An agent with long-term memory can answer from a record it should no longer use, such as a plan the user later cancelled. We ask what a memory store must expose for an agent to know this before retrieving anything, and we score that judgment on its own, as metamemory monitoring. Readers see only a value-free summary: record counts by lifecycle state and a list of attribute names. Each of 148 questions is asked against three versions of one store that differ in one attribute, so wording cannot give the answer away. What a model adds depends on the length of that list. On the short lists the benchmark's ground truth produces (2.6 names on average), none of five language models reliably beats a cosine-similarity lookup over the names, and the three stronger ones are equivalent to it within 0.05. Under a control that fixes counts and list length, none is reliably above it and none is shown equivalent. On lists longer than the stores we tested write, padded to 60 names, the lookup loses 0.18; GPT-5.6 Luna and Sol lose 0.06 to 0.07 and lead it by 0.13 to 0.14, while the other three fall with it. The lead holds for Sol against a sibling padded to the same length and counts. Reworded to share no word with the names, the lookup loses 0.02 and one stronger reader edges 0.04 ahead. On the short lists the three stronger readers lead only where the summary counts a state without naming it, and one count feature closes that lead within a question, though not when items are pooled. The backends we tested omit or misstate this information: under FR-Bank's own metadata, GPT-4.1 mini, Haiku 4.5 and the lookup fall from 0.73, 0.82 and 0.75 to 0.60, 0.71 and 0.59. In these tests, what the store wrote down limited every reader we gave it to. All stores are synthetic; control, whether a better judgment yields a better answer, is left to a later study.
Figures & tables
| A retraction triple. Question: What should I expect during my next Brazil Carnival trip? | |||
| summary line | present version | pn version | absent version |
| retraction total | 0 | 1 | 0 |
| category: entries, active, retracted | 55, 55, 0 | 56, 55, 1 | 54, 54, 0 |
| category names | ..., trip_plan | ..., trip_plan | ... (no trip_plan ) |
| superseded names | (none) | (none) | (none) |
| name lookup score | |||
| Reader | Named states | AUROC | persona interval | matched | W/T/L |
| mini a | superseded | ||||
| mini a | all | ||||
| Haiku | superseded | ||||
| Haiku | all | ||||
| Luna | superseded | ||||
| Luna | all | – | – | – | – |
| What we tried to break | What it removes | What happened |
|---|---|---|
| One question, three store versions | Question-wording leakage | A question-only reader scores by construction |
| Random tokens for the names p | Name meaning, length kept | ; the real lookup is higher |
| Sibling-name control | List length and counts | Lookup , models to , none reliably above the lookup |
| Lifecycle totals permuted | The summary’s two totals | No model moves by more than , none reliably |
| Count moved to a sibling version p | Where the count sits | Luna, Sol, Opus fall to ; mini, Haiku no reliable change |
| Every count deleted | The counts | Mini and Haiku within ; Luna loses , Opus |
| Backend | Mode | Census measurement |
|---|---|---|
| FR-Bank | Over-mark | Of entries and events, marks entries stale against in the reference (about ) and retains an active obsolete assertion after of supersession events, at the stored-versus-obsolete matching threshold of the census. |
| Mem0 | Overwrite | At the retrieval surface we evaluated ( search and get_all , mem0ai 1.0.5), of entries expose updated_at and nothing else, because UPDATE overwrites the record it exposes and ingestion is silently lossy; its per-memory history call was not consulted. |
| A-MEM | Unresolved links | In our integration ( agentic-memory 0.0.1), all link references across notes resolve to note identifiers, on the scored personas (not checked against the reference implementation). |
| Policy | Forbidding records in view (of ) | rate | Mean lines per question | Licensed record in view (of ) |
|---|---|---|---|---|
| Top-five retrieval | ||||
| Similarity, matched budget | – | |||
| Newest per attribute, matched | – | |||
| Always-on fetch, matched | – | |||
| Lookup rule | ||||
| Model rule |
Appendix figures & tables100 assets
Supplementary material from the paper’s appendix.
Appendix
| Population | (positives) | Models (calls per question) |
|---|---|---|
| the counterfactual set | items: questions asked of three versions of one store that differ in one attribute ( obligations: supersession, retraction, expiry) | GPT-4.1 mini (1, and 5), Haiku 4.5 (5), Opus 5 (5), GPT-5.6 Sol (5), GPT-5.6 Luna (5) |
| the expiry extension | items: the eligible expiry pairs the first draw left undrawn, built by the same construction ( expiry pairs pooled with the set’s ) | GPT-4.1 mini (1), Haiku 4.5 (5), Opus 5 (1) |
| the paraphrase set | items: the of the questions whose rewrite, sharing no content word with any printed name, passed a word screen and an author’s read, asked of the same three store versions ( obligations: supersession, retraction, expiry) | GPT-4.1 mini (1), Haiku 4.5 (5), Opus 5 (1), GPT-5.6 Luna (5), GPT-5.6 Sol (5); the last two under the default summary only |
| the counterfactual set, lists padded to names | items: the same questions asked of the same three store versions with the superseded-names list padded to names (148 obligations: supersession, retraction, expiry) | GPT-4.1 mini (5), Haiku 4.5 (5), Opus 5 (1), GPT-5.6 Luna (5), GPT-5.6 Sol (5); the default summary only |
| the supersession pairs, a sibling padded to names | items: the supersession pairs’ padded pn version against a fourth version with one name swapped for a same-category sibling (identical pad and counts) | GPT-4.1 mini (5), GPT-5.6 Luna (5), GPT-5.6 Sol (5); Haiku 4.5 and Opus 5 not run |
| the full obligation set | questions ( obligations: supersession, retraction, expiry): the hard tier’s and the easy tier’s together | GPT-4.1 mini (1); Haiku 4.5, Opus 5 and GPT-5.6 Luna (5); GPT-5.6 Sol and Kimi K3 stopped at the budget |
| Model | Vendor | Counterfactual ( ) | Full obligation ( ) | Coverage ( ) | Cross-model ( ) | Scoring |
|---|---|---|---|---|---|---|
| GPT-4.1 mini | OpenAI | ran | ran (the main study) | ran | ran | one call, temperature ; on the counterfactual set also five calls at the provider default (Table 29 ) |
| Haiku 4.5 | Anthropic | ran | ran | ran | ran | five calls, temperature |
| Opus 5 | Anthropic | ran | ran | ran | ran | five calls, provider default; one call on the expiry extension |
| GPT-5.6 Luna | OpenAI | ran | ran | ran | ran h | five calls, provider default; the cross-model set through the agent harness at reasoning effort low |
| GPT-5.6 Sol | OpenAI | ran | stopped at the budget | stopped at the budget | ran h | five calls; the cross-model set through the agent harness at reasoning effort low, the counterfactual set through the raw API |
| Kimi K3 | Moonshot | not run | stopped at the budget | stopped at the budget | ran | five calls |
| Model | Judgment | Positive | Blind | Indexed | ( CI) | Clustered CI | |
| mini | FOK | — | |||||
| mini | NOK | — † | — † | ||||
| GPT-5.6 Luna h | FOK | ||||||
| GPT-5.6 Luna h | NOK | ||||||
| GPT-5.6 Sol h | FOK | ||||||
| GPT-5.6 Sol h | NOK |
| Model | Judgment | Blind | Indexed | ( CI) | Clustered CI | |
|---|---|---|---|---|---|---|
| mini | NOK | — | — | |||
| mini | FOK | |||||
| GPT-5.6 Luna | NOK | |||||
| GPT-5.6 Luna | FOK | |||||
| Opus 5 | NOK | |||||
| Opus 5 | FOK |
| FOK by backend, GPT-4.1 mini | ||||||
| Backend | / pos. | Blind | Indexed (ground-truth; constrained on Mem0, A-MEM) | item interval | persona interval | |
| FR-Bank hard tier | / | |||||
| Mem0 | / | |||||
| A-MEM | / | |||||
| NOK by model, FR-Bank ; the name lookup scores (rounded from ) on the same questions | ||||||
| Model | / pos. | Blind | Indexed (ground-truth summary) | item interval | persona interval | |
| Registered item | Observation | Population | Outcome |
| P1: luna blind NOK includes chance | , CI includes | cross-model set h , | held |
| P2: blind FOK increases with scale | stratified set h , per model | held | |
| P3: indexed FOK gain decreases with scale | cross-model set h , | held | |
| P4: luna has a positive NOK gain | , CI | cross-model set h , | held |
| P5: Sol blind FOK remains below | stratified set h , | failed | |
| h Rows elicited through the vendor’s agent harness (Appendix A ). | |||
| Item | Registered reading and observation | Outcome |
|---|---|---|
| XV-1 | Registered: every new blind-NOK interval includes chance. Observed: Opus 5 is above, Haiku 4.5 below, and K3 includes chance | failed |
| XV-2 | Positive indexed-minus-blind NOK delta with interval excluding zero on Opus 5 and K3 | held |
| XV-3 | Opus 5 blind FOK exceeds Haiku 4.5 blind FOK; cross-generation price/capability ordering | held |
| XV-4 | Indexed FOK lies within the registered OpenAI-centered band on all three new models | held |
| XV-5 | K3 blind NOK includes chance under its always-reasoning configuration | reports |
| Probe and population | PN | AUROC ( CI) | Reading | |
| Supervised TF–IDF, cross-model set | text carries label signal | |||
| Supervised TF–IDF, hard row (FOK target PN) ‡ | text carries label signal | |||
| Zero-shot mini, cross-model set | includes chance | |||
| Supervised TF–IDF, powered matched set | not controlled for the question signal | |||
| Supervised TF–IDF, FOK target, pooled hard | text carries coverage-label signal | |||
| ‡ The hard tier contains no present items, so the FOK target (non- absent ) and the PN target are label-identical on this row; the registered coverage-target probe re-derives the same point estimate, and the recorded PN-probe interval is the one printed. | ||||
| Model | Judgment | Blind ( CI) | Indexed | ( CI) |
| Monitors | ||||
| mini | NOK | |||
| Opus 5 | NOK | |||
| Haiku 4.5 | NOK | |||
| mini | FOK | |||
| Opus 5 | FOK | |||
| Item-construction category | pn | Classes | ||
|---|---|---|---|---|
| Stable identity | single | |||
| Expired logistics | both | |||
| Superseded preference | both ( negative) | |||
| Selective forgetting | single | |||
| Natural absence | single | |||
| Multi-version fact | single |
| Question: What is my current work style or preference? | |||
|---|---|---|---|
| summary line | version 1: present | version 2: pn | version 3: absent |
| lifecycle total, superseded | 0 | 1 | 0 |
| PREFERENCES_HABITS : entries, active, superseded | 46, 46, 0 | 47, 46, 1 | 45, 45, 0 |
| names carrying superseded content | (none) | work_preference | (none) |
| the name lookup’s score | |||
| the label | a current value, licensed | the old value must not be used | no such record |
| Reader | whole | supersession | retraction | expiry | persona interval (whole) |
| Category-aware reader | |||||
| State-aware reader | |||||
| Totals reader | |||||
| Category-guessing reader | |||||
| Post hoc and privileged: the category-guessing reader’s share read at the manifest category | |||||
| Category-given share reader | |||||
| Reader | row | sup. only | moved-count control | difference (item; persona) |
|---|---|---|---|---|
| GPT-4.1 mini ( ) | whole set | ; | ||
| GPT-4.1 mini | retraction ( pairs) | ; | ||
| GPT-4.1 mini | expiry ( pairs) | ; | ||
| Haiku 4.5 ( ) | whole set | ; | ||
| Haiku 4.5 | retraction ( pairs) | ; | ||
| Haiku 4.5 | expiry ( pairs) | ; |
| Reader | summary | AUROC | persona interval | minus intact | item interval | persona interval | retraction | expiry |
|---|---|---|---|---|---|---|---|---|
| Haiku 4.5 | intact (recorded) | |||||||
| counts only | ||||||||
| names only | ||||||||
| counts only minus names only | ||||||||
| GPT-4.1 mini | intact (recorded) | |||||||
| counts only |
| Reader | summary | AUROC | minus intact | item interval | persona interval | matched | ret. | exp. |
| GPT-5.6 Luna, draw 1 | intact (recorded) | |||||||
| names only | ∗ | |||||||
| counts only | ∗ | |||||||
| counts minus names | ∗ | |||||||
| GPT-5.6 Luna, five draws | intact (recorded) | |||||||
| names only | ∗ |
| Reader, summary | scope | PN vs PRESENT | persona interval | pooled | W/T/L | strict-win | persona interval |
|---|---|---|---|---|---|---|---|
| GPT-4.1 mini, sup. only | whole | ||||||
| sup. | |||||||
| ret. | |||||||
| exp. | |||||||
| GPT-4.1 mini, all three | whole | ||||||
| sup. |
| Reader | pooled | state-balanced | W/T/L | matched | triple interval | persona interval | matched sup. / ret. / exp. | vs absent |
| The default summary (superseded content named) | ||||||||
| Name lookup | / / | |||||||
| GPT-4.1 mini, one draw | / / | |||||||
| GPT-4.1 mini, five draws | / / | |||||||
| Haiku 4.5 | / / | |||||||
| GPT-5.6 Luna | / / | |||||||
| Reader | pooled | within a triple ( ) | same state, across questions ( ) | across states ( ) |
|---|---|---|---|---|
| The default summary | ||||
| Name lookup | ||||
| GPT-4.1 mini, one draw | ||||
| GPT-4.1 mini, five draws | ||||
| Haiku 4.5 | ||||
| GPT-5.6 Luna | ||||
| Reader minus its lookup, whole set | difference | triple interval | persona interval | retraction | expiry |
|---|---|---|---|---|---|
| GPT-4.1 mini, one draw | |||||
| GPT-4.1 mini, five draws | |||||
| Haiku 4.5 | |||||
| GPT-5.6 Luna | ∗ | ||||
| GPT-5.6 Sol | ∗ | ||||
| Opus 5 | ∗ |
| Reader, summary | W/T/L | matched | supersession | retraction | expiry |
|---|---|---|---|---|---|
| GPT-4.1 mini, names only | |||||
| GPT-4.1 mini, names only, dash rule | |||||
| GPT-4.1 mini, counts only | |||||
| Haiku 4.5, names only | |||||
| Haiku 4.5, counts only | |||||
| GPT-4.1 mini, totals randomized |
| Store version | empty lists | mean length | length: pairs |
|---|---|---|---|
| pn | 1: 23, 2: 17, 3: 25, 4: 17, 5: 5, 6: 1 | ||
| present | 0: 24, 1: 16, 2: 31, 3: 12, 4: 4, 5: 1 | ||
| absent | 0: 24, 1: 16, 2: 31, 3: 12, 4: 4, 5: 1 |
| Reader | pop. | rubric | same-store AUROC | persona interval | pair interval | W/T/L |
| GPT-4.1 mini ( ) | A | main-study | 16/26/45 | |||
| A | intent-explicit | 22/34/31 | ||||
| B | main-study | 0/13/13 | ||||
| B | intent-explicit | 7/5/14 | ||||
| Haiku 4.5 ( ) | A | main-study | 18/16/53 | |||
| draw 1 only | A | main-study | 16/22/49 |
| Reader | pop. | rubric | interaction | persona interval | pair interval | pooled AUROC |
| GPT-4.1 mini ( ) | A | main-study | ||||
| A | intent-explicit | |||||
| B | main-study | |||||
| B | intent-explicit | |||||
| Haiku 4.5 ( ) | A | main-study | ||||
| draw 1 only | A | main-study |
| Reader | row | whole set | persona interval | minus the lookup | item interval | persona interval |
|---|---|---|---|---|---|---|
| GPT-5.6 Luna ( ) | main-study (recorded) | |||||
| intent-explicit | ||||||
| intent-explicit minus main-study | ||||||
| GPT-4.1 mini ( ) | main-study (recorded) | |||||
| intent-explicit | ||||||
| intent-explicit minus main-study |
| Reader | row | whole set | persona interval | minus the lookup | item interval | persona interval |
|---|---|---|---|---|---|---|
| default (superseded-only) | one draw, temperature (recorded) | |||||
| five draws, provider default | ||||||
| five minus one | ||||||
| all-states | one draw, temperature (recorded) | |||||
| five draws, provider default | ||||||
| five minus one |
| Model | whole set | supersession | minus the lookup | item interval | persona interval | both exclude zero |
|---|---|---|---|---|---|---|
| GPT-4.1 mini, one draw (recorded) | yes | |||||
| GPT-4.1 mini, five draws (provider default) | no | |||||
| Haiku 4.5 (five draws) | no | |||||
| Opus 5 (five draws) | no | |||||
| GPT-5.6 Sol (five draws) | no | |||||
| GPT-5.6 Luna (five draws) | no |
| Whole set, ; blind by construction; gain is AUROC less | |||||
| Reader, summary | AUROC | item interval | persona interval | first draw | first-draw gain |
| GPT-4.1 mini, sup. only | |||||
| GPT-4.1 mini, all three | |||||
| GPT-4.1 mini, totals rand. | — | ||||
| Haiku 4.5, sup. only | |||||
| Haiku 4.5, all three | |||||
| Model | Summary | AUROC | persona interval | mean, present | mean, absent | mean, pn |
|---|---|---|---|---|---|---|
| GPT-4.1 mini | sup. only | |||||
| GPT-4.1 mini | all three | |||||
| GPT-4.1 mini | totals rand. | |||||
| Haiku 4.5 | sup. only | |||||
| Haiku 4.5 | all three | |||||
| Haiku 4.5 | totals rand. |
| Lookup, summary | whole set | supersession | retraction | expiry |
|---|---|---|---|---|
| Lexical, sup. only | ||||
| Lexical, all three | ||||
| Embedding, sup. only | ||||
| Embedding, all three | ||||
| Model minus the lexical lookup on the same summary (paired bootstrap): difference, item interval, persona interval | ||||
| GPT-4.1 mini, sup. only | ||||
| Encoder | population, summary | AUROC | persona interval | minus MiniLM | item interval | persona interval |
| all-MiniLM-L6-v2 (recorded) | , default | – | – | – | ||
| , all-states | – | – | – | |||
| , committed | – | – | – | |||
| , marker-free | – | – | – | |||
| all-mpnet-base-v2 | , default | |||||
| , all-states |
| Scope | hybrid | persona interval | minus the lookup | item interval | persona interval |
|---|---|---|---|---|---|
| whole set | |||||
| supersession | – | ||||
| retraction | – | ||||
| expiry | – | ||||
| whole set, minus the category-guessing reader |
| Reader | W/T/L | pn vs sibling | triple interval | persona interval | pn vs present | sibling vs present |
|---|---|---|---|---|---|---|
| Name lookup | ||||||
| GPT-4.1 mini (five draws) | ||||||
| Haiku 4.5 (five draws) | ||||||
| GPT-5.6 Luna (five draws) | ||||||
| GPT-5.6 Sol (draw 1) | ||||||
| Opus 5 (draw 1) |
| Reader | list | AUROC | vs recorded | item interval | persona interval |
| Padding with correct names, the questions, ground-truth summary | |||||
| Name lookup | names | – | – | – | |
| Name lookup | names | ||||
| Name lookup | names | ||||
| Name lookup | names | ||||
| Name lookup | names | ||||
| Reader | States | original | paraphrase | difference | item interval | persona interval | reading | per state, paraphrase |
| Name lookup | superseded | holds | / / | |||||
| Lexical lookup | superseded | ∗ | collapses | / / | ||||
| Name lookup | all | ∗ | collapses | / / | ||||
| Lexical lookup | all | ∗ | collapses | / / | ||||
| GPT-4.1 mini | superseded | holds | / / | |||||
| GPT-4.1 mini | all | ∗ | collapses | / / |
| Reader | States | original W/T/L | matched | paraphrase W/T/L | matched | difference | triple interval | persona interval |
| Name lookup | superseded | |||||||
| Lexical lookup | superseded | ∗ | ||||||
| Name lookup | all | |||||||
| Lexical lookup | all | ∗ | ||||||
| GPT-4.1 mini | superseded | ∗ | ||||||
| GPT-4.1 mini | all |
| Model | States | paraphrases | item | persona | originals | item | persona |
| GPT-4.1 mini | superseded | ∗ | ∗ | ||||
| GPT-4.1 mini | all | ∗ | ∗ | ||||
| Haiku 4.5 | superseded | ||||||
| Haiku 4.5 | all | ∗ | |||||
| Opus 5 (draw 1) a | superseded | ∗ | ∗ | ||||
| Opus 5 (draw 1) a | all | ∗ | ∗ |
| Model | States | questions | Pooled AUROC | item interval | persona interval |
| GPT-4.1 mini | superseded | paraphrases | ∗ | ||
| originals | ∗ | ||||
| GPT-4.1 mini | all | paraphrases | ∗ | ||
| originals | ∗ | ||||
| Haiku 4.5 | superseded | paraphrases | ∗ | ||
| originals |
| Reader | States | original | paraphrase | difference | item interval | persona interval | reading |
| Name lookup | superseded | holds | |||||
| Lexical lookup | superseded | ∗ | collapses | ||||
| Name lookup | all | ∗ | collapses | ||||
| Lexical lookup | all | ∗ | collapses | ||||
| GPT-4.1 mini | superseded | ∗ | collapses | ||||
| GPT-4.1 mini | all | ∗ | collapses |
| Reader | States | rule 1, flagged | unflagged | originals, flagged | unflagged | rule 2, flagged | unflagged |
|---|---|---|---|---|---|---|---|
| Name lookup | superseded | ||||||
| Lexical lookup | superseded | ||||||
| Name lookup | all | ||||||
| Lexical lookup | all | ||||||
| GPT-4.1 mini | superseded | ||||||
| GPT-4.1 mini | all |
| Pooled AUROC | |||||
|---|---|---|---|---|---|
| Reader | unpadded | padded | difference | item interval | persona interval |
| Name lookup | ∗ | ||||
| Lexical lookup | ∗ | ||||
| Random-token lookup | ∗ | ||||
| GPT-4.1 mini | ∗ | ||||
| Haiku 4.5 | ∗ | ||||
| Padded lists | ||||||
|---|---|---|---|---|---|---|
| Model | pooled | item interval | persona interval | matched | triple interval | persona interval |
| GPT-4.1 mini | ||||||
| Haiku 4.5 | ||||||
| GPT-5.6 Luna | ∗ | ∗ | ||||
| GPT-5.6 Sol | ∗ | ∗ | ||||
| Opus 5 a | ||||||
| Pooled AUROC | |||||
|---|---|---|---|---|---|
| Reader | unpadded | padded | difference | item interval | persona interval |
| Name lookup | ∗ | ||||
| Lexical lookup | ∗ | ||||
| Random-token lookup | ∗ | ||||
| GPT-4.1 mini | ∗ | ||||
| Haiku 4.5 | ∗ | ||||
| Reader | retraction, pooled | retraction, matched | expiry, pooled | expiry, matched |
|---|---|---|---|---|
| Name lookup | / / | / / | / / | / / |
| Lexical lookup | / / | / / | / / | / / |
| Random-token lookup | / / | / / | / / | / / |
| GPT-4.1 mini | / / | / / | / / | / / |
| Haiku 4.5 | / / | / / ∗ | / / | / / |
| GPT-5.6 Luna | / / | / / | / / | / / |
| minus the lexical lookup | minus the random-token lookup | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | padded, pooled | matched | unpadded, pooled | matched | padded, pooled | matched | unpadded, pooled | matched |
| GPT-4.1 mini | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ||
| Haiku 4.5 | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ||
| GPT-5.6 Luna | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ |
| GPT-5.6 Sol | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ |
| Opus 5 a | ∗ | ∗ | ∗ | ∗ | ∗ | ∗ | ||
| Reader | rank against pn score | rank against outcome |
|---|---|---|
| Name lookup | ||
| Lexical lookup | ||
| Random-token lookup | ||
| GPT-4.1 mini | ||
| Haiku 4.5 | ||
| GPT-5.6 Luna |
| Reader | scope, metric | difference | 95% item/triple | 95% persona | 90% item/triple | 90% persona | reliable | equivalent |
| default summary | ||||||||
| GPT-4.1 mini, one draw | supersession, matched | yes | no | |||||
| GPT-4.1 mini, five draws | supersession, matched | yes | no | |||||
| Haiku 4.5 | supersession, matched | no | no | |||||
| GPT-5.6 Luna | supersession, matched | no | yes | |||||
| GPT-5.6 Sol | supersession, matched | no | yes | |||||
| Reader | scope, metric | difference | 95% item/triple | 95% persona | 90% item/triple | 90% persona | reliable | equivalent |
|---|---|---|---|---|---|---|---|---|
| paraphrase set | ||||||||
| GPT-4.1 mini, paraphrases | whole, pooled | yes | no | |||||
| GPT-4.1 mini, originals | whole, pooled | yes | no | |||||
| GPT-4.1 mini, paraphrases | supersession, pooled | yes | no | |||||
| GPT-4.1 mini, originals | supersession, pooled | yes | no | |||||
| GPT-4.1 mini, paraphrases | supersession, matched | yes | no | |||||
| Summary | scope | items | triples | pooled | persona interval | W/T/L | matched | triple interval | persona interval |
|---|---|---|---|---|---|---|---|---|---|
| padded to | whole | ||||||||
| padded to | supersession | ||||||||
| padded to | retraction | ||||||||
| padded to | expiry | ||||||||
| default (short list) | whole | ||||||||
| default (short list) | supersession |
| Reader | W/T/L | matched | triple interval | persona interval | pooled | sibling vs present |
|---|---|---|---|---|---|---|
| Name lookup (embedding) | ||||||
| Lexical lookup | ||||||
| Random-token lookup | ||||||
| Length-only reader | ||||||
| Count reader, guessed category (share) | ||||||
| GPT-5.6 Luna (five draws) |
| Reader | summary | expiry ( pairs) | extension ( pairs) | pooled ( pairs) |
|---|---|---|---|---|
| Name lookup | sup. only | |||
| All-states lookup | all three | |||
| GPT-4.1 mini | sup. only | |||
| GPT-4.1 mini | all three | |||
| GPT-4.1 mini | totals rand. | |||
| Haiku 4.5 | sup. only |
| Intent | Label | superseded | retracted | expired | active only | unaligned | none | total |
|---|---|---|---|---|---|---|---|---|
| current | pn | |||||||
| current | present | |||||||
| current | absent | |||||||
| upcoming | pn | |||||||
| upcoming | present | |||||||
| upcoming | absent |
| Reader | group | mean, question only | mean, with summary | AUROC, with summary | item interval | persona interval | |
|---|---|---|---|---|---|---|---|
| GPT-4.1 mini | obligations | ||||||
| (a) | |||||||
| (b) | |||||||
| (c) | |||||||
| Haiku 4.5 | obligations | ||||||
| (a) |
| Record | CI | Point or composition | Constrains | |
|---|---|---|---|---|
| Design records | ||||
| Scoreable corpus | n/a | personas across three backends, with preregistered Mem0 exclusions | design record | |
| Hard tier | n/a | FOK-positive items; external negatives: A-MEM and Mem0 | design record | |
| FR-Bank hard tier | n/a | natural-absent; slot-absent; PN | design record | |
| Intent audit | not recorded | route precision ; intent conditioning makes items ( ) ineligible for PN by definition | design record | |
| Inherited retrieval premise ( Mullick & Tüzün, 2026 ) | not recorded | not recorded | retraction-record pass | design record |
| Manipulation | Judgment | Intact | Manipulated | CI | ||
| Another persona’s summary | FOK | |||||
| Another persona’s summary | NOK | |||||
| Own summary, categories shuffled | FOK | |||||
| Own summary, categories shuffled | NOK | |||||
| The shuffle permutes the per-category count rows and the per-category attribute-name lists under a fixed seed, preserving byte length exactly on all summaries and leaving column alignment intact, so the manipulation is correspondence rather than damaged formatting. It preserves the global multiset of names and counts and leaves the flat superseded-names section untouched, so it bounds per-category correspondence and nothing more. Each is the recorded paired-bootstrap estimate of manipulated minus intact ( item-level draws, seed ; single-class draws discarded) rather than the difference of the two displayed columns, from which it differs by Monte Carlo error only, at most on every row. | ||||||
| Row | Item-level CI | Persona-clustered CI | ||
|---|---|---|---|---|
| FOK, hard headline | ||||
| FOK, FR-Bank hard | ||||
| FOK, easy contrast | ||||
| NOK, FR-Bank | ||||
| FOK, Mem0 | ||||
| FOK, A-MEM |
| Condition | AUROC | vs. intact | CI | ECE | |
|---|---|---|---|---|---|
| Blind | — | — | |||
| Intact summary | — | — | |||
| Counts jittered | |||||
| Counts coarsened | |||||
| Names only |
| Condition | AUROC | vs. intact | CI | |
|---|---|---|---|---|
| Intact summary | — | — | ||
| Counts jittered | ||||
| Counts coarsened | ||||
| Names only |
| Model | Backend/population | Blind | Indexed | ( CI) | |
|---|---|---|---|---|---|
| mini | Mem0 | ||||
| mini | A-MEM | ||||
| GPT-5.6 Sol h | external pooled | ||||
| GPT-5.6 Luna h | external pooled |
| Store | Condition | Indexed AUROC | vs blind | CI | |
|---|---|---|---|---|---|
| Mem0 | intact (recorded) | ||||
| Mem0 | count removed | ||||
| Mem0 | markers removed | ||||
| Mem0 | count markers removed | ||||
| A-MEM | intact (recorded) | ||||
| A-MEM | count removed |
| Store | Blind | Intact | Marker as absence | intact ( CI) | blind ( CI) | |
|---|---|---|---|---|---|---|
| Mem0 | ||||||
| A-MEM |
| Store | Blind | Constrained | Names | Names blind ( CI) | Names constrained ( CI) | NB-4 | NB-5 | |
|---|---|---|---|---|---|---|---|---|
| A-MEM | ||||||||
| Mem0 |
| Store | Condition | distinct values | variance | mean | mean, positives | mean, negatives |
|---|---|---|---|---|---|---|
| Mem0 ( ) | blind | |||||
| Mem0 ( ) | indexed | |||||
| A-MEM ( ) | blind | |||||
| A-MEM ( ) | indexed |
| Reader | store | summary | AUROC | persona interval | minus blind | item interval | persona interval | modal share |
|---|---|---|---|---|---|---|---|---|
| GPT-4.1 mini | Mem0 | blind | ||||||
| structural | ||||||||
| omitted | ||||||||
| neutral | ||||||||
| GPT-4.1 mini | A-MEM | blind | ||||||
| structural |
| Population | summary | monitor | NB-1 | NB-2 | NB-4 | NB-5 | NB-3 idx | NB-6 | best | ||
| FR-Bank FOK | ground-truth | – | |||||||||
| FR-Bank FOK | store-written | – | |||||||||
| FR-Bank NOK | ground-truth | ||||||||||
| FR-Bank NOK | store-written | ||||||||||
| Mem0 FOK | constrained ‡ | – | |||||||||
| Mem0 FOK | store-written | – |
| Obligation arises from | NB-6 ( CI) | monitor ( CI) | monitor NB-6 | |
|---|---|---|---|---|
| Supersession | ||||
| Expiry | ||||
| Retraction | ||||
| Whole population |
| Model | Obligation arises from | blind | indexed | indexed blind ( CI) | |
|---|---|---|---|---|---|
| GPT-5.6 Luna | Supersession | ||||
| GPT-5.6 Luna | Expiry | ||||
| GPT-5.6 Luna | Retraction | ||||
| Opus 5 | Supersession | ||||
| Opus 5 | Expiry | ||||
| Opus 5 | Retraction |
| Model | Obligation arises from | AUROC ( CI) | lookup ( CI) | |
|---|---|---|---|---|
| GPT-5.6 Luna | Supersession | |||
| GPT-5.6 Luna | Expiry | |||
| GPT-5.6 Luna | Retraction | |||
| Opus 5 | Supersession | |||
| Opus 5 | Expiry | |||
| Opus 5 | Retraction |
| Model | Lookup variant | model | lookup | difference | item interval | persona interval |
|---|---|---|---|---|---|---|
| GPT-4.1 mini | embedding | |||||
| GPT-4.1 mini | lexical | |||||
| GPT-5.6 Luna | embedding | |||||
| GPT-5.6 Luna | lexical | |||||
| Opus 5 | embedding | |||||
| Opus 5 | lexical |
| Population | / pos. | blind | indexed | lexical | embedding | model embedding | item interval | persona interval |
|---|---|---|---|---|---|---|---|---|
| FR-Bank hard tier | / | |||||||
| FR-Bank easy tier | / | |||||||
| Easy tier, all backends | / | |||||||
| Easy tier, whole map | / |
| Rend. | own name | Haiku, store-written | ground-truth | lookup | store-written minus ground-truth | Haiku minus lookup | |
|---|---|---|---|---|---|---|---|
| capped | present | ||||||
| item | |||||||
| persona | |||||||
| capped | absent | ||||||
| item | |||||||
| persona |
| Reader | Ground-truth summary | Store-written, capped | Store-written, cap lifted |
|---|---|---|---|
| GPT-4.1 mini | |||
| Haiku 4.5 | |||
| retention a | |||
| Name lookup |
| Obligations | GPT-4.1 mini | Haiku 4.5 | Name lookup | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| truth | capped | lifted | truth | capped | lifted | truth | capped | lifted | ||
| all (the printed rows) | ||||||||||
| alignment marks the forbidding record ( bar) | ||||||||||
| the same at the bar | ||||||||||
| the same at the bar | ||||||||||
| not marked at (the ) | ||||||||||
| Population | / pos. | Ground-truth | Own name removed | item-level | by persona | Lookup, ground-truth | Lookup, removed | |
|---|---|---|---|---|---|---|---|---|
| whole population | 661 / 104 | |||||||
| supersession-derived | 620 / 63 | |||||||
| expiry-derived | 572 / 15 | |||||||
| retraction-derived | 583 / 26 |
| Monitor | Population | / pos. | Truthful | Monitor | item-level | by persona | |
| Retrieved view | obligation, whole | 661 / 104 | |||||
| Retrieved view | obligation, supersession | 620 / 63 | |||||
| Retrieved view | obligation, expiry | 572 / 15 | |||||
| Retrieved view | obligation, retraction | 583 / 26 | |||||
| Retrieved view | coverage, hard tier | 307 / 35 | |||||
| Summary + retrieved view | obligation, whole | 661 / 104 |
| Monitor, tag source | AUROC | (against) | item interval | persona interval |
|---|---|---|---|---|
| Summary alone | ||||
| Tagged view, ground-truth tags | (the summary alone) | |||
| on expiry, 15 obligations | (the summary alone, ) | |||
| on supersession, 63 obligations | (the summary alone, ) | |||
| Summary plus tagged view, ground-truth tags | (the summary alone) | |||
| Tagged view, the store’s own tags | (the view under ground-truth tags) |
| Row | first | second | item interval | persona interval | |
| Haiku 4.5, view (ground-truth tags) minus its summary alone | |||||
| Haiku 4.5, summary plus view (ground-truth tags) minus its summary alone | |||||
| Ground-truth tags: the GPT-4.1 mini monitor on the same input minus the reader with no generative-model call | |||||
| View monitor minus tag density | |||||
| View monitor minus tagged-name lookup | |||||
| View monitor minus forbidding-record indicator | |||||
| Policy | abstains | sup. (of ) | ret. (of ) | exp. (of ) | abstain, absent (of ) |
|---|---|---|---|---|---|
| Top-5 only | |||||
| Oracle | |||||
| Model rule | |||||
| Lookup rule | |||||
| Model rule, all states named | |||||
| Lookup rule, all states named |
| Policy | answered | stale at | stale at , tagged / untagged | present answered | correct at |
| Top-5 only | of / of | ( ) | |||
| Oracle | of / of | ( ) | |||
| Model rule | of / of | ( ) | |||
| Lookup rule | of / of | ( ) | |||
| Model rule, all states named | of / of | ( ) | |||
| Lookup rule, all states named | of / of | ( ) |
| Answerer | Policy | Reached | Stale at | Stale at | Correct (licensed) | Rate at | Scorable, whole answer | Scorable, first sentence |
|---|---|---|---|---|---|---|---|---|
| mini | Top-5 | of | of | of | ||||
| mini | Model | of | of | of | ||||
| mini | Lookup | of | of | of | ||||
| mini | Model, all states | of | of | of | ||||
| mini | Lookup, all states | of | of | of | ||||
| mini | Literal | of | of | of |
| Policy | Lines | Record in view, of (Wilson) | Correct at (Wilson) | at | Among in view | Abstain, licensed | Abstain, absent, of (Wilson) |
| GPT-4.1 mini | |||||||
| Flat | of | ||||||
| Flat | of | ||||||
| Flat | of | ||||||
| Flat | of | ||||||
| Flat | of | ||||||
| Answerer | Policy (prompt) | Licensed correct | Reported (by the successor) | Absent abstained: rule answerer | Composite | Rule-only |
|---|---|---|---|---|---|---|
| GPT-4.1 mini | Top-5 (header) | ( ) | ||||
| GPT-4.1 mini | Top-5 (tagged) | ( ) | ||||
| GPT-4.1 mini | Literal rule (header) | ( ) | ||||
| GPT-4.1 mini | Literal rule (tagged) | ( ) | ||||
| GPT-4.1 mini | Model rule (header) | ( ) | ||||
| GPT-4.1 mini | Lookup rule (header) | ( ) |
| Measure | GT tags, primary | GT tags, secondary | own tags, primary | own tags, secondary |
|---|---|---|---|---|
| Views changed, of | ||||
| Lines propagated | ||||
| Audited duplicate items fully tagged, of | ||||
| False propagation on the licensed items’ active lines | / ( ) | / ( ) | / ( ) | / ( ) |
| Re-answers on the changed views under own tags (before after; stale at over answered obligation views; correct licensed answers) | ||||
| GPT-4.1 mini: top-five: / / ; correct ; model: / / ; correct ; oracle: / / ; correct | ||||
| Policy | Forbidding records in view (of ) | Stale answers (of ) | Stale, of the reached scorable obligations | Stale, of the answered subset; Wilson | Correct (of ) |
|---|---|---|---|---|---|
| Top-5 retrieval | of , | of , | |||
| Model rule | of , | of , | |||
| Lookup rule | of , | of , | |||
| Model rule, all states named and scored | of , | of , | |||
| Literal lookup rule (descriptive) | of , | of , | |||
| Oracle | of , | of , |
| Policy | fires | reach rate; Wilson interval of the stale rate | mean view | tagged | coverage (of ) |
|---|---|---|---|---|---|
| Top-5 retrieval | ; | ||||
| Model rule | ; | ||||
| Lookup rule | ; | ||||
| Model rule, all states named and scored | ; | ||||
| Literal lookup rule (descriptive) | ; | ||||
| Oracle | ; |
| reach (of ) | rate | supersession (of ) | retraction (of ) | expiry (of ) | |
|---|---|---|---|---|---|
| Policy | in view (of ) | Wilson | sup. / retr. / exp. | minus the model rule | item interval | persona interval |
|---|---|---|---|---|---|---|
| Model rule (recorded) | = | – | 53 / 20 / 12 | – | – | – |
| Always-open fetch, five lines plus the fetch | = | 54 / 20 / 12 | ||||
| Always-open fetch, matched | – | – | ||||
| Similarity view, seven lines (recorded rows) | = | 27 / 23 / 12 | ||||
| Similarity view, matched | – | – | ||||
| Recency-aware view, seven lines | = | 17 / 0 / 11 |
| Number (where) | Population | Label source | Record alignment | Tag source | Evaluator | Validation |
|---|---|---|---|---|---|---|
| Counterfactual NOK AUROC, lookup and models to (Section 5 , Table 1 ) | items, triples | manifest; pn labels hand-verified; question fixed across versions | not needed: versions built from the manifest | ground-truth summary | AUROC, persona-grouped bootstrap | registered predictions held; PN-vs-PRESENT decomposition post hoc |
| Paraphrase check, the lookup’s loss of (Section 5.2 , Table 38 ) | items, of the triples | manifest; labels transferred from the original triples, whose store versions are unchanged; rewrites screened by a word rule and read by an author with disclosed AI assistance | not needed: versions built from the manifest | ground-truth summary | AUROC, paired bootstrap (item and persona) | descriptive registration, no prediction; readings as they came out; the matched metric and the supersession-only comparison post hoc |
| Long-list check, the lookup’s loss of and the two leads of to (Section 5.2 , Tables 2 and 44 ) | items, triples, with -name lists; the supersession triples registered | manifest; labels unchanged, the store versions unchanged, the list lengthened with the persona’s own names | not needed: versions built from the manifest | ground-truth summary, the superseded-names list padded | pooled AUROC primary and the matched score secondary, both fixed in advance; paired bootstrap (item and persona; triple and persona) | registration committed before the run; two of four predictions held; Opus 5 at one draw |
| Own-name ablation, to (Section 5.1 ) | , supersession obligations | manifest; hand-verified | join, for the names removed | ground-truth summary, edited | AUROC, paired bootstrap | registered prediction held |
| Moved-count control (Section 5.1 ) | counted-state triples | manifest | not needed | ground-truth summary, one count moved | AUROC, paired bootstrap; band reading | descriptive; fired as neither |
| Tagged view , , (Section 7 , Table 79 ) | , obligations | manifest; hand-verified | join (ground-truth tags: of forbidding records marked); the store’s own tags for the row | ground-truth tags; FR-Bank’s own tags | AUROC, paired bootstrap | L2 predictions failed; the own-tags row descriptive |
| Artifact | Commit hash | Freeze day | What it froze | Outcome |
|---|---|---|---|---|
| E0 preregistration | 0f2fde2c3b | day 0 | pilot universe, labels, summary, prompts, decision rules, bootstrap, leakage; both hypotheses held | held |
| E0 Amendment 1 | d4ef3bee6a | day 0 | contamination sensitivity, absence split, PN verification; a method freeze carrying no prediction | no prediction |
| E0 Amendment 2 | c603f4a5ce | day 0 | blind-deduplicated sensitivity; a method freeze carrying no prediction | no prediction |
| E1 preregistration | 875f726fab | day 0 | taxonomy, tiers, summary source, prompts, decision rules, metrics; all four hypotheses held | held |
| E1 Amendment 1 | 307865f414 | day 0 | two-channel presence control and FR-only NOK scope; a method freeze carrying no prediction | no prediction |
| Frontier addendum | 3c3ef7dd6d | day 0 | matched frontier condition and interpretation rule, and three mechanism hypotheses. The two carrying a bar held; the third was registered as descriptive and its reading was not supported, so the registration’s own rule declared the mechanism unresolved | held |
| Artifact | Commit hash | Freeze day | What it froze | Outcome |
|---|---|---|---|---|
| Side experiments: name-bearing summary (S1) | e8b2630ce3 | day 33 | a manifest-built, name-bearing summary for the two structureless stores, the entry map with its hand check, a coverage floor, and two predictions on A-MEM, both of which held; Mem0 fell below the floor and prints as descriptive | held / descriptive |
| Side experiments: store-written-state cap (S2) | e8b2630ce3 | day 33 | two filter rules on the store’s own metadata and a precision bar neither reached, so the repair was withdrawn before any call; one descriptive condition, the store-written summary with its fifteen-name cap lifted, carrying no prediction | no prediction |
| R94: CF-2 store-counterfactual set | f016f033e9 | day 33 | question and slot pairs drawn under predicates fixed before any verdict, three store versions each, three summaries, three readers, five predictions and their branch texts; all five held, two of them printing either way: the lookups within tolerance and every reader inside the totals-randomization band; five recorded deviations | held |
| RESP: mechanism tests | 17e6328c81 | day 33 | the own-name ablation, the retrieved-view and hybrid monitors, and the rubric-gate test on the full obligation population, six predictions and their failure texts: L1-P1 held; L1-P2, L2-P1, L2-P2, L2-P3, L3-P1 failed, all printed | L1-P1 held; L1-P2, L2-P1, L2-P2, L2-P3, L3-P1 failed. |
| DS: downstream experiment | a5a2036c3b | day 34 | six policies over the obligation and coverage scores already on record, their assembled views, reach and abstention, and one answer call per distinct view; five predictions: DS-P3 and DS-P5 held; DS-P1, DS-P2 and DS-P4 failed; two bars set under the 5% premise the study corrected at L2; one prediction unfulfillable by its own gate, recorded as a deviation | held / failed |
| DS Amendment 4: a view header binding the entries to the persona and pinning the date | fb4b2ca8c8 (file hash) | day 35 | the same views answered once more by GPT-4.1 mini and Haiku 4.5 with a header naming the persona and the current date; the same detector with first-sentence and source columns beside; two predictions on the licensed items in view, with failure texts | two predictions; both held |
| Artifact | sha256 (first bytes) | Freeze day | What it froze | Outcome |
|---|---|---|---|---|
| Q4: the marker read as absence | 8b6ee9e3 | day 34 | one instruction line appended to the otherwise unchanged structureless summaries; the recorded coverage items of both stores; the paired differences against the recorded intact and blind rows (Table 64 ) | no prediction |
| DS Amendment 1: the gate elicited under the all-states summary | 72ab84da | day 34 | GPT-4.1 mini’s obligation score re-elicited under the summary naming all three states on the ; one further policy reading it; the counted-state reach beside the recorded gate (Table 81 ) | no prediction |
| List-length padding | 93e33022 | day 34 | the superseded-names list of each of the summaries padded with correct attribute names to , and ; both lookups and GPT-4.1 mini at each length, paired against the -name row; three branches enumerated | no prediction |
| Paraphrased counterfactual set | 45cad63d | day 34 | each of the questions rewritten to share no content token with the slot path or any printed name, screened and read by us; both lookups and three models on the same store versions, paired against the originals; four branches enumerated | no prediction; run on day on the accepted pairs (Table 38 ) |
| Paraphrased counterfactual set: the dated notes and the amendment | 2848f1ad (the whole file) | day 42 | below the registered text’s marker, dated and never editing the text above: the read disclosed as AI-assisted (an assistant flagged rows with reasons and a coding session printed persona evidence; every verdict an author’s); the floor amended from to the accepted pairs before any scoring call, with the screen-passing pairs as a sensitivity analysis, not registered; GPT-5.6 Luna and GPT-5.6 Sol added as readers, not registered, declared after the lookup result and before any model result | no prediction |
| Expiry extension of the counterfactual set | 2f555c1a | day 35 | the eligible expiry pairs the counterfactual draw left undrawn, fixed by the candidate census ( rows/cf3/CF3_CENSUS.md in the artifact); the same construction, renderer, leakage screen and readers; a pooled expiry row with the ’s pairs (Table 54 ) | no prediction |
| Artifact | Commit hash | Freeze day | What it froze | Outcome |
|---|---|---|---|---|
| Graphiti, three personas: the ingestion trial and its readings | f2d7688a54 | day 20 | the unmodified release on three personas, its census, and five readings written before any run and read at forty personas: two with a bar, that the obligation gain under the store-written summary excludes zero and that the superseded-names lookup sits within tolerance of the model or above it; one uncomputable on that population; two descriptive (Appendix G.4 ) | GR-1 (the gain) failed, GR-2 (coverage) not computable, GR-3 (the lookup) held, GR-4 and GR-5 descriptive, all read at forty personas |
| Moved-count control, Amendment 1: Opus 5 at five draws | 177697a4ae | day 40 | Opus 5 at five draws on the moved-count texts against its recorded five-draw row; one prediction, that the one-draw movement on retraction and expiry replicates reliably (Table 17 ) | held |
| Moved-count control, Amendment 2 | b0bee9329d | day 40 | GPT-5.6 Sol and GPT-5.6 Luna at five draws against their five-draw rows; one prediction, that each falls reliably on both counted states (Table 17 ) | failed for both on retraction |
| Names against counts on the counterfactual set | 177697a4ae | day 40 | the default summary with its counts removed and with its names removed, GPT-4.1 mini and Haiku 4.5 against their recorded one-draw and five-draw rows; two predictions, that names only sits within tolerance of the recorded row and that counts only falls reliably below names only, and one descriptive reading (Table 18 ) | both held |
| The marker ablation on the structureless stores | 177697a4ae | day 40 | the constrained rendering with its marker omitted and reworded, GPT-4.1 mini on both stores and GPT-5.6 Luna through the raw API; one prediction, that the coverage loss disappears under the neutral wording, and one descriptive reading (Table 67 ) | failed on Mem0, held on A-MEM |
| DS Amendment 8: label propagation inside the view | 177697a4ae | day 40 | a view-only rule propagating a tagged line’s tag to same-attribute lines that repeat it, under ground-truth and own tags; five predictions on the views changed, false propagation and the flagged rate (Table 86 ) | one held (false propagation under ground-truth tags at most ); three failed; one not defined |
| Artifact | Commit hash | Freeze day | What it froze | Outcome |
|---|---|---|---|---|
| The sibling-name control | 5cd8f809f2 | day 41 | a fourth store version per supersession pair with a same-category sibling in place of the target’s name, the name lookup, five models, the random-token lookup and the count reader on the matched pn -against-sibling comparison (Table 36 ); four readings with a bar, two of them (P-SB1, P-SB3) disclosed at the freeze as recorded values from the scorer’s dry run and not counted as predictions | P-SB2 (every model at or above ) and P-SB4 (no model reliably above the lookup) held; P-SB1 and P-SB3 as recorded |
| Names against counts, Amendment 1: the three stronger models | 5cd8f809f2 | day 41 | E2’s names-only and counts-only renderings on Opus 5, GPT-5.6 Luna and GPT-5.6 Sol (Table 19 ); three predictions: names only within tolerance of the intact row, counts only reliably below names only, counts only above on both counted states under the matched metric; Sol’s lane dropped by the registration’s own budget rule | the first failed on GPT-5.6 Luna and held on Opus 5; the second held on both; the third failed on both; not evaluable on Sol |
| The long-list check on the counterfactual set | 06c611f | day 43 | the items’ superseded-names lists padded to names, one pad per triple drawn from the persona’s own printed attribute names, the three lookups and the five models at their Table 1 settings (Opus 5 at one draw against its recorded five-draw row), pooled AUROC primary and the matched score secondary on the supersession triples (Table 44 ); four predictions: each of GPT-5.6 Luna, GPT-5.6 Sol and Opus 5 reliably above the lookup (P-L1), neither GPT-4.1 mini nor Haiku 4.5 reliably above it (P-L2), the lookup reliably losing more than against its unpadded row (P-L3), each of the three stronger models losing less than the lookup (P-L4) | P-L2 and P-L3 held; P-L1 and P-L4 failed on Opus 5 alone, Luna and Sol as predicted |
| The sibling-name control at names | f224177 | day 44 | the supersession pairs’ padded pn version against a fourth version with one name swapped (identical pad and counts), the no-model readers recorded before the freeze, GPT-5.6 Luna, GPT-5.6 Sol and GPT-4.1 mini at five draws (Table 53 ); four predictions: Luna reliably above the name lookup (P-S1), Sol reliably above it (P-S2), mini not reliably above it (P-S3), Luna and Sol at or above matched with the persona interval above (P-S4) | P-S2, P-S3 and P-S4 held; P-S1 failed |
| Reader, summary | flagged, whole | flagged, by state | unflagged, whole | unflagged, by state |
| Name lookup (sup. only) | / / | / / | ||
| All-states lookup (all three) | / / | / / | ||
| GPT-4.1 mini, sup. only | / / | / / | ||
| GPT-4.1 mini, all three | / / | / / | ||
| Haiku 4.5, sup. only | / / | / / | ||
| Haiku 4.5, all three | / / | / / |
| Population | lookup, slice | ground-truth | all faults | missing names | wrong names | length |
|---|---|---|---|---|---|---|
| Counterfactual set ( ) | embedding, whole set | |||||
| Counterfactual set ( ) | embedding, supersession | |||||
| Counterfactual set ( ) | lexical, whole set | |||||
| Counterfactual set ( ) | lexical, supersession | |||||
| Full population ( ) | embedding, whole set | |||||
| Full population ( ) | embedding, supersession |
| Census | recovered | stale-active | extraction miss | recall (bands / ) | precision | |
|---|---|---|---|---|---|---|
| Manifest supersession events | ( / of ) | of , | ||||
| Forbidding records, obligations of the full store (the main-text reading) | of | of , | ||||
| the same over distinct forbidding-record sets | of | the same | ||||
| Forbidding records, all obligation rows (the prediction’s population) | of | ( / of ); over the by attack vector | the same |
| Input or reader | NOK AUROC | Contrast and standing |
|---|---|---|
| Store-written summary, capped | Minus ground truth : ; . P2 holds. | |
| Store-written summary, lifted | Descriptive; the lifted name cap is . | |
| Ground truth on Graphiti’s schema, capped / lifted | / | Descriptive schema controls. |
| Ground-truth summary / question only | / | Recorded comparators on the same population. |
| Store’s superseded-name lookup | Minus capped monitor (from the unrounded scores): ; . P3 fails; GR-3 holds. | |
| Whole-map lookup, capped / lifted | / | Descriptive whole-map matcher. |
| Obligations | store-written capped | store-written lifted | store’s name lookup | whole-map lookup | five edges untagged | native tags | tagged + summary | FR-Bank top-five | FR-Bank ground truth | |
|---|---|---|---|---|---|---|---|---|---|---|
| all (the printed rows) | ||||||||||
| retained at (recovered or stale-active) | ||||||||||
| recovered (a non-active mark on the edge) | ||||||||||
| stale-active | ||||||||||
| extraction miss | ||||||||||
| retained at |
| Condition | GPT-4.1 mini | Haiku 4.5 |
|---|---|---|
| Repaired top five | ||
| Obligation-gated stale fetch | ||
| Obligation-gated successor assembly | ||
| Successor assembly without obligation gate | ||
| Matched depth |
LLM Agent Memory
Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.
See all 373 papers