MemShift: Separating Detection from Repair in LLM-Derived Personal Memory

Zichuan Xu; Haochen Zhang; Yuyang Dong; Makoto Onizuka; Chuan Xiao

cs.CLNov 2, 2026

MemShift: Separating Detection from Repair in LLM-Derived Personal Memory

Authors: Zichuan Xu, Haochen Zhang, Yuyang Dong, Makoto Onizuka, Chuan Xiao

Organizations: Osaka University · Initial S · Nagoya University

Abstract

AI assistants store values that a language model derived from a person's records under the person's own definitions, such as their important contacts. Keeping such a value correct takes two steps: detection, noticing that an input changed, and repair, producing the correct new value. Benchmarks that score only the final answer cannot tell which step failed, and they state every change as new text. Seeing this limitation, we present MemShift, a benchmark of five shifts: source revision, deletion, redefinition, writer switch, and export. The first three fork a synthetic world into branches that differ only in the shift. Gold values and dependencies are fixed before any text, and a deterministic scorer replaces an LLM judge. For a memory that reports its invalidations, MemShift shows for each changed item whether the memory noticed the change and whether its repair was right. Moreover, we build a reference memory with known detection and repair to validate the measure. By evaluating four open-weight LLMs, we find that detection helps only as much as repair allows: the LLMs re-derive few values computed by the person's definitions correctly, and programs that the LLM writes over typed records close most of the gap to exact recomputation. Code and data are available at https://github.com/zichuanxu/memshift.

Figures & tables

Figure 1: Detection and repair are separate steps. (a) One definition item and the five shifts. A source revision and a deletion change records, and a redefinition, a writer switch, and an export add no text. (b) The four HoldMem cells $(D,B)$ after a redefinition. With a key ( $B{=}1$ ), the memory invalidates the item and re-derives it. Without a key ( $B{=}0$ ), it keeps the old list and derives a new item only when a new record arrives. The writer derives a wrong list ( $D{=}0$ ), and exact recomputation the right one ( $D{=}1$ ). Each bar splits the changed definition items as in Equation 1 : right (gray at $D{=}0$ , blue at $D{=}1$ ), wrong after a re-derivation (red), and wrong after a missed change (amber). $m$ : share of changes missed. $r$ : share of re-derived items that are right.

Figure 2:

B{\times}D

on definition items against the share of programs right before the shift (a) and against

r_{0}

(b). Computing the value helps most when the programs are right (a) and the writer’s own re-derivation is poor (b). (a) The 18 program sets of the 6 writers after a redefinition, one square per set. Gray hulls enclose the sets of Qwen3.5-9B, of gemma-4-12B, and of Llama-3.1-8B and gpt-oss-20b together. Blue line and band: exact recomputation and its 95% interval. (b)

B{\times}D

with exact recomputation for three writers of definition items after a redefinition (squares) and a source revision (circles). Bars: 95% intervals.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Update and Stability of each native memory system and of the HoldMem cell with its inferred setting, after each shift. Wide gray bars: the cell’s estimate, with its 95% interval as a band in the color of its

B

arm. Narrow bars: the native system, filled if its estimate lies inside the cell’s interval and outlined if not. Error bars: 95% intervals.

Figure 4:

B{\times}D

on Update against

r_{1}-r_{0}

of the same cell, one marker per writer, item class, and shift (source key). Hollow: the replication, held out of the least-squares fit over the four writers. Band: the fit’s 95% bootstrap band.

\rho

in the key: Spearman correlation within the item class. Inset: the framed region near the origin. A definition-item marker lies below the diagonal by the effect of

D

B{=}0

Figure 5: The changed item-times the citation key missed since the fork, by cause, per shift and item class, in the main study (20 worlds) and the replication (20 worlds). Most come from records that no set recorded at write time can contain. Orange: a record created after the derivation (

b

new), or records that left a time window or a narrowed scope (

c

). Gray: a record of the dependency set that changed and that the writer had not cited (

a

), or an existing record that entered the scope (

b

old). The classes are exclusive.

n

: the number missed. %: the orange share.

Figure 6: Effect of

B

(source key) on Update at

D{=}0

and

D{=}1

, and

B{\times}D

, by shift and item class, with Qwen3.5-9B as writer and reader. On definition items, the key helps mainly when exact recomputation repairs them. Bars: main study (20 worlds), with its

B{\times}D

on definition items printed above. Diamonds: replication on 20 fresh worlds. Error bars: 95% intervals over worlds. In brackets: the class’s share of the changed item-times in the main study.

Figure 7:

B{\times}D

on Update with each reader of the same stores, and its store part, by item class and shift (exact recomputation as

D{=}1

, source key). The stored values carry the interaction. Qwen3.5-9B loses it on the other items, and Claude Sonnet 5.5 keeps part of it. Wide gray bars: the store part, the same for both readers, with each reader’s bar in front of it. The gap between the top of the gray bar and a reader’s bar is what that reader adds or loses, printed for the other items after a redefinition. Error bars: 95% intervals, the store part’s between the two readers.

Figure 8: No-store BM25 against HoldMem (1,1) over horizons. (a) Accuracy before any shift, with 95% intervals as bands. The bracket gives the paired difference at twelve months with its 95% interval. (b) Tokens per query at twelve months, split into serving (dark) and amortized ingestion and maintenance (light). The bracket gives the ratio of the two totals. Oracle context was not costed and is not in (b). The axis breaks before the cost ceiling (dashed), which bounds serving at half of one full re-read, and the gray region beyond it fails the ceiling.

Figure 9: Survival of a switch against writer agreement, for the 2,446

D{=}0

B{=}1

items pooled over the three tiers. Points: the survival of each agreement bin (edges at 50, 80, 90, and 95%) at its mean agreement. The strip under each axis gives the items per bin, and its tint deepens with the bin’s share of the items. Line and band: the pooled logistic fit and its 90% band. Dashed: perfect calibration. The card in (a) gives the calibration slope of the fit and the AUROC of agreement with 90% intervals. (b) magnifies agreement from 80 to 100%, where nearly all items lie, and prints the survival of each bin.

Memory-Augmented Language Models

Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.

See all 300 papers

EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems
Sep 3, 2026Zeyu Liu, Jian Zhong, Rongduo Han +12LLM Memory LLM Agent Memory
Nankai University · vivo AI Lab · Tsinghua University +1
ConMem: Counterfactual Multi-Scenario Probing for Resolving LLM Memory Misalignment
Oct 24, 2026Shuning Zhang, Tenghao Ji, Jingruo Chen +3Memory Management for LLM Agents Long-Term Conversational Memory
Tsinghua University · Independent Researcher · Cornell University +2
MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use
Aug 20, 2026Mengru Wang, Haozhe Luo, Zhenqian Xu +6Memory-Augmented Language Models LLM Memory
Zhejiang University · National University of Singapore · Northeastern University +2

Sep 3, 2026J/K move · Enter open · S save

EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems

Zeyu Liu, Jian Zhong, Rongduo Han, Ziyang Wu+11

Nankai University · vivo AI Lab · Tsinghua University+1

LLM Memory LLM Agent Memory Long-Term Conversational Memory Memory Agent Benchmarks

Open paper AskPDF arXiv

Long-horizon interactions with LLM-based assistants require memory systems that preserve and update user states, preferences, and interaction histories. Existing evaluations report end-to-end QA accuracy and cannot determine whether errors arise from encoding, retrieval, or generation. We introduce EvalMem, an operation-level diagnostic framework with three parallel Examiners. For each query, the Encoding Examiner checks whether the target fact is stored, the Retrieval Examiner assesses whether the native retriever returns usable evidence, and the Generation Examiner tests whether the model can answer from oracle evidence. Their outputs form fine-grained multi-label defect codes. To improve store-level diagnosis, we adapt agentic RAG with a recall-first strategy that searches using both the query and source evidence, increasing recall of present evidence on LoCoMo from 70.2% to 95.6%. Evaluations of seven memory systems on LoCoMo, LongMemEval-S, and dynamic DynaMem-Bench identify retrieval as the most frequently attributed failure layer; in default LoCoMo, retrieval defects reach 22.1%, compared with 7.7% for encoding and 6.5% for generation. Guided by this diagnosis, MemWiki, a search-friendly auxiliary structure built from each system's memory export, improves mean accuracy by 2.5 and 2.3 percentage points on LoCoMo and LongMemEval-S, respectively.

Abstract

Memory-Augmented Language Models

Explore similar work

EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems

ConMem: Counterfactual Multi-Scenario Probing for Resolving LLM Memory Misalignment

MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use