Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability

Ankit Goyal; Jaideep Ray

cs.AISep 4, 2026

Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability

Authors: Ankit Goyal, Jaideep Ray

Organizations: LinkedIn

Abstract

Model upgrades are routine; memory migrations are not. An agent can keep the same memory store and still forget: a new model may interpret old notes differently, mixed embedding versions may break retrieval, and repair may fail without the original evidence. We compare memory as the same history is preserved verbatim for long-context reading (LC-RAW), divided into chunks for retrieval-augmented generation (RAG), compressed by a model into natural-language notes (NOTES), or normalized into a fixed-schema knowledge graph (KG-fixed). The study uses 48 synthetic histories with randomized answer codes, exact scoring, and two open-weight models with sub 10 billion parameters. Our measurements show that fixed-schema structures transfer reliably, with KG-fixed accuracy changing by only $+0.0004 \pm 0.0020$ following a writer swap. Conversely, compressed NOTES exhibit high model coupling, with accuracy shifting asymmetrically by $+9.91$ or $-13.28$ percentage points depending on the specific migration direction. In RAG systems, partial embedding migrations using a 50/50 mixed index capture only a 4.96-point accuracy improvement, forfeiting the majority of the 11.90-point gain achieved through full re-embedding. Diagnostic decomposition attributes 80% ( $0.467 \pm 0.014$ ) of the NOTES accuracy deficit to information lost during initial construction, whereas retrieval failures drive 81% ( $0.364 \pm 0.012$ ) of the RAG deficit. Finally, store-only repair of NOTES fails to reach a 90% performance recovery target in all 48 test cases, whereas retaining the raw source history enables successful recovery in 34 of 48 cases for one tested direction. These findings highlight the necessity of direction-specific migration testing, strict embedding space isolation, and the retention of source histories for memory repair.

Figures & tables

Figure 1: Comparison of the four memory formats. Each format preserves different information and introduces a different upgrade risk.

Figure 2: Study overview. Fixed synthetic histories are converted into four memory formats. The evaluation changes one component at a time—the model, embedding index, repair source, or diagnostic evidence—and reports exact-match accuracy, RPAS, and CTR.

Test	Contrast	A positive value means	Run ID
H1	$C_{\mathrm{H1}}$	NOTES loses retained performance after a writer swap.	E1
H2	$C_{\mathrm{H2}}$	The mixed index is less accurate than full re-embedding.	E3
H4a	$C_{\mathrm{H4a}}$	The NOTES writer-swap penalty exceeds the KG-fixed penalty.	E1
H7a	$C_{\mathrm{H7a}}$	Raw-history repair is more accurate than store-only repair at matched budgets.	E6

Table 1: Definitions of the four contrasts recorded in the signed Git tag. The Run ID column preserves the original artifact identifiers rather than the sequential presentation order used in Section 3.6 . Each test uses

H_{0}:C_{h}\leq 5~\mathrm{pp}

against

H_{A}:C_{h}>5~\mathrm{pp}

. Repeated directions, readers, and budget rungs are averaged within each history before histories are compared.

Figure 3: Key empirical results. NOTES transfer changes sharply by migration direction, a mixed embedding index leaves seven percentage points unrecovered, NOTES loss is concentrated in memory construction while RAG loss is concentrated in retrieval, and store-only NOTES repair never reaches the 90% target.

Test	One-sided alternative	Estimate	Margin vs. 5 pp	Holm $p$	Decision
H1	$C_{\mathrm{H1}}>5$	$-1.47$ pp	$-6.47$ pp	$1.0$	Not supported
H2	$C_{\mathrm{H2}}>5$	$+6.95$ pp	$+1.95$ pp	$1.3\times 10^{-6}$	Supported
H4a	$C_{\mathrm{H4a}}>5$	$-1.51$ pp	$-6.51$ pp	$1.0$	Not supported
H7a	$C_{\mathrm{H7a}}>5$	$+8.90$ pp	$+3.90$ pp	$3.9\times 10^{-7}$	Supported

Table 2: Results for the planned tests across 48 histories. “Margin vs. 5 pp” is the estimate minus the action threshold, so positive values clear the threshold. All estimates use the positive-support orientation defined in Table 1 . The displayed

p

-values are Holm-adjusted across the four one-sided tests. The later bootstrap gives the same decisions. Later, unplanned checks place H1 and H4a within

\pm 5

pp (

p=0.035

and

0.037

). H7a exceeds 5 pp but not 10 pp.

Memory format	Reader	Own-store accuracy	Inherited-store accuracy	Swap $\Delta$ (pp)
NOTES	Llama	0.3762 $\,{\scriptstyle\pm\,0.0164}$	0.4753 $\,{\scriptstyle\pm\,0.0214}$	$+9.91$
NOTES	Qwen	0.4719 $\,{\scriptstyle\pm\,0.0187}$	0.3391 $\,{\scriptstyle\pm\,0.0164}$	$-13.28$
KG-fixed	Llama	0.8456 $\,{\scriptstyle\pm\,0.0072}$	0.8445 $\,{\scriptstyle\pm\,0.0075}$	$-0.11$
KG-fixed	Qwen	0.9878 $\,{\scriptstyle\pm\,0.0036}$	0.9880 $\,{\scriptstyle\pm\,0.0024}$	$+0.02$
Writer-free controls (one accuracy per reader)
LC-RAW	Llama	0.7121 $\,{\scriptstyle\pm\,0.0084}$		—

Table 3: Accuracy across 48 histories, reorganized around migration impact. Own-store memory is written by the reader model; inherited-store memory is written by the other model. Swap

\Delta

is inherited minus own accuracy in percentage points, so negative values indicate migration loss. KG-fixed is stable, whereas NOTES changes sharply and asymmetrically.

Index	Answer accuracy	Gain vs. old (pp)	recall@k	MRR
Old embedding index	0.4257 $\,{\scriptstyle\pm\,0.0099}$	—	0.4846 $\,{\scriptstyle\pm\,0.0089}$	0.3237 $\,{\scriptstyle\pm\,0.0056}$
Full re-embed	0.5447 $\,{\scriptstyle\pm\,0.0093}$	$+11.90$	0.5998 $\,{\scriptstyle\pm\,0.0094}$	0.4068 $\,{\scriptstyle\pm\,0.0060}$
50/50 mixed index	0.4753 $\,{\scriptstyle\pm\,0.0088}$	$+4.96$	0.5208 $\,{\scriptstyle\pm\,0.0077}$	0.3643 $\,{\scriptstyle\pm\,0.0057}$
Ideal routing upper bound	0.5960 $\,{\scriptstyle\pm\,0.0086}$	$+17.03$	0.6505 $\,{\scriptstyle\pm\,0.0083}$	0.4390 $\,{\scriptstyle\pm\,0.0057}$

Table 4: Results of upgrading bge-large-en from v1.0 to v1.5 across 48 histories and two readers. The added impact column reports the accuracy gain over the old index. Full re-embedding gains 11.90 points, whereas the mixed index gains only 4.96. Ideal routing assumes that the system already knows which index holds the answer and is only an upper bound.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Test group	MAE	RMSE	Average bias	Slope
Unseen history	0.0945	0.1157	$+0.0001$ $\,{\scriptstyle\pm\,0.0095}$	0.849
Unseen model pair	0.1004	0.1217	$-0.0004$ $\,{\scriptstyle\pm\,0.0099}$	0.847
Unseen memory format	0.1203	0.1444	$+0.0310$ $\,{\scriptstyle\pm\,0.0115}$	0.821

Appendix

Table 5: How well the 20-probe check predicts accuracy on the other 140 probes (576 held-out predictions). It transfers across histories and model pairs, but is biased when applied to a different memory format.

Memory format	Total loss	Retrieval	Store construction	Reader residual
NOTES	$0.584$ $\,{\scriptstyle\pm\,0.013}$	$0.036$ $\,{\scriptstyle\pm\,0.009}$ (6%)	$\mathbf{0.467}$ $\,{\scriptstyle\pm\,0.014}$ ( 80% )	$0.081$ $\,{\scriptstyle\pm\,0.004}$ (14%)
RAG	$0.450$ $\,{\scriptstyle\pm\,0.012}$	$\mathbf{0.364}$ $\,{\scriptstyle\pm\,0.012}$ ( 81% )	$0.005$ $\,{\scriptstyle\pm\,0.005}$ (1%)	$0.081$ $\,{\scriptstyle\pm\,0.004}$ (18%)

Appendix

Table 6: Diagnostic decomposition of end-to-end loss, pooled across 48 histories. The three component means add to total loss; parenthetical percentages are descriptive ratios of pooled means. Separate confidence intervals for these ratios require joint history-level resampling and cannot be recovered from the marginal intervals alone, so the inferential quantities reported here are the absolute components and their 95% interval half-widths. KG-fixed is excluded because supplying isolated KG-fixed items performed worse than its normal traversal read, so the controlled condition was not a valid replacement for that access pattern.

Old writer: Llama; new reader and repairer: Qwen
	90% target		95% target		99% target
Repair method	Success	Med. cost	Success	Med. cost	Success	Med. cost
NOTES raw-retained	34/48	$0.76	28/48	$0.75	22/48	$0.75
NOTES store-only	0/48	—	0/48	—	0/48	—
RAG re-embed	48/48	$0.013	48/48	$0.013	48/48	$0.013
KG-fixed schema rebuild	48/48	$\approx$ $0	48/48	$\approx$ $0	45/48	$\approx$ $0

Appendix

Table 7: Repair success across 48 histories, with success counts and conditional median costs separated into columns. “

n/48

” is the number of histories that reached the target at any tested budget. Cost is the median cheapest cost among those

n

successful histories; dashes mark methods with no successful histories. Failures have

\mathrm{CTR}_{i}(X)=\infty

. Success sets differ across thresholds.

Explore similar work

When Evidence Changes: Evaluating Memory Repair and Re-reading in Language-Model Agents
Oct 2, 2026Wenhui ChuAgent Memory Management LLM Agent Evaluation
University at Albany, State University of New York · Department of Cybersecurity College of Emergency Preparedness, Homeland Security and Cybersecurity University at Albany, State University of New York Albany, NY 12222, USA
GitOfThoughts: Version-Controlled Reasoning and Agent Memory You Can Replay, Diff, and Merge
Jun 12, 2026Pavan C Shekar, Abhishek H S, Aswanth KrishnanLLM Agent Memory Persistent Memory for Language Models
QpiAI, Bengaluru, India
Useful Memories Become Faulty When Continuously Updated by LLMs
May 13, 2026Dylan Zhang, Yanshan Lin, Zhengkun Wu +4AI Agent Reliability Persistent Memory for Language Models
University of Illinois Urbana-Champaign · IIIS, Tsinghua University

Oct 2, 2026, cs.CLJ/K move · Enter open · S save

When Evidence Changes: Evaluating Memory Repair and Re-reading in Language-Model Agents

Wenhui Chu

University at Albany, State University of New York · Department of Cybersecurity College of Emergency Preparedness, Homeland Security and Cybersecurity University at Albany, State University of New York Albany, NY 12222, USA

Agent Memory Management LLM Agent Evaluation Memory Agent Benchmarks

Open paper AskPDF arXiv

When documents supporting an agent's derived facts are revoked or replaced, should it repair memory or re-read current evidence? We introduce an evidence-revision evaluation on medication- and problem-list tasks from public ICU records. Under revocation, replacement and control events, we compare full and source-filtered re-reading with caching, rebuilding and graph-local repair across two 7B models. Memory is supplied in full without retrieval, and costs include ingest, revision and every use. On short records, local repair uses 5-10

\times

fewer revision tokens than rebuilding, yet every memory pipeline costs at least twice full re-reading in held-out conditions. In a small pre-specified development sweep, adding task-ineligible documents extended records to about 10,000 tokens; at that length, memory's mean cumulative cost fell below full re-reading's after 2-14 uses, partly through truncated extraction, while source-filtered re-reading remained cheapest. In the replacement study, none of the four primary confirmatory tests reached statistical significance. These results show why the cost of agent memory after evidence revision must be assessed against source-filtered re-reading over the full pipeline.

Abstract

Explore similar work

When Evidence Changes: Evaluating Memory Repair and Re-reading in Language-Model Agents

GitOfThoughts: Version-Controlled Reasoning and Agent Memory You Can Replay, Diff, and Merge

Useful Memories Become Faulty When Continuously Updated by LLMs