Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability

Ankit Goyal; Jaideep Ray

cs.AISep 4, 2026

Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability

Authors: Ankit Goyal, Jaideep Ray

Organizations: LinkedIn

Abstract

Model upgrades are routine; memory migrations are not. An agent can keep the same memory store and still forget: a new model may interpret old notes differently, mixed embedding versions may break retrieval, and repair may fail without the original evidence. We compare memory as the same history is preserved verbatim for long-context reading (LC-RAW), divided into chunks for retrieval-augmented generation (RAG), compressed by a model into natural-language notes (NOTES), or normalized into a fixed-schema knowledge graph (KG-fixed). The study uses 48 synthetic histories with randomized answer codes, exact scoring, and two open-weight models with sub 10 billion parameters. Our measurements show that fixed-schema structures transfer reliably, with KG-fixed accuracy changing by only $+0.0004 \pm 0.0020$ following a writer swap. Conversely, compressed NOTES exhibit high model coupling, with accuracy shifting asymmetrically by $+9.91$ or $-13.28$ percentage points depending on the specific migration direction. In RAG systems, partial embedding migrations using a 50/50 mixed index capture only a 4.96-point accuracy improvement, forfeiting the majority of the 11.90-point gain achieved through full re-embedding. Diagnostic decomposition attributes 80% ( $0.467 \pm 0.014$ ) of the NOTES accuracy deficit to information lost during initial construction, whereas retrieval failures drive 81% ( $0.364 \pm 0.012$ ) of the RAG deficit. Finally, store-only repair of NOTES fails to reach a 90% performance recovery target in all 48 test cases, whereas retaining the raw source history enables successful recovery in 34 of 48 cases for one tested direction. These findings highlight the necessity of direction-specific migration testing, strict embedding space isolation, and the retention of source histories for memory repair.

Figures & tables

Figure 1: Comparison of the four memory formats. Each format preserves different information and introduces a different upgrade risk.

Figure 2: Study overview. Fixed synthetic histories are converted into four memory formats. The evaluation changes one component at a time—the model, embedding index, repair source, or diagnostic evidence—and reports exact-match accuracy, RPAS, and CTR.

Test	Contrast	A positive value means	Run ID
H1	$C_{\mathrm{H1}}$	NOTES loses retained performance after a writer swap.	E1
H2	$C_{\mathrm{H2}}$	The mixed index is less accurate than full re-embedding.	E3
H4a	$C_{\mathrm{H4a}}$	The NOTES writer-swap penalty exceeds the KG-fixed penalty.	E1
H7a	$C_{\mathrm{H7a}}$	Raw-history repair is more accurate than store-only repair at matched budgets.	E6

Table 1: Definitions of the four contrasts recorded in the signed Git tag. The Run ID column preserves the original artifact identifiers rather than the sequential presentation order used in Section 3.6 . Each test uses

H_{0}:C_{h}\leq 5~\mathrm{pp}

against

H_{A}:C_{h}>5~\mathrm{pp}

. Repeated directions, readers, and budget rungs are averaged within each history before histories are compared.

Figure 3: Key empirical results. NOTES transfer changes sharply by migration direction, a mixed embedding index leaves seven percentage points unrecovered, NOTES loss is concentrated in memory construction while RAG loss is concentrated in retrieval, and store-only NOTES repair never reaches the 90% target.

Test	One-sided alternative	Estimate	Margin vs. 5 pp	Holm $p$	Decision
H1	$C_{\mathrm{H1}}>5$	$-1.47$ pp	$-6.47$ pp	$1.0$	Not supported
H2	$C_{\mathrm{H2}}>5$	$+6.95$ pp	$+1.95$ pp	$1.3\times 10^{-6}$	Supported
H4a	$C_{\mathrm{H4a}}>5$	$-1.51$ pp	$-6.51$ pp	$1.0$	Not supported
H7a	$C_{\mathrm{H7a}}>5$	$+8.90$ pp	$+3.90$ pp	$3.9\times 10^{-7}$	Supported

Table 2: Results for the planned tests across 48 histories. “Margin vs. 5 pp” is the estimate minus the action threshold, so positive values clear the threshold. All estimates use the positive-support orientation defined in Table 1 . The displayed

p

-values are Holm-adjusted across the four one-sided tests. The later bootstrap gives the same decisions. Later, unplanned checks place H1 and H4a within

\pm 5

pp (

p=0.035

and

0.037

). H7a exceeds 5 pp but not 10 pp.

Memory format	Reader	Own-store accuracy	Inherited-store accuracy	Swap $\Delta$ (pp)
NOTES	Llama	0.3762 $\,{\scriptstyle\pm\,0.0164}$	0.4753 $\,{\scriptstyle\pm\,0.0214}$	$+9.91$
NOTES	Qwen	0.4719 $\,{\scriptstyle\pm\,0.0187}$	0.3391 $\,{\scriptstyle\pm\,0.0164}$	$-13.28$
KG-fixed	Llama	0.8456 $\,{\scriptstyle\pm\,0.0072}$	0.8445 $\,{\scriptstyle\pm\,0.0075}$	$-0.11$
KG-fixed	Qwen	0.9878 $\,{\scriptstyle\pm\,0.0036}$	0.9880 $\,{\scriptstyle\pm\,0.0024}$	$+0.02$
Writer-free controls (one accuracy per reader)
LC-RAW	Llama	0.7121 $\,{\scriptstyle\pm\,0.0084}$		—

Table 3: Accuracy across 48 histories, reorganized around migration impact. Own-store memory is written by the reader model; inherited-store memory is written by the other model. Swap

\Delta

is inherited minus own accuracy in percentage points, so negative values indicate migration loss. KG-fixed is stable, whereas NOTES changes sharply and asymmetrically.

Index	Answer accuracy	Gain vs. old (pp)	recall@k	MRR
Old embedding index	0.4257 $\,{\scriptstyle\pm\,0.0099}$	—	0.4846 $\,{\scriptstyle\pm\,0.0089}$	0.3237 $\,{\scriptstyle\pm\,0.0056}$
Full re-embed	0.5447 $\,{\scriptstyle\pm\,0.0093}$	$+11.90$	0.5998 $\,{\scriptstyle\pm\,0.0094}$	0.4068 $\,{\scriptstyle\pm\,0.0060}$
50/50 mixed index	0.4753 $\,{\scriptstyle\pm\,0.0088}$	$+4.96$	0.5208 $\,{\scriptstyle\pm\,0.0077}$	0.3643 $\,{\scriptstyle\pm\,0.0057}$
Ideal routing upper bound	0.5960 $\,{\scriptstyle\pm\,0.0086}$	$+17.03$	0.6505 $\,{\scriptstyle\pm\,0.0083}$	0.4390 $\,{\scriptstyle\pm\,0.0057}$

Table 4: Results of upgrading bge-large-en from v1.0 to v1.5 across 48 histories and two readers. The added impact column reports the accuracy gain over the old index. Full re-embedding gains 11.90 points, whereas the mixed index gains only 4.96. Ideal routing assumes that the system already knows which index holds the answer and is only an upper bound.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Test group	MAE	RMSE	Average bias	Slope
Unseen history	0.0945	0.1157	$+0.0001$ $\,{\scriptstyle\pm\,0.0095}$	0.849
Unseen model pair	0.1004	0.1217	$-0.0004$ $\,{\scriptstyle\pm\,0.0099}$	0.847
Unseen memory format	0.1203	0.1444	$+0.0310$ $\,{\scriptstyle\pm\,0.0115}$	0.821

Appendix

Table 5: How well the 20-probe check predicts accuracy on the other 140 probes (576 held-out predictions). It transfers across histories and model pairs, but is biased when applied to a different memory format.

Memory format	Total loss	Retrieval	Store construction	Reader residual
NOTES	$0.584$ $\,{\scriptstyle\pm\,0.013}$	$0.036$ $\,{\scriptstyle\pm\,0.009}$ (6%)	$\mathbf{0.467}$ $\,{\scriptstyle\pm\,0.014}$ ( 80% )	$0.081$ $\,{\scriptstyle\pm\,0.004}$ (14%)
RAG	$0.450$ $\,{\scriptstyle\pm\,0.012}$	$\mathbf{0.364}$ $\,{\scriptstyle\pm\,0.012}$ ( 81% )	$0.005$ $\,{\scriptstyle\pm\,0.005}$ (1%)	$0.081$ $\,{\scriptstyle\pm\,0.004}$ (18%)

Appendix

Table 6: Diagnostic decomposition of end-to-end loss, pooled across 48 histories. The three component means add to total loss; parenthetical percentages are descriptive ratios of pooled means. Separate confidence intervals for these ratios require joint history-level resampling and cannot be recovered from the marginal intervals alone, so the inferential quantities reported here are the absolute components and their 95% interval half-widths. KG-fixed is excluded because supplying isolated KG-fixed items performed worse than its normal traversal read, so the controlled condition was not a valid replacement for that access pattern.

Old writer: Llama; new reader and repairer: Qwen
	90% target		95% target		99% target
Repair method	Success	Med. cost	Success	Med. cost	Success	Med. cost
NOTES raw-retained	34/48	$0.76	28/48	$0.75	22/48	$0.75
NOTES store-only	0/48	—	0/48	—	0/48	—
RAG re-embed	48/48	$0.013	48/48	$0.013	48/48	$0.013
KG-fixed schema rebuild	48/48	$\approx$ $0	48/48	$\approx$ $0	45/48	$\approx$ $0

Appendix

Table 7: Repair success across 48 histories, with success counts and conditional median costs separated into columns. “

n/48

” is the number of histories that reached the target at any tested budget. Cost is the median cheapest cost among those

n

successful histories; dashes mark methods with no successful histories. Failures have

\mathrm{CTR}_{i}(X)=\infty

. Success sets differ across thresholds.

Explore similar work

Oct 2, 2026cs.CL

When Evidence Changes: Evaluating Memory Repair and Re-reading in Language-Model Agents

When documents supporting an agent's derived facts are revoked or replaced, should it repair memory or re-read current evidence? We introduce an evidence-revision evaluation on medication- and problem-list tasks from public ICU records. Under revocation, replacement and control events, we compare full and source-filtered re-reading with caching, rebuilding and graph-local repair across two 7B models. Memory is supplied in full without retrieval, and costs include ingest, revision and every use. On short records, local repair uses 5-10

\times

fewer revision tokens than rebuilding, yet every memory pipeline costs at least twice full re-reading in held-out conditions. In a small pre-specified development sweep, adding task-ineligible documents extended records to about 10,000 tokens; at that length, memory's mean cumulative cost fell below full re-reading's after 2-14 uses, partly through truncated extraction, while source-filtered re-reading remained cheapest. In the replacement study, none of the four primary confirmatory tests reached statistical significance. These results show why the cost of agent memory after evidence revision must be assessed against source-filtered re-reading over the full pipeline.

Wenhui Chu

University at Albany, State University of New York · Department of Cybersecurity College of Emergency Preparedness, Homeland Security and Cybersecurity University at Albany, State University of New York Albany, NY 12222, USA

Agent Memory Management LLM Agent Evaluation Memory Agent Benchmarks

Ask PDF

Jun 12, 2026cs.AI

GitOfThoughts: Version-Controlled Reasoning and Agent Memory You Can Replay, Diff, and Merge

Large language model reasoning leaves no trace once it is done. The steps of a chain of thought disappear when the context window closes, a pruned search branch is just gone, and memory buffers cannot be diffed, merged, or audited. Code, infrastructure, and experiments are all version-controlled. Reasoning is not. GitOfThoughts stores an agent's reasoning tree as a git repository. Every scored thought becomes a commit, scores become notes, outcomes become tags, and retrieval is just git log over the agent's own history. We use this to test something simple. Does giving an agent memory from past problems actually make it more accurate? We tried five memory stores (none, a markdown file, a vector database, a graph, and git) across two benchmarks, two model sizes, and several pre-registered repeat experiments. The answer, on new problems, is no, including one promising early result that did not hold up when we repeated it. Memory only helps once the problem being solved is nearly identical to something already in memory (cosine similarity above about 0.8); below that, it does nothing. In other words, the model is finding the answer rather than learning the method. Even a model 4.5x larger still cannot pull a reusable method out of a worked example; it just gets better at spotting near-copies. The only thing that reliably helped on new problems was generating several answers and picking the most common one (self-consistency). So the case for using git as the memory store is not that it retrieves better. It is that it gives auditability, history, and the ability to merge two agents' memories, at no cost to accuracy.

Pavan C Shekar, Abhishek H S, Aswanth Krishnan

QpiAI, Bengaluru, India

LLM Agent Memory Persistent Memory for Language Models Agent Memory Management

Ask PDF

May 13, 2026cs.AI

Useful Memories Become Faulty When Continuously Updated by LLMs

Learning from past experience benefits from two complementary forms of memory: episodic traces -- raw trajectories of what happened -- and consolidated abstractions distilled across many episodes into reusable, schema-like lessons. Recent agentic-memory systems pursue the consolidated form: an LLM rewrites past trajectories into a textual memory bank that it continuously updates with new interactions, promising self-improving agents without parameter updates. Yet we find that such consolidated memories produced by today's LLMs are often faulty even when derived from useful experiences. As consolidation proceeds, memory utility first rises, then degrades, and can fall below the no-memory baseline. More surprisingly, even when consolidating from ground-truth solutions, GPT-5.4 fails on 54% of a set of ARC-AGI problems it had previously solved without memory. We trace the regression to the consolidation step rather than the underlying experience: the same trajectories yield qualitatively different memories under different update schedules, and an episodic-only control that simply retains those trajectories remains competitive with the consolidators we test. In a controlled ARC-AGI Stream environment that exposes Retain, Delete, and Consolidate actions, agents preserve raw episodes by default and double the accuracy of their forced-consolidation counterparts; disabling consolidation entirely (episodic management only) matches this auto regime. Practically, robust agent memory should treat raw episodes as first-class evidence and gate consolidation explicitly rather than firing it after every interaction. Looking forward, reliable agentic memory will require LLMs that can consolidate without overwriting the evidence they depend on.

Dylan Zhang, Yanshan Lin, Zhengkun Wu +4

University of Illinois Urbana-Champaign · IIIS, Tsinghua University

AI Agent Reliability Persistent Memory for Language Models Episodic Memory Memory Consolidation Agent Memory Management LLM Agent Reliability

Ask PDF