When Evidence Changes: Evaluating Memory Repair and Re-reading in Language-Model Agents

Wenhui Chu

cs.CLOct 2, 2026

When Evidence Changes: Evaluating Memory Repair and Re-reading in Language-Model Agents

Authors: Wenhui Chu

Organizations: University at Albany, State University of New York · Department of Cybersecurity College of Emergency Preparedness, Homeland Security and Cybersecurity University at Albany, State University of New York Albany, NY 12222, USA

Abstract

When documents supporting an agent's derived facts are revoked or replaced, should it repair memory or re-read current evidence? We introduce an evidence-revision evaluation on medication- and problem-list tasks from public ICU records. Under revocation, replacement and control events, we compare full and source-filtered re-reading with caching, rebuilding and graph-local repair across two 7B models. Memory is supplied in full without retrieval, and costs include ingest, revision and every use. On short records, local repair uses 5-10 $\times$ fewer revision tokens than rebuilding, yet every memory pipeline costs at least twice full re-reading in held-out conditions. In a small pre-specified development sweep, adding task-ineligible documents extended records to about 10,000 tokens; at that length, memory's mean cumulative cost fell below full re-reading's after 2-14 uses, partly through truncated extraction, while source-filtered re-reading remained cheapest. In the replacement study, none of the four primary confirmatory tests reached statistical significance. These results show why the cost of agent memory after evidence revision must be assessed against source-filtered re-reading over the full pipeline.

Figures & tables

Figure 1: The shared harness at the small size. D1–D3 are the patient’s medication orders, problem list and laboratory results, with their sources; only pharmacy documents are allowed for the medication task. D4 is the event document: a donor patient’s pharmacy record revoked as misfiled after the first use (E1, E1b), a pharmacy order replaced by a version with one changed strength (E2), or an added non-allowed treatment record (E4, ehr ); E0 has no revision. Ingest runs one extraction call per document; each claim node (c) records the document version it was shown as its parent, and a per-document summary (s) and a global summary (G) complete the graph. A revision invalidates exactly the transitive successors of the revised version, except under the no-propagation control, which keeps old nodes eligible; the others differ in what they recompute (§ 4 ). Every use starts from an immutable snapshot that supplies the prompt and the read set validated at commit.

model	pipeline	strict completion		stale (E1)	retained (E1b)	unsupported (E4)	tokens/traj.
		E1	E4				E1	E4
Qwen	re-read	18	18	0	18/20	0	1827	1911
Qwen	cache	15	16	0	18/20	2	4129	4173
Qwen	rebuild	18	18	0	19/20	0	9717	6958
Qwen	local repair	17	17	0	18/20	1	7375	6971
Mistral	re-read	14	11	0	19/20	4	2018	2122

Table 1: Held-out run: medication use after the revision or, in E4, the added record (small size; 20 sealed patients per model; counts of 20; retained is shown as a fraction). Stale = unique-support-revoked key listed; retained = still-supported key present; unsupported = non-allowed E4 drug listed; tokens = mean prompt+completion per trajectory. All conditions with intervals: Table 3 (App. C ).

Figure 2: Length sweep (E1, 20 later uses; 8, 3, 5 and 6 patients at S, M, L and XL, each level on its own patients). (a, b) Per-use tokens relative to re-reading (mean of per-patient ratios, 95 %

t

-intervals), with levels placed at their re-reading tokens per use on a log scale; re-reading used 0.6k–10.1k tokens per use from S to XL. TF, RRTF: task-filtered local repair and re-reading (allowed sources only). (c) Mean cumulative token difference of cache, rebuild and local repair from re-reading at XL; markers at the observed break-even (filled Qwen, open Mistral).

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

model	memory pipeline	memory	re-read	discordant m./r.	$p$ (one-sided)	Holm thr.	rejected
Qwen	cache	2/20	0/20	2 / 0	0.25	0.0167	no
Qwen	rebuild	0/20	0/20	0 / 0	1.0	0.05	no
Qwen	local repair	1/20	0/20	1 / 0	0.5	0.025	no
Mistral	cache	1/20	4/20	1 / 4	0.9688	0.05	no
Mistral	rebuild	4/20	4/20	3 / 3	0.6562	0.0167	no
Mistral	local repair	4/20	4/20	3 / 3	0.6562	0.025	no

Appendix

Table 2: Pre-specified held-out E4 primary test: each memory pipeline is compared with re-reading on patient-level unsupported-source additions (20 sealed patients per model). The table reports the memory/re-read counts, discordant pairs (m./r. = memory-only/re-read-only), one-sided exact McNemar

p

values and Holm-adjusted thresholds (thr.) for the three within-model comparisons. Non-rejection is not evidence of equivalence.

model	cond.	config	strict [CI]	stale	unsup. [CI]	fabr.	miss.	ret.	dx	tokens
Qwen	E0	re-read	18 [0.68–0.99]	—	0 [0.00–0.17]	2	2	—	18	1764
Qwen	E0	cache	15 [0.51–0.91]	—	0 [0.00–0.17]	3	5	—	15	3797
Qwen	E0	rebuild	17 [0.62–0.97]	—	0 [0.00–0.17]	2	3	—	9	6295
Qwen	E0	local repair	19 [0.75–1.00]	—	0 [0.00–0.17]	1	1	—	9	6250
Qwen	E1	re-read	18 [0.68–0.99]	0	0 [0.00–0.17]	2	2	—	18	1827
Qwen	E1	cache	15 [0.51–0.91]	0	0 [0.00–0.17]	2	5	—	14	4129

Appendix

Table 3: Held-out run, full table: 20 sealed patients, small size, four conditions, four pipelines, two models; medication use after the revision point (E0 and E4 have no revision). config = pipeline; strict = strict completion; counts of 20 with exact 95 % Clopper–Pearson intervals for the proportion; stale is defined only for E1 and retained only for E1b (— = not defined in that condition); ret. = retained, i.e. still-supported key present after the E1b revocation; unsup., fabr. and miss. = patients whose list named a drug found only in non-allowed documents, named a drug found in no document, or lacked an expected key, respectively; dx = problem-list (diagnosis) use, scored here by the frozen leaf-match proxy; tokens = mean prompt+completion per trajectory.

model	cond.	pipeline	ingest	use prompt	use compl.	total	$\times$ re-read (total)	$\times$ re-read (use prompt)
Qwen	E0	re-read	0	1466	298	1764	1.00	1.00
Qwen	E0	cache	1540	1969	289	3797	2.15	1.34
Qwen	E0	rebuild	2862	3146	286	6295	3.57	2.15
Qwen	E0	local repair	2850	3118	283	6250	3.54	2.13
Qwen	E1	re-read	0	1517	310	1827	1.00	1.00
Qwen	E1	cache	1811	2017	301	4129	2.26	1.33

Appendix

Table 4: Held-out run, per-phase mean prompt+completion tokens per trajectory (20 sealed patients per cell; raw llm_call usage). use prompt / use compl. = prompt and completion tokens of the three use calls; the last two columns are the ratios to re-reading of the total and of the use-stage prompt tokens. Phase means are rounded separately, so phases can differ from the total by one token; ratios use unrounded means.

Figure 3: E1 development token costs, recomputed from raw call usage: eight patients per cell, shared across settings. Bars separate ingestion, initial use, revision and the two later uses; labels show total cost relative to re-reading within each panel. These are descriptive means, not independent replications or projected reuse curves. Exact values and call counts: Table 5 .

setting	pipeline	ingest	use 0	revise	use 1 +use 2	total	calls	inv./rebuilt
Qwen large	re-read	0	1578	0	2970	4548	3	—
Qwen large	cache	6159	2593	0	5060	13812	10.8	1/0
Qwen large	rebuild	10152	3772	9721	7404	31050	34	3/14.5
Qwen large	local repair	10253	3857	1296	7453	22858	20.5	3/1
Mistral large	re-read	0	1772	0	3345	5117	3	—
Mistral large	cache	8402	1699	0	3230	13331	10.8	1/0

Appendix

Table 5: E1 development cells: mean prompt+completion tokens per trajectory by phase (use 0 = initial use before the revision; use 1 , use 2 = the two later uses) and model calls; inv./rebuilt = derived nodes invalidated / recomputed at the revision (Fig. 1 ). Re-reading performs no ingest and no revision work. Phase means are rounded separately, so a row’s phases can differ from its total by one token.

		frozen	fidelity	fidelity	contract	amend. 1	amend. 2	amend. 2
model	pipeline	proxy	strict	lenient	strict	strict	strict	lenient
Qwen	RR	18	12	19	16	16	18	19
Qwen	C	14	10	19	13	13	15	19
Qwen	RB	9 ∗	6	13	6 ∗	6 ∗	6 ∗	10
Qwen	LR	9 ∗	6	12	6 ∗	6 ∗	6 ∗	9
Mistral	RR	15	11	19	13	13	15	19

Appendix

Table 6: Post-hoc lexical scorer-sensitivity analysis of held-out problem-list outputs (§ 6.5 ), condition E1 (20 sealed patients per model). RR = re-read, C = cache, RB = rebuild, LR = local repair. Successful uses /20; ∗ marks a memory pipeline separated from re-reading after Holm-12 correction over all four conditions within the model. These values are not a clinical or confirmatory quality endpoint. All conditions: Table 7 .

		frozen proxy				fidelity strict				fidelity lenient				contract strict
model	cond.	RR	C	RB	LR	RR	C	RB	LR	RR	C	RB	LR	RR	C	RB	LR
Qwen	E0	18	15	9 ∗	9 ∗	12	11	6	6	19	19	13	12	16	14	6 ∗	6 ∗
Qwen	E1	18	14	9 ∗	9 ∗	12	10	6	6	19	19	13	12	16	13	6 ∗	6 ∗
Qwen	E1b	18	15	9 ∗	9 ∗	12	11	6	6	19	19	13	12	16	14	6 ∗	6 ∗
Qwen	E4	18	15	11	9 ∗	12	10	8	6	19	18	14	12	16	12	8 ∗	6 ∗
Mistral	E0	15	7 ∗	3 ∗	2 ∗	11	4	2 ∗	1 ∗	19	12	13	14	13	5 ∗	1 ∗	1 ∗

Appendix

Table 7: Held-out problem-list outputs under seven scoring rules, all four conditions (20 sealed patients per model; post hoc), part A: frozen proxy, fidelity strict, fidelity lenient, contract strict. RR, C, RB, LR as in Table 6 ; ∗ = memory pipeline separated from re-reading after Holm-12 within the model; last row = number of significant comparisons per rule. Part B: Table 8 .

		amendment 1 strict				amendment 2 strict				amendment 2 lenient
model	cond.	RR	C	RB	LR	RR	C	RB	LR	RR	C	RB	LR
Qwen	E0	16	14	6 ∗	6 ∗	18	16	6 ∗	6 ∗	19	19	10	9
Qwen	E1	16	13	6 ∗	6 ∗	18	15	6 ∗	6 ∗	19	19	10	9
Qwen	E1b	16	14	6 ∗	6 ∗	18	16	6 ∗	6 ∗	19	19	10	9
Qwen	E4	16	12	8 ∗	6 ∗	18	14	9 ∗	7 ∗	19	17	11	9
Mistral	E0	13	5 ∗	1 ∗	1 ∗	15	6 ∗	1 ∗	1 ∗	19	11	12	14

Appendix

Table 8: Part B of Table 7 : amendment 1 strict, amendment 2 strict, amendment 2 lenient (same outputs, same conventions).

model	cond.	branch	scored	dx strict	dx lenient	med strict	missing/invalid
Qwen	E0	F	16+16	11/16	14/16	10/16	0/0
Qwen	E0	SF	16+16	11/16	15/16	13/16	0/0
Qwen	E0	FS	16+16	11/16	16/16	13/16	0/0
Qwen	E0	D	16+16	11/16	15/16	10/16	0/0
Qwen	E4	F	16+16	11/16	14/16	9/16	0/0
Qwen	E4	SF	16+16	10/16	15/16	14/16	0/0

Appendix

Table 9: Batch A: diagnosis record fidelity (dx strict / lenient, fidelity scorer) and medication strict completion (med strict) per model, condition and branch (F, SF, FS, D as in § 6.5 ).

model	cond.	A $-$ B	metric	A: succ. (CP 95 %)	B: succ. (CP 95 %)	disc. A/B	diff	$p$	Holm $p$
Primary family (Holm over 4): SF $-$ F on diagnosis record fidelity, strict
Qwen	E0	SF $-$ F	dx strict	11/16 (.41–.89)	11/16 (.41–.89)	1/1	+0.00	1.000	1.000
Qwen	E4	SF $-$ F	dx strict	10/16 (.35–.85)	11/16 (.41–.89)	1/2	$-$ 0.06	1.000	1.000
Mistral	E0	SF $-$ F	dx strict	6/16 (.15–.65)	8/16 (.25–.75)	0/2	$-$ 0.12	0.500	1.000
Mistral	E4	SF $-$ F	dx strict	7/16 (.20–.70)	8/16 (.25–.75)	1/2	$-$ 0.06	1.000	1.000
Secondary family, Qwen (Holm over 12): FS $-$ SF, SF $-$ D, F $-$ D on strict and lenient

Appendix

Table 10: Batch A: primary and secondary patient-paired comparisons on diagnosis record fidelity (fidelity scorer) and, in the last family, on medication strict completion. A

-

B = the two branches compared; CP = exact Clopper–Pearson.

model	cond.	SF (amend. 2)	F (amend. 2)	disc. SF-only/F-only	diff.	exact $p$	Holm-4 $p$	SF/F (amend. 1)	SF/F (contract)	SF/F (fidelity)
Qwen	E0	13/16	13/16	1/1	+0.000	1.000	1.000	13/13	13/13	11/11
Qwen	E4	11/16	13/16	0/2	$-$ 0.125	0.500	1.000	11/13	11/13	10/11
Mistral	E0	4/16	10/16	0/6	$-$ 0.375	0.031	0.125	4/10	3/10	6/8
Mistral	E4	4/16	10/16	1/7	$-$ 0.375	0.070	0.211	4/10	3/10	7/8

Appendix

Table 11: Batch A under amendment 2 strict: SF vs F per model and condition (patient-paired exact McNemar; Holm-4; disc. = discordant pairs); amendment-1, contract and fidelity counts for reference, where SF/F gives the SF count and the F count of passing patients (two counts of 16, not a fraction).

seed	model	cond.	SF	F	disc. SF-only/F-only	diff.	exact $p$	Holm-8 $p$
1	Qwen	E0	11/16	11/16	1/1	+0.000	1.000	1.000
1	Qwen	E4	10/16	11/16	1/2	$-$ 0.0625	1.000	1.000
1	Mistral	E0	7/16	8/16	1/2	$-$ 0.0625	1.000	1.000
1	Mistral	E4	7/16	8/16	1/2	$-$ 0.0625	1.000	1.000
2	Qwen	E0	11/16	11/16	1/1	+0.000	1.000	1.000
2	Qwen	E4	10/16	11/16	1/2	$-$ 0.0625	1.000	1.000

Appendix

Table 12: Batch B: SF

-

F on fidelity-strict record fidelity per model, condition and seed (one Holm family of 8; disc. = discordant pairs). Medication strict completion, SF

-

F (Holm-8): seed 1 Qwen E0 13 vs 10 (Holm 1.000); seed 1 Qwen E4 13 vs 10 (Holm 1.000); seed 1 Mistral E0 14 vs 10 (Holm 1.000); seed 1 Mistral E4 11 vs 12 (Holm 1.000); seed 2 Qwen E0 13 vs 10 (Holm 1.000); seed 2 Qwen E4 14 vs 9 (Holm 0.500); seed 2 Mistral E0 14 vs 11 (Holm 1.000); seed 2 Mistral E4 12 vs 11 (Holm 1.000). Per-cell flips against seed 0 (problem list / medication): Qwen seed 1 1/128 / 6/128; Qwen seed 2 1/128 / 3/128; Mistral seed 1 2/128 / 5/128; Mistral seed 2 1/128 / 4/128.

seed	model	cond.	SF (amend. 2)	F (amend. 2)	disc. SF-only/F-only	diff.	exact $p$	Holm-8 $p$	SF/F (amend. 1)	SF/F (contract)	SF/F (fidelity)
1	Qwen	E0	13/16	13/16	1/1	+0.000	1.000	1.000	13/13	13/13	11/11
1	Qwen	E4	11/16	13/16	0/2	$-$ 0.125	0.500	1.000	11/13	11/13	10/11
1	Mistral	E0	4/16	10/16	1/7	$-$ 0.375	0.070	0.562	4/10	3/10	7/8
1	Mistral	E4	4/16	10/16	1/7	$-$ 0.375	0.070	0.562	4/10	3/10	7/8
2	Qwen	E0	13/16	13/16	1/1	+0.000	1.000	1.000	13/13	13/13	11/11
2	Qwen	E4	11/16	13/16	0/2	$-$ 0.125	0.500	1.000	11/13	11/13	10/11

Appendix

Table 13: Batch B under amendment 2 strict per seed (Holm-8; disc. = discordant pairs); amendment-1, contract and fidelity counts for reference, where SF/F gives the SF count and the F count of passing patients (two counts of 16, not a fraction).

model	configuration	success	new adopted	old retained	both present	others retained
Qwen	no-prop.	0/13 (0.00–0.25)	0/13	8/13	0/13	12/13
Qwen	quarantine	0/13 (0.00–0.25)	0/13	0/13	0/13	10/13
Qwen	re-read	9/13 (0.39–0.91)	13/13	0/13	0/13	9/13
Qwen	rebuild	7/13 (0.25–0.81)	8/13	0/13	0/13	12/13
Qwen	local repair	8/13 (0.32–0.86)	9/13	0/13	0/13	12/13
Mistral	no-prop.	0/13 (0.00–0.25)	0/13	8/13	0/13	11/13

Appendix

Table 14: Replacement batch: 13 development patients per model, one decoding; configuration = pipeline, no-prop. = no-propagation; exact Clopper–Pearson 95 % intervals on success; patient-paired exact McNemar with Holm correction as pre-declared; both present = old and new value listed together. Development evidence; counterfactual field changes; non-significance is not equivalence.

stratum	model	pipeline	adopted	old kept	others ret.	success	success ${}_{\text{strict}}$
A	Qwen	no-prop.	0/8	5/8	7/8	0/8	0/8
A	Qwen	quarantine	0/8	0/8	7/8	0/8	0/8
A	Qwen	re-read	8/8	0/8	8/8	8/8	8/8
A	Qwen	rebuild	5/8	0/8	7/8	4/8	4/8
A	Qwen	local repair	4/8	0/8	6/8	3/8	3/8
B	Qwen	no-prop.	0/11	8/11	11/11	0/11	0/11

Appendix

Table 15: Confirmatory replacement run: per-stratum, per-model counts (stratum A: 8 unused calibration patients; stratum B: 11 held-out patients, second use). No-prop. = no-propagation. Adopted = new strength value listed; old kept = superseded value listed; others ret. = every other valid medication identity retained; success = adopted

\wedge

others retained; success

{}_{\text{strict}}

additionally excludes unsupported identities. Strata are never pooled.

Figure 4: C2 adoption and adoption with retention of all other valid medication identities, with exact 95 % Clopper–Pearson intervals. A: 8 previously unused protocol patients; B: 11 reused patients. Counts, models and strata are never pooled. Intervals describe individual proportions; displayed

p

values use the paired one-sided adoption test, and no primary null hypothesis was rejected at

\alpha=0.05

. RR = re-reading, LR = local repair. No-propagation and quarantine have zero adoption; full counts and the stricter success definition appear in Table 15 .

batch	patients	events	pipelines / branches	traj.	GPU-s	role
screening (2 sizes)	8 dev	E0, E1	re-read, cache, rebuild, local	256	4,206	dev.
E1b/E4 controls (2 sizes)	same 8 dev	E1b, E4	same four	256	4,136	dev.
held-out (pre-specified)	20 sealed	E0, E1, E1b, E4	same four	640	5,050	confirm.
batch A same-facts ablation	16 dev	E0, E4	F, SF, FS, D	64 units	878	dev.
batch B decoding replay	same 16 dev	E0, E4	same; $T{=}0.2$ , seeds 1–2	128 units	632	dev.
batch C replacement	13 dev	E2	no-prop., quarantine, re-read, rebuild, local	130	1,696	dev.

Appendix

Table 16: Model-run batches that produced reported results, and the four failed 32B launches, with trajectories (traj.) and GPU-seconds (GPU-s). dev = development; calib. = calibration; local = local repair; no-prop. = no-propagation;

T

= temperature; batch-A/B unit = one snapshot per patient, condition and model; L1–L3 = EXP-L1–L3. Role: dev. = development evidence; p.s. = a separate analysis plan hashed in advance, as stated in the relevant appendix; confirm. = pre-specified confirmatory test; expl. = exploratory 32B supplement, run after the 7B results were known under a hashed plan (App. P ). Pilots, smoke tests, cancelled jobs and the serving checks are not listed; see the reconciliation below. Patients are distinct people; the 8 development targets are shared by the screening and E1b/E4 batches; the 16 batch-A and 13 batch-C patients are development patients (7 in common); batch E uses the 13 batch-C patients and 4 of the 8 screening targets; 20 sealed patients were used in the original held-out run, 11 of them again for C2, and all 20 again by the 32B supplements (batch ledger in App. P ).

model	pipeline	E2 success	E2 adopted	E2 old kept	E1 strict	E1 stale	calls	tokens
Qwen	re-read	9/13	13/13	0/13	6/8	0/8	3.0	1906
Qwen	rebuild	8/13	9/13	0/13	7/8	0/8	20.2	10021
Qwen	local repair	7/13	8/13	0/13	7/8	0/8	14.2	7679
Qwen	adapted selector ( $\lambda=0.3$ )	8/13	9/13	0/13	7/8	0/8	14.2	7679
Qwen	adapted selector ( $\lambda=10$ )	8/13	9/13	0/13	7/8	0/8	13.2	6768
Mistral	re-read	10/13	13/13	0/13	4/8	0/8	3.0	2121

Appendix

Table 17: Batch E (pre-specified development batch; 13 replacement and 8 revocation units per model from 17 distinct patients; memory pipelines forked from one shared ingest; one decoding). E2 success = new value adopted and other identities retained; calls and tokens are per trajectory with the shared ingest counted for every memory pipeline.

model	pipeline	fidelity		amendment 1		amendment 2
		strict	lenient	strict	lenient	strict	lenient
Qwen	re-read	15/21	20/21	18/21	20/21	18/21	20/21
Qwen	rebuild	13/21	19/21	14/21	19/21	14/21	19/21
Qwen	local repair	13/21	19/21	14/21	19/21	14/21	19/21
Qwen	adapted selector ( $\lambda=0.3$ )	13/21	19/21	14/21	19/21	14/21	19/21
Qwen	adapted selector ( $\lambda=10$ )	13/21	18/21	13/21	17/21	13/21	17/21

Appendix

Table 18: Batch E problem-list use under fidelity, amendment 1 and amendment 2 (strict and lenient): 21 condition–patient units per pipeline and model from 17 distinct patients (13 replacement + 8 revocation units); descriptive counts only, no test is computed on these pooled units.

Figure 5: Reuse sweep: cumulative tokens of each memory pipeline relative to re-reading,

C_{\mathrm{mem}}(k)/C_{\mathrm{reread}}(k)

, for

k=1

20

later uses (mean over eight paired development patients with 95 %

t

-intervals; one panel per model and size; dashed line at 1.0). No mean curve reaches 1.0.

model	size	pipeline	$k{=}1$	$k{=}5$	$k{=}10$	$k{=}20$	per-use $\Delta$ (tokens)
Qwen	large	cache	3.62 [3.31, 3.92]	2.33 [2.12, 2.54]	2.03 [1.84, 2.23]	1.86 [1.68, 2.04]	+1036.6
Qwen	large	local repair	6.23 [5.78, 6.68]	3.77 [3.47, 4.07]	3.20 [2.93, 3.47]	2.87 [2.62, 3.12]	+2275.5
Qwen	large	rebuild	8.90 [8.23, 9.57]	4.68 [4.30, 5.06]	3.70 [3.37, 4.02]	3.13 [2.84, 3.42]	+2265.4
Qwen	small	cache	2.66 [2.54, 2.78]	1.76 [1.69, 1.82]	1.54 [1.49, 1.60]	1.42 [1.37, 1.47]	+175.3
Qwen	small	local repair	4.86 [4.66, 5.07]	2.93 [2.78, 3.07]	2.47 [2.34, 2.60]	2.20 [2.08, 2.33]	+568.0
Qwen	small	rebuild	6.69 [6.46, 6.93]	3.56 [3.42, 3.71]	2.82 [2.69, 2.95]	2.39 [2.28, 2.51]	+573.2

Appendix

Table 19: Reuse sweep: cumulative cost relative to re-reading at

k=1,5,10,20

(mean [95 %

t

-interval] over eight paired patients) and the mean per-use token difference

U_{\mathrm{mem}}-U_{\mathrm{reread}}

over uses 1–20.

	pre-specified prediction	outcome
P1	At S, the per-use difference is positive for C, LR and RB in both models.	Holds: Qwen $+176$ , $+568$ , $+580$ ; Mistral $+230$ , $+681$ , $+689$ tokens per use.
P2	For C, LR and RB, the per-use ratio $U_{\mathrm{mem}}/U_{\mathrm{reread}}$ decreases from S to XL in both models.	Holds for S versus XL in 6 of 6 cells (Qwen 1.28–1.93 to 0.50–0.65; Mistral 1.34–2.03 to 0.14–0.23); a monotone reading fails for Qwen, whose M cell is above S for all three. Computed after the run (see text).
P3	At XL, the per-use difference is negative for C and LR in at least one model.	Holds in both models, and for RB; every 95 % interval excludes zero.
P4	Where both exist, $\|k^{}_{\mathrm{pred}}-k^{}_{\mathrm{obs}}\|\leq\max(2,0.25\,k^{*}_{\mathrm{obs}})$ .	Holds in 23 of 23 cells: 15 cells with a crossing after use 1 (largest error 0.9 uses) and 8 RRTF cells cheaper from the start ( $k^{}_{\mathrm{pred}}=0$ , $k^{}_{\mathrm{obs}}=1$ ).
P5	RRTF has the lowest cumulative cost at $k=20$ at every level in both models.	Holds in 8 of 8 model–level cells; largely implied by the design, which added only non-allowed documents.

Appendix

Table 20: Length sweep: the five pre-specified predictions and their outcomes (8, 3, 5 and 6 patients at S, M, L and XL). P1 and P3–P5 are the analysis program’s checks; P2 was computed afterwards from its per-cell output. RR = re-reading; C = cache; RB = rebuild; LR = local repair; TF = task-filtered local repair; RRTF = re-reading restricted to allowed sources.

		S	M patients ( $n=3$ )		L patients ( $n=5$ )		XL patients ( $n=6$ )
model	pipeline	all 8	S	M	S	L	S	XL
Qwen	RR tokens/use	623	592	1,232	652	4,156	630	9,584
	cache	1.28	1.31	1.45	1.28	0.95	1.30	0.50
	rebuild	1.93	1.86	2.09	1.93	1.30	1.93	0.64
	local repair	1.91	1.86	2.08	1.90	1.29	1.93	0.65
	TF	1.00	0.96	0.46	0.98	0.15	1.00	0.07

Appendix

Table 21: Length sweep, plan-specified subset report: re-reading tokens per use and per-use ratio to re-reading at S on all eight patients and on each longer level’s own patients, next to that level’s value (hashed program run unchanged on patient subsets; descriptive). RR, TF and RRTF as in Table 20 .

model	level	pipeline	$n$	ingest	use 0	revise	per use	total@20	per-use $\Delta$ [95% CI]	ratio@20 [95% CI]	$k^{}_{\mathrm{obs}}$ / $k^{}_{\mathrm{pred}}$	strict
Qwen	S	RR	8	0	684	0	623	13142	—	1.00	—	6/8
Qwen	S	C	8	1821	848	0	798	18639	+176 [+153, +198]	1.42 [1.37, 1.47]	none/none	4/8
Qwen	S	RB	8	3362	1280	2910	1202	31604	+580 [+492, +667]	2.40 [2.31, 2.50]	none/none	7/8
Qwen	S	LR	8	3366	1285	508	1191	28978	+568 [+473, +663]	2.21 [2.08, 2.33]	none/none	7/8
Qwen	S	TF	8	3366	702	507	623	17040	+0 [ $-$ 61, +62]	1.30 [1.20, 1.39]	none/none	8/8
Qwen	S	RRTF	8	0	505	0	446	9425	$-$ 177 [ $-$ 209, $-$ 145]	0.72 [0.66, 0.77]	1/0	8/8

Appendix

Table 22: Length sweep, full results as written by the hashed analysis program (Qwen here, Mistral on the next page; rows unchanged, split only for layout). Tokens are means over patients (prompt+completion); @

k

= after

k

later uses; per use = mean of uses 1–20; per-use

\Delta

(pipeline

-

RR) and ratio@20 are means of per-patient paired values with 95 %

t

-intervals against RR; use 0 = initial use; pipeline abbreviations as in Table 20 ;

k^{*}

as defined in the text (observed/predicted, the prediction rounded to the nearest use; “none”: not reached within 20 uses, or no negative per-use difference); strict = strict completion after the revision. Each column is rounded separately, so ingest + use 0 + revise + 20

\times

per use can differ from total@20 by rounding. M rows have

n=3

and are descriptive. No trajectory listed the revoked key or a non-allowed item (stale and unsupported-source listings 0 in every cell); no strict-completion comparison with RR was significant (Holm

p=1.0

in every cell).

model	level	pipeline	$n$	ingest	use 0	revise	per use	total@20	per-use $\Delta$ [95% CI]	ratio@20 [95% CI]	$k^{}_{\mathrm{obs}}$ / $k^{}_{\mathrm{pred}}$	strict
Mistral	S	RR	8	0	762	0	678	14320	—	1.00	—	4/8
Mistral	S	C	8	2365	963	0	908	21484	+230 [+209, +251]	1.51 [1.45, 1.56]	none/none	5/8
Mistral	S	RB	8	4126	1501	3551	1367	36517	+689 [+623, +755]	2.56 [2.42, 2.70]	none/none	4/8
Mistral	S	LR	8	4122	1492	610	1359	33404	+681 [+610, +753]	2.34 [2.22, 2.47]	none/none	5/8
Mistral	S	TF	8	4132	809	622	709	19743	+31 [ $-$ 22, +85]	1.38 [1.30, 1.46]	none/none	5/8
Mistral	S	RRTF	8	0	553	0	477	10091	$-$ 201 [ $-$ 237, $-$ 165]	0.70 [0.65, 0.76]	1/0	7/8

Appendix

Table 23: Length sweep, full results (continued): Mistral. Columns and conventions as on the previous page.

model	level	pipeline	no cache	$c=0.5$	$c=0.25$	$c=0.1$
Qwen	S	RR	1.00	1.00	1.00	1.00
Qwen	S	C	1.42	1.44	1.46	1.48
Qwen	S	RB	2.40	2.54	2.68	2.86
Qwen	S	LR	2.21	2.26	2.32	2.39
Qwen	S	TF	1.30	1.46	1.65	1.88
Qwen	S	RRTF	0.72	0.77	0.84	0.91

Appendix

Table 24: Length sweep, pre-specified cached-input pricing (pipeline abbreviations as in Table 20 ): cumulative tokens at

k=20

relative to re-reading when input tokens that repeat the previous call’s prefix cost

c

of an uncached token. Ratios of mean totals (not means of paired ratios), so the no-cache column can differ from Table 22 in the last digit. Mistral’s cached ratios (

c=0.5

0.25

0.1

) are not shown (—): the program counted Mistral’s repeated prefixes with the Qwen2.5 tokenizer, which favours memory, so those values are invalid and no claim uses them (App. O ). All other cells are as written by the hashed program.

model	level	pipeline	$n$	ratio@1	ratio@5	ratio@10	ratio@20	strict [95 % CP]	committed
Qwen	S	RR	8	—	—	—	—	6/8 [0.35–0.97]	8/8
Qwen	S	cache	8	2.66	1.76	1.54	1.42	4/8 [0.16–0.84]	8/8
Qwen	S	rebuild	8	6.71	3.57	2.83	2.40	7/8 [0.47–1.00]	8/8
Qwen	S	local repair	8	4.86	2.93	2.47	2.21	7/8 [0.47–1.00]	8/8
Qwen	S	TF	8	3.98	2.03	1.56	1.30	8/8 [0.63–1.00]	8/8
Qwen	S	RRTF	8	0.73	0.72	0.72	0.72	8/8 [0.63–1.00]	8/8

Appendix

Table 25: Length sweep, remaining pre-specified endpoints: cumulative tokens relative to re-reading at

k=1

, 5, 10 and 20 (means of per-patient paired ratios), strict completion with exact Clopper–Pearson (CP) 95 % intervals, and trajectories whose uses reached the commit gate (committed); RR, TF and RRTF as in Table 20 .

		original documents (D1–D4)			added documents
model	level	calls	at cap	parse errors	calls	at cap	parse errors
Qwen	S	152	0	0	0	0	0
Qwen	M	57	0	0	55	0	0
Qwen	L	95	0	0	105	35	0
Qwen	XL	114	0	0	155	75	0
Mistral	S	152	0	0	0	0	0

Appendix

Table 26: Length sweep, exploratory extraction diagnostic: ingest and revise extraction calls of cache, rebuild, local repair and TF combined, calls that stopped at the 2,048-token output cap, and outputs that failed to parse after the labelled repair, for the original and the added documents.

setting	pipeline	ingest	use 0	revise	later uses	total	calls	inv./rebuilt	$\times$ re-read
Qwen2.5-32B small	re-read	0	699	0	1162	1861	3.0	—	1.00
Qwen2.5-32B small	cache	1878	853	0	1499	4230	7.0	1/0	2.27
Qwen2.5-32B small	rebuild	3433	1286	2961	2304	9984	19.0	3/7	5.36
Qwen2.5-32B small	local repair	3428	1279	516	2301	7524	13.0	3/1	4.04
Qwen2.5-32B large	re-read	0	1598	0	2976	4575	3.0	—	1.00
Qwen2.5-32B large	cache	6098	2545	0	4904	13546	10.8	1/0	2.96

Appendix

Table 27: 32B supplement (E1, eight development patients per cell): mean prompt+completion tokens per trajectory by phase (use 0 = initial use before the revision), model calls, derived nodes invalidated / recomputed at the revision (as in Table 5 ), and total relative to re-reading at the same size. Phase means are rounded separately, so phases can differ from the total by one token.

cell	pipeline	adopted	old kept	others ret.	success	success ${}_{\text{strict}}$
dev13	no-prop.	0/13	3/13	12/13	0/13	0/13
dev13	quarantine	0/13	0/13	13/13	0/13	0/13
dev13	re-read	13/13	0/13	13/13	13/13	13/13
dev13	rebuild	2/13	0/13	12/13	2/13	2/13
dev13	local repair	4/13	0/13	13/13	4/13	4/13
calib8	no-prop.	0/8	4/8	8/8	0/8	0/8

Appendix

Table 28: 32B replacement supplement (E2; frozen field-level scorer): per-cell counts for the five batch-C pipelines; dev13, calib8 and sealed11 = 13 development, 8 calibration and 11 sealed patients; no-prop. = no-propagation. Adopted = new strength value listed; old kept = superseded value listed; others ret. = every other valid medication identity retained; success = adopted

\wedge

others retained; success

{}_{\text{strict}}

additionally excludes unsupported identities. Cells are never pooled.

stratum	cond.	pipeline	n	committed	strict	stale	retained	unsupported	tokens
A	E0	re-read	6	6	6	—	—	—	1612
A	E0	cache	6	6	5	—	—	—	3672
A	E0	rebuild	6	6	5	—	—	—	5910
A	E0	local repair	6	6	5	—	—	—	5908
A	E1	re-read	6	6	6	0	—	—	1675
A	E1	cache	6	6	5	0	—	—	3985

Appendix

Table 29: 32B revocation supplement (small size; frozen medication grade): per-stratum, per-condition counts for the four pipelines (stratum A = 6 calibration, B = 20 sealed patients; cond. = condition). committed = uses that reached the commit gate; strict = listed generic-key set equals the expected set; stale = unique-support-revoked key listed (E1); retained = still-supported key present (E1b); unsupported = the non-allowed E4 drug listed; — = endpoint not defined in that condition; tokens = mean prompt+completion per trajectory. Strata are never pooled.

Explore similar work

Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability
Sep 4, 2026Ankit Goyal, Jaideep RayAgent Memory Management Persistent Memory for Language Models
LinkedIn
TEPA: Revoking Stale Memories for Conflict-Robust Language Agents
Aug 7, 2026Yan Zhou, Yue Ouyang, Kaiyang Zheng +1LLM Agent Memory Persistent Memory for Language Models
School of Mathematics and Statistics, Changsha University of Science and Technology, Changsha, China · School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China.
RECON: Benchmarking Agent Memory for Compositional Reasoning over Long Contexts
Jul 18, 2026Mihir Shriniwas AryaCompositional Reasoning LLM Agent Memory
Department of Computer Science and Engineering RV College of Engineering

Sep 4, 2026, cs.AIJ/K move · Enter open · S save

Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability

Ankit Goyal, Jaideep Ray

Agent Memory Management Persistent Memory for Language Models RAG Robustness

Open paper AskPDF arXiv

Model upgrades are routine; memory migrations are not. An agent can keep the same memory store and still forget: a new model may interpret old notes differently, mixed embedding versions may break retrieval, and repair may fail without the original evidence. We compare memory as the same history is preserved verbatim for long-context reading (LC-RAW), divided into chunks for retrieval-augmented generation (RAG), compressed by a model into natural-language notes (NOTES), or normalized into a fixed-schema knowledge graph (KG-fixed). The study uses 48 synthetic histories with randomized answer codes, exact scoring, and two open-weight models with sub 10 billion parameters. Our measurements show that fixed-schema structures transfer reliably, with KG-fixed accuracy changing by only

+0.0004 \pm 0.0020

following a writer swap. Conversely, compressed NOTES exhibit high model coupling, with accuracy shifting asymmetrically by

+9.91

-13.28

percentage points depending on the specific migration direction. In RAG systems, partial embedding migrations using a 50/50 mixed index capture only a 4.96-point accuracy improvement, forfeiting the majority of the 11.90-point gain achieved through full re-embedding. Diagnostic decomposition attributes 80% (

0.467 \pm 0.014

) of the NOTES accuracy deficit to information lost during initial construction, whereas retrieval failures drive 81% (

0.364 \pm 0.012

) of the RAG deficit. Finally, store-only repair of NOTES fails to reach a 90% performance recovery target in all 48 test cases, whereas retaining the raw source history enables successful recovery in 34 of 48 cases for one tested direction. These findings highlight the necessity of direction-specific migration testing, strict embedding space isolation, and the retention of source histories for memory repair.

Abstract

Explore similar work

Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability

TEPA: Revoking Stale Memories for Conflict-Robust Language Agents

RECON: Benchmarking Agent Memory for Compositional Reasoning over Long Contexts