When Retrieval Is Not Enough: Obsolete Facts in Agent Memory

Oleeviya Babu Poikarayil; Tony Basil Babu; Abdulrahman Nahhas; Het Darshan Mehta; Jawid Ahmad Baktash; Ernesto William De Luca; Klaus Turowski

cs.AIOct 26, 2026

When Retrieval Is Not Enough: Obsolete Facts in Agent Memory

Authors: Oleeviya Babu Poikarayil, Tony Basil Babu, Abdulrahman Nahhas, Het Darshan Mehta, Jawid Ahmad Baktash, Ernesto William De Luca, Klaus Turowski

Organizations: Faculty of Computer Science, Otto-von-Guericke-Universität Magdeburg, Germany · University of Kerala, India · Otto-von-Guericke-Universität Magdeburg, Germany · Technical University of Munich, Germany

Abstract

Memory-agent benchmarks often collapse retrieval and answer generation into one score, making it hard to tell whether a failure comes from missing an update or from not using an update that was already retrieved. We examine this distinction on the single-hop FactConsolidation task of MemoryAgentBench. With Qwen2.5-7B-Instruct on 6K-history BM25 accuracy drops from 45% to 32% when the retrieved evidence budget grows from 64 to 512 tokens, even though a gold-assisted post-hoc audit finds the newest statement in every retrieved set. Most of the errors occur on questions that also contain an obsolete statement version. Under a frozen prompt and equal four-statement evidence size, replacing one unrelated statement with the obsolete competing statement reduces accuracy from 100% to 44.6% for Qwen2.5 and from 100% to 87.8% for Qwen3-14B, with all 50 harmful paired flips classified as stale. A 32K-history replication likewise favors BM25-64 over BM25-512 (69% versus 58%). These results suggest that retrieving the correct fact is not enough. An obsolete competing statement can be more harmful than an unrelated piece of evidence of the same size.

Figures & tables

History	Budget	Accuracy	Conflict acc.	Gold/latest support	Mean stmts.
6K	64	45	25.7	100/100	4.17
	128	40	20.3	100/100	8.57
	256	36	16.2	100/100	16.99
	512	32	12.2	100/100	33.60
32K	64	69	55.6	100/100	3.96
32K	512	58	34.9	100/100	32.96

Table 1: Evidence-budget results. “Conflict” uses reconstructed chains. Gold-assisted support is chain-based at 6K and direct literal benchmark-gold presence at 32K after two ambiguous chain labels. The validated 32K conflict subset has 63/63 chain support at both budgets.

Figure 1: Equal-size intervention on 74 conflict questions. Replacing one unrelated statement with the obsolete target fact harms both models.

Question	Latest / obsolete	Larger or conflict answer	Audit verdict
Author of The Marriage of Figaro ?	Thomas Kyd / Pierre Beaumarchais	Pierre Beaumarchais	Direct stale selection
Sport associated with flanker?	rugby / rugby union	rugby union	Metric false positive; stale value
Position of Lisa Leslie?	goaltender / center	goalie	Semantic match; metric false negative

Table 2: Representative audited outputs. The benchmark truth is the later synthetic memory, not ordinary world knowledge.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Purpose	Setup	Size	Main observation	Status
02	Full-context smoke	Qwen2.5; full 6K history	5 questions	2/5 (40%)	Preliminary
03	First-five evidence audit	Inspect full-history outputs	5 questions	Gold/latest text present in failed cases	Diagnostic
04	Oracle prompt diagnostic	Question-only / latest-only / all relevant	3 x 3	0/3; 3/3; 3/3	Diagnostic
05	Full-context baseline	Qwen2.5; full 6K history	100 calls	37/100	Valid baseline
06	Question-only baseline	Qwen2.5; no supplied memory	100 calls	19/100	Valid baseline; prompt differs
07	Full vs. question-only pairing	Paired 05 and 06	100 pairs	19 both; 18 full-only; 0 question-only-only	Diagnostic

Appendix

Table 3: Complete experiment inventory. Numbering follows the saved result files.

Condition	$N$	Correct	Acc. (%)	Prompt tokens	Interpretation
Full-context smoke	5	2	40.0	6,978–6,981	Preliminary subset
Full 6K context	100	37	37.0	6,975–6,986	Valid baseline
Question only	100	19	19.0	–	Different prompt; context-free
Gold-assisted isolated	100	97	97.0	81–103	Diagnostic; gold-derived evidence and different prompt

Appendix

Table 4: Preliminary and baseline results.

Q	Question	Gold	Model answer	Correct	Gold shown
1	Which sport is goaltender associated with?	pesäpallo	ice hockey	no	3
2	What is Nobuhiro Watsuki famous for?	The Fairly OddParents	Nobuhiro Watsuki is famous for Rurouni Kenshin.	no	1
3	Which country was rugby union created in?	India	England	no	5
4	What is Ferdowsi famous for?	Shahnameh	Ferdowsi is famous for Shahnameh.	yes	2
5	Which university was Joan Didion educated at?	University of California, Berkeley	University of California, Berkeley	yes	3

Appendix

Table 5: First-five full-context evidence audit. “Gold shown" counts literal occurrences displayed by the audit, not relation-specific support.

Condition	Correct	Total	Accuracy (%)
question_only	0	3	0.0
latest_only	3	3	100.0
all_relevant_chronological	3	3	100.0

Appendix

Table 6: Three-question oracle diagnostic.

Figure 2: Accuracy as a function of BM25 evidence budget. The 6K source provides the full fourbudget sweep, 32K repeats the endpoints.

Lower	Higher	Both correct	Lower-only	Higher-only	Both wrong	Diff. (pt)	Exact $p$
64	128	36	9	4	51	5.0	0.266846
128	256	34	6	2	58	4.0	0.289062
256	512	30	6	2	62	4.0	0.289062
64	256	33	12	3	52	9.0	0.035156
64	512	31	14	1	54	13.0	0.000977
128	512	30	10	2	58	8.0	0.038574

Appendix

Table 8: All paired 6K BM25 budget comparisons.

Trajectory	Count	Trajectory	Count	Trajectory	Count
0000	49	1111	29	1000	7
1110	4	1100	3	0100	3
1001	2	0010	2	0111	1

Appendix

Table 9: Per-question correctness trajectories across budgets 64, 128, 256, and 512. A one denotes correct.

Q	Question	Gold/latest	Answer 64	Answer 512	Direction
1	Which sport is goaltender associated with?	pesäpallo	pesäpallo	ice hockey	64-only
14	What is the country of citizenship of Cédric Klapisch?	Malaysia	Malaysia	France	64-only
18	Which country was association football created in?	Italy	Italy	England	64-only
24	Who is the director of American Broadcasting Company?	Ernst Heinkel	Ernst Heinkel	Bob Iger	64-only
25	Which city did Gennady Rozhdestvensky die in?	United States of America	United States of America	Moscow	64-only
29	What type of music does Michael Mantler play?	post-punk	post-punk	jazz	64-only

Appendix

Table 10: All discordant 6K BM25-64 versus BM25-512 questions.

Condition	Conflict correct	Conflict acc.	No-conflict correct	No-conflict acc.	Gap
question_only	0/74	0.0	19/26	73.1	73.1
full_context	11/74	14.9	26/26	100.0	85.1
bm25_64	19/74	25.7	26/26	100.0	74.3
bm25_128	15/74	20.3	25/26	96.2	75.9
bm25_256	12/74	16.2	24/26	92.3	76.1
bm25_512	9/74	12.2	23/26	88.5	76.3

Appendix

Table 11: Complete 6K conflict stratification.

Stratum	Both correct	64-only	512-only	Both wrong	Net loss
conflict	8	11	1	54	10
no_conflict	23	3	0	0	3

Appendix

Table 12: Paired BM25-64 versus BM25-512 by chain stratum.

Condition	Group	Correct / $N$	Accuracy (%)
latest_only	all	53/100	53.0
latest_only	conflict	28/74	37.8
latest_only	no_conflict	25/26	96.2
latest_plus_3_unrelated	all	65/100	65.0
latest_plus_3_unrelated	conflict	39/74	52.7
latest_plus_3_unrelated	no_conflict	26/26	100.0

Appendix

Table 13: Exploratory controlled experiment under the earlier abstaining prompt.

Condition	Latest	Stale	Unknown	Other
latest_only	28	0	46	0
latest_plus_3_unrelated	39	0	34	1
old_plus_latest	11	16	45	2
old_plus_latest_plus_2_unrelated	9	27	38	0

Appendix

Table 14: Answer classifications for the 74 conflict questions under the earlier abstaining prompt.

Condition	Correct	Total	Accuracy (%)
latest_only	10	10	100.0
latest_plus_3_unrelated	10	10	100.0
old_plus_latest	5	10	50.0
old_plus_latest_plus_2_unrelated	3	10	30.0

Appendix

Table 15: Frozen-prompt pilot on the first 10 conflict questions.

Model	A acc.	B acc.	Both correct	A-only	B-only	Stale flips	Diff.	Exact $p$
Qwen2.5-7B	100.0	44.6	33	41	0	41	55.4	9.09495e-13
Qwen3-14B	100.0	87.8	65	9	0	9	12.2	0.00390625

Appendix

Table 16: Final frozen equal-size intervention. Condition A is latest plus three unrelated statements; B is old plus latest plus two unrelated statements.

Model	Q	Question	Latest	Obsolete	Answer A	Answer B
Qwen2.5-7B	2	What is Nobuhiro Watsuki famous for?	The Fairly OddParents	Rurouni Kenshin	The Fairly OddParents	Rurouni Kenshin
Qwen2.5-7B	6	Who is the author of The Marriage of Figaro?	Thomas Kyd	Pierre Beaumarchais	Thomas Kyd	Pierre Beaumarchais
Qwen2.5-7B	7	Which sport is Tunisia national football team associated with?	basketball	association football	basketball	association football
Qwen2.5-7B	8	Which sport is quarterback associated with?	Muay Thai	American football	Muay Thai	American football
Qwen2.5-7B	9	What is the official language of Japan?	Swedish	Japanese	Swedish	Japanese
Qwen2.5-7B	13	What position does Robert Parish play?	quarterback	center	quarterback	center

Appendix

Table 17: All 50 correct-to-stale flips in the final equal-size interventions.

Budget	Group	Correct/ $N$	Acc.	Auto support	Latest	Stale	Other	Mean stmts.
64	all	69/100	69.0	98/100	69	27	4	3.96
64	conflict	35/63	55.6	63/63	35	27	1	3.90
64	no_conflict	34/37	91.9	35/37	34	0	3	4.05
512	all	58/100	58.0	99/100	58	41	1	32.96
512	conflict	22/63	34.9	63/63	22	41	0	32.78
512	no_conflict	36/37	97.3	36/37	36	0	1	33.27

Appendix

Table 18: Complete 32K BM25 replication. Automated chain support is shown here; direct literal support is 100/100 for both budgets after manual audit.

Q	Question	Latest	Obsolete	Answer 64	Answer 512	Type
2	What position does Nick Rimando play?	flanker	goalkeeper	flanker	goalkeeper	stale
12	What is the country of citizenship of Colin Irwin?	Israel	United Kingdom	Israel	United Kingdom	stale
20	Which religion is Maria Amalia of Naples and Sicily affiliated with?	Church of Greece	Catholic Church	Church of Greece	Catholic Church	stale
21	What position does Tony Parker play?	shooting guard	point guard	shooting guard	point guard	stale
42	Who is the chief executive officer of Apple Inc.?	Vijay Mallya	Tim Cook	Vijay Mallya	Tim Cook	stale
48	Which continent is San Francisco located in?	Africa	North America	Africa	North America	stale

Appendix

Table 19: All 18 detrimental BM25-64 to BM25-512 flips in the 32K replication.

ID	Category	Model	Q	Latest	Obsolete	Answer A	Answer B
1	Qwen 2.5 correct-to-stale flip	Qwen2.5	2	The Fairly OddParents	Rurouni Kenshin	The Fairly OddParents	Rurouni Kenshin
2	Qwen 2.5 correct-to-stale flip	Qwen2.5	6	Thomas Kyd	Pierre Beaumarchais	Thomas Kyd	Pierre Beaumarchais
3	Qwen 2.5 correct-to-stale flip	Qwen2.5	9	Swedish	Japanese	Swedish	Japanese
4	Qwen 2.5 correct-to-stale flip	Qwen2.5	20	Steve Jobs	Satya Nadella	Steve Jobs	Satya Nadella
5	Qwen 2.5 correct-to-stale flip	Qwen2.5	35	Taipei	Jerusalem	Taipei	Jerusalem
6	Qwen 3 correct-to-stale flip	Qwen3	44	Oderzo	Canberra	Oderzo.	Canberra.

Appendix

Table 20: Complete 20-case manual audit.

Agent Memory

Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.

See all 191 papers

Beyond Downstream QA: Rethinking How We Evaluate Agent Memory
Oct 15, 2026Sebastian-Antonio Toma, Jean de Kernier, Mike Ply +1Agent Memory LLM Agent Memory
Amazon Web Services · Technical University of Cluj-Napoca, Romania
Temporal Validity in Retrieval Memory: Eliminating Stale-Fact Errors for AI Agents over Evolving Knowledge
Jun 25, 2026Neeraj YadavRetrieval-Augmented Generation Temporal IR
MemStrata.dev — Called It Inc. (Enterprise)
What Should an Agent Forget? Separating What Is Stored from What Is Used
Sep 9, 2026Yuhang Li, Yuchen LiPersistent Memory for Language Models Temporal IR
East China Normal University Shanghai, China

Oct 15, 2026J/K move · Enter open · S save

Beyond Downstream QA: Rethinking How We Evaluate Agent Memory

Sebastian-Antonio Toma, Jean de Kernier, Mike Ply, Oliviu Matei

Amazon Web Services · Technical University of Cluj-Napoca, Romania

Agent Memory LLM Agent Memory Memory Agent Benchmarks Agent Evaluation

Open paper AskPDF arXiv

Agent memory benchmarks evaluate whether an agent's responses reflect correct knowledge, measuring downstream question-answering accuracy as a proxy for memory quality. We show this evaluation paradigm is incomplete: it conflates memory correctness with retrieval and generation quality, producing both false positives where correct responses mask broken memory and false negatives where incorrect responses blame correct memory for retrieval failures. We introduce FEB-30, a benchmark of 30 fact-evolution trajectories spanning 6 diagnostic categories and 3 difficulty levels, constructed via paradigm inversion: documents are generated from known fact trajectories, yielding deterministic ground truth without human annotation. Each trajectory defines how a fact should evolve across multi-author artifacts (documents, emails, conversations), enabling direct inspection of the memory store after each ingestion step. Applying FEB-30 to a representative memory system, we demonstrate that intrinsic and extrinsic evaluation diverge by an average of 16 percentage points, with 44% of trajectories exceeding meaningful divergence thresholds. Diagnostic decomposition reveals that the system correctly classifies memory operations (create, update, reinforce, supersede) only 20% of the time, a failure entirely invisible to extrinsic evaluation, and that failure modes differ systematically by category and difficulty. We release the benchmark, document generation pipeline, and scoring framework to support reproducible evaluation of agent memory systems.

Abstract

Agent Memory

Explore similar work

Beyond Downstream QA: Rethinking How We Evaluate Agent Memory

Temporal Validity in Retrieval Memory: Eliminating Stale-Fact Errors for AI Agents over Evolving Knowledge

What Should an Agent Forget? Separating What Is Stored from What Is Used