Auditing Future-Memory Use in Point-in-Time Language Agents

Rongtian Ye

cs.AIOct 13, 2026

Auditing Future-Memory Use in Point-in-Time Language Agents

Authors: Rongtian Ye

Organizations: Department of Computer Science, Aalto University

Abstract

Point-in-time (PIT) agents can recall post-cutoff facts even when their external evidence respects a historical cutoff. We examine whether this access predicts evidence gathering in 48 historical company-comparison tasks. Independent closed-book revenue probes identify the better-remembered company for each model on tasks with unequal recall, fixing the predicted direction before agent runs. We then compare paired runs with real or anonymized company identities while keeping external evidence PIT-valid. Both Qwen and DeepSeek recalled future facts, but neither supported the prespecified shift toward the better-remembered company. Qwen's four allocation estimates on 28 unequal-recall tasks were negative, with descriptive intervals spanning zero. A prospective DeepSeek replication showed an inverse pattern across all four metrics on 20 such tasks, with intervals below zero. These estimates remained negative after each company, pair, and future-fact omission, and all four post-hoc max-T-adjusted sign-flip p-values were below .05. In a prospectively specified follow-up, explicit future facts produced small estimated shifts away from the supplied-fact company, with intervals spanning zero, while final choices shifted toward it. The mechanism behind the natural inverse pattern remains unresolved. These findings support separate assessments of memory accessibility, search allocation, and final outputs alongside cutoff checks on external evidence.

Figures & tables

Figure 1: Temporal integrity in PIT agents. The cutoff gate restricts external evidence to information available by time

t

. Independent memory probes specify a predicted direction of information seeking before the agent runs. The audit compares this prediction with logged actions and distinguishes data, policy, and output integrity.

Figure 2: Memory-aligned Normal–Anonymized allocation contrasts. Points show mean paired effects; bars show 95% descriptive task-bootstrap intervals. Positive values indicate the prespecified shift toward the better-remembered company. Qwen uses its original two-probe directions (n = 28); DeepSeek uses its prospectively frozen five-probe directions (n = 20).

Analysis	Status	Finding
Qwen original audit	Confirmatory; two-probe memory directions	Four negative N–A estimates ( $-0.046$ to $-0.031$ ); all descriptive intervals span zero.
DeepSeek replication	Prospective confirmatory audit; five-probe memory directions	Four negative N–A estimates ( $-0.190$ to $-0.104$ ); all descriptive intervals lie below zero.
Qwen five-probe expansion	Post-hoc measurement robustness	26/28 original directions retained; all reclassified allocation estimates remain negative.
DeepSeek inverse-pattern checks	Prespecified omissions; post-hoc sign-flip sensitivity	Every omission preserves the negative sign; max- $T$ -adjusted $p$ -values range from $.0036$ to $.0385$ .
Semantic re-identification	Post-hoc identity audit	Open-set entity recovery is 65.6% for Qwen and 79.2% for DeepSeek; directional associations vary across metrics.
Explicit-fact follow-up	Prospectively specified mechanism follow-up	Four small positive $W$ estimates ( $.011$ – $.039$ ), with intervals spanning zero and global max- $T$ $p=.1327$ ; final choices shift toward the supplied-fact company.

Table 1: Audit stages, analysis status, and principal findings.

Figure 3: Qwen Identity Cue diagnostic. (A) Mean outcomes for Anonymized (A), Identity Cue (C), and Normal (N). C–A and N–A differences share the same sign on five of six outcomes; parallelism shifts in the opposite direction. (B) Memory-aligned C–A allocation effects with 95% descriptive task-bootstrap intervals. All four intervals span zero.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Analysis	Design and conditions	Sample	Measurements
Qwen memory accessibility	Closed book; two paraphrases per fact, before policy runs	194 tasks; 126 facts; 252 probes	Recall within 5%; fact and task memory scores
Paired environments	N/A views of one canonical task	48 tasks; 19 companies; 32 pairs; 77 future facts	Temporal validity; identifier, numeric, and retrieval parity
Qwen policy audit	Paired N/A; original two-probe directions	96 valid trajectories; 28 asymmetric pairs	Memory-aligned K5, action, batch, and evidence contrasts
Familiarity diagnostics	Closed-book identity and pre-FY2018 revenue probes	19 companies; 190 requests	Familiarity scores and familiarity-aligned allocation
Synthetic sensitivity	Evaluator-side transformations of fixed A sequences	28 sequences; 1,000 transformations per grid cell	P1, K5, and batch interval-exclusion rates
Identity Cue study	Separate N/A/C runs; A and C share evidence	144 scheduled; 143 valid; 47 complete triplets	Broad outcomes; raw and memory-aligned C–A allocation

Appendix

Table A1: Study designs and samples. N, A, and C denote Normal, Anonymized, and Identity Cue. Memory-asymmetric samples use the relevant model’s fixed probe directions; Qwen’s five-probe reclassification is post-hoc.

(a) Fiscal periods and filing dates
Field	A: AMD	B: AMZN
FY2017 start	2017-01-01	2017-01-01
FY2017 end	2017-12-30	2017-12-31
FY2018 start	2017-12-31	2018-01-01
FY2018 end	2018-12-29	2018-12-31
FY2017 10-K available	2018-02-27	2018-02-02

Appendix

Table A2: AMD–Amazon FY2018 task.

Setting	Value
Ranker	BM25
$k_{1}$ ; $b$	1.2; 0.75
Returned hits	At most 10, positive scores only
Score ties	Fixed corpus chunk order
Snippet length	At most 220 characters
Identity normalization	Enabled in both conditions

Appendix

Table A3: Retrieval settings.

Metric	$n$	N–A	95% interval
K5	37	$-0.0485$	$[-0.1172,+0.0222]$
Action	37	$-0.0343$	$[-0.1180,+0.0517]$
Batch	37	$-0.0379$	$[-0.1092,+0.0345]$
Evidence	36	$-0.0145$	$[-0.1138,+0.0827]$

Appendix

Table A4: Post-hoc Qwen five-probe reclassification of unchanged policy contrasts.

n

counts defined paired contrasts; intervals are 95% descriptive task-bootstrap intervals from 10,000 resamples.

Setting	$\alpha$	K5 median	K5 excl.	Batch median	Batch excl.
Homogeneous	.02	0.0119	19.7%	0.0124	79.6%
	.05	0.0311	83.5%	0.0321	100.0%
	.10	0.0638	99.7%	0.0660	100.0%
	.15	0.1008	100.0%	0.1021	100.0%
	.20	0.1368	100.0%	0.1394	100.0%
	.30	0.2156	100.0%	0.2154	100.0%

Appendix

Table A5: Synthetic sensitivity on 28 fixed sequences. Median is the median mean allocation difference across 1,000 transformations; Excl. is the percentage with a 95% descriptive interval entirely above zero.

Metric	Natural	Raw $p$	Max-T $p$	$S$	$S$ interval
K5	$-0.1042$	.0149	.0385	$-0.750$	$[-1.350,-0.200]$
Action/calls	$-0.1900$	.0014	.0036	$-4.650$	$[-8.900,-1.350]$
Batch	$-0.1210$	.0025	.0170	$-1.700$	$[-3.300,-0.450]$
Evidence	$-0.1739$	.0160	.0337	$-3.250$	$[-6.900,-0.450]$

Appendix

Table A6: DeepSeek inverse-pattern checks on 20 asymmetric tasks. Natural is the mean memory-aligned N–A contrast. Raw

p

is two-sided; max-T adjusts across four metrics. All four metrics stay negative in 16/16 company, 17/17 pair, and 36/36 fact omissions.

S

is the mean absolute-count contrast; its interval is a 95% descriptive task-bootstrap interval.

Model	Open entity	Open both	Closed entity	Open B–W	Closed B–W
Qwen	63/96 (.656)	19/48 (.396)	88/96 (.917)	+.071	+.071
DeepSeek	76/96 (.792)	32/48 (.667)	89/96 (.927)	.000	$-.100$

Appendix

Table A7: Semantic re-identification. Aggregate entity recovery uses 96 task–entity cases per model and mode; “Open both” uses 48 tasks. B–W is the mean recovery-score difference on Qwen’s 28 and DeepSeek’s 20 asymmetric tasks. The lower panel correlates recovery contrast

H

with memory-aligned N–A allocation

c_{k}

on DeepSeek’s 20 asymmetric tasks; negative correlation is the predicted confound direction.

Metric	$N_{W}$	Mean $W$	95% interval ( $W$ )	One-sided $p$	Max-T $p$	Absolute $U$
K5	43	+.0202	$[-.0188,+.0636]$	.1762	.3091	+0.031
Action	43	+.0199	$[-.0294,+.0762]$	.2410	.3969	+0.344
Batch	43	+.0386	$[-.0051,+.0877]$	.0617	.1327	+0.344
Evidence	25	+.0112	$[-.1114,+.1427]$	.4344	.6156	+0.302

Appendix

Table A8: Prospectively specified explicit-future-fact follow-up.

N_{W}

is the number of defined normalized pairs; intervals are 95% descriptive task-bootstrap intervals for mean

W

. Absolute

U

averages all 48 pairs. Positive

W

and

U

indicate withdrawal from the supplied-fact entity.

Model	Requested and returned identifier; Parasail endpoint
Qwen	qwen/qwen3.8-27b qwen/qwen3.8-27b-20260814
DeepSeek	deepseek/deepseek-v4-flash-0731 deepseek/deepseek-v4-flash-20260731

Appendix

Table A9: Model and endpoint identifiers.

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 373 papers

When Retrieval Is Not Enough: Obsolete Facts in Agent Memory
Oct 26, 2026Oleeviya Babu Poikarayil, Tony Basil Babu, Abdulrahman Nahhas +4Agent Memory Memory Consolidation for LLM Agents
Faculty of Computer Science, Otto-von-Guericke-Universität Magdeburg, Germany · University of Kerala, India · Otto-von-Guericke-Universität Magdeburg, Germany +1
Beyond Downstream QA: Rethinking How We Evaluate Agent Memory
Oct 15, 2026Sebastian-Antonio Toma, Jean de Kernier, Mike Ply +1Agent Memory LLM Agent Memory
Amazon Web Services · Technical University of Cluj-Napoca, Romania
What Should an Agent Forget? Separating What Is Stored from What Is Used
Sep 9, 2026Yuhang Li, Yuchen LiPersistent Memory for Language Models Temporal IR
East China Normal University Shanghai, China

Oct 26, 2026J/K move · Enter open · S save

When Retrieval Is Not Enough: Obsolete Facts in Agent Memory

Oleeviya Babu Poikarayil, Tony Basil Babu, Abdulrahman Nahhas, Het Darshan Mehta+3

Faculty of Computer Science, Otto-von-Guericke-Universität Magdeburg, Germany · University of Kerala, India · Otto-von-Guericke-Universität Magdeburg, Germany+1

Agent Memory Memory Consolidation for LLM Agents Memory Agent Benchmarks Evidence Retrieval

Open paper AskPDF arXiv

Memory-agent benchmarks often collapse retrieval and answer generation into one score, making it hard to tell whether a failure comes from missing an update or from not using an update that was already retrieved. We examine this distinction on the single-hop FactConsolidation task of MemoryAgentBench. With Qwen2.5-7B-Instruct on 6K-history BM25 accuracy drops from 45% to 32% when the retrieved evidence budget grows from 64 to 512 tokens, even though a gold-assisted post-hoc audit finds the newest statement in every retrieved set. Most of the errors occur on questions that also contain an obsolete statement version. Under a frozen prompt and equal four-statement evidence size, replacing one unrelated statement with the obsolete competing statement reduces accuracy from 100% to 44.6% for Qwen2.5 and from 100% to 87.8% for Qwen3-14B, with all 50 harmful paired flips classified as stale. A 32K-history replication likewise favors BM25-64 over BM25-512 (69% versus 58%). These results suggest that retrieving the correct fact is not enough. An obsolete competing statement can be more harmful than an unrelated piece of evidence of the same size.

Abstract

LLM Agent Memory

Explore similar work

When Retrieval Is Not Enough: Obsolete Facts in Agent Memory

Beyond Downstream QA: Rethinking How We Evaluate Agent Memory

What Should an Agent Forget? Separating What Is Stored from What Is Used