Delivered, Not Used: Evaluating Agent Memory at Its Point of Use

Changran Xu; Zeju Li; Qiang Xu

cs.AINov 1, 2026

Delivered, Not Used: Evaluating Agent Memory at Its Point of Use

Authors: Changran Xu, Zeju Li, Qiang Xu

Organizations: Department of Computer Science and Engineering The Chinese University of Hong Kong

Abstract

Task success, retrieval quality, and cost do not show whether an LLM agent used a delivered memory or what followed from that use. We introduce MemUse, which evaluates memory at its point of use in recorded trajectories: each memory delivery is compared with what the agent could already see and followed through the agent's later actions and their consequences. An agentic judge rates each delivered entry's relevance, accuracy, novelty, influence, and observed effect; programmatic checks enforce complete reading of cited evidence, and fixed rules aggregate the ratings into verdicts linked to memory histories and delivery cost. On 8,000 SWE-bench Verified episodes from three memory systems, resolve rates stay within 0.4 percentage points of a no-memory baseline, yet MemUse shows that most delivered entries were irrelevant, most relevant entries went unused, and in two systems most deliveries restated content the agent could still see. It also surfaces effects that outcomes hide: 23.2% of episodes with beneficial memory use still failed, and 5 of 16 with harmful use still succeeded.

Figures & tables

Figure 1: Overview of MemUse. The memory bridge connects agents to memory systems and annotates memory operations. The trajectory recorder captures model calls, with token usage and cost when available. Within each episode, data processing groups identical repeated deliveries into injection spans while earlier copies remain visible and assigns each entry a credit window for attributing actions. The memory graph links deliveries to recorded entry histories across trajectories. For each eligible span, an agentic judge assesses each entry’s information quality, use, and observed effects from its exposed text, prior context, and subsequent trajectory. Programmatic checks enforce evidence coverage and credit windows, and fixed rules derive the span verdict.

Figure 2: Credit windows and judge routing. (a) A’s deliveries at steps 2 and 4 share

S_{1}

; re-entry after removal starts

S_{4}

with a fresh prior context. (b) The shorter copy

A^{\prime}

adds no text or identifier beyond the visible

A

, so

A^{\prime}

S_{3}

receives an empty window and

A

S_{1}

retains credit through step 9. Effects of credited actions are followed through the episode end. (c)

S_{1}

S_{2}

, and

S_{4}

are judged;

S_{3}

receives a rule-derived neutral triage label without a judge call. Empty deliveries receive a neutral placeholder and are excluded from triage rates.

Aspect	Signal	Values	Question answered
Information	Relevance	true, false	Does the exposed text apply to the task or the agent’s subgoal at delivery?
	Accuracy	corroborated, uncontradicted, contradicted	Does evidence confirm or refute the claim in the situation it describes?
	Novelty	novel, confirming, redundant, undetermined	Is the content new, does it settle an open hypothesis, or was it already established?
Use	Influence	acted on, selected, referenced, none, undetermined	What did the entry demonstrably shape within its credit window?
Consequence	Observed effect	improved, no effect, harmed, undetermined	Did the attributed actions help, make no difference, or cause harm by the episode end?

Table 1: Signals assigned to each entry in an eligible span. Undetermined values are recorded as null . Observed effect is assessed only for acted-on entries; all other entries also receive null .

Configuration	Resolved (%)	Difference (pp) estimate [95% interval]	Cost per episode (USD)	Cost increase (%)
MemGovern	$73.4\pm 0.6$	$+0.40\;[-0.70,1.50]$	0.0049	$+$ 8.8
Mem0 (L+G)	$72.7\pm 0.7$	$-0.25\;[-1.55,1.00]$	0.0062	$+$ 37.7
Mem0 (L)	$72.8\pm 0.6$	$-0.20\;[-1.50,1.05]$	0.0062	$+$ 38.1
TencentDB	$72.7\pm 0.8$	$-0.30\;[-1.50,0.85]$	0.0178	$+$ 293.2
No memory	$73.0\pm 1.5$	—	0.0045	—

Table 2: Task success and mean cost per episode. Resolve rates are mean

\pm

sample standard deviation across four runs. Differences from the no-memory baseline are in percentage points, with 95% confidence intervals from a paired bootstrap over tasks. Costs are computed from recorded agent and memory-system usage for retained executions.

Figure 3: Six-axis memory profiles, pooled over four runs per configuration, with each profiled span weighted equally. (a) The radar plot uses a common logarithmic radial scale from 0.001% at the center to 100% at the outer ring, with tenfold increases between adjacent rings. (b) Heatmap cells report the same means as percentages, with shading on the same logarithmic scale. Lower harm shares are better. Table 7 reports these shares alongside uncertainty and token efficiency.

Figure 4: Memory effects concealed by task outcomes. (a) A TencentDB skill for a different Napoleon bug directs the agent into the wrong component (steps 2–5). A reproduction rules out Napoleon at step 7, and the agent fixes sphinx/util/docfields.py at step 9; the task passes despite the detour. (b) A MemGovern card guides the agent to add a subset handler at step 8, changing the reported subset check from None to True . The equality error persists, and at step 14 the agent decides to leave it unfixed despite the card’s advice to repair it; the task fails. Timelines show complete episodes with steps numbered locally. Arrows mark first consumption, shading marks credit windows, and blue steps mark actions based on memory. Red and green annotations highlight problems and repairs. Memory effects are judge assessments; task outcomes come from the SWE-bench harness.

Figure 5: Delivered-token and beneficial-entry shares by delivery form. Circles show each form’s share of estimated delivered tokens, including restatements, and diamonds show its share of beneficial entries. Values pool four runs per configuration, with both Mem0 configurations combined. Forms with no beneficial entries and less than 0.1% of tokens are omitted from the plot but retained in the token denominators. Full cards, task recall, and full skills supply most beneficial entries at small token shares. Table 11 gives the detailed breakdown.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Evidence	Content and role
Task	Task description and user updates, with their positions in the trajectory.
Target delivery	Verbatim placements, exposed entry values, identifiers visible to the agent, and per-entry credit windows.
Prior context	Ordered visible input at first consumption, with target text removed and other visible deliveries retained.
Earlier copies	Previous and same-input deliveries of each target item, their remaining visibility, and overlap with the target text.
Trajectory	Agent responses, actions, and tool results through the episode end, indexed by global step.
Optional references	A reference solution or recorded task outcome, used only to corroborate the corresponding claims.

Appendix

Table 3: Evidence available for assessing one injection span. Source positions distinguish information available before delivery from actions and consequences observed afterwards.

Record at step 4	Novelty	Influence	Reason
The agent notes that the retry loop never waits and plans to add a backoff delay in fetch() .	redundant	none	The prior decision explains step 7. An explicit mention that the entry confirms this plan would instead be referenced.
The agent only asks whether the retries need a delay.	confirming	undetermined	An open possibility and the entry both precede step 7, and nothing in the record separates them.
The agent only prints the code of fetch() , which has no delay.	novel	acted on, anchored at step 7	The delay is memory-specific content, even though a capable agent could have derived it.

Appendix

Table 4: Attribution example from the system prompt. The entry and the action at step 7 are fixed; only the pre-delivery record at step 4 changes.

Claim	Required complete reads
Every answer	Task file; prior-context file; the other-messages file and the reference solution when present; the first consuming step and its preceding step in the episode.
Determined novelty or acted-on influence	Every step file that $B_{S}$ names as a source of visible text, and the complete section of every other delivery visible in $B_{S}$ .
None, referenced, or selected	Every agent step in the window of the entry.
Determined effect for an acted-on entry	Every step from the start of the window through the end of the episode.
Structured step citation	The complete text of every integer step reference in anchors or the structured evidence list.

Appendix

Table 5: Complete reads required before an answer is accepted. Requirements accumulate with the claim.

Setting	Value
Investigation budget	250 model calls per span
Context window	272,000 tokens
Maximum output	32,768 tokens per call
Answer repair	At most two follow-ups, with four model calls each
Parallel evaluation	Five spans within one run
Request idle timeout	300 seconds

Appendix

Table 6: Judge execution settings used in all 16 evaluated memory runs. Call limits include the forced final-answer call.

Mean entry share (%)
	MemGovern	Mem0 (L+G)	Mem0 (L)	TencentDB
Relevance	15.08	1.42	1.43	28.48
Corroboration	3.39	0.23	0.22	2.19
Information gain	11.12	1.03	1.03	26.13
Acted on	3.95	0.40	0.51	4.08
Benefit	2.94	0.29	0.35	2.30

Appendix

Table 7: Memory profiles, entry yields, uncertainty, and coverage, pooled over four runs. Profile and uncertainty shares average over profiled spans, with all entries of each span in the denominator. Yields count beneficial or harmful entries per 100 profiled spans. Coverage uses all 2,000 episodes per configuration; recurring items occur in at least 50 episodes of a run. Delivered tokens include restatements and spans without profiles. Unknown novelty share is zero in every configuration.

Figure 6: Run variation and effects conditional on use. (a) Markers show each of four runs per configuration on logarithmic scales; within a row, runs 1–4 are offset from top to bottom, and shading spans the four runs. Relevance is the mean entry share across profiled spans; token efficiency divides beneficial-entry counts by all estimated delivered tokens, including restatements and spans without profiles. (b) Effect counts are pooled across four runs and divided by the number of acted-on entries,

n

. Labels give improved and undetermined shares; every segment retains its proportional width.

Run	Resolved	Relevance	Acted on	Benefit	Harm	Efficiency	Spans
MemGovern
1	74.0	16.42	4.42	3.25	0.29	56.25	681
2	72.6	14.45	3.35	2.47	0.44	41.10	687
3	73.6	14.77	4.38	3.25	0.28	56.03	710
4	73.2	14.71	3.65	2.78	0.44	48.47	688
Mem0 (L+G)

Appendix

Table 8: Results of each run. Resolved, relevance, acted on, benefit, and harm are percentages, with entry shares averaged over the profiled spans of the run. Efficiency counts beneficial entries per million delivered tokens, and spans counts profiled spans. The no-memory baseline resolved 75.0%, 72.0%, 73.0%, and 71.8% of the tasks in its four runs.

Relevant entries
	MemGovern	Mem0 (L+G)	Mem0 (L)	TencentDB
Count	1,247	571	571	4,118
Unused (%)	51.32	79.68	77.06	75.06
Novel among unused (%)	58.91	49.45	48.18	89.52
Acted-on entries
Count	112	121	160	321

Appendix

Table 9: Use and observed effects of judged entries, pooled over four runs. Unused relevant entries have influence none , and novelty is judged against the agent’s visible context. Delays compare the first anchored action with the first consuming step. Episode counts exclude the three episodes without a harness result; one MemGovern episode belongs to both effect groups.

Baseline runs that resolved the task	0	1	2	3	4
Tasks	104	22	13	33	328
Memory episodes	1,664	352	208	527	5,246
resolved (%)	4.27	27.84	44.71	77.42	98.34
Episodes with an acted-on entry	144	36	17	36	411
Episodes with a beneficial entry	79	12	7	25	260
share of memory episodes (%)	4.75	3.41	3.37	4.74	4.96

Appendix

Table 10: Memory episodes of all four configurations, grouped by the number of baseline runs that resolved their task. The three episodes without a harness result are excluded.

		Share of system (%)		Entry rate (%)
Delivered form	Route	Tokens	Beneficial	Relevant	Acted on	Harmful (n)
MemGovern
Search preview	A	87.52	3.57	4.48	0.01	0
Full card	A	12.48	96.43	45.77	15.35	10
Mem0
Task recall	H	12.96	75.77	2.00	0.88	4

Appendix

Table 11: Delivery volume and entry outcomes by form, pooled over four runs and combining both Mem0 configurations. H denotes host-scheduled delivery and A agent-directed delivery. Token and beneficial-entry shares use each system’s totals, including 84, 194, and 143 beneficial entries for MemGovern, Mem0, and TencentDB. Entry rates use judged entries of each form. Copies with empty credit windows retain their token volume without receiving an assessment. Omitted empty and automatic-copy forms each contribute less than 0.1% of tokens and remain in the denominators.

	Mem0 (L+G)	Mem0 (L)	TencentDB
Beneficial entries	86	108	143
Traced origins	86	108	138
with known source outcome	86	108	137
from failed source episodes	23	26	24
from another repository	0	0	13
Median gap in tasks	40	53	19

Appendix

Table 12: Origins of beneficial entries, pooled over four runs. Traced origins are earlier episodes in the same run. Source-outcome counts exclude one TencentDB entry whose creation episode has no harness result. Gaps count task positions from creation to beneficial use among traced entries.

Evaluation units
	MemGovern	Mem0 (L+G)	Mem0 (L)	TencentDB	Total
Deliveries	2,766	27,381	27,670	51,242	109,059
restatements (%)	0.00	78.11	78.39	88.36	81.02
Spans	2,766	5,994	5,980	5,963	20,703
automatic neutral	0	1,582	1,626	3	3,211
empty	0	1,101	1,001	119	2,221

Appendix

Table 13: Evaluation units, span triage, and judge usage, pooled over four runs. Spans split into judged, automatic neutral, and empty; triage counts include automatic neutral spans and exclude empty spans. Restatements are later deliveries within a span. Input tokens are split into uncached and cached tokens as reported by the provider.

Agent Memory

Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.

See all 206 papers

Don't Just Trust Your Memory: Evidence-Linked Memory Reduces Memory-Induced Degradation in LLM Agents
Nov 10, 2026Yukyeong Lee, Yumin Choi, Jinheon Baek +1Agent Memory LLM Agent Memory
KAIST
MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
Jul 1, 2026Zhishang Xiang, Zerui Chen, Yunbo Tang +5Agent Memory LLM Sycophancy
Xiamen University · Jilin University
MEME: Multi-entity & Evolving Memory Evaluation
May 12, 2026Seokwon Jung, Alexander Rubinstein, Arnas Uselis +2Agent Memory Persistent Memory for Language Models
KAIST AI · Tübingen AI Center, University of Tübingen · NAVER AI Lab

Nov 10, 2026J/K move · Enter open · S save

Don't Just Trust Your Memory: Evidence-Linked Memory Reduces Memory-Induced Degradation in LLM Agents

Yukyeong Lee, Yumin Choi, Jinheon Baek, Sung Ju Hwang

KAIST

Agent Memory LLM Agent Memory Experience Learning for LLM Agents

Open paper AskPDF arXiv

External memory enables LLM agents to improve from experience by distilling trajectories of past tasks into concise lessons for new tasks. However, memory can also lead to memory-induced degradation, where an agent fails on tasks it would solve without memory, often because it follows a lesson that is incorrect, or misapplies or misreads a valid one. This is because lessons are inferred from trajectories labeled only with task success or failure, yet are typically provided without the trajectories they came from, leaving the agent no way to check them. To address this, we introduce WikiTrace, an evidence-linked memory system designed to let agents check each lesson against its original trajectory. Specifically, motivated by the finding that a trajectory often still holds the evidence needed to catch an erroneous lesson, WikiTrace stores each lesson as a concise insight linked to a summary of why its trajectory succeeded or failed and to the raw trajectory itself. Then, on a new task, the agent starts from all insights and, whenever one lacks details needed to act, conflicts with another, or comes from a failed task, descends through its links with terminal tools to fill in missing details or correct the insight before acting. We validate WikiTrace on five agentic benchmarks, showing that it consistently outperforms existing memory approaches and a baseline given the same insights alone, while also reducing memory-induced degradation. Our code is available at https://github.com/yukyeongleee/wikitrace

Abstract

Agent Memory

Explore similar work

Don't Just Trust Your Memory: Evidence-Linked Memory Reduces Memory-Induced Degradation in LLM Agents

MemSyco-Bench: Benchmarking Sycophancy in Agent Memory

MEME: Multi-entity & Evolving Memory Evaluation