Task success, retrieval quality, and cost do not show whether an LLM agent used a delivered memory or what followed from that use. We introduce MemUse, which evaluates memory at its point of use in recorded trajectories: each memory delivery is compared with what the agent could already see and followed through the agent's later actions and their consequences. An agentic judge rates each delivered entry's relevance, accuracy, novelty, influence, and observed effect; programmatic checks enforce complete reading of cited evidence, and fixed rules aggregate the ratings into verdicts linked to memory histories and delivery cost. On 8,000 SWE-bench Verified episodes from three memory systems, resolve rates stay within 0.4 percentage points of a no-memory baseline, yet MemUse shows that most delivered entries were irrelevant, most relevant entries went unused, and in two systems most deliveries restated content the agent could still see. It also surfaces effects that outcomes hide: 23.2% of episodes with beneficial memory use still failed, and 5 of 16 with harmful use still succeeded.
Figures & tables
Figure 1: Overview of MemUse. The memory bridge connects agents to memory systems and annotates memory operations. The trajectory recorder captures model calls, with token usage and cost when available. Within each episode, data processing groups identical repeated deliveries into injection spans while earlier copies remain visible and assigns each entry a credit window for attributing actions. The memory graph links deliveries to recorded entry histories across trajectories. For each eligible span, an agentic judge assesses each entry’s information quality, use, and observed effects from its exposed text, prior context, and subsequent trajectory. Programmatic checks enforce evidence coverage and credit windows, and fixed rules derive the span verdict.
Figure 2: Credit windows and judge routing. (a) A’s deliveries at steps 2 and 4 share S1 ; re-entry after removal starts S4 with a fresh prior context. (b) The shorter copy A′ adds no text or identifier beyond the visible A , so A′ in S3 receives an empty window and A in S1 retains credit through step 9. Effects of credited actions are followed through the episode end. (c) S1 , S2 , and S4 are judged; S3 receives a rule-derived neutral triage label without a judge call. Empty deliveries receive a neutral placeholder and are excluded from triage rates.
Aspect
Signal
Values
Question answered
Information
Relevance
true, false
Does the exposed text apply to the task or the agent’s subgoal at delivery?
Accuracy
corroborated, uncontradicted, contradicted
Does evidence confirm or refute the claim in the situation it describes?
Novelty
novel, confirming, redundant, undetermined
Is the content new, does it settle an open hypothesis, or was it already established?
Use
Influence
acted on, selected, referenced, none, undetermined
What did the entry demonstrably shape within its credit window?
Consequence
Observed effect
improved, no effect, harmed, undetermined
Did the attributed actions help, make no difference, or cause harm by the episode end?
Table 1: Signals assigned to each entry in an eligible span. Undetermined values are recorded as null . Observed effect is assessed only for acted-on entries; all other entries also receive null .
Configuration
Resolved (%)
Difference (pp) estimate [95% interval]
Cost per episode (USD)
Cost increase (%)
MemGovern
73.4±0.6
+0.40[−0.70,1.50]
0.0049
+ 8.8
Mem0 (L+G)
72.7±0.7
−0.25[−1.55,1.00]
0.0062
+ 37.7
Mem0 (L)
72.8±0.6
−0.20[−1.50,1.05]
0.0062
+ 38.1
TencentDB
72.7±0.8
−0.30[−1.50,0.85]
0.0178
+ 293.2
No memory
73.0±1.5
—
0.0045
—
Table 2: Task success and mean cost per episode. Resolve rates are mean ± sample standard deviation across four runs. Differences from the no-memory baseline are in percentage points, with 95% confidence intervals from a paired bootstrap over tasks. Costs are computed from recorded agent and memory-system usage for retained executions.
Figure 3: Six-axis memory profiles, pooled over four runs per configuration, with each profiled span weighted equally. (a) The radar plot uses a common logarithmic radial scale from 0.001% at the center to 100% at the outer ring, with tenfold increases between adjacent rings. (b) Heatmap cells report the same means as percentages, with shading on the same logarithmic scale. Lower harm shares are better. Table 7 reports these shares alongside uncertainty and token efficiency.
Figure 4: Memory effects concealed by task outcomes. (a) A TencentDB skill for a different Napoleon bug directs the agent into the wrong component (steps 2–5). A reproduction rules out Napoleon at step 7, and the agent fixes sphinx/util/docfields.py at step 9; the task passes despite the detour. (b) A MemGovern card guides the agent to add a subset handler at step 8, changing the reported subset check from None to True . The equality error persists, and at step 14 the agent decides to leave it unfixed despite the card’s advice to repair it; the task fails. Timelines show complete episodes with steps numbered locally. Arrows mark first consumption, shading marks credit windows, and blue steps mark actions based on memory. Red and green annotations highlight problems and repairs. Memory effects are judge assessments; task outcomes come from the SWE-bench harness.
Figure 5: Delivered-token and beneficial-entry shares by delivery form. Circles show each form’s share of estimated delivered tokens, including restatements, and diamonds show its share of beneficial entries. Values pool four runs per configuration, with both Mem0 configurations combined. Forms with no beneficial entries and less than 0.1% of tokens are omitted from the plot but retained in the token denominators. Full cards, task recall, and full skills supply most beneficial entries at small token shares. Table 11 gives the detailed breakdown.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Evidence
Content and role
Task
Task description and user updates, with their positions in the trajectory.
Target delivery
Verbatim placements, exposed entry values, identifiers visible to the agent, and per-entry credit windows.
Prior context
Ordered visible input at first consumption, with target text removed and other visible deliveries retained.
Earlier copies
Previous and same-input deliveries of each target item, their remaining visibility, and overlap with the target text.
Trajectory
Agent responses, actions, and tool results through the episode end, indexed by global step.
Optional references
A reference solution or recorded task outcome, used only to corroborate the corresponding claims.
Appendix
Table 3: Evidence available for assessing one injection span. Source positions distinguish information available before delivery from actions and consequences observed afterwards.
Record at step 4
Novelty
Influence
Reason
The agent notes that the retry loop never waits and plans to add a backoff delay in fetch() .
redundant
none
The prior decision explains step 7. An explicit mention that the entry confirms this plan would instead be referenced.
The agent only asks whether the retries need a delay.
confirming
undetermined
An open possibility and the entry both precede step 7, and nothing in the record separates them.
The agent only prints the code of fetch() , which has no delay.
novel
acted on, anchored at step 7
The delay is memory-specific content, even though a capable agent could have derived it.
Appendix
Table 4: Attribution example from the system prompt. The entry and the action at step 7 are fixed; only the pre-delivery record at step 4 changes.
Claim
Required complete reads
Every answer
Task file; prior-context file; the other-messages file and the reference solution when present; the first consuming step and its preceding step in the episode.
Determined novelty or acted-on influence
Every step file that BS names as a source of visible text, and the complete section of every other delivery visible in BS .
None, referenced, or selected
Every agent step in the window of the entry.
Determined effect for an acted-on entry
Every step from the start of the window through the end of the episode.
Structured step citation
The complete text of every integer step reference in anchors or the structured evidence list.
Appendix
Table 5: Complete reads required before an answer is accepted. Requirements accumulate with the claim.
Setting
Value
Investigation budget
250 model calls per span
Context window
272,000 tokens
Maximum output
32,768 tokens per call
Answer repair
At most two follow-ups, with four model calls each
Parallel evaluation
Five spans within one run
Request idle timeout
300 seconds
Appendix
Table 6: Judge execution settings used in all 16 evaluated memory runs. Call limits include the forced final-answer call.
MemGovern
Mem0 (L+G)
Mem0 (L)
TencentDB
Mean entry share (%)
Relevance
15.08
1.42
1.43
28.48
Corroboration
3.39
0.23
0.22
2.19
Information gain
11.12
1.03
1.03
26.13
Acted on
3.95
0.40
0.51
4.08
Benefit
2.94
0.29
0.35
2.30
Appendix
Table 7: Memory profiles, entry yields, uncertainty, and coverage, pooled over four runs. Profile and uncertainty shares average over profiled spans, with all entries of each span in the denominator. Yields count beneficial or harmful entries per 100 profiled spans. Coverage uses all 2,000 episodes per configuration; recurring items occur in at least 50 episodes of a run. Delivered tokens include restatements and spans without profiles. Unknown novelty share is zero in every configuration.
Figure 6: Run variation and effects conditional on use. (a) Markers show each of four runs per configuration on logarithmic scales; within a row, runs 1–4 are offset from top to bottom, and shading spans the four runs. Relevance is the mean entry share across profiled spans; token efficiency divides beneficial-entry counts by all estimated delivered tokens, including restatements and spans without profiles. (b) Effect counts are pooled across four runs and divided by the number of acted-on entries, n . Labels give improved and undetermined shares; every segment retains its proportional width.
Run
Resolved
Relevance
Acted on
Benefit
Harm
Efficiency
Spans
MemGovern
1
74.0
16.42
4.42
3.25
0.29
56.25
681
2
72.6
14.45
3.35
2.47
0.44
41.10
687
3
73.6
14.77
4.38
3.25
0.28
56.03
710
4
73.2
14.71
3.65
2.78
0.44
48.47
688
Mem0 (L+G)
Appendix
Table 8: Results of each run. Resolved, relevance, acted on, benefit, and harm are percentages, with entry shares averaged over the profiled spans of the run. Efficiency counts beneficial entries per million delivered tokens, and spans counts profiled spans. The no-memory baseline resolved 75.0%, 72.0%, 73.0%, and 71.8% of the tasks in its four runs.
MemGovern
Mem0 (L+G)
Mem0 (L)
TencentDB
Relevant entries
Count
1,247
571
571
4,118
Unused (%)
51.32
79.68
77.06
75.06
Novel among unused (%)
58.91
49.45
48.18
89.52
Acted-on entries
Count
112
121
160
321
Appendix
Table 9: Use and observed effects of judged entries, pooled over four runs. Unused relevant entries have influence none , and novelty is judged against the agent’s visible context. Delays compare the first anchored action with the first consuming step. Episode counts exclude the three episodes without a harness result; one MemGovern episode belongs to both effect groups.
Baseline runs that resolved the task
0
1
2
3
4
Tasks
104
22
13
33
328
Memory episodes
1,664
352
208
527
5,246
resolved (%)
4.27
27.84
44.71
77.42
98.34
Episodes with an acted-on entry
144
36
17
36
411
Episodes with a beneficial entry
79
12
7
25
260
share of memory episodes (%)
4.75
3.41
3.37
4.74
4.96
Appendix
Table 10: Memory episodes of all four configurations, grouped by the number of baseline runs that resolved their task. The three episodes without a harness result are excluded.
Share of system (%)
Entry rate (%)
Delivered form
Route
Tokens
Beneficial
Relevant
Acted on
Harmful (n)
MemGovern
Search preview
A
87.52
3.57
4.48
0.01
0
Full card
A
12.48
96.43
45.77
15.35
10
Mem0
Task recall
H
12.96
75.77
2.00
0.88
4
Appendix
Table 11: Delivery volume and entry outcomes by form, pooled over four runs and combining both Mem0 configurations. H denotes host-scheduled delivery and A agent-directed delivery. Token and beneficial-entry shares use each system’s totals, including 84, 194, and 143 beneficial entries for MemGovern, Mem0, and TencentDB. Entry rates use judged entries of each form. Copies with empty credit windows retain their token volume without receiving an assessment. Omitted empty and automatic-copy forms each contribute less than 0.1% of tokens and remain in the denominators.
Mem0 (L+G)
Mem0 (L)
TencentDB
Beneficial entries
86
108
143
Traced origins
86
108
138
with known source outcome
86
108
137
from failed source episodes
23
26
24
from another repository
0
0
13
Median gap in tasks
40
53
19
Appendix
Table 12: Origins of beneficial entries, pooled over four runs. Traced origins are earlier episodes in the same run. Source-outcome counts exclude one TencentDB entry whose creation episode has no harness result. Gaps count task positions from creation to beneficial use among traced entries.
MemGovern
Mem0 (L+G)
Mem0 (L)
TencentDB
Total
Evaluation units
Deliveries
2,766
27,381
27,670
51,242
109,059
restatements (%)
0.00
78.11
78.39
88.36
81.02
Spans
2,766
5,994
5,980
5,963
20,703
automatic neutral
0
1,582
1,626
3
3,211
empty
0
1,101
1,001
119
2,221
Appendix
Table 13: Evaluation units, span triage, and judge usage, pooled over four runs. Spans split into judged, automatic neutral, and empty; triage counts include automatic neutral spans and exclude empty spans. Restatements are later deliveries within a span. Input tokens are split into uncached and cached tokens as reported by the provider.
External memory enables LLM agents to improve from experience by distilling trajectories of past tasks into concise lessons for new tasks. However, memory can also lead to memory-induced degradation, where an agent fails on tasks it would solve without memory, often because it follows a lesson that is incorrect, or misapplies or misreads a valid one. This is because lessons are inferred from trajectories labeled only with task success or failure, yet are typically provided without the trajectories they came from, leaving the agent no way to check them. To address this, we introduce WikiTrace, an evidence-linked memory system designed to let agents check each lesson against its original trajectory. Specifically, motivated by the finding that a trajectory often still holds the evidence needed to catch an erroneous lesson, WikiTrace stores each lesson as a concise insight linked to a summary of why its trajectory succeeded or failed and to the raw trajectory itself. Then, on a new task, the agent starts from all insights and, whenever one lacks details needed to act, conflicts with another, or comes from a failed task, descends through its links with terminal tools to fill in missing details or correct the insight before acting. We validate WikiTrace on five agentic benchmarks, showing that it consistently outperforms existing memory approaches and a baseline given the same insights alone, while also reducing memory-induced degradation. Our code is available at https://github.com/yukyeongleee/wikitrace
Memory has emerged as a cornerstone of modern LLM-based agents, supporting their evolution from single-turn assistants to long-term collaborators. However, memory is not always beneficial: retrieved memories often induce a critical issue of sycophancy, causing agents to over-align with the user at the cost of factual accuracy or objective reasoning. Despite this emerging risk, existing memory benchmarks primarily evaluate whether memories are correctly stored, retrieved, or updated, while overlooking how retrieved memories influence downstream reasoning and decision-making. To bridge this gap, we propose MemSyco-Bench, a comprehensive benchmark for evaluating memory-induced sycophancy in agent systems. MemSyco-Bench measures when memory should influence a decision and how valid memory should be used. Specifically, it covers five tasks that assess whether agents can reject memory as factual evidence, respect its applicable scope, resolve conflicts between memory and objective evidence, track memory updates, and use valid memory for personalization. All related resources are collected for the community at https://github.com/XMUDeepLIT/MemSyco-Bench.
LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. While prior benchmarks evaluate only single-entity updates, MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes, including three not scored by prior work: Cascade and Absence (dependency reasoning) and Deletion (post-removal state). Evaluating six memory systems spanning three memory paradigms on 100 controlled episodes, we find that all systems collapse on dependency reasoning under the default configuration (Cascade: 3%, Absence: 1% in average accuracy) despite adequate static retrieval performance. Prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs fail to close this gap. The largest gain comes from a file-based agent paired with Claude Opus 4.7 as its internal LLM, which partially closes the gap at roughly 70x the baseline cost. Code and data are available on the project page: https://seokwonjung-jay.github.io/meme-eval/
Seokwon Jung, Alexander Rubinstein, Arnas Uselis +2
KAIST AI · Tübingen AI Center, University of Tübingen · NAVER AI Lab