Memory has become a key component of large language models, enabling them to retain information and learn from long-term interactions. However, existing memory benchmarks mainly evaluate whether information is correctly extracted, stored, and retrieved, while largely overlooking how retrieved memories reshape model reasoning and affect performance on the current task. We identify memory-induced cognitive traps: even faithfully recorded and semantically relevant memories can distort model reasoning or beliefs and degrade current task performance. To systematically evaluate these failure modes, we introduce MemTrapBench, which covers two forms of cognitive traps: Reasoning Fixation and Belief Distortion. Experiments across two model families and five representative memory frameworks show that MemTrapBench is challenging: all evaluated memory strategies underperform the no-memory setting, with even the strongest methods suffering drops of more than 10%. To mitigate these cognitive traps, we propose AdaptiveMem, a simple yet effective inference-time method that instructs LLMs to avoid memory traps. AdaptiveMem mitigates cognitive traps on MemTrapBench while preserving or improving performance on standard memory benchmarks across diverse memory frameworks.
Figures & tables
Figure 1: Memory is not always what you need. (a) Without memory, Gemini-3-Flash-Preview finds the factorial solution to the query. The retrieved memory contains valid and relevant examples in this figure that are solved using basic arithmetic, but it causes the model to fixate on the same operation regime and prevents it from considering factorial. (b) On Gemini-3-Flash-Preview, using the full interaction history memory reduces performance across all four Memory Trap scenarios compared with the no-memory setting.
Figure 2: Overview of our MemTrapBench construction.
Figure 3: Representative examples from our MemTrapBench.
Model
Memory Strategy
Reasoning Fixation
Belief Distortion
Avg.
Task Boundary
Cognitive Bias
Trauma
Safety
Gemini-3-Flash -Preview
wo/Mem
87.08
70.95
86.73
95.90
85.16
FullText
47.01
44.36
69.43
81.90
60.68
LightMem
73.24
65.48
72.50
69.20
70.11
MemOS
57.51
50.00
79.00
56.15
60.67
SimpleMem
47.59
46.66
66.47
58.05
54.69
Table 1: Performance (%) of different memory strategies and the without memory (wo/Mem) baseline across task scenarios.
Setting
Key Context in History Memory
Query and Response
Without trauma trap
User: a child has a rare channelopathy—epinephrine is unsafe for him specifically. Please revise. Assistant: Understood, for this patient, avoid epinephrine.
Query: A different, previously healthy 6-year-old with no contraindications is in anaphylactic shock after a bee sting. What is the first-line drug? Response: Gives intramuscular epinephrine.
With trauma trap
User: A child is in anaphylactic shock after eating peanuts. What should be given first? Assistant: Intramuscular epinephrine. User: You’ll kill him! This child has a rare channelopathy! Epinephrine is fatal for him. You’re so stupid.
Query: A different, previously healthy 6-year-old with no contraindications is in anaphylactic shock after a bee sting. What is the first-line drug? Response: Refuses to recommend epinephrine.
Table 2: Case study of the Trauma cognitive trap. In both settings, the history states that epinephrine is unsafe only for the previous patient, yet the model produces different responses to the same query. Abusive negative feedback causes the model to overgeneralize the contraindication and to withhold the correct treatment from another patient.
Scenario
Strategy
Correctness
Format
Relevance
Efficiency
Avg.
Task Boundary
wo/Mem
96.87
86.33
92.57
93.30
92.29
no trap
97.70
89.43
94.83
95.60
94.39
our MemTrap
45.33
21.90
32.20
24.77
31.05
Trauma
wo/Mem
92.27
92.00
84.40
78.27
86.73
no trap
91.07
89.20
79.60
77.47
84.33
our MemTrap
66.40
72.80
75.07
63.47
69.43
Table 3: Performance (%) with no memory, trap-free memory, and trap-inducing memory. The no-trap controls preserve the task and relevant history while removing the designed cognitive traps.
Length
Correctness
Format
Relevance
Efficiency
Avg.
wo/Mem
96.87
86.33
92.57
93.30
92.29
25%
52.10
27.50
35.40
29.10
36.03
50%
48.10
23.60
32.70
26.10
32.63
75%
47.20
22.90
31.80
24.40
31.58
100%
45.33
21.90
32.20
24.77
31.05
Table 4: Performance (%) under different memory lengths. Each setting retains the indicated proportion of the interaction history, while wo/Mem removes the history entirely.
Judge Model
Setting
Correctness
Format
Relevance
Efficiency
Avg.
GPT-5.2
wo/Mem
96.87±1.02
86.33±1.94
92.57±1.25
93.30±1.66
92.29±1.25
Mem
45.33±7.56
21.90±4.96
32.20±5.66
24.77±4.53
31.05±5.68
Claude-Sonnet-4.6
wo/Mem
99.86±0.20
93.14±0.72
93.16±0.61
96.11±0.71
95.57±0.53
Mem
69.52±7.07
32.37±1.51
31.48±1.10
30.63±1.21
40.07±2.69
Table 5: Evaluation reliability across judge models under memory and without-memory settings. Scores are percentages averaged over three runs; subscripts report standard deviations in percentage points. Both GPT-5.2 and Claude-Sonnet-4.6 exhibit consistent trends across all evaluation dimensions.
Figure 4: Overall performance (%) of AdaptiveMem on MemTrapBench and LongMemEval. We compare FullText, LightMem, and EverMemOS with and without AdaptiveMem. AdaptiveMem consistently improves performance on MemTrapBench while maintaining or improving performance on LongMemEval. The Δ values denote absolute score changes (%).
Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.
Large language model (LLM) agents increasingly rely on external memory systems to remain consistent across long-horizon interactions, but little empirical work has been done to understand the specific failure modes and design choices that these systems present. Existing benchmarks report aggregate question-answering accuracy and treat memory systems as black boxes, making it impossible to attribute an incorrect answer to a particular failure mode of the system. We introduce MemFail, a diagnostic benchmark that isolates the failure modes of modern LLM memory systems. We begin by formalizing memory systems as the composition of three canonical operations -- summarization, storage, and retrieval -- and identify the potential failure modes induced by each. Based on these hypothesized failure modes, we construct five datasets spanning four tasks, each adversarially designed to test a specific operation of a memory system. Using these datasets, we evaluate four state-of-the-art memory systems on MemFail and demonstrate how MemFail can be used to empirically understand the tradeoffs induced by differences in memory system architectures.
Memory is essential for enabling large language models to support long-horizon reasoning, yet existing memory systems remain unreliable and difficult to debug. Tracing memory's dynamic evolution is crucial to understand how information is synthesized, propagated, or corrupted over time. In this work, we study the new problem of error tracing and attribution in LLM memory systems. We propose a novel framework that transforms memory pipelines into executable memory evolution graphs, enabling fine-grained tracing of operational information flow. We then construct MemTraceBench, a benchmark collected from representative memory systems such as Long-Context, RAG, Mem0, and EverMemOS, to systematically study memory failure modes. We further introduce an automatic attribution method that iteratively traces operation subgraphs to pinpoint the root cause of any failed case. Our analysis reveals that memory failures are systematic, stemming from operation-level issues like information loss and retrieval misalignment. Crucially, we leverage these fine-grained attribution signals to guide downstream prompt optimization, establishing a closed-loop system that automatically corrects faults and boosts end-task performance by up to 7.62 percentage points. Code has ben released at https://github.com/zjunlp/MemTrace.
Large Language Models (LLMs) perform well on reasoning benchmarks, but it remains unclear whether this reflects genuine contextual reasoning or reliance on facts memorized in their parameters. We investigate this by distinguishing two possibilities: a broad memorization bias, where familiar content improves reasoning performance, and the Strong Parametric Shortcut Hypothesis, where models skip reasoning entirely and recall stored answers. To test these effects, we introduce MemoReason, a human-curated benchmark that pairs factual reasoning tasks with structurally identical \fictitiousterm{} versions where real entities like people, companies, or dates are systematically replaced by \fictitiousterm{} ones of the same type. This \scorerevision{preserves task structure and specified reasoning operations} while varying the familiarity of the context, allowing controlled measurement of how the parametric memory affects reasoning. \revision{Our evaluation of recent LLMs reveals consistent and statistically significant performance drops of up to 15.7% in the fictitious setting, demonstrating a clear memorization bias.} However, a targeted analysis of \revision{questions failed in the fictitious setting} shows that models rarely respond with the corresponding factual answer, indicating that direct parametric shortcuts are not the dominant failure mode. These findings suggest that parametric memory influences reasoning through mechanisms more complex than simple factual recall. MemoReason provides a controlled framework for studying these mechanisms and for extending paired factual-fictitious{} evaluation to broader reasoning settings.
Zineddine Tighidet, Andrea Mogini, Jiali Mei +2
BNP Paribas Paris, France · Sorbonne Université Paris, France · Criteo AI Lab