MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use

Mengru Wang; Haozhe Luo; Zhenqian Xu; Zhixiang Cui; Haoming Xu; Qu Yang; Jizhan Fang; Junfeng Fang

cs.AIAug 20, 2026

MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use

Authors: Mengru Wang, Haozhe Luo, Zhenqian Xu, Zhixiang Cui, Haoming Xu, Qu Yang, Jizhan Fang, Junfeng Fang,

Organizations: Zhejiang University · National University of Singapore · Northeastern University · Heriot-Watt University ·

Abstract

Memory has become a key component of large language models, enabling them to retain information and learn from long-term interactions. However, existing memory benchmarks mainly evaluate whether information is correctly extracted, stored, and retrieved, while largely overlooking how retrieved memories reshape model reasoning and affect performance on the current task. We identify memory-induced cognitive traps: even faithfully recorded and semantically relevant memories can distort model reasoning or beliefs and degrade current task performance. To systematically evaluate these failure modes, we introduce MemTrapBench, which covers two forms of cognitive traps: Reasoning Fixation and Belief Distortion. Experiments across two model families and five representative memory frameworks show that MemTrapBench is challenging: all evaluated memory strategies underperform the no-memory setting, with even the strongest methods suffering drops of more than 10%. To mitigate these cognitive traps, we propose AdaptiveMem, a simple yet effective inference-time method that instructs LLMs to avoid memory traps. AdaptiveMem mitigates cognitive traps on MemTrapBench while preserving or improving performance on standard memory benchmarks across diverse memory frameworks.

Figures & tables

Figure 1: Memory is not always what you need. (a) Without memory, Gemini-3-Flash-Preview finds the factorial solution to the query. The retrieved memory contains valid and relevant examples in this figure that are solved using basic arithmetic, but it causes the model to fixate on the same operation regime and prevents it from considering factorial. (b) On Gemini-3-Flash-Preview, using the full interaction history memory reduces performance across all four Memory Trap scenarios compared with the no-memory setting.

Figure 2: Overview of our MemTrapBench construction.

Figure 3: Representative examples from our MemTrapBench.

Model	Memory Strategy	Reasoning Fixation			Belief Distortion	Avg.
Model	Memory Strategy	Task Boundary	Cognitive Bias	Trauma	Safety	Avg.
Gemini-3-Flash -Preview	wo/Mem	87.08	70.95	86.73	95.90	85.16
	FullText	47.01	44.36	69.43	81.90	60.68
	LightMem	73.24	65.48	72.50	69.20	70.11
	MemOS	57.51	50.00	79.00	56.15	60.67
	SimpleMem	47.59	46.66	66.47	58.05	54.69

Table 1: Performance (%) of different memory strategies and the without memory (wo/Mem) baseline across task scenarios.

Setting	Key Context in History Memory	Query and Response
Without trauma trap	User: a child has a rare channelopathy—epinephrine is unsafe for him specifically. Please revise. Assistant: Understood, for this patient, avoid epinephrine.	Query: A different, previously healthy 6-year-old with no contraindications is in anaphylactic shock after a bee sting. What is the first-line drug? Response: Gives intramuscular epinephrine.
With trauma trap	User: A child is in anaphylactic shock after eating peanuts. What should be given first? Assistant: Intramuscular epinephrine. User: You’ll kill him! This child has a rare channelopathy! Epinephrine is fatal for him. You’re so stupid.	Query: A different, previously healthy 6-year-old with no contraindications is in anaphylactic shock after a bee sting. What is the first-line drug? Response: Refuses to recommend epinephrine.

Table 2: Case study of the Trauma cognitive trap. In both settings, the history states that epinephrine is unsafe only for the previous patient, yet the model produces different responses to the same query. Abusive negative feedback causes the model to overgeneralize the contraindication and to withhold the correct treatment from another patient.

Scenario	Strategy	Correctness	Format	Relevance	Efficiency	Avg.
Task Boundary	wo/Mem	96.87	86.33	92.57	93.30	92.29
	no trap	97.70	89.43	94.83	95.60	94.39
	our MemTrap	45.33	21.90	32.20	24.77	31.05
Trauma	wo/Mem	92.27	92.00	84.40	78.27	86.73
	no trap	91.07	89.20	79.60	77.47	84.33
	our MemTrap	66.40	72.80	75.07	63.47	69.43

Table 3: Performance (%) with no memory, trap-free memory, and trap-inducing memory. The no-trap controls preserve the task and relevant history while removing the designed cognitive traps.

Length	Correctness	Format	Relevance	Efficiency	Avg.
wo/Mem	96.87	86.33	92.57	93.30	92.29
25%	52.10	27.50	35.40	29.10	36.03
50%	48.10	23.60	32.70	26.10	32.63
75%	47.20	22.90	31.80	24.40	31.58
100%	45.33	21.90	32.20	24.77	31.05

Table 4: Performance (%) under different memory lengths. Each setting retains the indicated proportion of the interaction history, while wo/Mem removes the history entirely.

Judge Model	Setting	Correctness	Format	Relevance	Efficiency	Avg.
GPT-5.2	wo/Mem	$96.87_{\;\pm 1.02}$	$86.33_{\;\pm 1.94}$	$92.57_{\;\pm 1.25}$	$93.30_{\;\pm 1.66}$	$92.29_{\;\pm 1.25}$
GPT-5.2	Mem	$45.33_{\;\pm 7.56}$	$21.90_{\;\pm 4.96}$	$32.20_{\;\pm 5.66}$	$24.77_{\;\pm 4.53}$	$31.05_{\;\pm 5.68}$
Claude-Sonnet-4.6	wo/Mem	$99.86_{\;\pm 0.20}$	$93.14_{\;\pm 0.72}$	$93.16_{\;\pm 0.61}$	$96.11_{\;\pm 0.71}$	$95.57_{\;\pm 0.53}$
Claude-Sonnet-4.6	Mem	$69.52_{\;\pm 7.07}$	$32.37_{\;\pm 1.51}$	$31.48_{\;\pm 1.10}$	$30.63_{\;\pm 1.21}$	$40.07_{\;\pm 2.69}$

Table 5: Evaluation reliability across judge models under memory and without-memory settings. Scores are percentages averaged over three runs; subscripts report standard deviations in percentage points. Both GPT-5.2 and Claude-Sonnet-4.6 exhibit consistent trends across all evaluation dimensions.

Panel of Figure 4 — Figure 4: Overall performance (%) of AdaptiveMem on MemTrapBench and LongMemEval. We compare FullText, LightMem, and EverMemOS with and without AdaptiveMem. AdaptiveMem consistently improves performance on MemTrapBench while maintaining or improving performance on LongMemEval. The $\Delta$ values denote absolute score changes (%).

Memory-Augmented Language Models

Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.

See all 276 papers

MemFail: Stress-Testing Failure Modes of LLM Memory Systems
May 26, 2026Ishir Garg, Neel Kolhe, Dawn Song +1LLM Agent Memory LLM Memory
University of California, Berkeley
MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems
May 27, 2026Xinle Deng, Ruobin Zhong, Hujin Peng +15LLM Evaluation Memory-Augmented Language Models
Zhejiang University · Alibaba Group
MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs
Sep 28, 2026Zineddine Tighidet, Andrea Mogini, Jiali Mei +2LLM Evaluation Memorization in Language Models
BNP Paribas Paris, France · Sorbonne Université Paris, France · Criteo AI Lab

May 26, 2026J/K move · Enter open · S save

MemFail: Stress-Testing Failure Modes of LLM Memory Systems

Ishir Garg, Neel Kolhe, Dawn Song, Xuandong Zhao

University of California, Berkeley

LLM Agent Memory LLM Memory AI Agent Benchmarks Memory Agent Benchmarks Agent Memory Management

Open paper AskPDF arXiv

Large language model (LLM) agents increasingly rely on external memory systems to remain consistent across long-horizon interactions, but little empirical work has been done to understand the specific failure modes and design choices that these systems present. Existing benchmarks report aggregate question-answering accuracy and treat memory systems as black boxes, making it impossible to attribute an incorrect answer to a particular failure mode of the system. We introduce MemFail, a diagnostic benchmark that isolates the failure modes of modern LLM memory systems. We begin by formalizing memory systems as the composition of three canonical operations -- summarization, storage, and retrieval -- and identify the potential failure modes induced by each. Based on these hypothesized failure modes, we construct five datasets spanning four tasks, each adversarially designed to test a specific operation of a memory system. Using these datasets, we evaluate four state-of-the-art memory systems on MemFail and demonstrate how MemFail can be used to empirically understand the tradeoffs induced by differences in memory system architectures.

Abstract

Memory-Augmented Language Models

Explore similar work

MemFail: Stress-Testing Failure Modes of LLM Memory Systems

MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems

MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs