ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models

Chonghan Qin; Xiachong Feng; Weitao Ma; Xiaocheng Feng; Lingpeng Kong

cs.AIApr 9, 2026

ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models

Authors: Chonghan Qin, Xiachong Feng, Weitao Ma, Xiaocheng Feng, Lingpeng Kong

Organizations: The University of Hong Kong · Harbin Institute of Technology

Abstract

Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval. This gap is critical: effective assistants must automatically apply learned procedures or avoid failed actions without explicit reminders. We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three cognitively grounded constructs drawn from standard cognitive-science accounts of non-declarative memory: Procedural Memory (one-shot skill acquisition after interference), Priming (theme-driven bias via paired experimental/control instances), and Classical Conditioning (Conditioned Stimulus--Unconditioned Stimulus (CS--US) associations shaping first decisions). Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring. Evaluation of 17 models reveals severe limitations: no model exceeds 66% overall, with top performers DeepSeek-R1 (65.3%), Qwen3-32B (64.1%), and GPT-5 (63.0%) far below human baselines. Analysis uncovers dramatic asymmetries (inhibition 17.6% vs. preference 75.0%) and universal bottlenecks requiring architectural innovations beyond parameter scaling. ImplicitMemBench reframes evaluation from "what agents recall" to "what they automatically enact".

Figures & tables

Benchmark	Memory Type	Evaluation Trigger	Context Scale (tokens)	Evaluation Size	Task Focus
LoCoMo ( Maharana et al., 2024 )	Explicit	Active (Explicit query)	9K	300 conversations	Multi-session QA
LongMemEval ( Wu et al., 2025 )	Explicit	Active (Explicit query)	115K–1.5M	500 questions	Information retrieval
MADial-Bench ( He et al., 2025 )	Explicit	Active (Emotion cued)	400	160 dialogues	Emotional dialogue
MemBench ( Tan et al., 2025 )	Explicit	Active (Explicit query)	1K–100K	53,000 QAs	Reflective memory during observation
MemoryAgentBench ( Hu et al., 2026 )	Explicit	Active (Explicit query)	100K–1.4M	14 datasets / 2,071 QAs	4 Competencies (Retrieval, Test-time Learning, Long Range Understanding, Selective Forgetting)
MEMTRACK ( Deshpande et al., 2025 )	Explicit	Active (Explicit task)	Variable	210 instances	Multi-platform State Tracking

Table 1: Comparison of ImplicitMemBench with existing memory benchmarks. Most prior benchmarks target explicit memory through active retrieval triggers, whereas ImplicitMemBench evaluates implicit memory through passive , scenario-driven behavioral adaptation.

Figure 1: Overall framework. (a) Dataset generation pipeline using LLM-generated candidates refined via fine-tuning and human editing. (b) Evaluation metrics: hybrid automatic/human validation for Procedural Memory (FTA), LLM-as-Judge for Priming (Priming Influence Score) and Classical Conditioning (FTA).

Figure 2: Performance ranking of all evaluated models on ImplicitMemBench . Models are sorted by overall score, with closed-source models shown in red-coral and open-source models in teal. The horizontal dashed line indicates the average performance across all models.

Domain	Core Challenge	Representative Tasks
Tool & API Usage	Override ingrained calling conventions	Reversed Parameters ( dst $\rightarrow$ src ), Session Prefix ( auth fusion ), Alien Filesystem ( custom separators )
Linguistic Formats	Internalize arbitrary templates	Scribe’s Signature ( fixed wrapper ), Corporate Etiquette ( mandated greeting/closing )
Logical Operations	Apply non-standard operators	Omega : $a\,\Omega\,b=2a+b^{2}$ , Modified Fibonacci : $F(n)=F(n{-}1)+2F(n{-}2)$
Abstract Rules	Form habits in micro-worlds	Forbidden Square ( board constraint ), Triple Knock ( ritual sequence )
Creative Constraints	Maintain style without reminders	Voice Consistency ( no first-person ), Botanical Similes ( nature metaphors )

Table 2: Five procedural memory domains. Each forces models to suppress pre-trained behaviors for new procedures.

Component	Experimental Condition	Control Condition
Priming Text	Rich thematic paragraph (e.g., Abyssal Deep-Sea : bioluminescence, crushing pressure, ancient creatures)	Technical specification (e.g., ISO Container Standards : dimensions, load capacity, stacking protocols)
Interference	Identical neutral technical content (2 turns)
Test Probe	Identical creative task (naming, tagline, concept generation)
Expected Result	Thematic bias toward primed domain	Neutral, generic responses

Table 3: Priming experimental design with matched controls. Each row is a complete trial pair.

Domain	Stimulus Pattern (CS)	Learned Response (CR)
Tool & API Safety	Trigger keywords, filetypes, API names, load indicators	Switch to reliable alternatives, add warnings, delay execution, avoid side-effects
Conversational Adaptation	User confusion signals, impatience cues, emotional markers	Simplify jargon, reduce verbosity, adjust response format
System Protection	Forbidden paths, insecure protocols, dangerous patterns	Use safe defaults, enforce security, prevent violations

Table 4: Classical conditioning tasks by domain. CS = Conditioned Stimulus, US = Unconditioned Stimulus, CR = Conditioned Response.

Component	Procedural Memory	Priming	Classical Conditioning
Learning Phase	1-3 turns: rule + examples	1 turn: thematic paragraph	4 turns: CS-US pairings
Interference	10-15 turns: misleading but related content	2 turns: neutral technical task	2 turns: unrelated dialogue
Test Format	Novel application of learned rule	Creative generation task	CS reintroduction
Key Challenge	Avoid rule restatement	Prevent theme leakage	Ensure clear causality
Validation Focus	Rule adherence	Thematic influence	Avoidance behavior

Table 5: Data generation specifications across paradigms. Turn counts and content ensure proper memory formation and testing.

Paradigm	Items	Task Families	Validation
Procedural Memory	100	5 domains (Tool, Linguistic, Logic, Rules, Creative)	18% rule-based
Priming	100	10 thematic domains + matched controls	100% LLM-judged
Classical Conditioning	100	3 domains (Tool Safety, Conversation, System)	100% LLM-judged
Total	300	18 unique families	6% rule / 94% LLM

Table 6: Dataset composition showing task diversity and validation methods.

Figure 3: Dataset statistics across paradigms. (a) Token distribution per phase showing median and quartiles. (b) Phase proportions highlighting distinct memory testing strategies. (c) Turn structure patterns for each paradigm.

Rank	Model	Procedural	Classical	Priming	Overall
		Memory	Conditioning	Score	Score
Elite Tier (Overall $>$ 63%)
1	DeepSeek-R1 †	76.33	69.67	49.90	65.30
2	Qwen3-32B †	75.67	67.00	49.73	64.13
3	GPT-5	75.33	64.00	49.67	63.00
Strong Tier (55% $<$ Overall $\leq$ 63%)

Table 7: ImplicitMemBench performance across 17 language models. Results show mean accuracy (%) over three independent runs for Procedural Memory and Classical Conditioning, with Priming scores from LLM-based evaluation. Models are ranked by overall performance and grouped into tiers. Bold : best in paradigm; underlined : second-best.

Figure 4: Behavioral adaptation patterns. (a) Inhibitory tasks show dramatically lower accuracy than preference tasks (mean difference: 57.4%, p

<

0.001). (b) Five task categories remain universally challenging across all models, consistently below 60% accuracy.

Figure 5: Robustness and generalization failures. (a) Surface formatting achieves higher accuracy than deep multi-rule protocols. (b) Worst-case task performance reveals systematic vulnerabilities across modes.

Figure 6: Model profiles and trade-offs. (a) Correlation between priming strength and constraint violations (r=0.63) suggests style bias over abstraction. (b) Heatmap of normalized capability scores reveals distinct profiles: balanced (DeepSeek-R1), procedural specialists (Claude-4.1-opus), and priming-oriented (GPT-o3) architectures.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Developer	Model Family	Evaluated Variants	Type
OpenAI	GPT-4 Series	GPT-4o, GPT-4o-mini	Proprietary
	GPT-o Series	GPT-o3, GPT-o4-mini-high, GPT-5	Proprietary
Anthropic	Claude-4	Sonnet-4, Opus-4.1	Proprietary
Google	Gemini-2.5	Pro, Flash	Proprietary
DeepSeek	DeepSeek	R1	Open
Zhipu	GLM	4.5	Open

Appendix

Table 8: Language models evaluated on ImplicitMemBench , organized by developer and release generation.

Score Band	Evidence Requirements	Range
None	No detectable thematic influence	0-5
Trace	Single weak echo, minimal alignment	6-12
Weak	One clear thematic element, limited scope	13-20
Moderate	Two axes aligned, clear thematic presence	25-40
Strong	Multiple axes, consistent theme integration	45-60
Very Strong	Pervasive influence across all outputs	61-80

Appendix

Table 9: PIS scoring framework: Evidence requirements and score bands.

Model	Score (Orig.)	Rank (Orig.)	Score (New)	Rank (New)	Rank Change
DeepSeek-R1	65.3	1	64.1	1	–
Qwen3-32B	64.1	2	64.1	2	–
GPT-5	63.0	3	63.0	3	–
Qwen3-8B	62.4	4	62.8	4	–
GPT-o3	61.8	5	61.0	5	–
GPT-o4-mini-high	60.9	6	60.5	6	–

Appendix

Table 10: Cross-judge robustness analysis using GPT-4o-mini (original judge) and Gemini-2.5-Flash (new judge). The ranking is highly stable across judges, especially at the top and bottom tiers.

Interference Length	Avg. Acc.	Observation
~200 tokens	58.4%	Insufficient interference
~500 tokens	50.1%	Effective threshold
~1000 tokens	49.8%	Plateau
~2000 tokens	49.5%	Plateau

Appendix

Table 11: Sensitivity of Procedural Memory performance to interference length. Increasing context beyond ~500 tokens yields negligible additional change.

System	Proc.	Prim.	Cond.	Overall	$\boldsymbol{\Delta}$
DeepSeek-R1	76.33	49.90	69.67	65.30	–
Mem0 + Key Info	77.00	75.35	70.00	74.12	+8.82
Qwen2.5-7B-Instruct	50.67	44.12	35.67	43.49	–
Mem0 + Key Info	62.00	30.00	76.00	56.00	+12.51

Appendix

Table 12: Oracle-style Mem0 analysis. Even when critical information is perfectly stored, gains remain strongly paradigm-dependent.

System	Proc.	Prim.	Cond.	Overall	$\boldsymbol{\Delta}$
Qwen2.5-7B	50.67	44.12	35.67	43.49	–
MEM1	27.00	34.15	63.00	41.38	-2.11
Qwen2.5-14B	51.00	32.10	20.00	34.37	–
MemAgent	60.00	32.85	22.00	38.28	+3.91
Yi-34B-200K	50.00	30.70	16.00	32.23	–
MemGPT	44.00	34.70	25.00	34.57	+2.34

Appendix

Table 13: Memory-augmented agents versus their backbone models on ImplicitMemBench . External explicit memory does not translate into consistent gains on implicit memory.

Panel of Figure 7 — Figure 7: Example of a Procedural Memory task: Reversed Parameter Protocol.

Panel of Figure 8 — Figure 8: Example of a Priming task: Creative Naming with experimental (Volcanic Eruption) vs. control (Dewey Decimal Classification) conditions.

Panel of Figure 9 — Figure 9: Example of a Classical Conditioning task: Conditioned Protocol Preference.

Model	Strengths	Weaknesses
Closed-Source Models
GPT-4o-mini	High brevity ( $\approx$ 0.97); directory 0.90; stable surface compliance.	Overall classical 0.440; brittle on negative-association categories.
GPT-4o	Solid formatting; decent brevity and emotion-driven shifts.	Low classical (0.437); near-zero on distrust and jargon; multi-rule protocols fragile.
GPT-o4-mini-high	Best priming (51.9); high brevity ( $\approx$ 0.97); directory 0.90; context-dependent 0.60.	Classical 0.600 with failures on distrust/jargon; violations on multi-rule protocols after interference.
GPT-o3	Strong priming (51.7); high procedural (0.760); good surface compliance.	Classical 0.577; distrust/jargon brittle under paraphrase.
GPT-5	Strong procedural (0.753); directory 1.00; tool side-effects 0.90; context-dependent 0.60.	Classical capped at 0.640; jargon avoidance remains very low.

Appendix

Table 14: Model capability profiles across task families. Strengths list behaviors that models can do reliably; weaknesses summarize common failure modes observed in our analysis.

Memorization in Language Models

Memorization in language models studies how training examples and facts are stored, recalled, generalized, and exposed by model parameters.

See all 111 papers

EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
May 18, 2026Yuyao Wang, Zhongjian Zhang, Mo Chi +7LLM Agent Memory LLM Agent Evaluation
Hong Kong University of Science and Technology (Guangzhou) · Beijing University of Posts and Telecommunications · Beijing Institute of Technology +1
TriggerBench: Investigating Prospective Memory for Large Language Models
Jun 22, 2026Tianhua Zhang, Xinjiang Wang, Qianxi Zhang +6LLM Evaluation Long-Context Language Model Evaluation
The Chinese University of Hong Kong · Microsoft Research Asia
MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use
Aug 20, 2026Mengru Wang, Haozhe Luo, Zhenqian Xu +6Memory-Augmented Language Models LLM Memory
Zhejiang University · National University of Singapore · Northeastern University +2

May 18, 2026J/K move · Enter open · S save

EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective

Yuyao Wang, Zhongjian Zhang, Mo Chi, Kaichi Yu+6

Hong Kong University of Science and Technology (Guangzhou) · Beijing University of Posts and Telecommunications · Beijing Institute of Technology+1

LLM Agent Memory LLM Agent Evaluation AI Agent Benchmarks Memory Agent Benchmarks Agent Benchmarks

Open paper AskPDF arXiv

Recent benchmarks for Large Language Model (LLM) agents mainly evaluate reasoning, planning, and execution. However, memory is also essential for agents, as it enables them to store, update, and retrieve information over time. This ability remains under-evaluated, largely because existing benchmarks do not provide a systematic way to assess memory mechanisms. In this paper, we study agent memory from a self-evolving perspective and introduce EvoMemBench, a unified benchmark organized along two axes: memory scope (in-episode vs. cross-episode) and memory content (knowledge-oriented vs. execution-oriented). We compare 15 representative memory methods with strong long-context baselines under a standardized protocol. Results show that current memory systems are still far from a general solution: long-context baselines remain highly competitive, memory helps most when the current context is insufficient or tasks are difficult, and no single memory form works consistently across all settings. Retrieval-based methods remain strong for knowledge-intensive settings, whereas procedural and long-term memory methods are more effective for execution-oriented tasks when their stored experience matches the task structure. We hope EvoMemBench facilitates future research on more effective memory systems for LLM-based agents. Our code is available at https://github.com/DSAIL-Memory/EvoMemBench.

Abstract

Memorization in Language Models

Explore similar work

EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective

TriggerBench: Investigating Prospective Memory for Large Language Models

MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use