Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations

Dongming Jiang; Yi Li; Songtao Wei; Jinxin Yang; Ayushi Kishore; Alysa Zhao; Dingyi Kang; Xu Hu

cs.CLFeb 22, 2026

Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations

Authors: Dongming Jiang, Yi Li, Songtao Wei, Jinxin Yang, Ayushi Kishore, Alysa Zhao, Dingyi Kang, Xu Hu, +3 more

Organizations: University of Texas at Dallas · University of California, Davis · Texas A&M University

Abstract

Agentic memory systems enable large language model (LLM) agents to maintain state across long interactions, supporting long-horizon reasoning and personalization beyond fixed context windows. Despite rapid architectural development, the empirical foundations of these systems remain fragile: existing benchmarks are often underscaled, evaluation metrics are misaligned with semantic utility, performance varies significantly across backbone models, and system-level costs are frequently overlooked. This survey presents a structured analysis of agentic memory from both architectural and system perspectives. We first introduce a concise taxonomy of MAG systems based on four memory structures. Then, we analyze key pain points limiting current systems, including benchmark saturation effects, metric validity and judge sensitivity, backbone-dependent accuracy, and the latency and throughput overhead introduced by memory maintenance. By connecting the memory structure to empirical limitations, this survey clarifies why current agentic memory systems often underperform their theoretical promise and outlines directions for more reliable evaluation and scalable system design.

Figures & tables

Survey	Taxonomy Focus	Memory Mgmt. & Policy	Benchmark Saturation	Metric Validity	Backbone Sensitivity	System Cost & Latency
The AI Hippocampus Jia et al. (2026)	Brain-inspired: implicit, explicit, agentic	(✓)	×	×	×	×
Memory in the Age of AI Agents Hu et al. (2025)	Forms–functions–dynamics	(✓)	×	×	×	×
Toward Efficient Agents Yang et al. (2026b)	Efficiency-focused: memory, tool learning, planning	✓	×	×	×	✓
Rethinking Memory Mechanisms Huang et al. (2026)	Substrate–cognition–subject	✓	(✓)	×	×	×
From Storage to Experience Luo et al. (2026)	Evolutionary: storage–reflection–experience	(✓)	×	×	×	×
Graph-based Agent Memory Yang et al. (2026a)	Graph-oriented lifecycle	✓	(✓)	×	×	×

Table 1: Comparison with related surveys on memory for LLM-based agents. ✓ indicates the topic is systematically discussed; (✓) indicates partial or brief coverage; × indicates the topic is not addressed.

Benchmark	Scalability Dimensions			Theoretical Saturation Risk
Benchmark	Avg. Volume	Interaction Depth	Entity Diversity	Theoretical Saturation Risk
HotpotQA Yang et al. (2018)	$\sim$ 1k Tokens	Single Turn	Low	High (Trivial for Context Window)
LoCoMo Maharana et al. (2024)	$\sim$ 20k Tokens	35 Sessions	High	Moderate (Requires Reasoning)
LongMemEval-S Wu et al. (2024)	103k Tokens	5 Core Abilities	High	Moderate (Borderline)
LongMemEval-M Wu et al. (2024)	$>$ 1M Tokens	5 Core Abilities	High	Low (Requires External Memory)
MemBench Tan et al. (2025a)	$\sim$ 100k Tokens	Fact/Reflection	Medium	High (Fits in 128k Window)

Table 2: Structural saturation risk of memory benchmarks. Benchmarks are analyzed based on intrinsic statistics rather than model performance. Saturation Risk is a heuristic estimate of whether a long-context LLM may solve the benchmark through direct prompting.

Method	Lexical Metric		Semantic Judge Score (Rank)
Method	F1-Score	Rank	Prompt 1 (MAGMA)	Prompt 2 (Nemori)	Prompt 3 (SimpleMem)
AMem Xu et al. (2025b)	0.116	5	0.480 (4)	0.512 (4)	0.482 (4)
MemoryOS Kang et al. (2025a)	0.413	3	0.553 (3)	0.589 (3)	0.552 (3)
Nemori Nan et al. (2025)	0.502	1	0.602 (2)	0.781 (1)	0.649 (2)
MAGMA Jiang et al. (2026a)	0.467	2	0.670 (1)	0.741 (2)	0.665 (1)
SimpleMEM Liu et al. (2026)	0.268	4	0.294 (5)	0.298 (5)	0.289 (5)

Table 3: Robustness of system ranking across evaluation protocols. We compare Lexical metrics (F1) against LLM-based semantic evaluation using three distinct prompt sources: MAGMA, Nemori, and SimpleMem.

Backbone	Method	Answer Score	Format Error
gpt-4o-mini	SimpleMem	0.289	1.20%
gpt-4o-mini	Nemori	0.781	17.91%
Qwen-2.5-3B	SimpleMem	0.102	4.82%
Qwen-2.5-3B	Nemori	0.447	30.38%

Table 4: Backbone Sensitivity Analysis. Frequency of recoverable format deviations during memory operations is used. Higher values indicate greater reliance on fallback parsing due to inconsistent structured outputs.

Method	User-Facing Latency (per turn)			Construction Cost (Offline)
Method	Retrieval ( $T_{read}$ )	Generation ( $T_{gen}$ )	Total (s)	Time (h)	Tokens (k)
Full Context	N/A	1.726	1.726	N/A	N/A
LOCOMO Maharana et al. (2024)	0.415	0.368	0.783	0.86	1,623
AMem Xu et al. (2025b)	0.062	1.119	1.181	15.00	1,486
MemoryOS Kang et al. (2025a)	31.247	1.125	32.372	7.83	4,043
Nemori Nan et al. (2025)	0.254	0.875	1.129	3.25	7,044

Table 5: The "Agency Tax": Efficiency Profiling. We evaluate the trade-off between runtime user latency and offline construction cost. User Latency(

T_{read}+T_{gen}

) dictates the interactive experience, while Construction Cost reflects the scalability and economic feasibility of the system. Note that Maintenance Cost is omitted as it is often handled asynchronously.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

System	Build Strategy (Memory Construction)	Query Strategy	Answer Strategy (Synthesis)
MemoryOS	Profile-based (User profiling, Knowledge extraction)	N/A (Direct Semantic Search)	Role-playing & Profile-enriched
AMem	Flat/Turn-based (Content analysis)	LLM Keyword Extraction	Retrieved Memory Context
Nemori	Episodic (Boundary detection, Episode generation)	N/A (Direct Semantic Search)	Episode-based Retrieval
MAGMA	Graph-based (Event extraction, Multi-hop reasoning)	Multi-hop Entity Parsing	Graph Traversal Synthesis
SimpleMem	Minimalist/Turn-level	Keyword Generation	Context-based

Appendix

Table 6: Taxonomy of System Operation Prompts across Memory Architectures.

Method	Embedding Model	LLM Temp.	Final Answer Top- $k$	System-Specific Defaults (Max Tokens & Structure)
Full Context	N/A	0.3	N/A	Max Tokens: 128k ( gpt-4o-mini )
LOCOMO	MiniLM-L6-v2	0.3	10	Max Tokens: Default; Buffer Size: Default
AMem	MiniLM-L6-v2	0.3	10	Max Tokens: Default; Keyword Extractor Temp: Default
MemoryOS	MiniLM-L6-v2	0.3	10	Max Tokens: Default; Update Frequency: Default
Nemori	MiniLM-L6-v2	0.3	10	Max Tokens: Default; Boundary Temp: 0.1
MAGMA	MiniLM-L6-v2	0.3	10	Max Tokens: Default; Consolidation Threshold: Default

Appendix

Table 7: Key hyper-parameter configurations for the evaluated memory architectures. To ensure fair comparison, embedding models, LLM generation temperatures, and final answer retrieval scopes are strictly standardized across all baselines, while structural capacity parameters (e.g., max tokens) follow repository defaults.

Method	Memory Structure	Update Policy	Retrieval Scope
A-MEM	Linked Node Graph (Atomic Units + Tags)	Evolutionary: LLM-based node rewriting & dynamic linking	Dense Embedding Similarity (Top- $k$ )
MemoryOS	Hierarchical Tiers (STM $\rightarrow$ LTM)	Rule-based: Frequency/Recency-based promotion	Cascading Hierarchy Search
MAGMA	Multi-relational Graph + Vector Index	Asynchronous: Dual-stream consolidation (Long-term)	Intent-guided Subgraph Traversal
Nemori	Dual Memory (Episodic Tree + Semantic Graph)	Gradient-inspired: Contextual memory modification	Hybrid (Top- $k$ Episodes + Semantic Facts)
SimpleMem	Hybrid Index (Dense/Sparse) of Compressed Units	Synchronous: On-the-fly synthesis & deduplication	Planner-guided Multi-view Search

Appendix

Table 8: Overview of memory systems and experimental configurations. We use gpt-4o-mini as the primary controller for all methods in the main benchmark to normalize reasoning costs.

Query & Gold Truth	Model Answer	Failure Type	F1	Judge	Analysis
Q: What is the duration of the event? Gold: 18 days	The total duration was 18 days.	Surface Variation	0.50	1.00	Additional phrasing lowers lexical precision despite identical meaning.
Q: What time does the event start? Gold: 14:00	2 PM	Semantic Equivalence Gap	0.00	1.00	Equivalent time representations yield zero token overlap.
Q: Describe the price level. Gold: cheap	inexpensive	Semantic Equivalence Gap	0.00	1.00	Synonym substitution is not captured by lexical matching.
Q: Is the software compatible with Mac? Gold: compatible with Mac	not compatible with Mac	Polarity Flip	0.857	0.00	Negation reverses meaning while preserving token overlap.
Q: Who completed the project? Gold: John completed the project	Sarah completed the project	Entity Drift	0.75	0.00	Incorrect entity maintains structure but changes semantics.
Q: How many items were included? Gold: three items	five items	Entity Drift	0.50	0.00	Numerical substitution is partially rewarded due to shared tokens.

Appendix

Table 9: Mechanism Oriented Failure Cases of Lexical Metrics. Lexical scores (F1) are contrasted with semantic judgments to highlight systematic mismatches.

Explore similar work

Are We Ready For An Agent-Native Memory System?
Jun 23, 2026Wei Zhou, Xuanhe Zhou, Shaokun Han +5Agent Memory LLM Agent Memory
Shanghai Jiao Tong University · Tsinghua University · MemTensor (Shanghai) Technology Co., Ltd
AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications
Feb 26, 2026Yujie Zhao, Boqin Yuan, Junbo Huang +9Agent Memory Long-Horizon Agent Evaluation
UCSD · Recursive.
Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers
Mar 8, 2026Pengfei DuLLM Agent Memory Agent Memory Management
Hong Kong Research Institute of Technology, HongKong, China

Jun 23, 2026J/K move · Enter open · S save

Are We Ready For An Agent-Native Memory System?

Wei Zhou, Xuanhe Zhou, Shaokun Han, Hongming Xu+4

Shanghai Jiao Tong University · Tsinghua University · MemTensor (Shanghai) Technology Co., Ltd

Agent Memory LLM Agent Memory Agent Memory Management

Open paper AskPDF arXiv

Memory for large language model (LLM) agents has rapidly evolved from simple retrieval-augmented mechanisms into a data management system that supports persistent information storage, retrieval, update, consolidation, and dynamic lifecycle governance throughout agent execution. Despite this evolution, existing evaluations still benchmark agent memory mainly through end-to-end task success metrics (e.g., F1, BLEU), while treating the underlying system as a monolithic black box. As a result, critical system-level concerns, including operational costs, architectural trade-offs across memory modules, and robustness under dynamic knowledge updates, remain insufficiently explored. In this paper, we present a systematic experimental study of agent memory from a data management perspective. We propose an analytical framework that decomposes agent memory into four core modules: memory representation and storage, extraction, retrieval and routing, and maintenance. Under this framework, we evaluate 12 representative memory systems and two reference baselines across five benchmark workloads spanning 11 datasets. Our extensive end-to-end evaluation shows that no single architecture dominates across all scenarios; instead, effectiveness depends heavily on how well the memory structure aligns with the workload bottleneck. Furthermore, through fine-grained ablation studies, we quantify their individual effects on representation fidelity, retrieval precision, update correctness, and long-horizon stability. Finally, we reveal cost-performance trade-offs under realistic workloads, showing localized maintenance is more cost-efficient than global reorganization. Based on these findings, we identify promising directions towards building truly agent-native memory systems. The code is publicly available at https://github.com/OpenDataBox/MemoryData.

Abstract

Explore similar work

Are We Ready For An Agent-Native Memory System?

AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications

Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers