Toward a Theory of Hierarchical Memory for Language Agents

Yashar Talebirad; Ali Parsaee; Csongor Y. Szepesvari; Amirhossein Nadiri; Osmar Zaiane

cs.IRMar 23, 2026

Toward a Theory of Hierarchical Memory for Language Agents

Authors: Yashar Talebirad, Ali Parsaee, Csongor Y. Szepesvari, Amirhossein Nadiri, Osmar Zaiane

Organizations: University of Alberta · York University

Abstract

Many recent long-context and agentic systems address context-length limitations by adding hierarchical memory: they extract atomic units from raw data, build multi-level representatives by grouping and compression, and traverse this structure to retrieve content under a token budget. Despite recurring implementations, there is no shared formalism for comparing design choices. We propose a unifying theory in terms of three operators. Extraction ( $α$ ) maps raw data to atomic information units; coarsening ( $C = (π, ρ)$ ) partitions units and assigns a representative to each group; and traversal ( $τ$ ) selects which units to include in context given a query and budget. We identify a self-sufficiency spectrum for the representative function $ρ$ and show how it constrains viable retrieval strategies (a coarsening-traversal coupling). Finally, we instantiate the decomposition on eleven existing systems spanning document hierarchies, conversational memory, and agent execution traces, showcasing its generality.

Figures & tables

Figure 1: The $(\alpha,C,\tau)$ pipeline: $D$ is extracted ( $\alpha$ ) into atoms $G_{0}$ ; coarsening $C_{1},C_{2},C_{3}$ yields layers $G_{1},G_{2},G_{3}$ ; traversal $\tau$ takes a query and budget and returns a subset $S$ of atoms.

System	Type	$\alpha$	$\pi$ ; $\rho$ (SS)	$\tau$
RAPTOR	D	100-tok chunks	UMAP+GMM; LLM summary (high)	collapsed / top-down
GraphRAG	D	entity/rel extract	Leiden; community report (high)	global / entity fan-out
xMemory	D	episode boundaries	guided split/merge; fact distill (mid)	top-down + expand
H-MEM	D	episode + profile	LLM 4-level; domain labels (low)	top-down FAISS
SimpleMem	D	window + density	online synthesis; consolidated (high)	multi-view parallel
Mastra’s OM	D	token threshold	all-in-one; Reflector (high)	full log

Table 1: Data and agent execution-trace systems as

(\alpha,C,\tau)

. Type: D = data, T = trace.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

System	Affinity $W(u,v)$	Partition method
RAPTOR	$\cos(\phi_{\mathrm{emb}}(u),\phi_{\mathrm{emb}}(v))$	UMAP + GMM
GraphRAG	connectivity in $E$	Leiden community detection
H-MEM	LLM-judged domain co-membership	LLM 4-level classification
xMemory	cosine + sparsity-semantic score	guided split/merge
SimpleMem	semantic + temporal proximity	online LLM synthesis
Mastra’s OM	temporal contiguity	token-count threshold

Appendix

Table 3

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 373 papers

Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents
Jun 10, 2026Hao-Lun Hsu, Nikki Lijing Kuang, Boyi Liu +2Hierarchical Memory Long-Horizon Agent Tasks
Duke University · Snowflake AI Research
Memora: A Harmonic Memory Representation Balancing Abstraction and Specificity
Feb 3, 2026Menglin Xia, Xuchao Zhang, Shantanu Dixit +6Agent Memory Retrieval-Augmented Generation
M365 Research, Microsoft.
Contextual Agentic Memory is a Memo, Not True Memory
Apr 30, 2026Binyan Xu, Xilin Dai, Kehuan ZhangAgent Memory Continual Learning for LLM Agents
The Chinese University of Hong Kong, Hong Kong, China · Zhejiang University, Hangzhou, China

Jun 10, 2026J/K move · Enter open · S save

Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents

Hao-Lun Hsu, Nikki Lijing Kuang, Boyi Liu, Zhewei Yao+1

Duke University · Snowflake AI Research

Hierarchical Memory Long-Horizon Agent Tasks Agent Memory Management Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Large language model (LLM) agents struggle with long-horizon tasks due to their inherent statelessness, requiring all task-relevant information to be encoded in growing input contexts. The resulting degraded reasoning quality, increased inference cost, and higher latency necessitate efficient working memory mechanisms. However, existing approaches either rely on lossy compression or similarity-based retrieval, which often fail to capture temporal structure and causal dependencies required for multi-step agentic tasks. In this work, we present HORMA, a Hierarchical Organize-and-Retrieve Memory Agent that organizes experience into a file-system-like hierarchical structure, where summarized entities are linked to the corresponding raw trajectories, enabling efficient access without losing detailed information. HORMA decomposes working memory into two stages: structured memory construction and navigation-based retrieval. The construction module iteratively refines how experiences are structured by distinguishing between failures caused by missing information and those caused by misleading or overloaded context. The navigation module retrieves task-relevant context by traversing the hierarchy using a lightweight agent trained with reinforcement learning to select minimal yet sufficient context, thereby reducing latency along the critical execution path. Across ALFWorld, LoCoMo, and LongMemEval, HORMA improves task performance under constrained context budgets while requiring at most 22.17% of the baseline token usage in long conversation tasks. Compared to existing methods, it consistently achieves better efficiency-performance trade-offs and generalizes effectively to unseen tasks.

Abstract

LLM Agent Memory

Explore similar work

Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents

Memora: A Harmonic Memory Representation Balancing Abstraction and Specificity

Contextual Agentic Memory is a Memo, Not True Memory