All-Mem: Agentic Lifelong Memory via Dynamic Topology Evolution

Can Lv; Heng Chang; Shengyu Tao; Mingju Chen; Zhaoxin Fan; Ziwei Zhang; Yuchen Guo; Shiji Zhou

cs.IRMar 20, 2026

All-Mem: Agentic Lifelong Memory via Dynamic Topology Evolution

Authors: Can Lv, Heng Chang, Shengyu Tao, Mingju Chen, Zhaoxin Fan, Ziwei Zhang, Yuchen Guo, Shiji Zhou

Organizations: Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing · Beihang University · Tsinghua University · Chalmers University of Technology

Abstract

Lifelong interactive agents are expected to assist users over months or years, which requires continually writing long term memories while retrieving the right evidence for each new query under fixed context and latency budgets. Existing memory systems often degrade as histories grow, yielding redundant, outdated, or noisy retrieved contexts. We present All-Mem, an online/offline lifelong memory framework that maintains a topology structured memory bank via explicit, non destructive consolidation, avoiding the irreversible information loss typical of summarization based compression. In online operation, it anchors retrieval on a bounded visible surface to keep coarse search cost bounded. Periodically offline, an LLM diagnoser proposes confidence scored topology edits executed with gating using three operators: Split, Merge, and Update, while preserving immutable evidence for traceability. At query time, typed links enable hop bounded, budgeted expansion from active anchors to archived evidence when needed. Experiments on LoCoMo and LongMemEval-s show improved retrieval and QA over representative baselines. The code is available at https://github.com/LvCan926/All-Mem.

Figures & tables

Figure 1: Comparison of prior works and All-Mem along real-time constraints, structural drift, and safe maintenance (traceability).

Figure 2: All-Mem framework. (a) Online/offline decoupling with id-level buffering. (b) ATC performs parallel diagnosis with confidence-gating and non-destructive topology edits (Split $\rightarrow$ Merge $\rightarrow$ Update) while preserving versioned traceability. (c) Topology-aware retrieval anchors on the visible surface, performs budgeted typed-link expansion, and selects final memories for the agent.

	LoCoMo						LongMemEval-s
Method	4o-J	F1	B-1	R-L	R@5	N@5	4o-J	F1	B-1	R-L	R@5	N@5
Full History	41.39	29.57	26.76	29.03	-	-	47.00	15.30	17.18	25.62	-	-
Naive RAG	37.94	32.56	29.05	33.94	28.08	26.16	45.80	24.92	16.76	25.11	71.83	69.57
MemGPT Packer et al. (2024)	33.19	25.34	20.51	25.74	24.47	21.04	42.80	20.25	12.84	20.70	70.24	67.43
A-Mem Xu et al. (2025)	40.74	34.59	30.14	34.56	31.94	28.75	50.40	30.82	22.77	30.91	81.36	78.24
HippoRAG 2 Gutiérrez et al. (2025)	42.37	34.79	29.67	33.56	34.95	33.19	53.20	32.90	24.35	32.80	84.13	81.76

Table 1: Performance on LoCoMo and LongMemEval-s. Answer quality: 4o-J/F1/BLEU-1/ROUGE-L; retrieval: R@5/N@5. Caveat: LoCoMo uses turn-level ID matching, while LongMemEval-s uses session-level matching, so retrieval scores are not directly comparable across datasets.

Figure 3: Accuracy–tokens sweep on LongMemEval-s. Each point is one retrieval-budget setting. All-Mem and A-Mem vary

(K,k,L)

; Mem0 varies

(K,k)

Figure 4: Sensitivity analysis on LoCoMo. Left: varying the ATC threshold

\theta

affects F1 and visible-surface size

|V^{+}|

. Right: varying the semantic out-degree cap

d_{\sigma}

affects F1 and memory-module latency. Defaults

\theta{=}0.9

and

d_{\sigma}{=}8

give the best trade-off.

Component	Latency	Tokens
Online writing	2.38s / turn	539 / turn
Offline ATC event	12.79s / event	74.1k / event
Amortized offline	0.21s / turn	1237 / turn
Query-time retrieval	27.54ms / query	918 / query

Table 2: Overall cost profile of All-Mem on LongMemEval-s. Online/offline tokens count LLM input+output tokens for memory construction or maintenance; retrieval tokens denote mean generator input tokens per query.

Component	Total Count	Time / Event	Tok. / Event
Diagnosis	621k	1.57s	48.2k
Split	12.6k	1.77s	4.0k
Merge	87.9k	7.77s	18.2k
Update	25.1k	1.68s	3.8k
Edits total	125.6k	11.22s	25.9k
ATC total	8.37k	12.79s	74.1k

Table 3: Offline ATC cost decomposition on LongMemEval-s. Counts report total diagnoses, executed edits, or offline events over the full run; event-level costs are averaged over offline ATC events.

Variant	4o-J $\uparrow$	F1 $\uparrow$	R@5 $\uparrow$	Mem-Lat $\downarrow$
All-Mem (Full)	54.63	52.18	46.63	23.02
No-Visibility	50.87	47.63	41.84	34.58
w/o Merge	52.08	49.02	43.96	27.40
Anchors-only	51.76	49.27	43.19	14.27
No-recovery-links	52.74	50.06	44.07	22.67

Table 4: Selected ablations on LoCoMo. Mem-Lat is mean memory-module latency (Stage 1–3; excluding generator inference). Full ablations are in Appendix E (Table 7 ).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	LoCoMo	LongMemEval-s
Conversations	10	500
Avg. sessions / conv.	27.2	50.2
Avg. queries / conv.	198.6	1.0
Avg. tokens / conv.	20.1k	103.1k
Session dates	Yes	Yes
Retrieval labels	Yes	Yes

Appendix

Table 5: Dataset statistics for the evaluated benchmarks.

Component	Setting
Generator	GPT-4o-mini, temperature $0$
All-Mem LLM calls	GPT-4o-mini, temperature $0$
Embedding model	All-MiniLM-L6-v2
Stage-1 anchors	$k=10$
Hop budget	$H_{q}=4$
Expansion budget	$L=40$

Appendix

Table 6: Default configuration for All-Mem in the main experiments.

Variant	LoCoMo				LongMemEval-s
	4o-J	F1	R@5	N@5	4o-J	F1	R@5	N@5
All-Mem (Full)	54.63	52.18	46.63	41.02	60.20	45.19	94.68	93.27
No-Visibility	50.87	47.63	41.84	34.76	56.20	40.60	91.24	89.43
w/o Split	53.21	50.94	45.12	39.58	59.00	43.30	93.76	92.15
w/o Merge	52.08	49.02	43.96	38.14	57.80	42.10	92.46	90.73
w/o Update	52.67	49.74	44.41	39.02	58.20	42.60	92.98	91.47

Appendix

Table 7: Full ablation results on LoCoMo and LongMemEval-s. Answer quality is measured by 4o-J and F1; retrieval quality is measured by R@5 and N@5. For LongMemEval-s, retrieval is session-level and should be interpreted together with downstream QA.

Panel of Figure 5 — Figure 5: Efficiency diagnostics. (a) The searchable visible surface $|\mathcal{V}_{N}^{+}|$ grows more slowly than the append-only reference $|\mathcal{V}_{N}^{+}|=N$ . (b) Query-time memory latency is decomposed into Stage 1 dense retrieval, Stage 2 typed-link expansion, and Stage 3 re-ranking. (c) Stage 1 Top- $k$ latency when searching over the full bank $\mathcal{V}_{N}$ versus the visible surface $\mathcal{V}_{N}^{+}$ at consolidation checkpoints; shaded bands denote interquartile ranges.

Figure 6: Budgeted recoverability analysis. Coverage

\mathrm{Cov}(H)

of archived units reachable within hop budget

H

over the directed typed-link topology

\mathcal{E}_{N}

. The dashed line marks the hop budget achieving 95% coverage (

H_{0.95}=4

); the median distance is

H=2

Op.	Audit	Valid	Harmful
Split	40	90.0%	5.0%
Merge	80	93.8%	3.8%
Update	70	95.7%	2.9%
All	190	94.2%	3.7%

Appendix

Table 8: Human audit of accepted ATC edits. “Audit” is the number of accepted edits manually inspected. “Valid” is the fraction judged semantically correct. “Harmful” is the fraction judged likely to degrade future retrieval.

Figure 7: Case study: Update with budgeted recovery. A user first states that they tried three Korean restaurants, and later updates the fact to four . (a) Before consolidation, both the older and newer evidence units may compete during retrieval. (b) ATC applies Update : the newer unit remains on the visible surface $\mathcal{V}_{N}^{+}$ , while the superseded unit is archived and connected through a version link. (c) At query time, Stage 1 anchors on the visible surface and retrieves the current fact; Stage 2 can still recover the archived predecessor through typed-link expansion if supporting history is needed. Stage 3 prioritizes the up-to-date evidence, yielding the correct answer under the context budget.

Agent Memory Management

Agent memory management studies how AI agents store, update, retrieve, compress, and forget information across long-running tasks.

See all 446 papers

Infini Memory: Maintainable Topic Documents for Long-Term LLM Agent Memory
Jun 9, 2026Suozhao Ji, Baodong Wu, Zehao Wang +8Agentic RAG Persistent Memory for Language Models
1Infinigence AI · 2Tsinghua University · 3Shanghai Jiaotong University
LeanMem: Simple and Efficient Long-Term Memory for LLM Agents
Aug 4, 2026Yuxin Liao, Le Wu, Min Hou +3LLM Agent Memory Persistent Memory for Language Models
Hefei University of Technology
RaMem: Contextual Reinstatement for Long-term Agentic Memory
Jun 22, 2026Wei Yang, Bryce Kan, Shixuan Li +5Agent Memory LLM Agent Memory
University of Southern California

Jun 9, 2026J/K move · Enter open · S save

Infini Memory: Maintainable Topic Documents for Long-Term LLM Agent Memory

Suozhao Ji, Baodong Wu, Zehao Wang, Lei Xia+7

1Infinigence AI · 2Tsinghua University · 3Shanghai Jiaotong University

Agentic RAG Persistent Memory for Language Models Memory Consolidation Agent Memory Management LLM Agent Memory Memory Management for LLM Agents+1

Open paper AskPDF arXiv

Long-term LLM agents need persistent memory that can track changing facts and provide relevant evidence across sessions. Existing memory systems often store observations as isolated records, summaries, or indexed fragments, which makes evidence aggregation, fact revision, and memory maintenance difficult. We propose Infini Memory, a maintainable text-based persistent memory architecture that treats agent memory as topic-structured documents. Each topic document serves as a semantic unit for collecting related evidence, preserving metadata, and revising facts over time. New observations are first staged in a buffer and periodically consolidated into coherent textual contexts. At inference time, an agentic retrieval procedure lets the LLM read memory through iterative tool calls rather than a single retrieval step. On MemoryAgentBench, Infini Memory achieves 64.7% overall score. Ablations show that topic-structured maintenance and iterative evidence inspection improve complementary aspects of long-term memory use.

Abstract

Agent Memory Management

Explore similar work

Infini Memory: Maintainable Topic Documents for Long-Term LLM Agent Memory

LeanMem: Simple and Efficient Long-Term Memory for LLM Agents

RaMem: Contextual Reinstatement for Long-term Agentic Memory