NextMem: Towards Latent Factual Memory for LLM-based Agents

Zeyu Zhang; Rui Li; Xiaoyan Zhao; Yang Zhang; Wenjie Wang; Xu Chen; Tat-Seng Chua

cs.AIFeb 26, 2026

NextMem: Towards Latent Factual Memory for LLM-based Agents

Authors: Zeyu Zhang, Rui Li, Xiaoyan Zhao, Yang Zhang, Wenjie Wang, Xu Chen, Tat-Seng Chua

Organizations: Renmin University of China · The Chinese University of Hong Kong · National University of Singapore · University of Science and Technology of China

Abstract

Memory is critical for LLM-based agents to preserve past observations for future decision-making, where factual memory serves as its foundational part. However, existing approaches to constructing factual memory face several limitations. Textual methods impose heavy context and indexing burdens, while parametric methods suffer from catastrophic forgetting and high costs. To address these challenges, we introduce NextMem, a latent factual memory framework that utilizes an autoregressive autoencoder to efficiently construct latent memory while ensuring accurate reconstruction. For better optimization, we propose a two-stage training process, including autoregressive reconstruction alignment and progressive latent substitution. We also incorporate quantization to reduce storage overhead. Extensive experiments demonstrate that NextMem achieves superior performance, and excels in retrieval, robustness, and extensibility properties. We release our code and model checkpoints at https://github.com/nuster1128/NextMem.

Figures & tables

Figure 1: Comparison between task-oriented and factual memory.

Figure 2: Overview of NextMem framework.

Datasets	Methods	F1	ROUGE-1	ROUGE-L	METEOR	BLEU	BertScore
HotpotQA	DyPRAG	0.0305	0.0347	0.0337	0.0187	0.0000	0.7983
	DeepSeek-OCR	0.4540	0.5492	0.5374	0.3987	0.2432	0.8664
	ICAE	0.7890	0.8570	0.8340	0.7493	0.5782	0.9581
	NextMem-Dense	0.9820	0.9862	0.9854	0.9820	0.9633	0.9966
	NextMem-Sparse	0.9805	0.9842	0.9833	0.9810	0.9612	0.9962
RACE	DyPRAG	0.0696	0.0826	0.0689	0.0341	0.0000	0.8158

Table 1: Performance comparison of factual reconstruction ( i.e., task 1 for memory storage) across multiple datasets. In each assessment, values in bold represent the best results, and those with an underline represent the second-best results.

Methods	HotpotQA		SQuAD		LoCoMo		LongMemEval
Methods	Comp.	DeComp.	Comp.	DeComp.	Comp.	DeComp.	Comp.	DeComp.
DyPRAG	0.5000	0.3789	0.2659	0.2023	0.0191	0.0239	0.0800	0.0971
DeepSeek-OCR	0.5673	0.3744	0.2124	0.2225	0.0766	0.1435	0.1943	0.1543
ICAE	0.8565	0.8229	0.7775	0.7066	0.5407	0.5215	0.4971	0.5029
NextMem-Dense	0.5179	0.8072	0.3223	0.7572	0.2871	0.5407	0.1971	0.5400
NextMem-Sparse	0.4978	0.8184	0.3092	0.7630	0.2679	0.5263	0.2029	0.5486

Table 2: Performance comparison of contextual generation ( i.e., task 2 for memory utilization) across various datasets. For all non-oracle methods, the best results are highlighted in bold , and the second-best results are underlined .

Datasets	Methods	Hit@5	Recall@5	MRR@5	MAP@5	DCG@5	NDCG@5
HotpotQA	DeepSeek-OCR	0.3358	0.1487	0.1659	0.0730	0.2260	0.1171
	ICAE	0.4453	0.2217	0.3187	0.1555	0.4058	0.2126
	NextMem-Dense	0.7245	0.3925	0.5194	0.2793	0.6673	0.3680
	NextMem-Sparse	0.7208	0.4030	0.5107	0.2788	0.6683	0.3687
	*BGE	0.9585	0.6681	0.8063	0.5442	1.1756	0.6438
LoCoMo	DeepSeek-OCR	0.0676	0.0321	0.0269	0.0115	0.0383	0.0206

Table 3: Performance comparison of dense passage retrieval (task 3 for memory retrieval) across various datasets. For all reconstruction models, the best results are highlighted in bold , and the second-best results are underlined .

Methods	F1	ROUGE-L	METEOR	BertScore
Dense	0.8552	0.8580	0.8691	0.9735
w/o ST	0.3799	0.3804	0.4048	0.7307
w/o PT	0.0159	0.0138	0.0169	0.7686
w/o PS	0.7389	0.7358	0.7353	0.9502
Sparse	0.8554	0.8583	0.8705	0.9731
w/o SQ	0.0309	0.0290	0.0442	0.7521

Table 4: Results of ablation study. The best results are highlighted in bold , and the second-best results are underlined .

Figure 3: Results under varying compression ratios.

Figure 4: Robustness results of latent representations under varying levels of Gaussian noise ( $\sigma$ ) and NF4 quantization.

Figure 5: Semantic assignment analysis of latent memory.

Figure 6: Curves of training and evaluation loss.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Results of extensive experiments.

Quantization

See all 49 papers

ElasticMem: Latent Memory as a Learnable Resource for LLM Agents
May 29, 2026Tao Feng, Chongrui Ye, Tianyang Luo +5LLM Agent Memory Persistent Memory for Language Models
University of Illinois Urbana-Champaign · Nanyang Technological University
Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory
May 19, 2026Jingwei Sun, Jianing Zhu, Jiangchao Yao +2Memorization in Language Models LLM Agent Memory
TMLR Group, Hong Kong Baptist University · The University of Texas at Austin · Shanghai Jiao Tong University +1
MemTrain: Self-Supervised Context Memory Training
Jun 2, 2026Ziheng Li, Xingrun Xing, Haoqing Wang +2LLM Agent Memory Persistent Memory for Language Models
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Samsung Research, Beijing, China

May 29, 2026J/K move · Enter open · S save

ElasticMem: Latent Memory as a Learnable Resource for LLM Agents

Tao Feng, Chongrui Ye, Tianyang Luo, Jingjun Xu+4

University of Illinois Urbana-Champaign · Nanyang Technological University

LLM Agent Memory Persistent Memory for Language Models Agent Memory Management Memory-Augmented Language Model Agents Memory Management for LLM Agents

Open paper AskPDF arXiv

Long-term memory is essential for LLM agents to reason coherently across extended interactions, personalize responses, and reuse past experience. However, existing memory-augmented methods typically treat memory as a fixed resource: text-space approaches concatenate retrieved memories into the context window, causing substantial token overhead and sensitivity to noisy evidence, while latent-space approaches reduce textual cost but still rely on rigid retrieval or fixed-capacity memory interfaces. This creates a mismatch between query-dependent memory utility and fixed memory allocation. We propose ElasticMem, a memory-augmented LLM framework that learns to use memory as an elastic latent resource. ElasticMem builds an offline latent memory bank with retrieval keys and content caches, retrieves memories adaptively from the reasoner's hidden state, assigns each retrieved memory a variable latent budget through a learned policy, and injects selected latent states as soft memory tokens for generation. The full memory-use process is optimized with downstream task rewards through group-relative policy optimization. We evaluate ElasticMem on MemorySuite, covering memory-intensive QA and embodied agent control. Across Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct backbones, ElasticMem improves weighted average QA accuracy by 26.2% and 24.6%, and improves ALFWorld success rate by 66.3% and 27.2%, respectively, over the strongest baselines, while achieving the lowest ALFWorld token cost. Ablations and qualitative analyses further show that adaptive retrieval and elastic budget allocation help ElasticMem prioritize useful evidence and transferable plans beyond rigid cosine similarity. Our code for ElasticMem will be released at https://github.com/ulab-uiuc/ElasticMem.

Abstract

Quantization

Explore similar work

ElasticMem: Latent Memory as a Learnable Resource for LLM Agents

Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory

MemTrain: Self-Supervised Context Memory Training