LSTMem: Hierarchical Long Short-Term Online Memory for Large Language Models

Xianglong Shi; Ruijie Yang; Sirui Zhao; Shukang Yin; Zihao Bian; Tinghao Yi; Enhong Chen

cs.AISep 27, 2026

LSTMem: Hierarchical Long Short-Term Online Memory for Large Language Models

Authors: Xianglong Shi, Ruijie Yang, Sirui Zhao, Shukang Yin, Zihao Bian, Tinghao Yi, Enhong Chen

Organizations: University of Science and Technology of China

Abstract

Large language models increasingly serve as long-horizon assistants and agents, where they must both accumulate information across interactions and make the relevant parts available when later requests depend on them. Existing compact online memories typically use a single persistent state both to accumulate history and to serve readout, so what the memory stores cannot be controlled separately from what it exposes to the current computation. We propose LSTMem, an LSTM-inspired online memory that instead equips each layer of a frozen LLM with two matrix-valued states: a cell state that accumulates history and a hidden state whose readouts correct the backbone's attention. Input and forget gates control what the cell stores, while an output gate separately controls what the cell exposes through the hidden state. LSTMem further connects memory across depth through forward hidden-state propagation and block-end feedback, and uses higher-layer reconstruction gradients to refine lower-layer cell states before rebuilding hidden states from shallow to deep layers. Across memory benchmarks on Qwen3-4B-Instruct, LSTMem consistently improves MemoryAgentBench, LoCoMo, and HotpotQA over the plain backbone. Comparisons further show that the LSTM-based memory formulation outperforms an associative-memory counterpart, while removing cross-layer hidden-memory propagation degrades performance. These results demonstrate the benefits of separating memory accumulation from memory expression and organizing memory hierarchically across model depth. The code is available at https://github.com/Longchentong/LSTMem.

Figures & tables

External Memory
Model	HotpotQA $\uparrow$		MemoryAgentBench $\uparrow$
Model	EM	F1	Avg.	AR	TTL	LRU	SF
Qwen3-4B-Instruct	$42.35$	$56.00$	$29.54$	$35.30$	$26.14$	$\mathbf{47.08}$	$14.37$
+ BM25 RAG	$40.35_{\scriptstyle-2.00}$	$52.83_{\scriptstyle-3.17}$	$24.43_{\scriptstyle-5.11}$	$32.20_{\scriptstyle-3.10}$	$9.74_{\scriptstyle-16.40}$	$37.86_{\scriptstyle-9.22}$	$15.00_{\scriptstyle+0.63}$
+ LLMLingua-2	$36.93_{\scriptstyle-5.42}$	$50.03_{\scriptstyle-5.97}$	$15.63_{\scriptstyle-13.91}$	$21.45_{\scriptstyle-13.85}$	$1.43_{\scriptstyle-24.71}$	$38.45_{\scriptstyle-8.63}$	$8.62_{\scriptstyle-5.75}$
+ MemoryBank	—	—	$17.65_{\scriptstyle-11.89}$	$22.65_{\scriptstyle-12.65}$	$7.67_{\scriptstyle-18.47}$	$36.36_{\scriptstyle-10.72}$	$9.88_{\scriptstyle-4.49}$

Table 1: Main results across three base models. Bold marks the highest score within each model group; subscripts give point changes from its unadapted model. Baseline scores are from Lei et al. (2026b) . HotpotQA is scored by exact match (EM) and token-level F1 against the gold answer. AR, TTL, LRU and SF denote accurate retrieval, test-time learning, long-range understanding and selective forgetting. AR and SF are scored by accuracy, TTL by classification accuracy and movie-recommendation Recall@5, and LRU by summarization F1 and Detective QA accuracy. MAB averages use 2,000/700/171/800 question weights for AR/TTL/LRU/SF; baseline averages are approximated from rounded family scores.

Model	Avg.	Multi	Temp.	Open	Single
Qwen3-4B-Instruct	40.79	38.39	32.89	10.77	48.05
+ BM25 RAG	36.68	38.12	20.34	9.99	45.47
+ LLMLingua-2	40.98	39.07	30.13	10.98	49.19
+ MemoryBank	38.14	37.88	21.76	13.35	47.31
+ Context2LoRA	48.11	37.95	34.99	16.75	60.11
+ MemGen	40.05	32.93	33.30	12.67	48.13

Table 2: LoCoMo F1 (%). Avg. is weighted over 1,540 questions. Multi, Temp., Open and Single denote multi-hop, temporal, open-domain and single-hop questions.

Model	Avg.	Multi	Temp.	Open	Single
Qwen3-4B-Instruct	40.79	38.39	32.89	10.77	48.05
+ BM25 RAG	36.68	38.12	20.34	9.99	45.47
+ LLMLingua-2	40.98	39.07	30.13	10.98	49.19
+ MemoryBank	38.14	37.88	21.76	13.35	47.31
+ Context2LoRA	48.11	37.95	34.99	16.75	60.11
+ MemGen	40.05	32.93	33.30	12.67	48.13

Table 2: LoCoMo F1 (%). Avg. is weighted over 1,540 questions. Multi, Temp., Open and Single denote multi-hop, temporal, open-domain and single-hop questions.

Model	IFEval $\uparrow$	GPQA-D $\uparrow$
Qwen3-4B-Instruct	81.89	39.39
+ Context2LoRA	76.71	29.29
+ MemGen	39.37	38.89
+ MLP Memory	24.95	22.73
+ $\delta$ -Mem (MSW)	81.52	37.37
+ LSTMem (Ours)	81.70	42.42

Table 3: General-capability scores (%). IFEval uses strict prompt-level accuracy; GPQA-D uses accuracy.

Seed	Depth coupling	MAB Avg.	AR	TTL	LRU	SF
42	on	45.08	55.70	47.50	37.59	18.00
	off	43.16	54.30	41.29	38.21	18.00
	$\Delta$	$+1.92$	$+1.40$	$+6.21$	$-0.62$	$0.00$
43	on	45.45	54.90	52.36	37.44	17.50
	off	43.68	54.35	44.36	37.39	17.75
	$\Delta$	$+1.77$	$+0.55$	$+8.00$	$+0.05$	$-0.25$

Table 4: Depth ablation.

\Delta

= on

-

off. The coupling and feedback are removed in training and inference alike.

Seed	Depth coupling	MAB Avg.	AR	TTL	LRU	SF
42	on	45.08	55.70	47.50	37.59	18.00
	off	43.16	54.30	41.29	38.21	18.00
	$\Delta$	$+1.92$	$+1.40$	$+6.21$	$-0.62$	$0.00$
43	on	45.45	54.90	52.36	37.44	17.50
	off	43.68	54.35	44.36	37.39	17.75
	$\Delta$	$+1.77$	$+0.55$	$+8.00$	$+0.05$	$-0.25$

Table 4: Depth ablation.

\Delta

= on

-

off. The coupling and feedback are removed in training and inference alike.

Memory layout	Avg.	Multi	Temp.	Open	Single
All 36	53.60	46.25	40.94	19.92	64.75
Even 18	51.30	45.97	40.51	16.10	61.22
Odd 18	51.56	46.83	39.73	14.10	61.93
First 12	48.09	43.91	33.98	11.12	59.10
Middle 12	50.42	46.93	39.19	13.94	60.03
Last 12	50.58	47.50	41.37	14.28	59.28

Table 5: Memory placement ablation on LoCoMo. F1 (%) on Qwen3-4B-Instruct. Layer indices are one-based: even/odd select 18 alternating layers; first/middle/last select layers 1–12, 13–24 and 25–36, respectively.

Method	Training stage	Multi-hop	Temporal	Open-domain	Single-hop	Avg.
$\delta$ -Mem	1: QASPER	42.57	39.31	18.12	58.59	49.12
$\delta$ -Mem	2: + Long	46.73	38.75	16.20	59.36	50.06
LSTMem	1: QASPER	43.70	39.03	18.72	60.19	50.18
LSTMem	2: + Long	46.25	40.94	19.92	64.75	53.60

Table 6: Training-stage comparison on LoCoMo. Stage 1 trains on QASPER, and Stage 2 continues training on the Long dataset. Scores are F1 (%), with Avg. weighted by question count.

Block-end feedback	Avg.	Multi	Temp.	Open	Single
On	53.60	46.25	40.94	19.92	64.75
Off	51.89	46.31	40.78	17.48	61.93

Table 7: Block-end feedback ablation on LoCoMo. F1 (%) on Qwen3-4B-Instruct. Both rows keep forward depth coupling and differ only in whether block-end feedback is applied.

Explore similar work

TransMem: Transforming Hidden States into Memory for Large Language Models
Jul 31, 2026Haodong Lei, Junming Liu, Yirong Chen +4Large Language Model Memory Large Language Model Agents
Southeast University, Nanjing, Jiangsu, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China
$δ$ -mem: Efficient Online Memory for Large Language Models
May 12, 2026Jingdi Lei, Di Zhang, Junxian Li +7Peak Memory Efficient Long-Context Inference
Nanyang Technological University · Mind Lab · Fudan University +3
LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory
Jul 17, 2026Jing Yu, Yibo Zhao, Jiaming Zhang +1Selective
School of Data Science and Engineering, East China Normal University

Jul 31, 2026, cs.MAJ/K move · Enter open · S save

TransMem: Transforming Hidden States into Memory for Large Language Models

Haodong Lei, Junming Liu, Yirong Chen, Pinlong Cai+3

Southeast University, Nanjing, Jiangsu, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China

Large Language Model Memory Large Language Model Agents Hidden States Self-Distillation Framework

Open paper AskPDF arXiv

Large language model (LLM) agents increasingly operate over long interaction histories, where effective reasoning requires identifying and exploiting task-relevant evidence distributed across past observations and actions. However, useful information encoded in previously computed representations is often underutilized during subsequent generation. We propose \textbf{TransMem}, a lightweight inference-time parametric memory module that transforms sparse historical hidden states from a frozen LLM backbone into reusable memory representations. TransMem uses a lightweight gating network to dynamically apply the latent intervention to the current hidden states, without repeatedly encoding the preceding context. To learn transferable memory utilization rather than task-specific knowledge, we introduce evidence-conditioned self-distillation. A memory-augmented student processes the full context and matches the predictive distribution of an evidence-only teacher that shares the same frozen backbone. Experiments on LoCoMo, HotpotQA, and MemoryAgentBench demonstrate consistent improvements across different model architectures and scales. TransMem yields gains of 11.58--29.25

F_1

on LoCoMo and 10.20--13.03

F_1

on HotpotQA, while improving the average MemoryAgentBench accuracy from 29.54% to 40.00%. These results establish sparse historical hidden states as an effective and efficient memory substrate for long-context LLM agents. Our code is available at https://github.com/Haodong-Lei-Ray/TransMem.

Abstract

Explore similar work

TransMem: Transforming Hidden States into Memory for Large Language Models

δδδ-mem: Efficient Online Memory for Large Language Models

LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory

$δ$ -mem: Efficient Online Memory for Large Language Models