LSTMem: Hierarchical Long Short-Term Online Memory for Large Language Models

Xianglong Shi; Ruijie Yang; Sirui Zhao; Shukang Yin; Zihao Bian; Tinghao Yi; Enhong Chen

cs.AISep 27, 2026

LSTMem: Hierarchical Long Short-Term Online Memory for Large Language Models

Authors: Xianglong Shi, Ruijie Yang, Sirui Zhao, Shukang Yin, Zihao Bian, Tinghao Yi, Enhong Chen

Organizations: University of Science and Technology of China

Abstract

Large language models increasingly serve as long-horizon assistants and agents, where they must both accumulate information across interactions and make the relevant parts available when later requests depend on them. Existing compact online memories typically use a single persistent state both to accumulate history and to serve readout, so what the memory stores cannot be controlled separately from what it exposes to the current computation. We propose LSTMem, an LSTM-inspired online memory that instead equips each layer of a frozen LLM with two matrix-valued states: a cell state that accumulates history and a hidden state whose readouts correct the backbone's attention. Input and forget gates control what the cell stores, while an output gate separately controls what the cell exposes through the hidden state. LSTMem further connects memory across depth through forward hidden-state propagation and block-end feedback, and uses higher-layer reconstruction gradients to refine lower-layer cell states before rebuilding hidden states from shallow to deep layers. Across memory benchmarks on Qwen3-4B-Instruct, LSTMem consistently improves MemoryAgentBench, LoCoMo, and HotpotQA over the plain backbone. Comparisons further show that the LSTM-based memory formulation outperforms an associative-memory counterpart, while removing cross-layer hidden-memory propagation degrades performance. These results demonstrate the benefits of separating memory accumulation from memory expression and organizing memory hierarchically across model depth. The code is available at https://github.com/Longchentong/LSTMem.

Figures & tables

External Memory
Model	HotpotQA $\uparrow$		MemoryAgentBench $\uparrow$
Model	EM	F1	Avg.	AR	TTL	LRU	SF
Qwen3-4B-Instruct	$42.35$	$56.00$	$29.54$	$35.30$	$26.14$	$\mathbf{47.08}$	$14.37$
+ BM25 RAG	$40.35_{\scriptstyle-2.00}$	$52.83_{\scriptstyle-3.17}$	$24.43_{\scriptstyle-5.11}$	$32.20_{\scriptstyle-3.10}$	$9.74_{\scriptstyle-16.40}$	$37.86_{\scriptstyle-9.22}$	$15.00_{\scriptstyle+0.63}$
+ LLMLingua-2	$36.93_{\scriptstyle-5.42}$	$50.03_{\scriptstyle-5.97}$	$15.63_{\scriptstyle-13.91}$	$21.45_{\scriptstyle-13.85}$	$1.43_{\scriptstyle-24.71}$	$38.45_{\scriptstyle-8.63}$	$8.62_{\scriptstyle-5.75}$
+ MemoryBank	—	—	$17.65_{\scriptstyle-11.89}$	$22.65_{\scriptstyle-12.65}$	$7.67_{\scriptstyle-18.47}$	$36.36_{\scriptstyle-10.72}$	$9.88_{\scriptstyle-4.49}$

Table 1: Main results across three base models. Bold marks the highest score within each model group; subscripts give point changes from its unadapted model. Baseline scores are from Lei et al. (2026b) . HotpotQA is scored by exact match (EM) and token-level F1 against the gold answer. AR, TTL, LRU and SF denote accurate retrieval, test-time learning, long-range understanding and selective forgetting. AR and SF are scored by accuracy, TTL by classification accuracy and movie-recommendation Recall@5, and LRU by summarization F1 and Detective QA accuracy. MAB averages use 2,000/700/171/800 question weights for AR/TTL/LRU/SF; baseline averages are approximated from rounded family scores.

Model	Avg.	Multi	Temp.	Open	Single
Qwen3-4B-Instruct	40.79	38.39	32.89	10.77	48.05
+ BM25 RAG	36.68	38.12	20.34	9.99	45.47
+ LLMLingua-2	40.98	39.07	30.13	10.98	49.19
+ MemoryBank	38.14	37.88	21.76	13.35	47.31
+ Context2LoRA	48.11	37.95	34.99	16.75	60.11
+ MemGen	40.05	32.93	33.30	12.67	48.13

Table 2: LoCoMo F1 (%). Avg. is weighted over 1,540 questions. Multi, Temp., Open and Single denote multi-hop, temporal, open-domain and single-hop questions.

Model	Avg.	Multi	Temp.	Open	Single
Qwen3-4B-Instruct	40.79	38.39	32.89	10.77	48.05
+ BM25 RAG	36.68	38.12	20.34	9.99	45.47
+ LLMLingua-2	40.98	39.07	30.13	10.98	49.19
+ MemoryBank	38.14	37.88	21.76	13.35	47.31
+ Context2LoRA	48.11	37.95	34.99	16.75	60.11
+ MemGen	40.05	32.93	33.30	12.67	48.13

Table 2: LoCoMo F1 (%). Avg. is weighted over 1,540 questions. Multi, Temp., Open and Single denote multi-hop, temporal, open-domain and single-hop questions.

Model	IFEval $\uparrow$	GPQA-D $\uparrow$
Qwen3-4B-Instruct	81.89	39.39
+ Context2LoRA	76.71	29.29
+ MemGen	39.37	38.89
+ MLP Memory	24.95	22.73
+ $\delta$ -Mem (MSW)	81.52	37.37
+ LSTMem (Ours)	81.70	42.42

Table 3: General-capability scores (%). IFEval uses strict prompt-level accuracy; GPQA-D uses accuracy.

Seed	Depth coupling	MAB Avg.	AR	TTL	LRU	SF
42	on	45.08	55.70	47.50	37.59	18.00
	off	43.16	54.30	41.29	38.21	18.00
	$\Delta$	$+1.92$	$+1.40$	$+6.21$	$-0.62$	$0.00$
43	on	45.45	54.90	52.36	37.44	17.50
	off	43.68	54.35	44.36	37.39	17.75
	$\Delta$	$+1.77$	$+0.55$	$+8.00$	$+0.05$	$-0.25$

Table 4: Depth ablation.

\Delta

= on

-

off. The coupling and feedback are removed in training and inference alike.

Seed	Depth coupling	MAB Avg.	AR	TTL	LRU	SF
42	on	45.08	55.70	47.50	37.59	18.00
	off	43.16	54.30	41.29	38.21	18.00
	$\Delta$	$+1.92$	$+1.40$	$+6.21$	$-0.62$	$0.00$
43	on	45.45	54.90	52.36	37.44	17.50
	off	43.68	54.35	44.36	37.39	17.75
	$\Delta$	$+1.77$	$+0.55$	$+8.00$	$+0.05$	$-0.25$

Table 4: Depth ablation.

\Delta

= on

-

off. The coupling and feedback are removed in training and inference alike.

Memory layout	Avg.	Multi	Temp.	Open	Single
All 36	53.60	46.25	40.94	19.92	64.75
Even 18	51.30	45.97	40.51	16.10	61.22
Odd 18	51.56	46.83	39.73	14.10	61.93
First 12	48.09	43.91	33.98	11.12	59.10
Middle 12	50.42	46.93	39.19	13.94	60.03
Last 12	50.58	47.50	41.37	14.28	59.28

Table 5: Memory placement ablation on LoCoMo. F1 (%) on Qwen3-4B-Instruct. Layer indices are one-based: even/odd select 18 alternating layers; first/middle/last select layers 1–12, 13–24 and 25–36, respectively.

Method	Training stage	Multi-hop	Temporal	Open-domain	Single-hop	Avg.
$\delta$ -Mem	1: QASPER	42.57	39.31	18.12	58.59	49.12
$\delta$ -Mem	2: + Long	46.73	38.75	16.20	59.36	50.06
LSTMem	1: QASPER	43.70	39.03	18.72	60.19	50.18
LSTMem	2: + Long	46.25	40.94	19.92	64.75	53.60

Table 6: Training-stage comparison on LoCoMo. Stage 1 trains on QASPER, and Stage 2 continues training on the Long dataset. Scores are F1 (%), with Avg. weighted by question count.

Block-end feedback	Avg.	Multi	Temp.	Open	Single
On	53.60	46.25	40.94	19.92	64.75
Off	51.89	46.31	40.78	17.48	61.93

Table 7: Block-end feedback ablation on LoCoMo. F1 (%) on Qwen3-4B-Instruct. Both rows keep forward depth coupling and differ only in whether block-end feedback is applied.

Explore similar work

Jul 31, 2026cs.MA

TransMem: Transforming Hidden States into Memory for Large Language Models

Large language model (LLM) agents increasingly operate over long interaction histories, where effective reasoning requires identifying and exploiting task-relevant evidence distributed across past observations and actions. However, useful information encoded in previously computed representations is often underutilized during subsequent generation. We propose \textbf{TransMem}, a lightweight inference-time parametric memory module that transforms sparse historical hidden states from a frozen LLM backbone into reusable memory representations. TransMem uses a lightweight gating network to dynamically apply the latent intervention to the current hidden states, without repeatedly encoding the preceding context. To learn transferable memory utilization rather than task-specific knowledge, we introduce evidence-conditioned self-distillation. A memory-augmented student processes the full context and matches the predictive distribution of an evidence-only teacher that shares the same frozen backbone. Experiments on LoCoMo, HotpotQA, and MemoryAgentBench demonstrate consistent improvements across different model architectures and scales. TransMem yields gains of 11.58--29.25

F_1

on LoCoMo and 10.20--13.03

F_1

on HotpotQA, while improving the average MemoryAgentBench accuracy from 29.54% to 40.00%. These results establish sparse historical hidden states as an effective and efficient memory substrate for long-context LLM agents. Our code is available at https://github.com/Haodong-Lei-Ray/TransMem.

Haodong Lei, Junming Liu, Yirong Chen +4

Southeast University, Nanjing, Jiangsu, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China

Large Language Model Memory Large Language Model Agents Hidden States Self-Distillation Framework

Ask PDF

May 12, 2026cs.AI

$δ$ -mem: Efficient Online Memory for Large Language Models

Large language models increasingly need to accumulate and reuse historical information in long-term assistants and agent systems. Simply expanding the context window is costly and often fails to ensure effective context utilization. We propose

δ

-mem, a lightweight memory mechanism that augments a frozen full-attention backbone with a compact online state of associative memory.

δ

-mem compresses past information into a fixed-size state matrix updated by delta-rule learning, and uses its readout to generate low-rank corrections to the backbone's attention computation during generation. With only an

8\times8

online memory state,

δ

-mem improves the average score to

1.10\times

that of the frozen backbone and

1.15\times

that of the strongest non-

δ

-mem memory baseline. It achieves larger gains on memory-heavy benchmarks, reaching

1.31\times

on MemoryAgentBench and

1.20\times

on LoCoMo, while largely preserving general capabilities. These results show that effective memory can be realized through a compact online state directly coupled with attention computation, without full fine-tuning, backbone replacement, or explicit context extension.

Jingdi Lei, Di Zhang, Junxian Li +7

Nanyang Technological University · Mind Lab · Fudan University +3

Peak Memory Efficient Long-Context Inference Kimi Delta Attention Frozen Backbone Memoryagentbench

Ask PDF

Jul 17, 2026cs.AI

LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory

Long-term memory enables LLM agents to leverage past interactions, but dialogue histories quickly exceed the context window, forcing agents to retrieve relevant subsets at query time. Because useful evidence is sparse and scattered across verbose conversations, retrieval faces a fundamental tension: broadening recall improves coverage but floods downstream reasoning with noise, while compressing memories at write time eases retrieval but irreversibly discards details that future queries may need. We introduce LazyMem, which resolves this tension by deferring all memory construction to query time. Given a retrieved candidate pool, a lightweight model processes it in overlapping parallel windows, selectively retaining and compressing only query-relevant content. The model is trained with supervised fine-tuning followed by reinforcement learning, using a reward that jointly encourages the identification of relevant messages and the generation of compressions that are faithful to the source and useful for answering the query. On LongMemEval, LazyMem-4B achieves an LLM-judge accuracy of 0.85, outperforming the strongest non-oracle baseline while using only 213 answer-context memory tokens, 21.0 times fewer than the baseline. It further generalizes to LoCoMo without target-domain training and reduces mean latency relative to the prior query-time baseline. Code is available at https://github.com/allacnobug/LazyMem.

Jing Yu, Yibo Zhao, Jiaming Zhang +1

School of Data Science and Engineering, East China Normal University

Selective

Ask PDF

Abstract

Explore similar work

TransMem: Transforming Hidden States into Memory for Large Language Models

δδδ-mem: Efficient Online Memory for Large Language Models

LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory

$δ$ -mem: Efficient Online Memory for Large Language Models