Trained Persistent Memory for Frozen Encoder--Decoder LLMs: Six Architectural Methods

Hong Jeong

cs.LGMar 17, 2026

Trained Persistent Memory for Frozen Encoder--Decoder LLMs: Six Architectural Methods

Authors: Hong Jeong

Organizations: Inha University in Tashkent, Uzbekistan

Abstract

Frozen encoder--decoder language models are stateless: the latent representation is discarded after every forward pass, so no information persists across sessions. This paper presents a proof-of-concept pilot study showing that persistent memory in the continuous latent space of a frozen LLM is feasible -- even under severe resource constraints (a single frozen Flan-T5-XL backbone, small trainable adapters, a single dataset). We implement six architectural methods spanning three injection points and four write mechanisms; unlike text-level memory systems, every write and read is a differentiable operation on dense vectors. After training only the adapter, the memory bank continues to accumulate at inference time without gradients, enabling conversational learning. Under a forgetting-curve evaluation on LoCoMo at two capacity scales (1 $\times$ and 10 $\times$ ), the stateless baseline scores exactly zero; at 10 $\times$ all six trained adapters produce positive memory-recall curves; at 1 $\times$ three methods collapse, revealing capacity as a critical design parameter. Because the memory bank is a compact numerical array, it can be scaled to arbitrarily large capacity without altering the backbone. We argue that full end-to-end training with larger models, larger data, and orders-of-magnitude larger memory will yield substantially stronger results; this pilot study establishes the feasibility baseline and design-space taxonomy that such efforts require.

Figures & tables

Panel of Figure 1 — Figure 1: Frozen encoder–decoder baseline used as the stateless control. The latent representation is consumed within the current turn and then discarded.

	Method	Write rule ( $P_{t}\leftarrow$ )	Read rule (inject into decoder)	Type
0	Baseline	—	$D(Z_{t})$	—
1	Prefix	$\gamma P+A^{\top}V$ , $V{=}Z_{t}W_{V}$	$[Z_{t};\;P\,W_{P}]\to D$ (extra KV)	Delegated
2	XAttn	$\gamma P+A^{\top}V$ , $V{=}Z_{t}W_{V}$	$\operatorname{softmax}(s\,W_{Q}^{m}(P\,W_{K}^{m})^{\top}\!/\!\sqrt{d})\,(P\,W_{V}^{m})$	Explicit
3	KV Extension	$\gamma P+A^{\top}V$ , $V{=}Z_{t}W_{V}$	$K{=}[K_{Z};\,PW_{K,m}],\;V{=}[V_{Z};\,PW_{V,m}]$	Delegated
4	Hebbian	$\gamma M+(Z_{t}W_{K,H})^{\top}(Z_{t}W_{V,H})$	$(Z_{t}W_{Q,H})\,M\to W_{\mathrm{Mem}}\to D$ (extra KV)	Explicit
5	Gated	$\gamma P+A^{\top}V$ , $V{=}Z_{t}W_{V}$	$g_{t}\odot\mathrm{XAttn}(s,P)$ , $g_{t}{=}\sigma(W_{g}[s;c]+b_{g})$	Explicit

Table 1: Read and write operations for each method.

A=\operatorname{softmax}(Z_{t}W_{Q}(PW_{K})^{\top}/\sqrt{d})

. Delegated read means the frozen decoder cross-attention selects from the concatenated KV; explicit read means the adapter performs its own retrieval before passing to the decoder.

Panel of Figure 2 — Figure 2: M. 1 injects persistent memory as an encoder-input prefix and writes the current latent back into memory through an attention-coupled update.

Panel of Figure 3 — Figure 3: M. 2 preserves the frozen cross-attention route over the current encoder latent and adds a parallel decoder memory branch scaled by a learned coefficient.

Panel of Figure 4 — Figure 4: M. 3 extends decoder keys and values with learned projections of persistent memory while keeping the current encoder tokens on their original path.

Panel of Figure 5 — Figure 5: M. 4 stores associative structure in a Hebbian memory matrix that is queried by the current latent and passed to the decoder as additional memory.

Panel of Figure 6 — Figure 6: M. 5 adds a memory read branch inside the decoder and lets a learned gate decide when the auxiliary memory signal should influence the frozen path.

Panel of Figure 7 — Figure 7: M. 6 writes to a fixed set of memory slots sparsely and exposes the slot bank to the decoder as an explicit episodic memory.

Panel of Figure 8 — Figure 8: Two learning phases. Type 1 updates memory adapter parameters offline by backpropagation, whereas Type 2 updates the persistent memory online at each turn with frozen model weights.

	Method	Injection point	Primary-path safe	New params	Mem. cost	Write mechanism
1	Encoder-input prefix	Before $E$	✓	4.2M	const.	$A_{t}^{\top}V$
2	Parallel decoder XAttn	Inside $D$	✓	16.8M	const.	$A_{t}^{\top}V$
3	Decoder KV extension	Inside $D$ xattn	✓	4.2M	const.	$A_{t}^{\top}V$
4	Hebbian / associative	Decoder KV	✓	1.0M	$O(d_{h}^{2})$	Hebbian outer prod.
5	Context-gated decoder branch	Inside $D$	✓	21.0M	const.	$A_{t}^{\top}V$
6	Slot-based sparse write	Decoder KV	✓	4.2M	$O(Sd)$	Top- $k$ overwrite

Table 2: Six trained persistent-memory methods compared across key design dimensions. “Primary-path safe” means the original frozen encoder–decoder route through

Z_{t}

is preserved, even if an auxiliary decoder-side memory branch is added. “Memory cost” is per turn.

Panel of Figure 9 — Figure 9: Forgetting curves on LoCoMo at two memory-capacity scales. Each point is the memory recall rate from Eq. ( 40 ), smoothed by a weighted non-increasing isotonic fit. Left (1 $\times$ ): three methods (M.1, M.3, M.5) collapse; M.2 XAttn and M.6 Slot dominate. Right (10 $\times$ ): all six methods produce non-trivial curves; M.4 Hebbian is strongest at long lag. Higher and flatter curves indicate stronger memory.

1 $\times$ capacity ( $n_{p}\!=\!64$ , $d_{h}\!=\!256$ )
	Lag bucket (turns)
Method	0–31	32–63	64–127	128–255	256+	Mean
$n$ (samples)	28	24	62	130	395
M.0 Baseline	0.00	0.00	0.00	0.00	0.00	0.00
M.1 Prefix	0.02	0.02	0.02	0.02	0.02	0.02
M.3 KV Ext	0.00	0.00	0.00	0.00	0.00	0.00

Table 3: Memory recall rate (%) by lag bucket on LoCoMo (smoothed, non-increasing isotonic fit) at two capacity scales. The baseline is identically zero by construction.

n

gives the number of QA pairs per bucket. Higher

\to

stronger memory retention.

Method	$K_{30}$ (%)	$\Delta K$ (%)
M.0 Baseline	5.57	5.57
M.1 Prefix	0.00	0.00
M.3 KV Ext	0.00	0.00
M.5 Gated	0.17	0.17
M.2 XAttn	11.04	7.34
M.4 Hebbian	10.62	7.84

Table 4: Knowledge accumulation on LoCoMo (1

\times

capacity).

K_{30}

is the terminal knowledge score after all 30 sessions;

\Delta K=K_{30}-K_{1}

measures net knowledge gain. Higher values indicate stronger persistent memory.

	1 $\times$ capacity		10 $\times$ capacity
Method	Tax (%)	Benefit (%)	Tax (%)	Benefit (%)
M.1 Prefix	+2.38	$-$ 6.42	+4.23	+4.00
M.2 XAttn	+3.39	+6.76	+3.39	+5.09
M.3 KV Ext	+2.38	$-$ 6.44	+4.23	+6.26
M.4 Hebbian	+3.39	+3.10	+3.39	+5.46
M.5 Gated	+3.39	$-$ 6.30	+3.39	+1.83

Table 5: Adapter interference analysis. Tax

>0

means the adapter degrades baseline knowledge when memory is empty; Benefit

>0

means memory helps more than the adapter hurts. Baseline mean F1 = 6.44% at both scales (stateless, independent of adapter capacity).

Persistent Memory for Language Models

Persistent memory for language models studies how models retain, update, and use information across interactions and beyond their context windows.

See all 271 papers

Trained Persistent Memory for Frozen Decoder-Only LLMs
Mar 20, 2026Hong JeongPersistent Memory for Language Models Memory-Augmented Language Models
Inha University in Tashkent, Uzbekistan
HasMem: Hard-Origin Adaptively Softened Memory for Long-Term LLM Agents
Sep 25, 2026Zihong He, Junxiao Shen, Chen Liang +1Memory Management for LLM Agents Long-Term Conversational Memory
The Hong Kong University of Science and Technology (Guangzhou) · University of Bristol · The Hong Kong University of Science and Technology
Memorilla: Latent Semantic Memory for LLMs
Oct 15, 2026Adibvafa Fallahpour, Parsa Idehpour, Vignesh Kothapalli +4Memory-Augmented Language Models Persistent Memory for Language Models
Stanford University · University of Toronto · Vector Institute +3

Mar 20, 2026J/K move · Enter open · S save

Trained Persistent Memory for Frozen Decoder-Only LLMs

Hong Jeong

Inha University in Tashkent, Uzbekistan

Persistent Memory for Language Models Memory-Augmented Language Models LLM Memory Decoder-Only Language Models

Open paper AskPDF arXiv

Decoder-only language models are stateless: hidden representations are discarded after every forward pass and nothing persists across sessions. Jeong (2026a) showed that trained memory adapters give a frozen encoder-decoder backbone persistent latent-space memory, building on the lateral-memory framework of Jeong (2026b,c). Here we ask whether the same principle transfers to the decoder-only setting, where no cross-attention pathway exists and memory must enter through self-attention alone. We adapt six methods -- prefix, parallel cross-attention, KV extension, Hebbian memory, context-gated branch, and slot-based sparse write -- to a frozen GPT-2, training only a small adapter

θ_{mem}

. The write rule is shared; only the read injection changes from decoder cross-attention to self-attention KV prefix or parallel branch. On LoCoMo we find a striking inductive-bias dichotomy: at

1\times

capacity, three methods with strong architectural priors -- cross-attention (M.2), Hebbian (M.4), and slot write (M.6) -- achieve retained-memory scores of

7-18\%

and knowledge gains

ΔK

7-10

, while the other three fail (

< 0.4\%

). At

10\times

capacity all six converge, showing the gap is architectural, not fundamental. Together with the encoder-decoder results of Jeong (2026a) and the brain-inspired modules of Jeong (2026b,c), these findings establish persistent latent-space memory as a general paradigm spanning major transformer families.

Abstract

Persistent Memory for Language Models

Explore similar work

Trained Persistent Memory for Frozen Decoder-Only LLMs

HasMem: Hard-Origin Adaptively Softened Memory for Long-Term LLM Agents

Memorilla: Latent Semantic Memory for LLMs