Trained Persistent Memory for Frozen Decoder-Only LLMs

Hong Jeong

cs.LGMar 20, 2026

Trained Persistent Memory for Frozen Decoder-Only LLMs

Authors: Hong Jeong

Organizations: Inha University in Tashkent, Uzbekistan

Abstract

Decoder-only language models are stateless: hidden representations are discarded after every forward pass and nothing persists across sessions. Jeong (2026a) showed that trained memory adapters give a frozen encoder-decoder backbone persistent latent-space memory, building on the lateral-memory framework of Jeong (2026b,c). Here we ask whether the same principle transfers to the decoder-only setting, where no cross-attention pathway exists and memory must enter through self-attention alone. We adapt six methods -- prefix, parallel cross-attention, KV extension, Hebbian memory, context-gated branch, and slot-based sparse write -- to a frozen GPT-2, training only a small adapter $θ_{mem}$ . The write rule is shared; only the read injection changes from decoder cross-attention to self-attention KV prefix or parallel branch. On LoCoMo we find a striking inductive-bias dichotomy: at $1\times$ capacity, three methods with strong architectural priors -- cross-attention (M.2), Hebbian (M.4), and slot write (M.6) -- achieve retained-memory scores of $7-18\%$ and knowledge gains $ΔK$ of $7-10$ , while the other three fail ( $< 0.4\%$ ). At $10\times$ capacity all six converge, showing the gap is architectural, not fundamental. Together with the encoder-decoder results of Jeong (2026a) and the brain-inspired modules of Jeong (2026b,c), these findings establish persistent latent-space memory as a general paradigm spanning major transformer families.

Figures & tables

Panel of Figure 1 — Figure 1: Frozen decoder-only baseline. The hidden state $H_{t}$ is consumed within the current turn and then discarded; no information persists across sessions.

Panel of Figure 2 — Figure 2: Three decoder-only injection strategies. (a) KV prefix prepends memory-derived keys and values to the self-attention cache. (b) Parallel cross-attention inserts a new attention pathway alongside the existing self-attention. (c) Gated branch computes a memory readout and adds it through a learned content-dependent gate.

Method	Enc-Dec read path	Decoder-only read path	Strategy
M.1 Prefix	Encoder-input prefix	Self-attention KV prefix	KV prefix
M.2 XAttn	Parallel decoder XAttn	Parallel cross-attention	Inserted XAttn
M.3 KV Ext	Decoder KV extension	Self-attention KV extension	KV prefix
M.4 Hebbian	Decoder KV extension	Self-attention KV extension	KV prefix
M.5 Gated	Context-gated decoder	Gated additive branch	Gated branch
M.6 Slot	Decoder KV extension	Self-attention KV extension	KV prefix

Table 1: Mapping of the six memory methods from the encoder–decoder setting [ 5 ] to the decoder-only setting. The write rule is identical; only the read injection point changes.

Panel of Figure 3 — Figure 3: M.1: Memory is projected into soft key–value pairs and prepended to the self-attention cache at every layer. The write rule updates $P$ from $H_{t}$ .

Panel of Figure 4 — Figure 4: M.2: A parallel cross-attention layer is inserted after each frozen self-attention block. The scaling factor $\beta^{(\ell)}$ is initialised to zero for safe startup.

Panel of Figure 5 — Figure 5: M.3: Memory is projected into per-layer key–value pairs and concatenated with the frozen self-attention cache. Unlike M.1, both the read projections and the write update are layer-specific.

Panel of Figure 6 — Figure 6: M.4: The Hebbian matrix is queried by the current hidden state; the recalled vector is projected into additional KV pairs for self-attention.

Panel of Figure 7 — Figure 7: M.5: A gated memory branch reads from $P$ and contributes through a content-dependent gate. At initialisation $b_{g}<0$ keeps the gate nearly closed, preserving the frozen self-attention path.

Panel of Figure 8 — Figure 8: M.6: Memory is organised as $S$ addressable slots. Only the top- $k$ addressed slots are updated each turn (sparse write); all slots are read via self-attention KV extension.

Panel of Figure 9 — Figure 9: Two learning phases. Type 1 trains the memory adapter parameters offline by backpropagation through the frozen decoder. Type 2 updates the persistent memory online at each conversation turn with all model weights frozen.

	Method	Injection strategy	Self-attn safe	New params	Mem. cost	Write mechanism
1	Self-attention KV prefix	KV prefix	✓	$\sim$ 2.4M	const.	$A_{t}^{\top}V$
2	Parallel cross-attention	Inserted XAttn	✓	$\sim$ 4.1M	const.	$A_{t}^{\top}V$
3	Self-attention KV extension	KV prefix	✓	$\sim$ 2.4M	const.	$A_{t}^{\top}V$ (per-layer)
4	Hebbian / associative	KV prefix	✓	$\sim$ 0.8M	$O(d_{h}^{2})$	Hebbian outer prod.
5	Gated additive branch	Gated branch	✓	$\sim$ 4.7M	const.	gated XAttn
6	Slot-based sparse write	KV prefix	✓	$\sim$ 1.8M	$O(Sd)$	Top- $k$ overwrite

Table 2: Six trained persistent-memory methods compared for decoder-only models. “Self-attn safe” means the original frozen self-attention route is preserved. “Memory cost” is per turn.

Parameter	$\mathbf{1\times}$	$\mathbf{10\times}$
Memory bank size $n_{P}$	64	640
Hebbian dimension $d_{h}$	256	810
Slot count $S$	64	640
Top- $k$ writes	8	80

Table 3: Memory capacity conditions. All other hyperparameters (backbone, learning rate, decay

\gamma

, training epochs) are shared.

Panel of Figure 10 — Figure 10: Absolute forgetting curves on LoCoMo for the GPT-2 (124M) backbone at $1\times$ capacity. Three methods (M.2, M.4, M.6) produce substantial retained-memory scores; three (M.1, M.3, M.5) are indistinguishable from the baseline. Higher and flatter curves indicate stronger memory.

	$1\times$ capacity		$10\times$ capacity
Method	Ret. mem. (%)	$\Delta K$	Ret. mem. (%)	$\Delta K$
M.0 Baseline	0.00	5.57	0.00	5.57
M.1 Prefix	0.02	0.00	9.20	—
M.2 XAttn	9.02	7.34	9.88	—
M.3 KV Ext	0.00	0.00	9.69	—
M.4 Hebbian	9.23	7.84	10.32	—

Table 4: Retained-memory score (min across lag bins, %) and knowledge accumulation (

\Delta K

) for each method at

1\times

and

10\times

capacity on GPT-2. Bold: best in column. Knowledge accumulation at

10\times

is omitted: the forgetting-curve convergence already establishes the capacity-bridging result (§ 7.3 ).

Panel of Figure 11 — Figure 11: Absolute forgetting curves at $10\times$ capacity. All six methods now produce substantial retained-memory scores. Methods that failed at $1\times$ (M.1, M.3, M.5) converge to comparable performance with the strong-bias methods, demonstrating that the $1\times$ gap is architectural, not fundamental.

Panel of Figure 12 — Figure 12: Cumulative knowledge score $K_{s}$ over 30 sessions at $1\times$ capacity. Successful methods (M.2, M.4, M.6) accumulate knowledge substantially above the baseline; failing methods (M.1, M.3, M.5, not shown for clarity) produce $\Delta K<0.2$ .

Persistent Memory for Language Models

Persistent memory for language models studies how models retain, update, and use information across interactions and beyond their context windows.

See all 271 papers

Mar 17, 2026

Trained Persistent Memory for Frozen Encoder--Decoder LLMs: Six Architectural Methods

Frozen encoder--decoder language models are stateless: the latent representation is discarded after every forward pass, so no information persists across sessions. This paper presents a proof-of-concept pilot study showing that persistent memory in the continuous latent space of a frozen LLM is feasible -- even under severe resource constraints (a single frozen Flan-T5-XL backbone, small trainable adapters, a single dataset). We implement six architectural methods spanning three injection points and four write mechanisms; unlike text-level memory systems, every write and read is a differentiable operation on dense vectors. After training only the adapter, the memory bank continues to accumulate at inference time without gradients, enabling conversational learning. Under a forgetting-curve evaluation on LoCoMo at two capacity scales (1

\times

and 10

\times

), the stateless baseline scores exactly zero; at 10

\times

all six trained adapters produce positive memory-recall curves; at 1

\times

three methods collapse, revealing capacity as a critical design parameter. Because the memory bank is a compact numerical array, it can be scaled to arbitrarily large capacity without altering the backbone. We argue that full end-to-end training with larger models, larger data, and orders-of-magnitude larger memory will yield substantially stronger results; this pilot study establishes the feasibility baseline and design-space taxonomy that such efforts require.

Hong Jeong

Inha University in Tashkent, Uzbekistan

Persistent Memory for Language Models Memory-Augmented Language Models Continual Learning for LLMs

Ask PDF

Sep 15, 2026

Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization

We introduce a simple architectural modification to decoder-only transformers: a persistent recurrent state that observes hidden representations via cross-attention, updates itself through a GRU, and modulates subsequent processing via gated addition. Inserted between the lower and upper halves of a 6-layer transformer, this module adds only 3.7% additional parameters while reducing evaluation loss from

2.438 \pm 0.004

1.743 \pm 0.018

, corresponding to a 28.5% reduction on held-out language modeling data. The improvement is statistically significant across 5 random seeds (

p < 0.01

) and corresponds to reduced overfitting (generalization gap 0.12 vs 0.26). Through controlled ablations, we demonstrate that the improvement stems entirely from the persistent memory topology, not from auxiliary self-prediction objectives. A model with identical topology but no auxiliary loss performs equivalently, while a random auxiliary loss provides no benefit. Representation probing reveals that the persistent state encodes narrative position (52% vs 33% chance level)---information that standard attention maintains less efficiently. Our results suggest that bridging transformer layers with a lightweight recurrent memory is a simple, effective approach to improving generalization in small-scale language models.

Eduardo Novaes Hering

FITec Labs / Ericsson São Paulo

Memory-Augmented Language Models Decoder-Only Language Models Persistent Memory for Language Models Language Model Generalization Recurrent Transformers

Ask PDF

Jul 30, 2026

Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory

Decoder-only language models entangle long-term memory and reasoning in a single parameter set, making it difficult to scale memory capacity independently. Memory Decoder introduces a parametric long-term memory module but only studies it at a relatively small scale. In this work, we present Memory Decoder at Scale, scaling memory models up to 6.9B parameters and pretraining them on 300B tokens. At this data scale, the combined cost of indexing and search makes a standard Faiss pipeline infeasible. We address this bottleneck with a distributed pipeline for Faiss indexing and retrieval, together with sparse, batch-wise loading of kNN distributions. Across model scales, we find that allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone. On 17 benchmarks, pairing a 6.9B general memory with Pythia-410M raises its average score from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters. For Qwen3 Base models ranging from 0.6B to 14B, 1.7B domain memories improve the average score across the three domains by more than 9 points at every scale. Overall, our results demonstrate that independently scaling pretrained memory offers a more parameter efficient path to improving language model performance.

Rubin Wei, Jiaqi Cao, Jiarui Wang +4

LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Electronic Engineering, Tsinghua University

Language Model Pretraining Memory-Augmented Language Models Persistent Memory for Language Models Language Model Scaling Language Modeling

Ask PDF

Abstract

Persistent Memory for Language Models

Explore similar work

Trained Persistent Memory for Frozen Encoder--Decoder LLMs: Six Architectural Methods

Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization

Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory