cs.LGMar 20, 2026

Trained Persistent Memory for Frozen Decoder-Only LLMs

Authors: Hong Jeong

Organizations: Inha University in Tashkent, Uzbekistan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Decoder-only language models are stateless: hidden representations are discarded after every forward pass and nothing persists across sessions. Jeong (2026a) showed that trained memory adapters give a frozen encoder-decoder backbone persistent latent-space memory, building on the lateral-memory framework of Jeong (2026b,c). Here we ask whether the same principle transfers to the decoder-only setting, where no cross-attention pathway exists and memory must enter through self-attention alone. We adapt six methods -- prefix, parallel cross-attention, KV extension, Hebbian memory, context-gated branch, and slot-based sparse write -- to a frozen GPT-2, training only a small adapter θmemθ_{mem}. The write rule is shared; only the read injection changes from decoder cross-attention to self-attention KV prefix or parallel branch. On LoCoMo we find a striking inductive-bias dichotomy: at 1×1\times capacity, three methods with strong architectural priors -- cross-attention (M.2), Hebbian (M.4), and slot write (M.6) -- achieve retained-memory scores of 7−18%7-18\% and knowledge gains ΔKΔK of 7−107-10, while the other three fail (<0.4%< 0.4\%). At 10×10\times capacity all six converge, showing the gap is architectural, not fundamental. Together with the encoder-decoder results of Jeong (2026a) and the brain-inspired modules of Jeong (2026b,c), these findings establish persistent latent-space memory as a general paradigm spanning major transformer families.

Figures & tables

Figure 1: Frozen decoder-only baseline. The hidden state HtH_{t} is consumed within the current turn and then discarded; no information persists across sessions.
Figure 2: Three decoder-only injection strategies. (a) KV prefix prepends memory-derived keys and values to the self-attention cache. (b) Parallel cross-attention inserts a new attention pathway alongside the existing self-attention. (c) Gated branch computes a memory readout and adds it through a learned content-dependent gate.
MethodEnc-Dec read pathDecoder-only read pathStrategy
M.1 PrefixEncoder-input prefixSelf-attention KV prefixKV prefix
M.2 XAttnParallel decoder XAttnParallel cross-attentionInserted XAttn
M.3 KV ExtDecoder KV extensionSelf-attention KV extensionKV prefix
M.4 HebbianDecoder KV extensionSelf-attention KV extensionKV prefix
M.5 GatedContext-gated decoderGated additive branchGated branch
M.6 SlotDecoder KV extensionSelf-attention KV extensionKV prefix
Table 1: Mapping of the six memory methods from the encoder–decoder setting [ 5 ] to the decoder-only setting. The write rule is identical; only the read injection point changes.
Figure 3: M.1: Memory is projected into soft key–value pairs and prepended to the self-attention cache at every layer. The write rule updates PP from HtH_{t} .
Figure 4: M.2: A parallel cross-attention layer is inserted after each frozen self-attention block. The scaling factor β(ℓ)\beta^{(\ell)} is initialised to zero for safe startup.
Figure 5: M.3: Memory is projected into per-layer key–value pairs and concatenated with the frozen self-attention cache. Unlike M.1, both the read projections and the write update are layer-specific.
Figure 6: M.4: The Hebbian matrix is queried by the current hidden state; the recalled vector is projected into additional KV pairs for self-attention.
Figure 7: M.5: A gated memory branch reads from PP and contributes through a content-dependent gate. At initialisation bg<0b_{g}<0 keeps the gate nearly closed, preserving the frozen self-attention path.
Figure 8: M.6: Memory is organised as SS addressable slots. Only the top- kk addressed slots are updated each turn (sparse write); all slots are read via self-attention KV extension.
Figure 9: Two learning phases. Type 1 trains the memory adapter parameters offline by backpropagation through the frozen decoder. Type 2 updates the persistent memory online at each conversation turn with all model weights frozen.
 MethodInjection strategySelf-attn safeNew paramsMem. costWrite mechanism
1Self-attention KV prefixKV prefix✓∼\sim 2.4Mconst.At⊤VA_{t}^{\top}V
2Parallel cross-attentionInserted XAttn✓∼\sim 4.1Mconst.At⊤VA_{t}^{\top}V
3Self-attention KV extensionKV prefix✓∼\sim 2.4Mconst.At⊤VA_{t}^{\top}V (per-layer)
4Hebbian / associativeKV prefix✓∼\sim 0.8MO(dh2)O(d_{h}^{2})Hebbian outer prod.
5Gated additive branchGated branch✓∼\sim 4.7Mconst.gated XAttn
6Slot-based sparse writeKV prefix✓∼\sim 1.8MO(Sd)O(Sd)Top- kk overwrite
Table 2: Six trained persistent-memory methods compared for decoder-only models. “Self-attn safe” means the original frozen self-attention route is preserved. “Memory cost” is per turn.
Parameter1×\mathbf{1\times}10×\mathbf{10\times}
Memory bank size nPn_{P}64640
Hebbian dimension dhd_{h}256810
Slot count SS64640
Top- kk writes880
Table 3: Memory capacity conditions. All other hyperparameters (backbone, learning rate, decay γ\gamma , training epochs) are shared.
Figure 10: Absolute forgetting curves on LoCoMo for the GPT-2 (124M) backbone at 1×1\times capacity. Three methods (M.2, M.4, M.6) produce substantial retained-memory scores; three (M.1, M.3, M.5) are indistinguishable from the baseline. Higher and flatter curves indicate stronger memory.
 1×1\times capacity10×10\times capacity
MethodRet. mem. (%)ΔK\Delta KRet. mem. (%)ΔK\Delta K
M.0 Baseline0.005.570.005.57
M.1 Prefix0.020.009.20—
M.2 XAttn9.027.349.88—
M.3 KV Ext0.000.009.69—
M.4 Hebbian9.237.8410.32—
Table 4: Retained-memory score (min across lag bins, %) and knowledge accumulation ( ΔK\Delta K ) for each method at 1×1\times and 10×10\times capacity on GPT-2. Bold: best in column. Knowledge accumulation at 10×10\times is omitted: the forgetting-curve convergence already establishes the capacity-bridging result (§ 7.3 ).
Figure 11: Absolute forgetting curves at 10×10\times capacity. All six methods now produce substantial retained-memory scores. Methods that failed at 1×1\times (M.1, M.3, M.5) converge to comparable performance with the strong-bias methods, demonstrating that the 1×1\times gap is architectural, not fundamental.
Figure 12: Cumulative knowledge score KsK_{s} over 30 sessions at 1×1\times capacity. Successful methods (M.2, M.4, M.6) accumulate knowledge substantially above the baseline; failing methods (M.1, M.3, M.5, not shown for clarity) produce ΔK<0.2\Delta K<0.2 .
More in this topic

Persistent Memory for Language Models

Persistent memory for language models studies how models retain, update, and use information across interactions and beyond their context windows.

See all 271 papers

Explore similar work

CardsList

  1. Trained Persistent Memory for Frozen Encoder--Decoder LLMs: Six Architectural Methods

    Mar 17, 2026Hong JeongPersistent Memory for Language ModelsMemory-Augmented Language Models

    Inha University in Tashkent, Uzbekistan

  2. Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization

    Sep 15, 2026Eduardo Novaes HeringMemory-Augmented Language ModelsDecoder-Only Language Models

    FITec Labs / Ericsson São Paulo

  3. Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory

    Jul 30, 2026Rubin Wei, Jiaqi Cao, Jiarui Wang +4Language Model PretrainingMemory-Augmented Language Models

    LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Electronic Engineering, Tsinghua University