Frozen Memory Is Not Enough: Rethinking External Memory as Extraction

Mingyuan Li; Guangsheng Yu; Xu Wang; Shaoxiong Ji

cs.CLAug 17, 2026

Frozen Memory Is Not Enough: Rethinking External Memory as Extraction

Authors: Mingyuan Li, Guangsheng Yu, Xu Wang, Shaoxiong Ji

Organizations: ELLIS Institute Finland · University of Turku · University of Technology Sydney

Abstract

Methods for improving knowledge use in large language models typically fall into two regimes. Non-parametric retrieval offers flexible access to external knowledge, but adds retrieval latency, context overhead, and only shallow integration with the backbone. Parametric adaptation is efficient at inference time, but entangles knowledge with model weights and can be hard to update, audit, or transfer. Engram-style hashed memory occupies a middle regime: it stores learned information in an external, addressable table, yet consumes that table through a small learned reader. This raises a basic question: when such a memory is moved across backbones, what matters more, the frozen memory itself or the target-side reader? We study this question through cross-model frozen-memory extraction, in which a memory trained on a source model is frozen and attached to a different target model, with only a lightweight reader trained. Ablations show that learned memory content and correct addressing both matter, but the transferred table becomes useful only through a reader aligned to the target model. In downstream question answering tasks, a dual-layer, four-branch reader nearly closes the gap between same-model and cross-model reuse, achieving an average score of 38.8 under our controlled evaluation protocol. Moreover, when the provider reader is directly compatible with the target interface, the frozen artifact can provide substantial utility without target-side training, while optional reader adaptation yields further improvement. These results suggest that Engram can serve as a reusable external knowledge artifact, provided that the target has access to a compatible reader interface; target-side adaptation can further improve alignment when direct reader reuse is insufficient.

Figures & tables

Figure 1 : Overview of the cross-model memory transfer. A memory table trained with source model $A$ is frozen and attached to target model $B$ . Canonicalization keeps the address space fixed across tokenizers, and only the target-side reader (including multi-head and multi-layer variants) is trained. This setup operationalizes whether external memory remains reusable outside its source backbone.

Panel of Figure 2 — Figure 2 : The full 3x3 transfer matrix and target-scale transfer performance across different model types trained on Wikitext-103.

Variant	Budget	NQ	WebQA	TriviaQA	TruthQA	HotpotQA	Average
Base	–	20.6	29.3	57.7	32.1	21.0	32.1
Non-parametric Methods
RAG [ Lewis et al., 2020 ]	–	22.6 1.9	24.9 -4.4	54.2 -3.4	35.5 3.4	29.8 8.8	33.4 (+3.9%)
kNN [ Khandelwal et al., 2020 ]	–	21.1 0.4	30.5 1.2	57.8 0.1	32.3 0.2	21.2 0.2	32.6 (+1.4%)
Parametric Methods
CPT	–	12.2 -8.5	34.1 4.8	61.2 3.6	29.2 -2.9	16.0 -4.9	30.5 (-5.0%)

Table 1 : QA accuracy (%) on Mistral-7B-v0.3 for different reader variants and baselines. Budget notation follows Table 2 trained on Wikipedia-2021 [ Izacard et al., 2023 ] .

Variant	Budget	NQ	WebQA	TriviaQA	TruthQA	HotpotQA	Average
Base	–	20.6	29.3	57.7	32.1	21.0	32.1
Transfer Verification
Mistral $\{2,10\}$ -R4	10/20	30.2 9.5	34.0 4.7	70.0 12.3	30.8 -1.3	27.7 6.8	38.5 (+20.0%)
LLaMA $\{2,10\}$ -R4	10/20	30.3 9.7	33.7 4.4	69.9 12.3	30.9 -1.2	27.6 6.7	38.5 (+20.0%)
Frozen Mistral $\{2,10\}$ -R4	10/0	30.5 9.8	30.3 1.0	70.8 13.2	32.2 0.2	27.7 6.8	38.3 (+19.2%)
Frozen LLaMA $\{2,10\}$ -R4	10/0	30.6 10.0	30.3 1.0	70.8 13.1	32.2 0.1	27.6 6.7	38.3 (+19.2%)

Table 2 : QA accuracy (%) on Mistral-7B-v0.3 for transfer verification and ablation studies. Deltas are measured against the baseline. Budget

=

P1/P2 denotes source-memory and target-reader training tokens (millions) trained on Wikipedia-2021.

Figure 3 : Downstream task evaluation. Bars report the change in accuracy (

\Delta

) produced by transferred memory relative to the no-memory baseline across six downstream tasks. Results are averaged over five seeds. Positive values indicate that transferred memory improves the target model after target-side adaptation on FineWeb-Edu (FW-9B) [ Lozhkov et al., ] .

Task	Transferred	Random	Disabled	Ablated	$\Delta\log p$ (Trans. - Random)	$\Delta\log p$ (Trans. - Disabled)	$\Delta\log p$ (Trans. - Ablated)
NQ	25.1	20.1	18.3	26.2	+0.035	+0.032	-0.013
WebQA	32.3	27.4	31.1	33.6	+0.147	+0.197	-0.001
TriviaQA	72.5	65.5	58.9	72.5	+0.047	+0.072	-0.000
TruthQA	30.8	32.6	31.7	30.9	-0.581	-0.258	-0.029
HotpotQA	27.1	22.9	18.6	27.2	+0.022	+0.019	-0.009

Table 3 : QA memory-contribution trained on Wikipedia-2021 analysis for LLaMA-2-7B [ Touvron et al., 2023 ]

\to

Mistral-7B-v0.3 [ Jiang et al., 2023 ] with dual-layer injection. The

\Delta\log p

columns report mean per-example differences in length-normalized gold-answer log-probability. Positive values indicate that transferred memory makes the gold answer more likely.

Condition	Trainable Params	5M	20M	50M
Transferred	1.05M	$\mathbf{21.8}$	$\mathbf{21.5}$	$\mathbf{21.5}^{\dagger}$
From scratch	34.6M	$21.9$	$21.8$	$21.6$

Table 4 : Scaling analysis for frozen-memory transfer on Pythia-160M

\rightarrow

Pythia-410M. Results report test perplexity as a function of target-side training tokens, averaged over 3 seeds.

Panel of Figure 4 — Figure 4 : Target-side scaling for Qwen3.5-0.8B $\rightarrow$ Qwen3.5-2B with the source memory trained on WikiText-103. Transferred memory maintains a consistent intrinsic perplexity advantage, while downstream improvements remain positive but task-dependent.

Condition	Test PPL $\downarrow$	95% CI	NQ	WebQA	TriviaQA	TruthQA	HotpotQA	Avg $\uparrow$
No memory baseline	$9.9$	$[9.8,10.1]$	$20.6$	$29.3$	$57.7$	$32.1$	$21.0$	$32.1$
Transferred ( $R{=}4$ )	$8.7$	$[8.6,8.9]$	$30.3$	$33.7$	$69.9$	$30.9$	$27.6$	$38.5$
Interface Simplifications
No gate	$10.1$	$[9.9,10.3]$	$24.2$	$27.7$	$61.6$	$31.0$	$24.1$	$33.7$
Affine stitch	$9.5$	$[9.4,9.7]$	$25.0$	$29.2$	$62.8$	$30.4$	$23.9$	$34.3$
Content and Training Controls

Table 5 : Ablation study of frozen-memory transfer on LLaMA-2-7B

\rightarrow

Mistral-7B-v0.3, with the source memory trained on Wikipedia-2021. Results report held-out test perplexity and downstream QA performance.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$A,B$	Source model and target model, respectively.
$d_{A},d_{B}$	Hidden dimensions of source model $A$ and target model $B$ , respectively.
$\mathcal{E}_{A}$	Frozen Engram memory artifact learned with source model $A$ , comprising the tables $\{E_{n,k}\}$ .
$E_{n,k}$	Memory table associated with N-gram order $n$ and hash head $k$ , where $E_{n,k}\in\mathbb{R}^{M\times d_{\text{head}}}$ .
$N_{\max}$	Maximum N-gram order used for memory addressing, with $n\in\{2,\ldots,N_{\max}\}$ .
$K$	Number of independent hash heads for each N-gram order.

Appendix

Table 6 : Notation used throughout this paper.

Property	KNN-LM	RAG	RETRO	Mem. Layers	LLM Mod.	Engram	Ours
$O(1)$ retrieval	✗	✗	✗	✗	✗	✓	✓
Parametric (trained)	✗	✗	✓	✓	✓	✓	✓
Cross-model portable	✗	✓	✗	✗	$\circ$	✗	✓
Surgical deletion	✗	✓	✗	✗	✗	✓	✓
No context overhead	✗	✗	$\circ$	✓	✓	✓	✓
Reader-only integration	N/A	N/A	✗	✗	✗	✗	✓

Appendix

Table 7: Qualitative comparison of knowledge augmentation approaches. ✓ = fully supported,

\circ

= partially supported, ✗ = not supported.

Aspect	Original Engram	Transfer-oriented Engram Reader
Memory indexing	Layer-specific compressed-token $n$ -gram hashing: $r_{t}^{(l,n,j)}=\Bigl(\bigoplusop\displaylimits_{k=0}^{n-1}c(x_{t-k})\cdot a_{k}^{(l)}\Bigr)\bmod p_{l,n,j}$	Shared canonical hashing: $r_{t}^{(n,j)}=\bigl(H(\mathrm{canon\_ngram}_{t}^{(n)})\oplus s_{j}\bigr)\bmod M$
Memory representation	Layer-specific Engram embedding: $e_{t}^{(l)}=\mathrm{Concat}_{n,j}\mathbf{E}_{l,n,j}[r_{t}^{(l,n,j)}]$	Shared memory vector: $m_{t}=\mathrm{Concat}_{j}\mathbf{T}_{j}[r_{t}^{(j)}]\hskip 9.24994pt(d_{\mathrm{mem}}=512)$
Key / gate	$\displaystyle s_{t}$ $\displaystyle=\frac{\left\langle\mathrm{RMSNorm}(W_{k}e_{t}),\mathrm{RMSNorm}(h_{t})\right\rangle}{\sqrt{d}}$ $\displaystyle\alpha_{t}$ $\displaystyle=\sigma\!\left(\mathrm{sign}(s_{t})\sqrt{\|s_{t}\|}\right)$	$k_{t}^{(r)}=W_{k}^{(r)}m_{t},\hskip 9.24994ptv_{t}=W_{v}m_{t}$ $\alpha_{t}^{(r)}=\sigma\!\left(\frac{\langle\mathrm{RMSNorm}(h_{t}),\mathrm{RMSNorm}(k_{t}^{(r)})\rangle}{\sqrt{d}}+\beta_{r}\right)$
Output update	$u_{t}=\alpha_{t}W_{v}e_{t},\hskip 9.24994pto_{t}=u_{t}+\mathrm{ShortConv}(u_{t})$	$o_{t}=\frac{1}{R}\sumop\displaylimits_{r=1}^{R}\alpha_{t}^{(r)}v_{t},\hskip 9.24994pth_{t}\leftarrow h_{t}+o_{t}$
Branch structure	Native multi-branch inside backbone	Explicit reader branches ( $R$ controls capacity)
Injection	Internal transformer block component	Post-layer hook injection (e.g., layers 2 and 10)

Appendix

Table 8: Formula-level comparison between original Engram and our transfer-oriented reader. We replace the native within-block module with a target-side reader over frozen hashed memory.

Family	Model	Experimental role	Params.	$d$	Checkpoint	Hugging Face identifier
Pythia	Pythia-160M	Source: main matrix	160M	768	Base	EleutherAI/pythia-160m
Pythia	Pythia-410M	Target: main matrix	410M	1024	Base	EleutherAI/pythia-410m
Qwen3.5	Qwen3.5-0.8B	Source: main matrix, scaling Target: downstream	0.8B	1024	Base	Qwen/Qwen3.5-0.8B-Base
	Qwen3.5-2B	Target: scaling, downstream	2B	2048	Base	Qwen/Qwen3.5-2B-Base
	Qwen3.5-4B	Source: peer, downstream Target: main matrix, peer, scaling, downstream	4B	2560	Base	Qwen/Qwen3.5-4B-Base
	Qwen3.5-9B	Source: main matrix, downstream Target: scaling, downstream	9B	4096	Base	Qwen/Qwen3.5-9B-Base

Appendix

Table 9 : Models and checkpoints used in the transfer experiments. “Source” denotes memory construction, and “target” denotes reader fitting for a frozen memory. Here,

d

is the residual hidden size used by the memory interface.

Regime / phase	Corpus	Budget / cap	Seq. / batch	LR / warm-up	Trainable components and exceptions
Primary training settings
RQ1 4.1 Phase 1	WikiText-103	50M tokens	512 / 16	$3\times 10^{-5}$ 1,000	Source backbone, memory, and source reader are trainable. For Qwen3.5 sources, the memory learning rate is $10^{-3}$ .
RQ1 4.1 Phase 2	WikiText-103	20M tokens	512 / 16	$3\times 10^{-5}$ 500	Only the target reader is trainable; the target backbone and transferred memory remain frozen.
Figure 3 Phase 1	FineWeb-Edu	50M tokens	512 / 2	$10^{-3}$ / 1,000	Source backbone, memory, and source reader trained end-to-end; gradient checkpointing
Figure 3 Phase 2	FineWeb-Edu	20M tokens	512 / 4	$3\times 10^{-5}$ / 500	Target reader only; target backbone and transferred memory frozen; gradient checkpointing; early stopping patience 5
Table 1 to Table 5 Phase 1	Wikipedia-2021	10–30M tokens	2048 / 1	$3\times 10^{-5}$ 1,000	The source memory and reader are trainable; the source backbone is frozen. Early stopping uses patience 3.

Appendix

Table 10 : Principal training and evaluation settings. “LR / warm-up” reports the learning rate and number of warm-up steps. Unless otherwise noted, training uses cosine decay, AdamW with weight decay

0.01

, and gradient-norm clipping at

1.0

. Alternative-corpus experiments follow the intrinsic reader-fitting protocol with LAMBADA or English C4 replacing WikiText-103. OOD rows are evaluation-only.

Target	Baseline	Transferred	Random
Pythia-160M source
Pythia-410M	$21.9$	$\mathbf{21.6\pm 0.1}$ -1.6%	$21.9\pm 0.0$
Qwen3.5-4B	$10.8$	$10.1\pm 0.0$ -6.8%	$10.3\pm 0.0$
TinyLlama-1.1B	$10.6$	$9.5\pm 0.0$ -10.6%	$10.0\pm 0.0$
Qwen3.5-0.8B source
Pythia-410M	$22.9$	$21.4\pm 0.0$ -6.8%	$22.1\pm 0.0$

Appendix

Table 11: Full cross-architecture transfer matrix: test PPL (

\downarrow

) for 3 source memories

\times

3 target models. Random replaces the learned source memory with a size-matched randomly initialized memory under the same fitting protocol. Mean

\pm

std over 3 seeds. Deltas shown next to the transferred result are relative to the no-memory baseline; negative values indicate better PPL. A displayed standard deviation of

0.0

indicates a value below

0.05

after rounding to one decimal place.

Condition	Test PPL $\downarrow$	95% CI	Params / Cost
Baseline (no memory)	$21.9$	$[20.6,23.4]$	—
Transfer Conditions
Random memory	$21.9\pm 0.0$ -0.2%	$[20.6,23.3]$	1.05M
Transferred memory	$\mathbf{21.6\pm 0.1}$ -1.6%	$\mathbf{[20.2,23.0]}$	1.05M
Parameter-Matched Baselines
LoRA (rank 7) [ Hu et al., 2022 ]	$23.3\pm 0.8$ +6.5%	$[21.9,24.9]$	1.03M

Appendix

Table 12: Same-tokenizer transfer (Pythia-160M

\to

Pythia-410M). Mean

\pm

std over 3 seeds. Deltas in the PPL column are relative to the no-memory baseline; negative values indicate better PPL. LoRA and Cross-LoRA are iso-parameter baselines (1.03M parameters, same 20M token budget). KNN-LM uses a 5M-token datastore with

k{=}1024

retrieval at inference time.

Condition	Test PPL $\downarrow$	95% CI	Params
Baseline (no memory)	$10.6$	$[10.0,11.3]$	—
Transfer Conditions
Random memory	$10.0\pm 0.0$ -5.9%	$[9.4,10.7]$	2.10M
Transferred memory	$\mathbf{9.5\pm 0.0}$ -10.6%	$\mathbf{[8.9,10.2]}$	2.10M

Appendix

Table 13: Cross-tokenizer transfer (Pythia-160M

\to

TinyLlama-1.1B). Mean

\pm

std over 3 seeds. Deltas in the PPL column are relative to the no-memory baseline; negative values indicate better PPL. Reader parameters: 2.10M.

Direction	Baseline	Transferred	Random
Phi-4-mini $\rightarrow$ Qwen3.5-4B	$10.5$	$\mathbf{9.5\pm 0.0}$ -10.1%	$10.2\pm 0.0$
Qwen3.5-4B $\rightarrow$ Phi-4-mini	$12.6$	$11.5\pm 0.0$ -9.2%	$12.3\pm 0.0$

Appendix

Table 14: Peer-to-peer transfer between Phi-4-mini (3.8B) and Qwen3.5-4B. Source memories trained on WikiText-103 (50M tokens). Mean

\pm

std over 3 seeds. Deltas shown next to the transferred result are relative to the no-memory baseline; negative values indicate better PPL.

Target	Baseline	Transferred	Random
Qwen3.5-2B	$13.7$	$\mathbf{11.7\pm 0.0}$ -14.1%	$12.6\pm 0.0$
Qwen3.5-4B	$10.5$	$9.6\pm 0.0$ -8.9%	$10.2\pm 0.0$
Qwen3.5-9B	$9.3$	$8.5\pm 0.0$ -8.3%	$9.0\pm 0.0$

Appendix

Table 15: Target model scaling: Qwen3.5-0.8B source

\to

Qwen3.5-{2B, 4B, 9B}. Mean

\pm

std over 3 seeds. Deltas shown next to the transferred result are relative to the no-memory baseline; negative values indicate better PPL. Baselines are eval-only (no reader training).

Figure 5 : Linear CKA similarity between mean-pooled backbone representations at layer $\lfloor L/3\rfloor$ for the three models shown in the matrix.

Target	Source	RTE $\uparrow$	BoolQ $\uparrow$	OpenBookQA $\uparrow$	SciQ $\uparrow$	TruthfulQA $\uparrow$	RACE $\uparrow$
Qwen-0.8B	FW-avg	+6.1	-0.6	+0.7	+1.8	-0.1	-0.2
Qwen-0.8B	Nemo-avg	+7.9	+4.1 ∗	+1.1	+8.4 ∗	-0.8	+1.1 ∗
Qwen-2B	FW-avg	+2.7	+3.2	+0.3	+3.5	-0.4	-0.0
Qwen-2B	Nemo-avg	+8.7 ∗	+14.6 ∗	+0.5	-0.7	+0.0	+1.1 ∗
Qwen-4B	FW-avg	+1.3	-0.2	+0.6	+0.9	-0.5	+0.1
Qwen-4B	Nemo-avg	+0.4	+0.6	+0.2	-6.3	-2.9	-0.4

Appendix

Table 16 : Domain alignment ablation with corpus-matched Phase 2 readers. Positive gains are shown in green and negative changes in red .

Source corpus	BoolQ $\uparrow$	RTE $\uparrow$	OBQA $\uparrow$	SciQ $\uparrow$	TQA $\uparrow$	RACE $\uparrow$
HQ-DQA (8B, STEM Q&A)	+15.3 $\pm$ 0.3	+8.9 $\pm$ 4.7	+0.3 $\pm$ 0.2	+0.9 $\pm$ 3.0	+0.3 $\pm$ 0.2	+1.4 $\pm$ 0.4
HQ (26B, organic web)	+3.0 $\pm$ 0.4	+1.2 $\pm$ 2.1	+0.5 $\pm$ 0.1	-0.5 $\pm$ 0.0	+0.2 $\pm$ 0.3	-0.4 $\pm$ 0.2
FW-avg (reference)	+3.2	+2.7	+0.3	+3.5	-0.4	-0.0

Appendix

Table 17: Corpus structure vs. size on Qwen-2B: accuracy change (

\Delta\%

) under corpus-matched Phase 2 readers. Positive gains are shown in green and negative changes in red .

Source	Phase 2	BoolQ $\Delta\%\uparrow$	RTE $\Delta\%\uparrow$	SciQ $\Delta\%\uparrow$
Mismatched Phase 2 (WikiText-103)
10M	WikiText-103	+8.3 $\pm$ 0.5	+7.7 $\pm$ 3.9	-4.5 $\pm$ 0.5
50M	WikiText-103	+6.6 $\pm$ 0.4	+6.1 $\pm$ 4.7	-1.8 $\pm$ 0.6
200M	WikiText-103	+0.0 $\pm$ 0.0	+2.9 $\pm$ 4.7	-1.4 $\pm$ 0.6
Matched Phase 2 (HQ-DQA)
10M	HQ-DQA	+14.1 $\pm$ 1.2	+10.7 $\pm$ 4.0	-2.3 $\pm$ 3.0

Appendix

Table 18: Reader alignment: downstream accuracy change (

\Delta\%

) under mismatched vs. matched Phase 2 adaptation. Positive gains are shown in green and negative changes in red .

Source	Gate mean		Frac. $<0.1$
tokens	WikiText-103	BoolQ	WikiText-103	BoolQ
10M	0.6	0.5	5.6%	3.6%
50M	0.6	0.5	9.1%	2.9%
200M	0.5	0.4	17.5%	15.3%

Appendix

Table 19 : Gate activation statistics during BoolQ evaluation versus Phase 2 WikiText-103 training. Longer source training suppresses the gate globally; BoolQ shows a lower mean gate, while the near-closed-gate fraction rises on both distributions under mismatch.

Design	BoolQ $\uparrow$	RTE $\uparrow$	OBQA $\uparrow$	SciQ $\uparrow$	TQA $\uparrow$	RACE $\uparrow$	DQA-agg $\uparrow$	Br-agg $\uparrow$
Single-corpus references
HQ-DQA ref.	+15.3 $\pm$ 0.3	+8.9 $\pm$ 4.7	+0.3 $\pm$ 0.2	+0.9 $\pm$ 3.0	+0.3 $\pm$ 0.2	+1.4 $\pm$ 0.4	+8.4	+0.7
FW-Edu ref.	+2.5 $\pm$ 1.7	+1.9 $\pm$ 0.2	-0.0 $\pm$ 0.2	+3.2 $\pm$ 0.6	-0.2 $\pm$ 0.2	+0.1 $\pm$ 0.1	+2.5	-0.1
Mixed or broadened memories
50/50 mix	+14.3 $\pm$ 1.1	+11.7 $\pm$ 3.5	+0.4 $\pm$ 0.4	-0.5 $\pm$ 3.8	+0.3 $\pm$ 0.3	+0.5 $\pm$ 0.3	+8.5	+0.4
Sequential HQ $\to$ FW	+13.6 $\pm$ 1.0	+11.9 $\pm$ 2.8	-0.0 $\pm$ 0.2	-1.7 $\pm$ 2.7	+0.1 $\pm$ 0.4	+0.2 $\pm$ 0.3	+7.9	+0.1

Appendix

Table 20: Mixed-corpus memory study on Qwen3.5-2B. Positive gains are shown in green and negative changes in red . The two rightmost columns report aggregate scores: DQA-agg

=

mean(BoolQ, RTE, SciQ); Br-agg

=

mean(OBQA, TruthfulQA, RACE).

Target Model	Dataset	Baseline	Transferred	$\Delta$ (%)
Pythia-410M	LAMBADA	$41.0$	$41.0$	+0.0%
	WikiText-103	$22.6$	22.1	-2.4%
	C4	$24.9$	24.8	-0.3%
TinyLlama-1.1B	LAMBADA	$23.6$	23.5	-0.7%
	WikiText-103	$10.6$	10.1	-5.0%
	C4	$11.7$	11.6	-0.2%

Appendix

Table 21: Out-of-domain evaluation: test PPL (

\downarrow

) for baseline vs. transferred memory. Negative

\Delta

indicates improvement.

Reader corpus	Eval. corpus	No memory	Transfer	$\Delta$
WikiText	WikiText	22.616	22.113	-0.502
WikiText	C4	24.726	24.663	-0.062
C4	WikiText	22.616	22.538	-0.078
C4	C4	24.726	24.604	-0.122

Appendix

Table 22: Cross-corpus reader fitting and evaluation. Lower PPL is better.

Language	No-memory PPL	Transfer PPL	Rel. improvement
Chinese	20.1248	$19.6718\pm 0.0033$	2.251%
Japanese	15.3555	$15.1985\pm 0.0251$	1.023%

Appendix

Table 23: Character-span addressing for non-segmented languages. Results are averaged over three seeds.

Condition	Tokens	Train time	GPU-hours	Peak inf. memory	Prefill
No memory	0	0	0	$13.658\pm 0.020$ GiB	$55.10\pm 0.06$ ms
Phase-1 source artifact	4.096M	0.653 h	3.154	–	–
Fresh Mistral memory	19.968M	3.871 h	15.760	–	–
Matched FFN	19.968M	3.680 h	14.929	$13.736\pm 0.020$ GiB	$55.32\pm 0.49$ ms
Transfer, incl. Phase 1	4.096M+19.968M	4.451 h	18.623	$13.881\pm 0.020$ GiB	$57.69\pm 0.40$ ms

Appendix

Table 24: Measured cost for the LLaMA-2-7B

\rightarrow

Mistral-7B-v0.3 using QA experiment configuration on LUMI supercomputer.

Explore similar work

Jun 17, 2026cs.AI

User as Engram: Internalizing Per-User Memory as Local Parametric Edits

Personal memory in a language model is two problems: content and reasoning skill. The brain keeps the two apart (a sparse, local engram in the hippocampus for each episode, a slow neocortex for the shared skills that interpret it), so a new fact need not overwrite everything else. Most personalization today keeps a user's facts outside the weights, in a natural-language memory file or a retrieval index. When facts are written into the model instead, the standard recipe is the per-user LoRA adapter, which does the opposite of the brain, folding content and skill into one global weight delta. Writing a user's facts as a LoRA contaminates text unrelated to them; writing the same facts as local Engram rows leaves it mathematically untouched, resulting in a roughly 33,000x smaller memory footprint. We therefore propose User as Engram: store a user's content as surgical edits to the hash-keyed memory table of an Engram model, and carry the reasoning skill in one shared adapter. This layered design matches per-user LoRA's direct recall while delivering 5.6x higher indirect-reasoning accuracy on average, and never makes a single user worse at reasoning than the untouched base. The edit is a glass box: writing a fact switches on its lookup at exactly the trigger, adds the value the answer needs, leaves every other position unchanged to the last bit, and fails if written into the wrong layer. Because different users' facts land in disjoint hash slots, their edits compose: many users live in one shared table at once, stacking additively and losslessly, where a per-user LoRA, a single global weight delta, admits only one. Upon retrieval, a per-user Engram table does not grow with the population the retriever must search, so past ~100 facts it overtakes a retrieval pipeline on a 2.5x larger model.

Bojie Li

Pine AI

Large Language Model Memory Semantic Memory Reasoning Skills Edit

Ask PDF

May 14, 2026cs.CL

MeMo: Memory as a Model

Large language models (LLMs) achieve strong performance across a wide range of tasks, but remain frozen after pretraining until subsequent updates. Many real-world applications require timely, domain-specific information, motivating the need for efficient mechanisms to incorporate new knowledge. In this paper, we introduce MeMo (Memory as a Model), a modular framework that encodes new knowledge into a dedicated memory model while keeping the LLM parameters unchanged. Compared to existing methods, MeMo offers several advantages: (a) it captures complex cross-document relationships, (b) it is robust to retrieval noise, (c) it avoids catastrophic forgetting in the LLM, (d) it does not require access to the LLM's weights or output logits, enabling plug-and-play integration with both open and proprietary closed-source LLMs, and (e) its retrieval cost is independent of corpus size at inference time. Our experimental results on three benchmarks, BrowseComp-Plus, NarrativeQA, and MuSiQue, show that MeMo achieves strong performance compared to existing methods across diverse settings.

Ryan Wei Heng Quek, Sanghyuk Lee, Alfred Wei Lun Leong +6

Institute of Data Science, National University of Singapore, Singapore · Integrative Sciences and Engineering Programme, NUSGS, Singapore · Agency for Science, Technology, Research (A*STAR), Singapore +6

Large Language Model Memory

Ask PDF

Jul 8, 2026cs.CL

TF-Engram: A Train-Free Engram with SSD-Backed Memory for Large Language Models

Large Language Models (LLMs) store factual knowledge and domain-specific patterns implicitly in dense Transformer parameters, making knowledge expansion costly through pretraining, fine-tuning, retrieval augmentation, or longer contexts. Engram-style memory offers a compact hidden-state injection pathway, but existing GPU-resident designs often rely on hash-based compression, causing unrelated phrases to collide in shared slots and weakening phrase-level semantic fidelity. We present TF-Engram, a train-free Engram system that constructs phrase-specific semantic memory offline from external corpora, stores large memory tables across a GPU--DRAM--SSD hierarchy, and uses Early-Exit Guided Predictive Prefetching to hide external-memory latency during autoregressive decoding. On Qwen3-0.6B, TF-Engram improves the average downstream score from 57.6 to 59.4, outperforming both the frozen backbone and a parameter-matched LoRA baseline. System evaluation shows that large TF-Engram tables can be built with moderate offline cost, SSD-backed storage substantially reduces GPU memory demand, and predictive prefetching recovers much of the throughput loss caused by external memory access. These results demonstrate that static phrase memory can be integrated into LLM inference as a scalable, train-free, and low-overhead system component.

Yutang Ma, Kecheng Huang, Xikun Jiang +1

The Chinese University of Hong Kong · Beijing Institute of Technology, Zhuhai

Large Language Model Memory LLM Inference Optimization Preemption Hidden States

Ask PDF