Deep Fusion Memory: A Chunk-Level Parametric Memory for Large Language Models

Rubin Wei; Siyuan Sun; Hao Sun; Yuliang Liu; Xiang Shi; Jiarui Wang; Junming Zhang; Zhouhan Lin

cs.CLOct 18, 2026

Deep Fusion Memory: A Chunk-Level Parametric Memory for Large Language Models

Authors: Rubin Wei, Siyuan Sun, Hao Sun, Yuliang Liu, Xiang Shi, Jiarui Wang, Junming Zhang, Zhouhan Lin

Organizations: LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory

Abstract

Large language models need effective ways to acquire and use external knowledge beyond what they retain from pretraining. Continued pretraining risks catastrophic forgetting, while retrieval at inference time can add substantial overhead. External parametric memory avoids online retrieval, but constructing its training data requires extensive offline search, and output-level interpolation provides only shallow fusion. We introduce Deep Fusion Memory (DFM), a parametric memory architecture that reduces this construction workload and incorporates learned memory into intermediate hidden states. A causal memory predictor learns to generate memory from the preceding context, using retrieved chunk embeddings from the corpus as supervision. The predicted memory is shared across tokens within each chunk and fused into their hidden states through gated cross-attention. Memory prediction and fusion are trained end to end with the backbone frozen, and inference requires no explicit retrieval. Compared with existing parametric memory approaches, which often construct token-level memory targets, DFM requires $32\times$ fewer retrieval queries and $2048\times$ fewer retrieved items to construct its training data. Across six QA tasks, DFM performs better over the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base. Beyond QA, domain adaptation yields average gains of 6.51 and 4.38 points over Qwen3-4B-Base and Qwen3-8B-Base, respectively.

Figures & tables

Figure 1: Overview of Deep Fusion Memory. (a) Memory Prediction. The memory predictor maps pooled chunk embeddings to predicted memory, supervised by offline target memory. A shared projector maps predicted memory to the decoder’s hidden space. (b) LLM Backbone. Selected decoder layers incorporate memory for next-token prediction. (c) Memory Fusion. Memory predicted from the preceding context supports token predictions in the next chunk. Per-head gates control the memory cross-attention contribution alongside self-attention. Inference requires no retrieval.

Mistral-7B-v0.3
Method	Single-hop QA		Multi-hop QA				Avg
Method	NQ	WebQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg
Base	20.30	29.28	20.96	5.87	1.43	3.91	13.63
Non-parametric methods
RAG	24.03	27.90	29.09	31.02	9.87	18.56	23.41
xRAG	27.39	40.06	27.55	27.78	9.57	12.51	24.14

Table 1: Question answering performance on two single-hop benchmarks and four multi-hop benchmarks. Avg is the unweighted mean across all six tasks. Red subscripts show decreases from Base. Blue highlights DFM. Bold scores indicate the best results within each backbone.

Qwen3-4B-Base
Method	Code		Math	Law	Avg
Method	HumanEval	HumanEval+	GSM8K	LawBench
Base	62.04	57.32	83.40	29.96	58.18
RAG	62.80	55.49 ${}_{{\color[rgb]{1,0.2,0.2}\text{\textminus{}1.83}}}$	84.99	24.35 ${}_{{\color[rgb]{1,0.2,0.2}\text{\textminus{}5.61}}}$	56.91 ${}_{{\color[rgb]{1,0.2,0.2}\text{\textminus{}1.27}}}$
CPT	66.62	58.54	84.84	41.66	62.92
LoRA	59.45 ${}_{{\color[rgb]{1,0.2,0.2}\text{\textminus{}2.59}}}$	58.54	85.37	39.43	60.70

Table 2: Domain adaptation on Qwen3-4B-Base and Qwen3-8B-Base across Code, Math, and Law. Bold and underlined scores mark the best and second-best results within each backbone. Red subscripts show percentage-point decreases from Base.

Figure 2: Resource comparison with token-level MLP Memory. (a) Added parameters. (b) Training cost: MLP Memory and DFM take approximately 64 and 24 hours, respectively, on 16 A800 GPUs, a 62.5% reduction. (c) Retrieval workload: retrieval queries and total retrieved items (queries

\times K

) for a 4B-token corpus. Reference bars in (b,c) equal 100%; arrows show percentage reductions.

Figure 3: DFM ablations with predicted memory on Mistral-7B-v0.3. Points show single-run six-task QA means. Filled circles and dashed lines mark the main setting (chunk 32, layers

[9,16,24,31]

, Top-16). In (b), the diamond and triangle use four fusion layers at

[28,29,30,31]

and

[2,10,21,31]

, respectively.

Gate	NQ	WebQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg
No gate	34.08	42.72	29.80	24.94	10.08	19.35	26.83
$b_{g}=-6$	32.34	42.18	28.54	23.55	9.03	14.94	25.10
$b_{g}=-4$	32.70	42.22	29.39	25.98	10.42	19.70	26.73
$b_{g}=-2$	33.57	42.47	29.53	25.00	10.05	22.65	27.21
$b_{g}=0$	34.59	42.13	30.38	29.68	10.94	19.69	27.90

Table 3: Gate initialization and removal on Mistral-7B-v0.3. Gated variants change only the initial bias

b_{g}

;

b_{g}=-2

is the default. Bold marks column maxima; Avg is the six-task mean.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Split	Samples	Metric	Max. tokens
NQ	Test	3,610	F1	15
WebQA	Test	2,032	Substring match	100
HotpotQA	Validation	7,405	F1	15
2Wiki	Validation	12,576	F1	32
MuSiQue	Validation	2,417	F1	32
Bamboogle	Test	125	F1	32

Appendix

Table 4: Evaluation datasets and metrics for question answering. Max. tokens is the maximum number of newly generated tokens.

Method	Single-hop QA		Multi-hop QA				Avg
Method	NQ	WebQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg
OLMo-3-1025-7B
Base	21.69	40.31	27.99	11.05	4.67	2.25	17.99
RAG	22.57	31.50	21.97	7.91	2.58	2.75	14.88 ${}_{{\color[rgb]{1,0.2,0.2}\text{\textminus{}3.11}}}$
xRAG	23.64	34.79	26.33	23.15	11.95	16.65	22.75
RETRO	21.61	38.09	27.97	29.90	11.61	12.57	23.63

Appendix

Table 5: Question answering performance on OLMo-3-1025-7B across two single-hop and four multi-hop benchmarks. Avg is the six-task mean. Red subscripts show decreases from Base. Blue highlights DFM. Bold and underlined scores mark the best and second-best results.

Legal Knowledge Memorization
ID	Task	Description
1-1	Article Recitation	Recall a specified law article.
1-2	Knowledge Question Answering	Answer a legal multiple-choice question.
Legal Knowledge Understanding
2-1	Document Proofreading	Correct errors in a legal sentence.
2-2	Dispute Focus Identification	Identify issues disputed by the parties.

Appendix

Table 6: LawBench task IDs and brief definitions, following Fei et al. (2024) . The first digit identifies the legal knowledge category. These IDs are used in Tables 7 and 8 .

Task	Base	RAG	CPT	LoRA	DFM
1-1	18.76	11.41	42.46	31.70	22.23
1-2	6.20	20.00	27.00	28.20	31.60
2-1	16.83	20.06	26.82	35.36	24.13
2-2	12.20	0.60	31.40	23.80	35.60
2-3	32.73	16.10	32.75	30.16	38.97
2-4	0.00	0.20	34.00	33.40	30.80

Appendix

Table 7: LawBench task results on Qwen3-4B-Base under zero-shot evaluation. DFM uses the same checkpoint as the main table. Task definitions appear in Table 6 . Bold marks the highest score among the displayed methods. Macro is the equal-weight average over all 20 task metrics.

Task	Base	RAG	CPT	LoRA	DFM
1-1	16.72	8.99	54.57	54.15	29.71
1-2	70.60	17.20	32.20	30.00	53.80
2-1	20.74	21.89	38.60	27.72	30.85
2-2	0.20	0.40	14.60	9.60	36.80
2-3	16.01	16.41	46.21	40.52	52.40
2-4	0.00	0.00	33.80	35.00	34.40

Appendix

Table 8: LawBench task results on Qwen3-8B-Base under zero-shot evaluation. DFM uses the same checkpoint as the main table. Task definitions appear in Table 6 . Bold marks the highest score among the displayed methods. Macro is the equal-weight average over all 20 task metrics.

Figure 4: Task-level score changes relative to the main DFM configuration on Mistral-7B-v0.3. Blue and brown indicate increases and decreases, respectively. Avg. gives the change in the unweighted six-task mean. Fusion labels denote the number of fusion layers, while Last four and Start layer 2 use

[28,29,30,31]

and

[2,10,21,31]

, respectively. All differences are computed before rounding.

Learning rate	NQ	WebQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg
$2\times 10^{-5}$	33.47	41.93	29.18	28.60	10.43	19.00	27.10
$1\times 10^{-5}$	33.57	42.47	29.53	25.00	10.05	22.65	27.21

Appendix

Table 9: Learning-rate ablation from the main DFM configuration. All other settings are fixed. The blue

1\times 10^{-5}

row is the default and reproduces the DFM results in Table 1 . Bold marks the higher Avg.

Projector	NQ	WebQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg
Per-layer	32.98	43.21	29.88	26.35	10.18	18.70	26.88
Shared	33.57	42.47	29.53	25.00	10.05	22.65	27.21

Appendix

Table 10: Shared versus per-layer memory projectors on Mistral-7B-v0.3. All other training settings are fixed. Blue marks the default shared projector. Bold marks the higher Avg. Avg is the six-task mean.

Predictor	NQ	WebQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg
14 layers	34.27	41.83	30.14	26.31	10.30	21.03	27.31
56 layers	33.89	42.18	29.58	25.10	10.14	17.91	26.47
28 layers	33.57	42.47	29.53	25.00	10.05	22.65	27.21

Appendix

Table 11: Memory predictor depth on Mistral-7B-v0.3. Only the number of predictor layers changes. Blue marks the default 28-layer predictor. Bold marks the highest Avg. Avg is the six-task mean.

Objective	NQ	WebQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg
Cosine	33.02	42.86	29.40	25.49	10.04	20.42	26.87
Raw MSE	33.57	42.47	29.53	25.00	10.05	22.65	27.21

Appendix

Table 12: Memory prediction objectives on Mistral-7B-v0.3. Only the memory prediction loss changes; all other training settings are fixed. Blue marks the default raw MSE objective. Bold marks the higher Avg. Avg is the six-task mean.

Quantity	MLP Memory	DFM	Basis
Added parameters (B)	1.50	0.75	Configuration
A800 GPUs	16	16	Configuration
Measured training hours	64	24	Timing
Training GPU-hours	1,024	384	GPUs $\times$ hours
Target queries	4B	125M	Workload count
Neighbors per query	1,024	16	Configuration

Appendix

Table 13: Numerical accounting for the comparison in Figure 2 . Construction counts assume a common 4B-token corpus. Training hours are rounded measurements on 16 A800 GPUs for both methods.

Figure 5: Six selected HotpotQA answer comparisons using Mistral-7B-v0.3. Columns show Base (left), MLP Memory (middle), and DFM (right). Check and cross symbols indicate correct and incorrect normalized exact-match answers.

Figure 6: Seven selected offline retrieval examples. Each row shows the preceding query chunk, retrieved chunk text, and ground-truth continuation. Brown spans mark continuation-relevant information in the retrieved text. Text is shortened for display, and split words preserve chunk boundaries.

Memory-Augmented Language Models

Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.

See all 292 papers

Frozen Memory Is Not Enough: Rethinking External Memory as Extraction
Aug 17, 2026Mingyuan Li, Guangsheng Yu, Xu Wang +1Memory-Augmented Language Models Knowledge Augmentation for Language Models
ELLIS Institute Finland · University of Turku · University of Technology Sydney
MemDLM: Memory-Enhanced DLM Training
Mar 23, 2026Zehua Pei, Hui-Ling Zhen, Weizhe Lin +4Diffusion Language Model Training Diffusion Language Models
The Chinese University of Hong Kong · Huawei Technologies Co., Ltd
Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
Jul 30, 2026Rubin Wei, Jiaqi Cao, Jiarui Wang +4Language Model Pretraining Memory-Augmented Language Models
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Electronic Engineering, Tsinghua University

Aug 17, 2026J/K move · Enter open · S save

Frozen Memory Is Not Enough: Rethinking External Memory as Extraction

Mingyuan Li, Guangsheng Yu, Xu Wang, Shaoxiong Ji

ELLIS Institute Finland · University of Turku · University of Technology Sydney

Memory-Augmented Language Models Knowledge Augmentation for Language Models

Open paper AskPDF arXiv

Methods for improving knowledge use in large language models typically fall into two regimes. Non-parametric retrieval offers flexible access to external knowledge, but adds retrieval latency, context overhead, and only shallow integration with the backbone. Parametric adaptation is efficient at inference time, but entangles knowledge with model weights and can be hard to update, audit, or transfer. Engram-style hashed memory occupies a middle regime: it stores learned information in an external, addressable table, yet consumes that table through a small learned reader. This raises a basic question: when such a memory is moved across backbones, what matters more, the frozen memory itself or the target-side reader? We study this question through cross-model frozen-memory extraction, in which a memory trained on a source model is frozen and attached to a different target model, with only a lightweight reader trained. Ablations show that learned memory content and correct addressing both matter, while the amount of usable signal depends on how well the reader is aligned to the target model. In downstream question answering tasks, a dual-layer, four-branch reader nearly closes the gap between same-model and cross-model reuse, achieving an average score of 38.8 under our controlled evaluation protocol. Moreover, when the provider reader is directly compatible with the target interface, the frozen artifact can provide substantial utility without target-side training, while optional reader adaptation yields further improvement. These results suggest that Engram can serve as a reusable external knowledge artifact, provided that the target has access to a compatible reader interface; target-side adaptation can further improve alignment when direct reader reuse is insufficient.

Abstract

Memory-Augmented Language Models

Explore similar work

Frozen Memory Is Not Enough: Rethinking External Memory as Extraction

MemDLM: Memory-Enhanced DLM Training

Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory