Retrieval-augmented generation (RAG) helps large language models (LLMs) answer factual questions beyond their training data, but it adds storage and inference costs. Distilling previously retrieved information into parametric memories enables reuse without online retrieval. However, training the memory to mimic a retriever rewards accurate independent predictions, which do not necessarily improve the final output quality when combined with the LLM's existing predictions. To address this, we introduce ResMem, in which the LLM reads retrieved text offline to generate teacher targets. This design avoids storing massive token representations, unlike existing parametric memory approaches such as MLP Memory, which relies on k-nearest neighbor (kNN) datastores. Furthermore, by using the LLM as an offline teacher that reads privileged retrieved evidence alongside the input, we train a parametric memory to learn an additive logit residual. This residual aligns the predictions of a student sharing the same backbone with the teacher distribution, while self-distillation updates remain confined to the parametric memory. On a Wikipedia subset, ResMem reduces the storage retained for teacher construction by 94.7% compared with the evaluated kNN pipeline. Across five question answering datasets, ResMem raises mean token F1 from 32.83% for the frozen LLM to 34.48%. It also breaks 21% fewer of the LLM's correct answers than a kNN-trained memory. These findings validate our motivation of treating retrieval as a learning resource, with parametric memory shaped by how retrieved knowledge is used to inform the model's future predictions. Code and model weights are available at https://github.com/SSRand/From-Lookups-to-Lessons and https://huggingface.co/RandHan/ResMem-Mistral-7B-v0.3.
Figures & tables
Figure 1: Training pipelines. (a) MLP Memory fits a memory alone to kNN targets; the base enters only at inference (Eq. 5 ). (b) ResMem fits a logit residual so that base plus residual matches an evidence-reading teacher; inference drops the teacher and scales the residual by γ (Eq. 3 ).
Open-domain QA
Long-tail QA
Multi-hop QA
Method
EM ↑
F1 ↑
PPL ↓
EM ↑
F1 ↑
PPL ↓
EM ↑
F1 ↑
PPL ↓
Base
29.97
38.51
11.11
22.01
25.15
11.96
17.64
23.47
6.27
Non-parametric methods
+ BM25-RAG
24.54 -5.42
31.87 -6.64
10.02 ↓ 9.82%
22.65 +0.64
25.86 +0.71
12.29 ↑ 2.72%
18.51 +0.87
26.27 +2.80
5.88 ↓ 6.28%
Parametric methods
+ LoRA
30.00 +0.03
38.71 +0.20
10.60 ↓ 4.67%
22.43 +0.42
26.51 +1.36
11.61 ↓ 2.99%
17.63 -0.01
25.32 +1.85
5.93 ↓ 5.42%
Table 1: Knowledge QA on 38,627 questions. Open-domain QA averages NQ, WebQuestions, and TriviaQA; long-tail QA is PopQA; multi-hop QA is HotpotQA (per-dataset results in Appendix B ). Subscripts give changes from Base: percentage points for EM and F1, and the relative reduction for PPL. Bold marks the best value per column.
Figure 2: Retained storage of the kNN and RAG routes, measured on nested shard sets of English Wikipedia up to 652M positions.
Figure 3: Correction–harm trade-off on 38,627 QA questions. (a) Correction versus harm rates across coefficients; stars ( ∗ ) mark coefficients selected on development data, and the dotted line marks zero net gain. (b) Correction and harm contributions at the selected coefficients.
Figure 4: Distribution concentration. (a) Percentage of positions whose most likely token has probability at least 95%; the base prediction is truncated to its top 64. (b) Tokens covering 95% of the memory’s own softmax, averaged over positions, for memories trained on the kNN teacher from a shared initialization, either alone (as in MLP Memory) or through base plus memory (as in ResMem). The standalone memory’s softmax is its mixture component. The residual’s softmax is not used at inference and is shown for reference.
Native configuration
EM
F1
Gold NLL
Base
25.91
32.83
2.295
Standalone + mixture
25.87
33.12
2.291
Memory Decoder + mixture
25.90
32.87
2.329
Residual + additive
26.03
33.24
2.273
Table 2: Shared-teacher QA ablation with the BM25-RAG teacher, equal exposure, and eight paired seeds. Top: native configurations on 38,627 questions; Gold NLL is the token-weighted reference-answer NLL. Bottom: F1 at the development-selected coefficient / coefficient one on 4,096 diagnostic questions (Base 32.29).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
NQ
WebQ
TriviaQA
PopQA
HotpotQA
Method
( n =3,610)
( n =2,032)
( n =11,313)
( n =14,267)
( n =7,405)
EM ↑
Base
18.31
16.88
54.71
22.01
17.64
+ BM25-RAG
15.87
7.48
50.28
22.65
18.51
+ LoRA
18.55
16.73
54.71
22.43
17.63
+ Memory Decoder
18.50
17.18
54.90
22.12
17.77
Appendix
Table 3: Per-dataset knowledge-QA results behind Table 1 , whose group values are means of these columns before rounding. Coefficients are selected per dataset (Appendix A ). Bold marks the best value per column.
Method
Dialogue
QA
Summarization
Average
Mistral-7B-v0.3
58.02
56.04
50.42
54.83
+ BM25-RAG
63.57 +5.55
52.75 -3.29
50.74 +0.32
55.69 +0.86
+ MLP Memory
65.48 +7.46
63.08 +7.04
52.45 +2.03
60.34 +5.51
+ ResMem (Ours)
65.26 +7.24
70.08 +14.04
53.93 +3.51
63.09 +8.26
Appendix
Table 4: HaluEval hallucination-detection accuracy (%). Average weights the three tasks equally. Subscripts denote percentage-point changes from the base model; bold marks the column best.
Method
Memory
F1
TTFT (ms)
Total (ms)
Decode (tok/s)
Peak (GiB)
Base
—
32.79
413.30
1011.22
829.19
14.942
MLP Memory
FP32
33.68
420.77
1252.34
596.23
20.692
MLP Memory
BF16
33.70
416.16
1099.91
725.03
17.817
ResMem
FP32
34.77
420.79
1254.14
594.72
20.692
ResMem
BF16
34.75
415.95
1097.76
726.77
17.817
Memory Decoder
FP32
33.07
830.74
2029.42
413.20
17.921
Appendix
Table 5: Precision and deployment cost for the knowledge-QA checkpoints at fixed λ=0.25 (MLP Memory and Memory Decoder) and γ=0.4 (ResMem) on every dataset; because Table 1 selects coefficients per dataset by EM and decodes with eager attention, its F1 values differ. Quality covers 38,627 questions with EOS-or-12-token decoding; timing uses batch 16, 384 input tokens, and exactly 32 output tokens on one A100-SXM4-80GB with BF16 Base, SDPA and KV caching. Times and peak allocated memory are medians over ten steady batches; decode throughput uses pooled generated tokens per second. Memory Decoder rows were measured on a second identical host, where Base takes 1,014 ms in total. These checkpoints and runtime differ from Table 2 .
ResMem
ResMem + gate
Dataset
EM
F1
EM
F1
NQ
18.98
25.69
19.20
25.95
WebQ
17.32
31.94
17.13
31.65
TriviaQA
55.55
61.57
55.68
61.61
Open-domain mean
30.61
39.73
30.67
39.74
PopQA (long-tail)
22.85
26.91
22.91
26.81
Appendix
Table 6: Fixed-strength ResMem with and without the tokenwise gate. ResMem matches Table 1 .
Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.
Retrieval-augmented generation (RAG) improves language model (LM) performance by providing relevant context at test time for knowledge-intensive situations. In this work, we systematically study the trade-off between pretraining and retrieval by training OLMo-2-based LMs ranging from 30M to 3B parameters on up to 100B DCLM tokens, while varying pretraining data scale, retrieval store size, and retrieval store source (pretraining vs. new data) across reasoning, scientific QA, and open-domain QA benchmarks. We find that retrieval gains depend on model capacity and pretraining exposure and are strongly front-loaded, with a median 91% of the largest observed improvement realized by one retrieval token per model parameter. However, the interaction is objective-dependent: smaller models gain more in gold-answer perplexity, whereas larger, more-pretrained models gain more in accuracy. Retrieval from previously seen data also preserves most of the held-out retrieval gain. Retrieval is therefore a task-, regime-, and metric-dependent complement to parametric learning whose value also depends on datastore size and information novelty. Overall, this motivates the explicit partitioning of data between internalization and external access for LM design.
Karan Singh, Michael Yu, Varun Gangal +4
Stanford University · †DegenAI Labs · Independent Researcher +3
Large language models need effective ways to acquire and use external knowledge beyond what they retain from pretraining. Continued pretraining risks catastrophic forgetting, while retrieval at inference time can add substantial overhead. External parametric memory avoids online retrieval, but constructing its training data requires extensive offline search, and output-level interpolation provides only shallow fusion. We introduce Deep Fusion Memory (DFM), a parametric memory architecture that reduces this construction workload and incorporates learned memory into intermediate hidden states. A causal memory predictor learns to generate memory from the preceding context, using retrieved chunk embeddings from the corpus as supervision. The predicted memory is shared across tokens within each chunk and fused into their hidden states through gated cross-attention. Memory prediction and fusion are trained end to end with the backbone frozen, and inference requires no explicit retrieval. Compared with existing parametric memory approaches, which often construct token-level memory targets, DFM requires 32× fewer retrieval queries and 2048× fewer retrieved items to construct its training data. Across six QA tasks, DFM performs better over the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base. Beyond QA, domain adaptation yields average gains of 6.51 and 4.38 points over Qwen3-4B-Base and Qwen3-8B-Base, respectively.
Rubin Wei, Siyuan Sun, Hao Sun +5
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
Retrieval-augmented generation (RAG) improves factuality but adds latency and engineering overhead at serving time. We propose RING (Retrieval-Internalized Generation), a holistic paradigm spanning both architecture and training that injects large-scale external knowledge into a Mixture-of-Memory Experts and learns parametric search over this internal memory via reinforcement learning, removing the external retriever entirely. Training proceeds in three stages: continued pre-training injects new corpora into a Knowledge Expert via our novel Dual Causal Attention; supervised fine-tuning teaches a ``search-then-answer'' pattern; and reinforcement learning with hierarchical rewards optimizes the routing-and-search policy over the parametric memory. Unlike prior parametric injection methods that pair internal memory with a fixed or rule-based retriever, RING {learns} its retrieval policy directly from task signals. We further frame RING theoretically as a search-free approximation to the classical RAG objective. To evaluate large-scale injection of genuinely {new} knowledge without test-time leakage, we further construct News-2025, a benchmark built from news strictly post-dating the base LLM's pretraining cutoff. RING matches or surpasses both search-based RAG and parametric injection baselines in accuracy and efficiency.
Shicheng Xu, Liang Pang, Liyi Chen +7
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Xiaohongshu Inc.