Retrieval-augmented generation (RAG) helps large language models (LLMs) answer factual questions beyond their training data, but it adds storage and inference costs. Distilling previously retrieved information into parametric memories enables reuse without online retrieval. However, training the memory to mimic a retriever rewards accurate independent predictions, which do not necessarily improve the final output quality when combined with the LLM's existing predictions. To address this, we introduce ResMem, in which the LLM reads retrieved text offline to generate teacher targets. This design avoids storing massive token representations, unlike existing parametric memory approaches such as MLP Memory, which relies on k-nearest neighbor (kNN) datastores. Furthermore, by using the LLM as an offline teacher that reads privileged retrieved evidence alongside the input, we train a parametric memory to learn an additive logit residual. This residual aligns the predictions of a student sharing the same backbone with the teacher distribution, while self-distillation updates remain confined to the parametric memory. On a Wikipedia subset, ResMem reduces the storage retained for teacher construction by 94.7% compared with the evaluated kNN pipeline. Across five question answering datasets, ResMem raises mean token F1 from 32.83% for the frozen LLM to 34.48%. It also breaks 21% fewer of the LLM's correct answers than a kNN-trained memory. These findings validate our motivation of treating retrieval as a learning resource, with parametric memory shaped by how retrieved knowledge is used to inform the model's future predictions. Code and model weights are available at https://github.com/SSRand/From-Lookups-to-Lessons and https://huggingface.co/RandHan/ResMem-Mistral-7B-v0.3.
Figures & tables
Figure 1: Training pipelines. (a) MLP Memory fits a memory alone to kNN targets; the base enters only at inference (Eq. 5 ). (b) ResMem fits a logit residual so that base plus residual matches an evidence-reading teacher; inference drops the teacher and scales the residual by γ (Eq. 3 ).
Open-domain QA
Long-tail QA
Multi-hop QA
Method
EM ↑
F1 ↑
PPL ↓
EM ↑
F1 ↑
PPL ↓
EM ↑
F1 ↑
PPL ↓
Base
29.97
38.51
11.11
22.01
25.15
11.96
17.64
23.47
6.27
Non-parametric methods
+ BM25-RAG
24.54 -5.42
31.87 -6.64
10.02 ↓ 9.82%
22.65 +0.64
25.86 +0.71
12.29 ↑ 2.72%
18.51 +0.87
26.27 +2.80
5.88 ↓ 6.28%
Parametric methods
+ LoRA
30.00 +0.03
38.71 +0.20
10.60 ↓ 4.67%
22.43 +0.42
26.51 +1.36
11.61 ↓ 2.99%
17.63 -0.01
25.32 +1.85
5.93 ↓ 5.42%
Table 1: Knowledge QA on 38,627 questions. Open-domain QA averages NQ, WebQuestions, and TriviaQA; long-tail QA is PopQA; multi-hop QA is HotpotQA (per-dataset results in Appendix B ). Subscripts give changes from Base: percentage points for EM and F1, and the relative reduction for PPL. Bold marks the best value per column.
Figure 2: Retained storage of the kNN and RAG routes, measured on nested shard sets of English Wikipedia up to 652M positions.
Figure 3: Correction–harm trade-off on 38,627 QA questions. (a) Correction versus harm rates across coefficients; stars ( ∗ ) mark coefficients selected on development data, and the dotted line marks zero net gain. (b) Correction and harm contributions at the selected coefficients.
Figure 4: Distribution concentration. (a) Percentage of positions whose most likely token has probability at least 95%; the base prediction is truncated to its top 64. (b) Tokens covering 95% of the memory’s own softmax, averaged over positions, for memories trained on the kNN teacher from a shared initialization, either alone (as in MLP Memory) or through base plus memory (as in ResMem). The standalone memory’s softmax is its mixture component. The residual’s softmax is not used at inference and is shown for reference.
Native configuration
EM
F1
Gold NLL
Base
25.91
32.83
2.295
Standalone + mixture
25.87
33.12
2.291
Memory Decoder + mixture
25.90
32.87
2.329
Residual + additive
26.03
33.24
2.273
Table 2: Shared-teacher QA ablation with the BM25-RAG teacher, equal exposure, and eight paired seeds. Top: native configurations on 38,627 questions; Gold NLL is the token-weighted reference-answer NLL. Bottom: F1 at the development-selected coefficient / coefficient one on 4,096 diagnostic questions (Base 32.29).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
NQ
WebQ
TriviaQA
PopQA
HotpotQA
Method
( n =3,610)
( n =2,032)
( n =11,313)
( n =14,267)
( n =7,405)
EM ↑
Base
18.31
16.88
54.71
22.01
17.64
+ BM25-RAG
15.87
7.48
50.28
22.65
18.51
+ LoRA
18.55
16.73
54.71
22.43
17.63
+ Memory Decoder
18.50
17.18
54.90
22.12
17.77
Appendix
Table 3: Per-dataset knowledge-QA results behind Table 1 , whose group values are means of these columns before rounding. Coefficients are selected per dataset (Appendix A ). Bold marks the best value per column.
Method
Dialogue
QA
Summarization
Average
Mistral-7B-v0.3
58.02
56.04
50.42
54.83
+ BM25-RAG
63.57 +5.55
52.75 -3.29
50.74 +0.32
55.69 +0.86
+ MLP Memory
65.48 +7.46
63.08 +7.04
52.45 +2.03
60.34 +5.51
+ ResMem (Ours)
65.26 +7.24
70.08 +14.04
53.93 +3.51
63.09 +8.26
Appendix
Table 4: HaluEval hallucination-detection accuracy (%). Average weights the three tasks equally. Subscripts denote percentage-point changes from the base model; bold marks the column best.
Method
Memory
F1
TTFT (ms)
Total (ms)
Decode (tok/s)
Peak (GiB)
Base
—
32.79
413.30
1011.22
829.19
14.942
MLP Memory
FP32
33.68
420.77
1252.34
596.23
20.692
MLP Memory
BF16
33.70
416.16
1099.91
725.03
17.817
ResMem
FP32
34.77
420.79
1254.14
594.72
20.692
ResMem
BF16
34.75
415.95
1097.76
726.77
17.817
Memory Decoder
FP32
33.07
830.74
2029.42
413.20
17.921
Appendix
Table 5: Precision and deployment cost for the knowledge-QA checkpoints at fixed λ=0.25 (MLP Memory and Memory Decoder) and γ=0.4 (ResMem) on every dataset; because Table 1 selects coefficients per dataset by EM and decodes with eager attention, its F1 values differ. Quality covers 38,627 questions with EOS-or-12-token decoding; timing uses batch 16, 384 input tokens, and exactly 32 output tokens on one A100-SXM4-80GB with BF16 Base, SDPA and KV caching. Times and peak allocated memory are medians over ten steady batches; decode throughput uses pooled generated tokens per second. Memory Decoder rows were measured on a second identical host, where Base takes 1,014 ms in total. These checkpoints and runtime differ from Table 2 .
ResMem
ResMem + gate
Dataset
EM
F1
EM
F1
NQ
18.98
25.69
19.20
25.95
WebQ
17.32
31.94
17.13
31.65
TriviaQA
55.55
61.57
55.68
61.61
Open-domain mean
30.61
39.73
30.67
39.74
PopQA (long-tail)
22.85
26.91
22.91
26.81
Appendix
Table 6: Fixed-strength ResMem with and without the tokenwise gate. ResMem matches Table 1 .
Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.