Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time. We study this setting as document knowledge internalization: converting a fixed corpus into usable parametric knowledge for retrieval-free question answering. We propose IAR (Inject, Align, and Recover), a three-stage post-training framework that separates structured document knowledge injection, QA behavior alignment, and general ability recovery. Unlike conventional continued pretraining, Inject converts source documents into continuation, rewrite, and instruction-conditioned reconstruction objectives. Align then adapts the injected model with answer-only QA supervision, while Recover merges the domain-adapted model with the base instruction model to recover general capabilities. Across Common Corpus (CC) and CCI, and across Llama, Phi, Qwen, and SmolLM model families, IAR improves the domain-primary domain-general frontier for retrieval-free document internalization. In the main comparison, IAR improves over Vanilla SFT on all four reported metrics in 7 of 8 dataset-model settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance across IFEval, MMLU, and MSBench. Extended CC baselines show that LoRA and FAPM can win individual general metrics, but among methods that also reach leading or near-leading domain internalization, IAR retains one of the strongest general profiles.
Figures & tables
Figure 1: The overview of IAR . Vanilla SFT learns from generated QA pairs and covers only the facts selected by those questions. CPT+SFT models full document token streams before a separate QA alignment stage. IAR injects document knowledge through continuation, rewrite, and instruction-conditioned reconstruction objectives, aligns the injected model with answer-only QA supervision, and recovers general ability via post-hoc model merging. The final model is selected by balancing retrieval-free domain QA performance and general capability retention.
Model
Method
CC
CCI
Dom.
IFEval
MMLU
MSB.
Dom.
IFEval
MMLU
MSB.
Llama-3.2-3B
Base Instruct
11.2
77.1
50.8
54.5
29.2
77.1
50.8
54.5
Vanilla SFT
35.5
54.2
11.2
21.5
53.0
61.2
22.5
31.5
CPT+SFT
38.3
26.4
3.7
13.5
53.7
24.2
4.3
17.5
IAR
36.5
60.2
35.0
30.5
55.3
61.3
33.2
36.5
Phi-4-mini
Base Instruct
13.1
77.2
61.0
62.0
27.3
77.2
61.0
62.0
Table 1: Main comparison for RQ1. Scores are reported as percentages. Within each model block and dataset block, bold denotes the best result and underlining denotes the second-best result for each metric. CPT+SFT starts from the corresponding released Base checkpoint; “/” indicates that no such release is available, as for Phi-4-mini.
Model
Method
CC
CCI
Dom.
IFEval
MMLU
MSB.
Dom.
IFEval
MMLU
MSB.
Llama-3.2-3B
Vanilla SFT
35.5
54.2
11.2
21.5
53.0
61.2
22.5
31.5
BudgetMatch
40.4
53.4
17.3
22.0
53.0
45.3
22.5
24.0
IAR
36.5
60.2
35.0
30.5
55.3
61.3
33.2
36.5
Qwen3-4B
Vanilla SFT
42.4
51.1
8.8
51.0
75.1
45.6
26.3
49.5
BudgetMatch
46.8
49.4
31.8
45.0
72.2
56.1
29.5
52.5
Table 2: Token-budget ablation for RQ2 (higher is better; scores are percentages). BudgetMatch uses setting-specific QA-only epochs matched to the Inject+Align budget. IAR improves both domain accuracy and mean general performance in three settings; CC Llama is the remaining domain–general trade-off. Supplementary Material, Section C gives the full accounting.
Figure 2: BudgetMatch-to- IAR movement. Right is higher domain accuracy; up is a higher mean over IFEval, MMLU, and MSBench. IAR moves up and right in three of four settings; for CC Llama, it trades 3.9 domain points for an 11.0-point gain in mean general performance.
Figure 3: Pre-recovery domain gains from Inject+Align. CCI Qwen3-4B’s +0.4 pp is the high-base-prior boundary case.
Model
Method
Domain (%)
IFEval (%)
MMLU (%)
MSBench (%)
Qwen3-8B
Base Instruct
38.5
87.6
65.3
82.5
Vanilla SFT
48.7
56.4
14.0
52.5
Best IA
57.5
50.6
18.5
48.5
IAR (TIES d=0.3 )
56.8
62.2
26.7
73.5
Qwen3-14B
Base Instruct
40.4
90.0
72.5
81.5
Vanilla SFT
54.8
62.9
54.5
57.0
Table 3: Complete Qwen3 scaling ablation on CC. Bold and underline mark the best and second-best result within each model block. Across 8B/14B/32B, IAR stays within 1.1 points of Best IA domain accuracy while recovering 14.9–24.1 points in mean general performance.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Split
Count
Input at train time
Input at test time
Primary use
CC
train
14,258 QA
question, answer, derived document fields
question only
mixed-domain internalization
test
750 QA
/
question only
held-out domain test
CCI
train
10,926 QA
question, answer, derived document fields
question only
Chinese-domain internalization
test
575 QA
/
question only
held-out domain test
Appendix
Table 4: Dataset construction contract. The train/test split and retrieval-free input column define the experimental setting: models must answer from internalized parameters rather than from retrieved source passages. The files named eval_750.jsonl and eval_575.jsonl in the repository are treated as held-out test files in this paper.
Step
Prompt schema
Output contract
Anchor extraction
Given a text chunk, extract at most K independently referable core objects. Prefer explicit concepts, methods, mechanisms, modules, devices, or technical terms appearing in the text. Do not output deictic objects such as “this method” or “the above mechanism.”
JSON list of anchors.
Type applicability
Given a question-type description and the text chunk, decide whether the chunk can support a question of that type. The supported types are factual extraction, mechanism explanation, design rationale, condition/constraint, limitation/trade-off, and comparison/relation.
yes/no.
Question generation
Given the chunk, a selected anchor, a question type, the type description, and an expected answer schema, generate one natural question. The anchor must be explicitly named; the question must be understandable without the source document; deictic expressions such as “this,” “above,” or “according to the text” are forbidden; output only one question ending with a question mark.
Plain question text.
Question validation
Check whether the question is independently understandable, avoids document/deictic references, is semantically clear, and has an answer direction.
JSON with valid and reason .
Answer generation
Given the source chunk and generated question, answer strictly from the chunk. The answer must be faithful, accurate, professional, directly answer the question, avoid document/deictic references, and be written as a natural paragraph rather than a template with section headings.
JSON with answer .
Appendix
Table 5: Document-derived QA construction prompts. The schemas show how chunks are converted into self-contained questions and grounded answers while preventing deictic questions that require access to the original document.
Dataset
Input docs
Files filtered
Chunks kept
Valid questions
QA kept
Experiment QA
CC
4,001
1,944 (48.6%)
5,324/6,593 (80.8%)
37,397/77,224 (48.4%)
16,674/37,397 (44.6%)
15,008
CCI
7,000
310 (4.4%)
11,407/11,793 (96.7%)
100,096/140,148 (71.4%)
62,670/100,096 (62.6%)
11,501
Appendix
Table 6: QA-generation stage accounting. “Files filtered” gives the count and percentage of input documents rejected at file-level filtering; the chunk, question, and QA columns give retained/considered counts and micro rates. “Experiment QA” is the train-plus-test total after exact-question deduplication and fixed chunk-group sampling.
Objective
User input u
Assistant target y
Masked tokens
Recipe role
Intended exposure
Continuation
Continue/complete instruction plus a document prefix.
Held-out suffix.
System prompt, instruction, and prefix.
single or mixed
Prefix-conditioned document exposure.
Rewrite
Reconstruction instruction plus a generated summary, outline, or knowledge skeleton.
Full cleaned document.
System prompt, instruction, and compressed representation.
single or mixed
Recover document content from a compressed representation.
Instruction-formatted reconstruction
Short generic reading instruction.
Full cleaned document.
System prompt and instruction.
1:0:0 or mixed
Dense exposure through a full-document target.
Appendix
Table 7: Inject objective definitions. All three objectives use the instruction model’s chat template and assistant-target loss; the loss mask excludes every system/user token. Recipe ratios control the relative counts of the three objective streams, while realized shares can differ slightly after tokenizer-specific length filtering.
Parameter
Value
Parameter
Value
Optimizer
AdamW (PyTorch)
Learning rate
5×10−5
Adam β1,β2,ϵ
.9,.999,10−8
Scheduler / warmup
cosine / ratio .05
Weight decay / max grad norm
.01/1.0
Precision / max length
BF16 / 4096
Batch per GPU
1
Gradient accum. / GPUs
8/8
Effective global batch
64 examples/step
DeepSpeed
ZeRO-2, no offload
Termination / checkpoint
epoch based / final epoch
Gradient checkpointing
model-level enabled
Appendix
Table 8: Shared optimization settings recovered for the completed 3B/4B training runs and extended CC baselines. The effective global batch is per-device batch 1×8 accumulation steps ×8 GPUs.
Method / stage
Epochs
Data and objective
Method-specific setting
Vanilla SFT
3
QA; answer-only
Original Instruct initialization
BudgetMatch
14/17/21/11
Same QA and loss as Vanilla
CC Llama/Qwen; CCI Llama/Qwen order
Inject
3
Three assistant-target document-generation objectives
Selected mixtures and counts in Table 10
Align
3
QA; answer-only
Initializes from the Inject final epoch
SDFT
3
Model-specific synthetic QA; answer-only
14,258 CC examples per model
LoRA
3
CC QA; answer-only
r=16 , α=32 , dropout .05; merged for evaluation
Appendix
Table 9: Stage-specific training settings. Unless stated as an exception, each row uses Table 8 . BudgetMatch epoch counts are setting-specific rather than a shared 13-epoch approximation.
Dataset
Model
Selected Inject recipe
Inject rows
Align QA rows
CC
Llama-3.2-3B
Mixed 1:1:2
19,000
14,258
Phi-4-mini
Mixed 1:1:2
19,000
14,258
Qwen3-4B
Mixed 1:1:1
18,968
14,258
SmolLM3-3B
Mixed 1:1:1
18,960
14,258
CCI
Llama-3.2-3B
Mixed 1:1:2
19,000
10,926
Phi-4-mini
Mixed 1:1:2
19,000
10,926
Appendix
Table 10: Selected Inject configurations for the eight main settings. Counts are realized post-tokenization training rows; tokenizer-specific length filtering explains the small differences among nominally equal mixtures.
Setting
Inject recipe (samples)
Inject
Align
IA total
BudgetMatch
BM/IA
CC Llama-3.2-3B
Mixed 1:1:2 (19k)
45.736
12.917
58.653
60.280
102.8%
CC Qwen3-4B
Mixed 1:1:1 (18,968)
46.017
10.191
56.208
57.752
102.7%
CCI Llama-3.2-3B
Mixed 1:1:2 (19k long-doc.)
57.296
9.734
67.030
68.085
101.6%
CCI Qwen3-4B
1:0:0 (10k)
19.081
7.151
26.232
26.202
99.9%
Appendix
Table 11: Realized token accounting for BudgetMatch, in millions of non-padding model tokens. BM/IA compares the completed QA-only BudgetMatch run directly with the corresponding Inject+Align token volume.
Metric
Direction
Evaluation type
Reported unit
Domain accuracy
higher is better
V2 multi-judge correctness
percentage
IFEval inst-strict
higher is better
instruction-following evaluator
percentage
MMLU accuracy
higher is better
multiple-choice benchmark
percentage
MSBench accuracy
higher is better
LLM-judge benchmark
percentage
Appendix
Table 12: Evaluation metrics and their roles in operating-point selection. Domain QA is the primary retrieval-free internalization metric, while IFEval, MMLU, and MSBench serve as general-capability guardrails.
Scope
Records
Exact agree
Binary agree
Binary κ
Third judge
CC test artifacts
163,347
.732
.854
.691
.274
CCI test artifacts
78,308
.656
.836
.670
.345
Full audit
242,255
.707
.848
.691
.297
Appendix
Table 13: Domain QA judge reliability. Binary agreement collapses scores at ≥.5 . The full audit additionally contains 600 samples from a smaller CCI evaluation set. These statistics quantify judge reliability over the evaluated samples.
Judge
Votes
Mean score
P(s≥.5)
P(s=1)
gpt-oss-120b
242,031
.387
.454
.321
MiniMax2.5_Local
156,812
.303
.434
.172
deepseek-v3.2
131,172
.296
.465
.127
deepseek-v3.1
24,697
.378
.652
.103
Appendix
Table 14: Per-judge score distributions before aggregation. Vote counts differ because panel configurations vary across evaluations and later judges are invoked adaptively; the marginal means are therefore descriptive and should not be interpreted as a controlled judge ranking.
Operator family
Hyperparameter grid
Candidate count
SLERP
t∈{0.2,0.3,0.4}
3
Task Arithmetic
w∈{0.3,0.5,0.7}
3
TIES
d∈{0.3,0.5,0.7}
3
DARE
dr∈{0.1,0.3,0.5}
3
Total
fixed grid per IA checkpoint
12
Appendix
Table 15: Recover candidate grid. Every selected IAR row is chosen from this fixed set of post-hoc merge candidates rather than from an unreported per-row search space. Candidate choice is made on the validation split before held-out test reporting.
Figure 4: Recover candidate frontiers for the main dataset–model settings. Each panel plots held-out test performance of the fixed Recover candidates by retrieval-free domain QA accuracy and mean general performance across IFEval, MMLU, and MSBench. The black line marks non-dominated points in this two-dimensional projection, and the black ring highlights the candidate selected by the validation protocol for the main IAR table. The displayed test frontier is diagnostic and is not used to choose the selected candidate.
Dataset
Model
Selected checkpoint
Domain (%)
IFEval (%)
MMLU (%)
MSBench (%)
Selection note
CC
Llama-3.2-3B
TIES d=0.3
36.5
60.2
35.0
30.5
domain-primary feasible point
Phi-4-mini
Task Arithmetic w=0.7
34.1
49.0
57.0
43.0
balanced feasible point
Qwen3-4B
TIES d=0.3
50.5
59.8
19.5
63.0
domain-primary feasible point
SmolLM3-3B
TIES d=0.3
37.5
40.3
25.7
29.0
domain-primary feasible point
CCI
Llama-3.2-3B
TIES d=0.3
55.3
61.3
33.2
36.5
domain-primary feasible point
Phi-4-mini
TIES d=0.3
39.7
51.6
50.2
44.0
boundary trade-off point
Appendix
Table 16: Selected Recover settings for the main IAR rows. The table reports held-out test scores for the concrete operating point chosen from the fixed Recover grid by the validation protocol, making the domain-primary selection rule explicit rather than treating recovery as an unreported hyperparameter search.
Initial checkpoint
CCI QA (%)
Doc. mean BPB
Corpus BPB
Qwen–peer macro Δ [95% CI]
Qwen3-4B-Instruct
70.6
0.744
0.729
–
Llama-3.2-3B-Instruct
29.2
1.021
0.998
−0.273 [ −0.285 , −0.261 ]
Phi-4-mini-instruct
27.3
0.968
0.948
−0.220 [ −0.231 , −0.208 ]
SmolLM3-3B
34.1
0.838
0.819
−0.093 [ −0.102 , −0.084 ]
Appendix
Table 17: Conditional BPB on identical CCI source-document continuations. Corpus BPB is byte weighted; the primary effect first averages paired Qwen-minus-peer BPB differences within each of seven domains and then weights domains equally. Negative Δ means lower BPB for Qwen. All intervals exclude zero and all seven domain means agree in direction.
Model
MMLU
MMLU-Pro
BBH
MGSM
MMMLU
INCLUDE
Qwen3-1.7B-Base
62.63
36.76
54.47
50.71
63.27
45.57
Qwen3-4B-Base
72.99
50.58
72.59
67.74
71.42
56.29
Qwen3-8B-Base
76.89
56.73
78.40
76.02
75.72
59.40
Qwen3-14B-Base
81.05
61.03
81.07
79.20
79.69
64.55
Qwen3-32B-Base
83.61
65.54
87.38
83.06
83.83
67.87
Appendix
Table 18: Officially disclosed Qwen3 dense-family base-model scores from the Qwen3 technical report. These are public general and multilingual benchmark results, not CCI QA results, and are included only to contextualize the high-base-prior interpretation.
Model
Setting
CCI domain (%)
Qwen3-1.7B
Base Instruct
60.0
Vanilla SFT
57.0
Mixed 1:1:1 + Stage2
56.7
Qwen3-4B
Base Instruct
70.6
Vanilla SFT
75.1
Context-aware SFT
65.0
Appendix
Table 19: Qwen3 CCI high-base-prior diagnostics. The 1.7B and 4B rows show that Qwen3 starts unusually high on CCI, so this setting is better interpreted as preserving and recovering a strong prior than as creating a large new domain gain.
Operator
Coefficient
CCI domain (%)
Operator
Coefficient
CCI domain (%)
DARE
dr=0.1
74.1
SLERP
t=0.2
69.0
DARE
dr=0.3
74.3
SLERP
t=0.3
72.7
DARE
dr=0.5
73.7
SLERP
t=0.4
73.2
Task Arithmetic
w=0.3
71.1
TIES
d=0.3
76.3
Task Arithmetic
w=0.5
74.1
TIES
d=0.5
72.5
Task Arithmetic
w=0.7
76.3
TIES
d=0.7
75.0
Appendix
Table 20: Qwen3-4B CCI Recover sweep. Multiple merge operators stay in the high-domain range, with Task Arithmetic and TIES reaching the main selected value; this supports the boundary-case reading of the Qwen3 CCI row.
Model
Method
Domain (%)
IFEval (%)
MMLU (%)
MSBench (%)
Llama-3.2-3B
Vanilla SFT
35.5
54.2
11.2
21.5
SDFT
39.9
58.4
23.5
33.5
LoRA
22.5
64.0
53.2
30.0
Replay
30.8
56.7
30.7
37.5
Vanilla-FAPM
22.3
75.3
51.8
62.0
IA-FAPM
22.4
74.5
53.7
59.0
Appendix
Table 21: Complete CC extended-baseline comparison for Llama-3.2-3B and Phi-4-mini.
Model
Method
Domain (%)
IFEval (%)
MMLU (%)
MSBench (%)
Qwen3-4B
Vanilla SFT
42.4
51.1
8.8
51.0
SDFT
44.1
55.9
14.0
53.0
LoRA
31.3
68.2
48.5
53.0
Replay
42.8
69.7
34.7
66.5
Vanilla-FAPM
41.2
83.8
61.3
83.5
IA-FAPM
44.1
82.6
65.5
85.0
Appendix
Table 22: Complete CC extended-baseline comparison for Qwen3-4B and SmolLM3-3B.
Dataset
Model
Domain (%)
IFEval (%)
MMLU (%)
MSBench (%)
CC
Llama-3.2-3B
38.7
43.5
9.8
19.5
CC
Phi-4-mini
31.2
43.3
40.7
31.0
CCI
Llama-3.2-3B
52.9
29.0
8.3
23.0
Appendix
Table 23: Available Instruct-initialized CPT+SFT diagnostics. CC Llama exposes a domain–general trade-off rather than uniform IAR dominance, while the CC Phi and CCI Llama diagnostics remain below the corresponding best IA domain scores. Coverage is limited to completed archived runs and is not extrapolated to Qwen3 or SmolLM3.
Dataset
Model
Vanilla SFT (%)
Vanilla-FAPM (%)
IA-FAPM (%)
CC
Llama-3.2-3B
35.5
22.3
22.4
Phi-4-mini
24.4
17.1
16.9
Qwen3-4B
42.4
41.2
44.1
SmolLM3-3B
32.1
24.7
26.1
CCI
Llama-3.2-3B
53.0
38.6
40.7
Phi-4-mini
40.2
27.3
28.7
Appendix
Table 24: FAPM domain results at sparsity 0.9. The domain-only comparison shows that pruning-based recovery often sacrifices internalized document knowledge, even when it is useful for general-capability restoration.
Dataset
Model
Variant
Domain (%)
IFEval (%)
MMLU (%)
MSBench (%)
CC
Llama-3.2-3B
Vanilla-FAPM
22.3
75.3
51.8
62.0
IA-FAPM
22.4
74.5
53.7
59.0
Phi-4-mini
Vanilla-FAPM
17.1
56.4
58.2
64.5
IA-FAPM
16.9
53.8
61.3
63.0
Qwen3-4B
Vanilla-FAPM
41.2
83.8
61.3
83.5
IA-FAPM
44.1
82.6
65.5
85.0
Appendix
Table 25: FAPM domain-general results at sparsity 0.9. The table exposes the recovery trade-off directly: FAPM often improves general benchmarks, but the domain column explains why it is not a drop-in replacement for IAR .
Model
Vanilla
Mix 1:1:1
Continue
1:0:0
Rewrite
Mix 1:1:2
Llama-3.2-3B
35.5
36.1
35.1
36.8
34.4
38.3
Phi-4-mini
24.4
27.5
27.2
29.9
27.6
32.1
Qwen3-4B
42.4
47.7
40.9
44.8
43.5
46.9
SmolLM3-3B
32.1
36.8
32.7
33.6
31.6
36.1
Appendix
Table 26: CC Inject recipe sensitivity before Recover. Mixed objectives are usually robust on CC, but the best pre-recovery domain checkpoint still varies by model family.
Model
Vanilla
Mix 1:1:1
Continue
1:0:0
Rewrite
Mix 1:1:2
Llama-3.2-3B
53.0
57.2
55.8
57.2
54.4
58.6
Phi-4-mini
40.2
45.0
42.1
36.7
37.7
46.3
Qwen3-4B
75.1
73.7
72.9
75.5
74.4
72.5
SmolLM3-3B
52.3
54.1
52.2
53.6
51.1
54.6
Appendix
Table 27: CCI Inject recipe sensitivity before Recover. Mixed 1:1:2 remains strongest for most model families, while Qwen3-4B is the high-base-prior boundary case where the 1:0:0 reconstruction recipe slightly exceeds mixed objectives.
Model
Method
Domain (%)
IFEval (%)
MMLU (%)
MSBench (%)
Qwen3-8B
Base Instruct
38.5
87.6
65.3
82.5
Vanilla SFT
48.7
56.4
14.0
52.5
Best IA
57.5
50.6
18.5
48.5
IAR (TIES d=0.3 )
56.8
62.2
26.7
73.5
Qwen3-14B
Base Instruct
40.4
90.0
72.5
81.5
Vanilla SFT
54.8
62.9
54.5
57.0
Appendix
Table 28: Complete Qwen3 scaling ablation on CC. Bold and underline mark the best and second-best result within each model block.
Retrieval-augmented generation (RAG) improves factuality but adds latency and engineering overhead at serving time. We propose RING (Retrieval-Internalized Generation), a holistic paradigm spanning both architecture and training that injects large-scale external knowledge into a Mixture-of-Memory Experts and learns parametric search over this internal memory via reinforcement learning, removing the external retriever entirely. Training proceeds in three stages: continued pre-training injects new corpora into a Knowledge Expert via our novel Dual Causal Attention; supervised fine-tuning teaches a ``search-then-answer'' pattern; and reinforcement learning with hierarchical rewards optimizes the routing-and-search policy over the parametric memory. Unlike prior parametric injection methods that pair internal memory with a fixed or rule-based retriever, RING {learns} its retrieval policy directly from task signals. We further frame RING theoretically as a search-free approximation to the classical RAG objective. To evaluate large-scale injection of genuinely {new} knowledge without test-time leakage, we further construct News-2025, a benchmark built from news strictly post-dating the base LLM's pretraining cutoff. RING matches or surpasses both search-based RAG and parametric injection baselines in accuracy and efficiency.
Shicheng Xu, Liang Pang, Liyi Chen +7
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Xiaohongshu Inc.
Long input sequences are central to document understanding and multi-step reasoning in Large Language Models, yet the quadratic cost of attention makes inference both memory-intensive and slow. Context distillation mitigates this by compressing contextual information into model parameters, and recent work such as Doc-to-LoRA amortizes context distillation into a single forward pass that generates one LoRA adapter per document. However, producing a single monolithic adapter for all queries leads to irrelevant-query interference, limited compositional recall, and poor scalability to long-document reasoning. To address these challenges, we propose Doc-to-Atom (Doc2Atom), a compositional parametric memory framework that decomposes each document into semantically typed knowledge atoms. Each atom is compiled into an independent micro-LoRA adapter and a provenance retrieval key. At inference time, a lightweight query router selects and assembles only the relevant atoms into a query-specific adapter, which is then injected into a frozen base model. The entire system is trained end-to-end through a multi-objective distillation framework. Experiments on six diverse QA benchmarks demonstrate that Doc2Atom outperforms Doc-to-LoRA baselines while reducing the memory cost of document internalization.
Xingjian Diao, Wenbo Li, Yashas Malur Saidutta +3
AI Center-Mountain View, Samsung Electronics · Dartmouth College
Retrieval-augmented generation (RAG) typically treats retrieval and generation as separate systems. We ask whether an attention-based encoder-decoder can instead retrieve directly from its own internal representations. We introduce INTRA (INTrinsic Retrieval via Attention), a framework where decoder attention queries score pre-encoded evidence chunks that are then directly reused as context for generation. By construction, INTRA unifies retrieval and generation, eliminating the retriever-generator mismatch typical of RAG pipelines. This design also amortizes context encoding by reusing precomputed encoder states across queries. On question-answering benchmarks, INTRA outperforms strong engineered retrieval pipelines on both evidence recall and end-to-end answer quality. Our results demonstrate that attention-based models already possess a retrieval mechanism that can be elicited, rather than added as an external module.
Elad Hoffer, Yochai Blau, Edan Kinderman +3
NVIDIA · Department of Electrical Engineering, Technion, Haifa, Israel