Retrieval-augmented generation (RAG) improves language model (LM) performance by providing relevant context at test time for knowledge-intensive situations. In this work, we systematically study the trade-off between pretraining and retrieval by training OLMo-2-based LMs ranging from 30M to 3B parameters on up to 100B DCLM tokens, while varying pretraining data scale, retrieval store size, and retrieval store source (pretraining vs. new data) across reasoning, scientific QA, and open-domain QA benchmarks. We find that retrieval gains depend on model capacity and pretraining exposure and are strongly front-loaded, with a median 91% of the largest observed improvement realized by one retrieval token per model parameter. However, the interaction is objective-dependent: smaller models gain more in gold-answer perplexity, whereas larger, more-pretrained models gain more in accuracy. Retrieval from previously seen data also preserves most of the held-out retrieval gain. Retrieval is therefore a task-, regime-, and metric-dependent complement to parametric learning whose value also depends on datastore size and information novelty. Overall, this motivates the explicit partitioning of data between internalization and external access for LM design.
Figures & tables
Figure 1: Overview of the controlled pretraining–retrieval scaling study. Left: We train OLMo-2 models from 30M to 3B parameters while varying DCLM pretraining exposure, datastore size, and whether indexed data is held out from or reused during pretraining. Center: We fit a scaling law over model size, pretraining tokens, and retrieval tokens; fixed-token-budget slices support performance-side comparisons among allocations. Right: The interaction with model scale depends on the evaluation target. Gold-answer perplexity gains are front-loaded and larger for smaller models, whereas decision-accuracy gains favor larger models on the multiple-choice benchmarks studied.
Figure 2: Parametric scaling baselines without RAG ( R=0 ). Left: Empirical measurements across model sizes and data budgets, overlaid with iso-loss contours from the power-law model. Each point corresponds to a trained model configuration, colored by observed perplexity. The blue line denotes the compute-efficient frontier and the vertical dashed lines, discrete training budgets. Right: Iso-compute slices of the scaling surface, showing predicted loss as a function of model size ( N ). Empirical observations are overlaid for reference.
Benchmark
CV ARE (%)
LOMO ARE (%)
α
β
L0
Reasoning
CommonsenseQA
6.87
10.76
0.1558
0.0869
2.1619
HellaSwag
2.18
4.94
0.3923
0.4713
1.3714
StrategyQA
12.28
14.93
2.0000
0.4647
7.6735
Scientific/Open QA
SciQ
13.07
24.03
0.5267
0.2606
0.9522
OpenBookQA
4.78
9.41
0.3688
0.2120
1.6579
AI2C-ARC Easy
7.10
13.67
0.3566
0.2195
1.0299
Table 1: Power-law fit quality and scaling exponents for parametric baselines ( R=0 ). We report cross-validation average relative error (CV ARE) [interpolation error under random splits] and leave-one-model-size-out ARE (LOMO ARE) [extrapolation error to unseen model size]. Lower ARE indicate better fit quality. Exponents α and β , when combined with L0 , summarize how loss scales with model size and pretraining data in the baseline regime. ARE should be interpreted relative to the inherent noise and discreteness of benchmarks, with smoother, likelihood-based tasks yielding low errors, and reasoning-heavy tasks showing higher variance and ARE.
Benchmark
CV ARE
LOMO
α
β
C×103
δ
ζ
L0
(%)
(%)
Reason.
CommonsenseQA
5.10
8.77
0.169
0.087
27.33
−0.433 [-0.51, -0.33]
−0.247 [-0.30, -0.19]
2.162
HellaSwag
1.76
3.31
0.476
0.505
0.13
1.295 [1.17, 1.44]
0.277 [0.26, 0.29]
1.484
StrategyQA
14.33
18.73
1.178
0.573
72.10
−0.109 [-0.13, -0.09]
−0.236 [-0.25, -0.22]
7.599
Scientific/Open QA
SciQ
7.91
14.36
0.545
0.302
11.50
0.401 [0.20, 0.67]
0.072 [-0.05, 0.22]
0.952
OpenBookQA
3.14
7.13
0.376
0.239
4.60
0.731 [0.55, 0.97]
0.164 [0.04, 0.29]
1.952
Table 2: Per-benchmark fits of Eq. 2 . CV ARE is the average relative error on held-out whole (N,D) retrieval curves, evaluated on retrieval points ( R>0 ). LOMO is leave-one-model-size-out error. δ and ζ carry 95% bootstrap intervals obtained by resampling benchmark-evaluation noise; δ>0 indicates larger absolute retrieval gains for smaller models and ζ>0 larger gains in less-saturated pretraining regimes. Setting δ=ζ=0 recovers an additive retrieval law in which the gain is independent of N and D . The retrieval-rate parameter η reaches its upper bound of 106 on every benchmark, and profile fits continue improving or remain flat as this bound is increased. Thus, our datastore grid does not identify a finite retrieval-onset scale: the observed positive- R response is already approximately linear in logR .
Figure 3: Decomposing retrieval gains into reused and novel information. Left: Mean perplexity reduction relative to no retrieval, decomposed into the benefit from re-accessing pretraining data and the additional novelty premium from using held-out data ( Held-out - Reused ). Labels report each component’s share of the total held-out retrieval gain. Right: The novelty premium as a function of pretraining tokens per parameter ( D/N ). Points show benchmark-level values, lines show mean for each model size, and shaded regions show the bootstrapped 95% confidence interval.
Figure 4: Trade-off between pretraining and retrieval under a fixed data budget. Left: We quantify the substitutability between retrieval and pretraining via the number of pretraining tokens saved per retrieval token, computed by fitting scaling laws and determining, for each pretraining scale, the amount of retrieval required to match baseline performance without retrieval. The dotted line represents a linear line-of-best-fit across all model scales. Right: We measure the marginal benefit of retrieval as perplexity improvement per billion retrieval tokens (higher is better) for models trained near their optimal pretraining ratio.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Configuration
Embedding model
Qwen3-Embedding-8B
Embedding dimension
4096
Pooling
Last non-padded token
Normalization
L2-normalized
Tokenizer
cl100k_base
Chunk length
900 tokens
Appendix
Table 3: Embedding and chunking configuration for retrieval corpus construction.
Component
Configuration
Index type
IVFPQ (inner product)
Sub-quantizers
128
Bits per code
8
Search parameter
nprobe = 64
Training set size
max(nlist×50,105)
Batch size (add)
500K vectors
Appendix
Table 4: FAISS index configuration.
Corpus Split
nlist
Shards
5%
32,768
32
10%
32,768
64
20%
65,536
128
30%
65,536
128
Appendix
Table 5: Per-split FAISS index configuration.
Model Size
Layers
Hidden Dim
Number of Heads
Query Groups
Intermediate Size
Block Size
30M
8
256
4
4
512
4096
136M
8
512
8
8
2048
4096
233M
16
640
10
10
2560
4096
728M
18
1280
10
10
5120
4096
1B
24
1408
11
11
5632
4096
3B
26
2560
20
20
10240
4096
Appendix
Table 6: Pretraining hyperparameters across model sizes
Generator
Dmax
Rmax
Ftrain
Fembed
Embed / Train
(B tokens)
(B tokens)
(FLOPs)
(FLOPs)
30M
4.50
0.60
8.10×1017
9.60×1018
11.85
136M
20.40
2.72
1.66×1019
4.35×1019
2.61
233M
34.95
4.66
4.89×1019
7.46×1019
1.53
728M
72.80
14.56
3.18×1020
2.33×1020
0.73
1B
100.00
20.00
6.00×1020
3.20×1020
0.53
Appendix
Table 7: Approximate generator-pretraining and one-time datastore-embedding costs at the largest evaluated scales. Estimates use 6ND for training and 2NeR for embedding with Ne=8 B.
Benchmark
CV ARE
LOMO
α
β
C×103
δ
ζ
L0
(%)
(%)
Reas.
CommonsenseQA
4.10
10.53
0.157
0.098
1.06
−1.462 [ − 1.65, − 1.30]
−0.699 [ − 0.80, − 0.60]
0.255
HellaSwag
4.16
7.81
0.100
0.069
0.40
−2.000 ‡ [ − 2.00, − 2.00]
−0.924 [ − 0.97, − 0.88]
0.262
Scientific QA
SciQ
17.30
51.96
0.618
0.359
0.34
−1.216 [ − 1.51, − 0.99]
−0.751 [ − 0.97, − 0.62]
0.046
OpenBookQA
2.23
4.95
0.068
0.047
1.69
−1.112 [ − 1.49, − 0.85]
−0.595 [ − 0.80, − 0.42]
0.364
AI2-ARC Easy
7.27
20.32
0.259
0.171
2.32
−0.894 [ − 0.98, − 0.82]
−0.474 [ − 0.52, − 0.43]
0.148
Appendix
Table 8: Eq. 2 refit to downstream error rate ( L=1−accuracy ) instead of gold-answer perplexity, on the six multiple-choice benchmarks for which accuracy is defined and produces stable fits.
Figure 5: Interaction exponents across evaluation targets. Fitted capacity exponent δ (left) and pretraining-exposure exponent ζ (right) under gold-answer perplexity, choice-normalized Brier loss, and decision error. Positive values indicate larger retrieval gains for smaller models ( δ ) or models with fewer pretraining tokens per parameter ( ζ ). Bars show 95% intervals obtained by resampling complete (N,D) retrieval curves. Estimates at ±2 lie on an optimization bound.
Benchmark
30M
136M
233M
728M
1B
3B
All folds
Excl. 30M
CommonsenseQA
19.6
4.4
3.7
4.1
5.2
15.5
8.8
6.6
HellaSwag
10.2
1.5
2.6
1.7
1.7
2.7
3.4
2.1
StrategyQA
93.5
16.3
14.4
17.3
7.2
7.6
26.1
12.6
SciQ
28.8
10.3
8.9
7.1
7.4
2.7
10.9
7.3
OpenBookQA
18.4
2.7
3.9
3.8
4.1
5.6
6.4
4.0
AI2-ARC Easy
18.4
8.1
4.7
4.1
6.7
3.1
7.5
5.4
Appendix
Table 9: LOMO average relative error (%) on held-out retrieval points ( R>0 ), by which model size was withheld.
Model
Retrieval gain G
#par.
CV ARE
Δ
wins
LOMO
Δ
(%)
(%)
(%)
(%)
M∅
0 (no retrieval term)
0
9.24
+32.5
0/9
14.91
+18.1
M0
Clog(1+ηr)
2
7.04
—
—
12.85
—
MN
Cn−δlog(1+ηr)
3
6.67
−7.2
8/9
11.36
−12.7
MS
Cs−ζlog(1+ηr)
3
6.98
−1.4
7/9
12.82
−0.5
MNS
Cn−δs−ζlog(1+ηr)
4
6.29
−12.7
9/9
10.96
−16.3
Appendix
Table 10: Comparison of retrieval-gain specifications. Results are averaged across the benchmarks. All specifications share the same five-parameter baseline P(N,D) . Δ is the relative change from the additive law M0 and wins counts benchmarks with lower grouped-CV error than M0 .
Benchmark
Mean σ
\sigma@1×
\sigma@10×
\sigma@100×
Mean κ
\kappa@1×
\kappa@10×
\kappa@100×
AI2C-ARC Challenge
4.73
0.06
9.14
11.20
0.4778
0.1582
2.4505
0.4216
AI2C-ARC Easy
22.54
0.38
27.83
64.40
0.3701
0.1048
0.4348
0.5736
CommonsenseQA
677.51
677.93
1216.95
1951.00
0.5738
0.2715
1.4521
1.0438
NQ-Open
5.28
0.23
2.72
13.63
0.4421
0.4118
0.5427
0.3192
OpenBookQA
6.65
0.32
6.05
-10.49
2.2450
0.2014
3.8579
0.4100
SciQ
24.21
0.20
11.69
78.62
0.9830
0.3722
0.2576
0.4918
Appendix
Table 11: Replacement cost σ and marginal benefit κ across benchmarks and training regimes.
Figure 6: Stability of scaling-law fits across random seeds. We report mean ± standard deviation of cross-validation ARE (CV ARE) and leave-one-model-out ARE (LOMO ARE) across 27 separate fits (every possible combination of 3 model families × 3 seeds each).
Figure 7: Effect of retrieval query formulation on performance. Comparison of standard generation on SimpleQA without retrieval (Baseline) to RAG under two query formulations: (i) RAG (Query) , which retrieves top- k passages using only the question, and (ii) RAG (Query + Gold) , which includes the gold answer in the query too (an oracle-style ablation). SimpleQA is not multiple-choice (no answer choices), so we do not report RAG (Query + Choices) here. All methods use a shared corpus index constructed from 20% of the data, retrieving the top-5 passages per query. Left: OLMo-2 136M. Right: OLMo-2 1B.
Figure 8: Effect of retrieval query formulation on LAMBADA. Similar to Figure 7 above.
Figure 9: Effect of retrieval query formulation across GSM8K and CommonsenseQA. Similar to Figure 7 . For CommonsenseQA, we additionally compare querying both choices and gold answer (RAG Query + Choices + Answer). Both panels show OLMo-2 1B as a function of pretraining tokens per parameter.
Figure 10: Calibration plots for 3D scaling law fits across benchmarks. We show the alignment between predicted and observed gold-answer perplexity across six benchmarks for the interaction model MNS (Eq. 2 ), which augments the Hoffmann power-law formulation with a capacity- and saturation-modulated logarithmic retrieval term. The tight grouping around the diagonal indicates that the model effectively captures retrieval-augmented scaling behavior. Reported MSE and ARE values are computed in-sample over the plotted points; held-out cross-validation errors are reported separately in Table 2 .
Retrieval augmented generation RAG is widely deployed to improve factual accuracy in language models yet it remains unclear whether smaller models of size 7B parameters or less can effectively utilize retrieved information. To investigate this question we evaluate five model sizes from 360M to 8B across three architecture families SmolLM2 Qwen2.5 and Llama 3.1 under four retrieval conditions including no retrieval BM25 dense retrieval using E5 large v2 and oracle retrieval where the retrieved passage is guaranteed to contain the answer. We introduce a parametric knowledge split that separates questions a model can already answer from those that require external knowledge which allows us to isolate utilization failure from retrieval quality failure. We find three main results. First even with oracle retrieval models of size 7B or smaller fail to extract the correct answer 85 to 100 percent of the time on questions they cannot answer alone which indicates a fundamental utilization bottleneck. Second adding retrieval context destroys 42 to 100 percent of answers the model previously knew suggesting a distraction effect driven by the presence of context rather than its quality. Third an error analysis of 2588 oracle failures shows that the dominant failure mode is irrelevant generation where the model ignores the provided context entirely. These patterns hold across multiple prompt templates and retrieval methods. The results indicate that for models below 7B parameters the main limitation of RAG is context utilization rather than retrieval quality and that deploying RAG at this scale can lead to a net negative trade off under standard evaluation conditions.
Sanchit Pandey
BITS Pilani, Hyderabad, India · BITS Pilani, Hyderabad Campus, Telangana, India
Large reasoning models such as DeepSeek-R1 and OpenAI o1 generate extended chains of thought spanning thousands of tokens, yet their integration with retrieval-augmented generation (RAG) remains fundamentally misaligned. Current RAG systems optimize for providing context before reasoning begins, while reasoning models require evidence injection during multi-step inference chains. We introduce ReaLM-Retrieve, a reasoning-aware retrieval framework that addresses this mismatch through three key innovations: (1) a step-level uncertainty detector that identifies knowledge gaps at reasoning-step granularity rather than token or sentence level; (2) a retrieval intervention policy that learns when external evidence maximally benefits ongoing reasoning; and (3) an efficiency-optimized integration mechanism that reduces per-retrieval overhead by 3.2x compared to naive integration. Experiments on MuSiQue, HotpotQA, and 2WikiMultiHopQA demonstrate that ReaLM-Retrieve achieves on average 10.1% absolute improvement in answer F1 over standard RAG (range: 9.0-11.8% across the three benchmarks) while reducing retrieval calls by 47% compared to fixed-interval approaches like IRCoT (all improvements significant at p<0.01, paired bootstrap). On the challenging MuSiQue benchmark requiring 2-4 hop reasoning, our method achieves 71.2% F1 with an average of only 1.8 retrieval calls per question. Analysis shows that ReaLM-Retrieve also improves retrieval quality itself, achieving 81.3% Recall@5 with consistently higher precision and MRR than fixed-interval baselines on supporting evidence, establishing new state-of-the-art efficiency-accuracy trade-offs for reasoning-intensive retrieval tasks.
Dongxin Guo, Jikun Wu, Siu Ming Yiu
The University of Hong Kong Hong Kong, China · Brain Investing Limited Hong Kong, China · Stellaris AI Limited Hong Kong, China
Retrieval-augmented generation (RAG) typically treats retrieval and generation as separate systems. We ask whether an attention-based encoder-decoder can instead retrieve directly from its own internal representations. We introduce INTRA (INTrinsic Retrieval via Attention), a framework where decoder attention queries score pre-encoded evidence chunks that are then directly reused as context for generation. By construction, INTRA unifies retrieval and generation, eliminating the retriever-generator mismatch typical of RAG pipelines. This design also amortizes context encoding by reusing precomputed encoder states across queries. On question-answering benchmarks, INTRA outperforms strong engineered retrieval pipelines on both evidence recall and end-to-end answer quality. Our results demonstrate that attention-based models already possess a retrieval mechanism that can be elicited, rather than added as an external module.
Elad Hoffer, Yochai Blau, Edan Kinderman +3
NVIDIA · Department of Electrical Engineering, Technion, Haifa, Israel