Retrieval augmented generation RAG is widely deployed to improve factual accuracy in language models yet it remains unclear whether smaller models of size 7B parameters or less can effectively utilize retrieved information. To investigate this question we evaluate five model sizes from 360M to 8B across three architecture families SmolLM2 Qwen2.5 and Llama 3.1 under four retrieval conditions including no retrieval BM25 dense retrieval using E5 large v2 and oracle retrieval where the retrieved passage is guaranteed to contain the answer. We introduce a parametric knowledge split that separates questions a model can already answer from those that require external knowledge which allows us to isolate utilization failure from retrieval quality failure. We find three main results. First even with oracle retrieval models of size 7B or smaller fail to extract the correct answer 85 to 100 percent of the time on questions they cannot answer alone which indicates a fundamental utilization bottleneck. Second adding retrieval context destroys 42 to 100 percent of answers the model previously knew suggesting a distraction effect driven by the presence of context rather than its quality. Third an error analysis of 2588 oracle failures shows that the dominant failure mode is irrelevant generation where the model ignores the provided context entirely. These patterns hold across multiple prompt templates and retrieval methods. The results indicate that for models below 7B parameters the main limitation of RAG is context utilization rather than retrieval quality and that deploying RAG at this scale can lead to a net negative trade off under standard evaluation conditions.
Figures & tables
Model
Params
Quant.
Backend
SmolLM2-360M-Inst.
360M
NF4 4-bit
Local T4
Qwen2.5-1.5B-Inst.
1.5B
NF4 4-bit
Local T4
Qwen2.5-3B-Inst.
3.0B
NF4 4-bit
Local T4
Qwen2.5-7B-Inst.
7.0B
NF4 4-bit
Local T4
Llama-3.1-8B-Inst. †
8.0B
FP16
Groq API
Table 1: Models evaluated. † Llama-3.1-8B uses FP16 via API; absolute values not directly comparable, but the qualitative pattern is consistent (see Section 4.7 ).
Method
@1
@5
@10
@20
BM25
4.9
10.5
13.1
16.9
Dense
8.7
16.3
19.5
22.9
Hybrid (RRF)
6.2
14.9
19.0
21.9
Oracle
100.0
100.0
100.0
100.0
Table 2: Retrieval hit rates (%) on the 1,000-question evaluation set (NQ + HotpotQA).
Model
None
BM25
Dense
Hybrid
SmolLM2-360M
1.5
0.0
0.0
0.0
Qwen2.5-1.5B
14.5
5.5
3.5
6.0
Qwen2.5-3B
19.5
5.5
12.0
6.5
Qwen2.5-7B
16.5
–
14.0
–
Table 3: Pilot study EM (%) across retrieval methods ( n=200 ). No retrieval outperforms all retrieval methods for every model.
Split
Retrieval
360M
1.5B
3B
7B
Unknown
None
0.0
0.0
0.0
0.0
Noisy (dense)
0.0
4.6 [3.0–6.3]
6.2 [4.4–8.1]
8.0 [5.9–10.2]
Oracle
0.0
10.0 [7.8–12.2]
12.8 [10.5–15.5]
14.6 [11.9–17.6]
Known
None
100.0
100.0
100.0
100.0
Noisy (dense)
0.0
36.0 [26.7–46.5]
46.4 [36.8–55.2]
48.8 [41.0–56.6]
Oracle
0.0
43.0 [32.6–53.5]
54.4 [45.6–62.4]
58.4 [51.2–65.7]
Table 4: Exact Match (%) by parametric split, retrieval condition, and model size [95% CI]. Corpus-matched oracle only ( n=756 ). All non-zero differences from None are significant at p<0.001 (McNemar, Bonferroni-corrected), except SmolLM2-360M. The 360M Known cell is based on n=11 only and should be interpreted with extreme caution.
Figure 1: Retrieval utilization across model scale. Left: For Unknown questions, oracle retrieval achieves at most 14.6% EM at 7B, meaning 85%+ of retrieval effort is wasted. Right: For Known questions, all retrieval methods destroy 42–64% of previously correct answers. Error bars are 95% bootstrap CIs.
Figure 2: The distraction effect: percentage of Known answers destroyed by adding retrieval context. For models ≤ 3B, oracle and noisy retrieval cause statistically indistinguishable harm—context presence, not quality, drives distraction.
Model
pkn
Δ Unk
Δ Kn
Net
360M ∗
0.013
+0.0
− 100.0
− 1.3
1.5B
0.090
+4.6
− 64.0
− 1.6
3B
0.136
+6.2
− 53.6
− 1.9
7B
0.185
+8.0
− 51.2
− 2.9
Table 5: Net EM trade-off for noisy (dense) retrieval. The net effect is negative for every model tested. ∗ 360M: pkn based on n=11 ; interpret with caution.
Prompt
Overall EM
In ctx.
Not in ctx.
v1 (forced)
12.0
27.1
3.8
v2 (permissive)
12.5
28.6
3.8
v3 (minimal)
9.0
18.6
3.8
No retrieval
19.5
37.1
10.0
Table 6: Prompt ablation on Qwen2.5-3B × dense ( n=200 ). No prompt recovers the no-retrieval baseline (19.5% EM).
Retrieval
EM (%)
F1 (%)
None
24.0
34.5
BM25
16.5
24.2
Dense
17.5
25.3
Hybrid
16.5
25.2
Table 7: Llama-3.1-8B-Instruct (FP16, Groq API) on 200 NQ questions. Same qualitative pattern: none outperforms all retrieval methods. Not directly comparable to 4-bit local models in absolute terms.
Figure 3: Oracle failure taxonomy (% of failures per category, corpus-matched only, n=2,588 ). Irrelevant generation is dominant at all scales (61–100%). Refusal is unexpectedly high at 7B (24%). Inter-annotator agreement: 89% (96 manually reviewed samples).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Dataset
@1
@5
@10
@20
BM25
NQ
7.6
15.8
20.0
26.8
BM25
HotpotQA
7.2
15.6
19.4
23.8
BM25
PopQA
0.0
0.0
0.0
0.0
Dense
NQ
15.8
28.2
32.6
38.2
Dense
HotpotQA
10.2
20.8
26.0
30.6
Dense
PopQA
0.0
0.0
0.0
0.0
Appendix
Table 8: Retrieval hit rates (%) by dataset.
Component
Latency (ms/query)
BM25 retrieval
3.5
FAISS search (exact IP)
180.2
E5-large-v2 query encoding
∼ 50
Dense total
∼ 230
Hybrid (BM25+dense+RRF)
∼ 234
Appendix
Table 9: Latency measured on Kaggle T4 GPU with 500K-passage corpus. BM25 is ≈ 66 × faster than dense retrieval.
Model
None EM
Dense EM
Δ EM
SmolLM2-360M
1.5
0.0
− 1.5
Qwen2.5-1.5B
14.5
3.5
− 11.0
Qwen2.5-3B
19.5
12.0
− 7.5
Qwen2.5-7B
16.5
14.0
− 2.5
Appendix
Table 10: Pilot study full results ( n=200 , Δ EM = dense − none).
Figure 4: Oracle utilization gap: the pink region shows the fraction of retrieval effort that is wasted (answer in passage, model still fails). Even at 7B, 85% of oracle retrievals are unused.
Figure 5: Error category breakdown as stacked proportions. Irrelevant generation decreases as a share with scale but remains the dominant failure mode. Refusal is non-monotone across Qwen2.5 sizes, likely reflecting instruction-tuning recipe differences.
Retrieval-augmented generation (RAG) improves language model (LM) performance by providing relevant context at test time for knowledge-intensive situations. In this work, we systematically study the trade-off between pretraining and retrieval by training OLMo-2-based LMs ranging from 30M to 3B parameters on up to 100B DCLM tokens, while varying pretraining data scale, retrieval store size, and retrieval store source (pretraining vs. new data) across reasoning, scientific QA, and open-domain QA benchmarks. We find that retrieval gains depend on model capacity and pretraining exposure and are strongly front-loaded, with a median 91% of the largest observed improvement realized by one retrieval token per model parameter. However, the interaction is objective-dependent: smaller models gain more in gold-answer perplexity, whereas larger, more-pretrained models gain more in accuracy. Retrieval from previously seen data also preserves most of the held-out retrieval gain. Retrieval is therefore a task-, regime-, and metric-dependent complement to parametric learning whose value also depends on datastore size and information novelty. Overall, this motivates the explicit partitioning of data between internalization and external access for LM design.
Karan Singh, Michael Yu, Varun Gangal +4
Stanford University · †DegenAI Labs · Independent Researcher +3
Retrieval-Augmented Generation (RAG) grounds Large Language Models in external knowledge, yet current evaluations rely on discrete heuristics that suffer from ''epistemic blindness'' - failing to distinguish genuine contextual information extraction from parametric memory recall. To address this, we introduce the Normalized Context Utilization (NCU) metric, leveraging continuous token log-probabilities across zero-shot, oracle, and adversarial conditions to strictly quantify contextual information gain. Evaluating architectures ranging from 1.5B to 72B parameters alongside a proprietary commercial API reveals that for strict factual extraction (without Chain-of-Thought reasoning), traditional scaling laws exhibit extreme diminishing returns: highly efficient Small Language Models (SLMs) match or outperform high-capacity architectures. Furthermore, we demonstrate that ``Prior Dominance'' correlates with model scale and proprietary alignments. The evaluated commercial API not only overrode explicit external evidence in nearly half of adversarial conflicts, but also frequently suffered from systemic confidence collapse (Negative Transfer) when its parametric priors were contradicted. Our findings highlight the structural epistemic advantage and superior contextual adherence of SLMs in strict extraction workflows.
Barak Or
ArtificialGate Ltd. · Google & Reichman Tech School · Technion – Israel Institute of Technology +1
While large language models have been dominating the research landscape recently, small language models remain highly relevant across various domains; yet, they receive far less attention. In this study, we investigate how smaller language models perform during the generation stage within a Retrieval-Augmented Generation (RAG) system. To benchmark these models effectively, we utilised both open-source and proprietary datasets covering diverse subject areas and question types. Our findings demonstrate that a RAG system with small language models can be executed directly on-device without requiring any GPU hardware within a reasonable time. The experimental code and links to the supplementary materials can be accessed through the GitHub repository: https://github.com/SibNN/SLM-RAG-EVAL.