Retrieval Augmented Generation Compositionality

Ido Levy; Segev Shlomov; Leshem Choshen

cs.CLOct 21, 2026

Retrieval Augmented Generation Compositionality

Authors: Ido Levy, Segev Shlomov, Leshem Choshen

Organizations: IBM · Weizmann Institute of Science

Abstract

Retrieval Augmented Generation (RAG) grounds large language models in external knowledge. Modern pipelines combine embedding, chunking, retrieval, reranking, query augmentation, context compression, and agentic verification, yet these techniques are typically evaluated one at a time. It remains unclear whether their isolated gains survive composition, whether retrieval quality and answer faithfulness favor the same design choices, and whether conclusions from small corpora hold at large scale. We pre-declare three hypotheses and introduce LASAGNA, an open-source framework for controlled RAG evaluation. We test 450 configurations across eight benchmarks and corpora ranging from 5K to 200K documents. Three findings emerge. First, individually strong techniques from different phases of a RAG pipeline do not necessarily compose: three retrieval techniques predict a +47% combined improvement but deliver only +17% when used together. Second, component gains stack differently for retrieval and faithfulness, and optimizing one can hurt the other. Third, the best composition changes as the corpus grows: the combination that wins on a small corpus may not remain best at large scale. These findings show that gains measured in isolation are not reliable predictors of realistic performance, and that new RAG techniques should be tested with the full surrounding stack. We call on researchers to do so and supply LASAGNA to simplify these experiments.

Figures & tables

Panel of Figure 1 — Figure 1: lasagna overview. (a) Seven functional blocks. (b) Eight dimensions in a 167-configuration D-optimal design. (c) Faithfulness versus latency for four operational profiles; faded points show other configurations.

Panel of Figure 2 — Figure 2: lasagna research workflow: register a component, declare a factorial sweep in YAML, and run end-to-end for a paired statistical test.

Figure 3: Main effects and pairwise interactions across eight RAG pipeline factors. The upper triangle shows retrieval (

167

configurations, NDCG@10), the lower triangle faithfulness (

122

configurations, RAGAS). Darker cells mean stronger effects; an interaction cell shows how much one factor’s effect depends on another factor’s setting. Color encodes marginal (unadjusted)

\eta^{2}

, the share of variance a factor or interaction explains, so the two metrics can be compared descriptively. Embedding and enrichment matter most for retrieval, whereas query augmentation and agentic gating matter most for faithfulness. Because the D-optimal design is unbalanced, values can overlap and should not be summed into a total explained variance.

Panel of Figure 4 — Figure 4: Scaling behavior vs. corpus size (log scale), four SOTA profiles, 3 seeds per size. Dashed: log-linear fit over 5K–200K; dotted beyond 200K: projections, not measurements . In (a) , balanced overtakes standard between 15K–25K, significant at 200K.

Profile	NDCG@10	Faithfulness	Lat. mean (s)	HV ( $30$ s ref)
standard	0.860 $\pm$ 0.003	0.739 $\pm$ 0.016	5.30 $\pm$ 0.05	0.523
balanced	0.840 $\pm$ 0.003	0.911 $\pm$ 0.014	12.25 $\pm$ 3.55	0.453
max-reliable	0.804 $\pm$ 0.004	0.916 $\pm$ 0.022	14.16 $\pm$ 6.44	0.389
speed	0.541 $\pm$ 0.000	0.524 $\pm$ 0.031	0.70 $\pm$ 0.02	0.277

Table 2: End-to-end profile comparison on a SQuAD v2 / MuSiQue / HotpotQA mixture over three seeds (GPT-5.1 judge; faithfulness uses each profile’s deployment-native answer prompt; per-profile regime and joint-HV sensitivity to that choice in Tables 26 and C.2 ). Mean

\pm

std; bold : column best; green : highest joint retrieval–faithfulness–latency volume.

\mathrm{HV}(L_{\mathrm{ref}})=\mathrm{NDCG@10}\times\mathrm{Faith}\times\max(0,1-\mathrm{latency}/L_{\mathrm{ref}})

( Zitzler et al., 2003 ) , with

L_{\mathrm{ref}}=30

s; reference-latency sensitivity is in Table 39 .

Appendix figures & tables66 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Configs	Queries	Seeds	Source
H1 retrieval ANOVA	167	93	1–3	Table 32
H1 generation ANOVA	122	56	1–3	Section D.2
H2 Pareto (SQuAD/MuSiQue/HotpotQA)	4	43 gen / 56 retrieval	3	Table 2
E1 focused $5{\times}4$	20	43	3	Section D.9
H3 scaling sweep	4	93	1–3	Table 23
Non-factorial ablations	138	93	1–3	Appendix F

Appendix

Table 3: Notation and counts by experiment. “Configs” = distinct pipeline configurations executed; “Queries” = distinct evaluation queries. Query-config-seed triples aggregate across factorial and ablation runs; individual query evaluations include repeated RAGAS evaluation tasks per sample.

Date of pre-registration:	Design matrix, hypotheses, statistical tests, and success criteria were fixed prior to the confirmatory experiments; the design matrix is content-addressed by the SHA-256 below. No third-party (OSF/AsPredicted) timestamp was filed for this study; all analyses are reported alongside their pre-specified test statistics, and H1/H2/H3 are labeled explicitly throughout as “planned confirmatory analyses” rather than as timestamp-verified pre-registrations.
Design matrix SHA-256:	31a0dc197b46...df1728d6
Software:	Python 3.11, NumPy 2.3.5, SciPy 1.16.3, seed $=$ 42

Appendix

Table 4: Pre-registration metadata: date and mode of hypothesis commitment, design-matrix content hash, and software environment.

	Research question	What confirming rejects	Pre-registered test
H1	Do components combine additively?	Single-component ablation as evidence	$\eta^{2}\geq 0.20$ , Type-III ANOVA F-test
H2	Does a balanced configuration sweep a wider tradeoff envelope?	“You must pick one axis to optimize”	Hypervolume, Wilcoxon signed-rank
H3	Do RAG profile rankings flip with corpus scale?	Small-corpus ablations predict production deployment	Per-profile power-law slopes, bootstrap CIs, Kendall’s $\tau$

Appendix

Table 5: Summary of the three pre-registered hypotheses, their research questions, what confirming each hypothesis would rule out, and the pre-registered test statistic.

Role	Primary	Alternatives	Selection rationale
Generator	GPT-5.1 (OpenAI)	Claude Sonnet 4.5 (Anthropic), Gemini 2.0 (Google)	Held constant; cost vs. capability
Judge	GPT-5.1 (OpenAI)	Claude Sonnet 4.5 (Anthropic), Llama-3-70B (Meta)	Determinism at $T{=}0$ ; cost for ${\sim}$ 1.7M calls
Embedding	MiniLM-L6 (384d), BGE-M3 (1024d), E5-large (1024d), BGE-large (1024d)		Span lightweight to SOTA multilingual
Reranker	ms-marco-MiniLM-L6-v2	BGE-reranker-v2	Standard BEIR cross-encoder baseline

Appendix

Table 6: Model candidates. The generator is held constant across all configurations; the judge evaluates all RAGAS metrics. Embedding models are factorial dimension D1.

Dim	Stage	$k$	Levels
D1	Embedding	4	MiniLM-L6 ∗ , BGE-M3, E5-large, BGE-large
D2	Chunking	4	fixed ∗ , semantic, sentence-window, hierarchical
D3	Enrichment	4	none ∗ , contextual-retrieval, RAPTOR, GraphRAG
D4	Retrieval	3	dense ∗ , hybrid-BM25, hybrid-SPLADE
D5	Query aug.	5	none ∗ , HyDE, RAG-Fusion, decomposition, step-back
D6	Reranking	4	none ∗ , single-CE, ensemble, MMR

Appendix

Table 7: The eight factorial dimensions. Baseline levels (marked ∗ ) are the simplest, lowest-cost options.

ID	Dataset	Corpus	Eval queries	Relevance
DS1	MS MARCO v1	8.8M passages	500 sampled	BM25 sparse (TREC)
DS2	HotpotQA	5.2M paragraphs	500 sampled	Supporting facts
DS3	BEIR/FiQA	57K posts	500 sampled	Crowdsourced
DS4	BEIR/SciFact	5K abstracts	300 (all)	Expert binary
DS5	CUAD (legal)	510 contracts	500 sampled	Clause-level

Appendix

Table 8: Planned benchmark datasets. Total evaluation queries per configuration: 2,300.

Experiment	Configs	Queries/config	Est. cost (USD)
Factorial sweep	242	2,300	$2,478
Scaling experiment	15	500	$30
Pareto experiment	20	500	$45
Total			$\mathbf{\sim\$ 2{,}553}$

Appendix

Table 9: Estimated compute budget.

Dim	Stage	# Levels	Levels	Role
D1	Embedding	4	MiniLM-L6 ∗ , BGE-M3, E5-large, BGE-large	Repr. capacity
D2	Chunking	4	fixed ∗ , semantic, sent-window, hierarchical	Segmentation
D3	Enrichment	4	none ∗ , contextual-retr., RAPTOR, GraphRAG	Index-time augm.
D4	Retrieval	3	dense ∗ , hybrid-BM25, hybrid-SPLADE	Sparse–dense mix
D5	Query aug.	5	none ∗ , HyDE, RAG-Fusion, decomp., step-back	Pre-retrieval
D6	Reranking	4	none ∗ , single-CE, ensemble, MMR	Post-retrieval

Appendix

Table 10: The eight factorial dimensions. # Levels is the number of configuration options per dimension. Baseline levels (marked ∗ ) are the simplest, lowest-cost options used when other features are toggled. Descriptions, citations, and implementations appear in Appendix K .

Configuration	NDCG@10	Latency (ms)
no_graph_baseline	0.4780	486
graph_1hop	0.4774	1,021
graph_2hop	0.4664	1,138
graph_3hop	0.4661	1,210

Appendix

Table 11: Isolated GraphRAG ablation on the retrieval track. Graph depth increases latency without improving NDCG@10 over the non-graph baseline.

Pre-registered	Executed	Status	Implication
242-config D-optimal fractional factorial	167 configs executed (69% of design); additionally 138 non-factorial ablation configurations across 24 experiments ( Appendix F )	Partially executed (factorial); fully executed (ablations)	Full ANOVA estimable for main effects and 28 two-way interactions; 3 of 4 enrichment levels covered. Ablations cover 20 rerankers, 15 embeddings, and all factorial axes at finer granularity.
8-dimension full ANOVA ( $\eta^{2}$ for all 28 two-way interactions)	Full retrieval ANOVA (167 configs) + faithfulness ANOVA (122 configs)	Executed	H1 testable on both metrics; joint 28-two-way ANOVA does not reach $\eta^{2}\geq 0.20$ at per-sample granularity ( Section 5 , Section D.3 )
5 corpus sizes (10K, 25K, 50K, 100K, 1M)	7 corpus levels (5K, 10K, 15K, 25K, 50K, 100K, 200K)	Executed	H3 fully testable across 5K–200K; per-profile power-law fits and rank crossover observed ( Section 5 )
H1: $\eta^{2}\geq 20\%$ from 2-way interactions (faithfulness); pre-registered as joint 28-pair $F$ -test and per-pair Bonferroni ( $\alpha{=}0.05/28$ ), with the $0.20$ bar justified per-term (Cohen large-effect)	Both pre-registered procedures executed: joint 28-two-way ANOVA (two-track, per-sample + aggregated, nested-SSR joint SS) and per-pair Bonferroni across all 28 pairs ( Section D.3 )	Not confirmed at pre-registered $20\%$ joint threshold; pre-registered per-pair Bonferroni identifies D5 $\times$ D8 as large-effect follow-up. On the pre-registration’s own variance-floor decision rule ( $\eta^{2}<5\%$ $\rightarrow$ reject H1 / additivity holds), the executed E1 $\eta^{2}=0.033$ and per-sample joint $\eta^{2}=0.040$ sit on the additivity-holds side; a parallel significance test rejects strict-zero interactions ( $p{<}10^{-9}$ aggregated). We report both readings (see Appendix A )	Joint per-config ( $n{=}113$ ) unidentifiable (232 params); per-sample $\eta^{2}{=}0.040$ retracted as pseudo-replication; aggregated $\eta^{2}{=}0.428$ sensitivity only (BIC $\Delta{=}+347$ favors mains-only). Pre-registered per-pair follow-up: D5 $\times$ D8 $\eta^{2}{=}0.201$ , $F(12,\,306){=}6.40$ , $p{=}3.8\!\times\!10^{-10}$ ( Section D.3 ).
H2: balanced has higher faithfulness–latency–cost hypervolume than max-reliable on at least three of five benchmarks	Four-profile retrieval–faithfulness–response-time comparison on one mixed-benchmark slice, with three seeded runs and reference-time sensitivity	Not confirmed as pre-registered ; informative executed comparison	The five-benchmark test was not executed. On the available SQuAD v2 / MuSiQue / HotpotQA mixture, no profile leads every metric. At the fixed $30$ s reference, standard has the highest joint HV ( $0.523$ ); within the agentic pair, balanced has $4.5\%$ higher NDCG@10 and $13.5\%$ lower mean response time than max-reliable , with a $0.005$ faithfulness difference ( Tables 2 and 39 ).
H3: profile rankings flip with corpus scale	Per-profile power-law fits across 5K–200K (3 seeded runs at 25K–200K)	Supported	Cross-profile slope range $b_{\max}-b_{\min}\approx 0.10$ per decade ( $\geq 0.05$ threshold cleared); rank crossover visible from 25K, statistically significant at 200K (Welch’s $t$ , $p{=}0.004$ ); intermediate 50K/100K $p>0.17$

Appendix

Table 12: Pre-registration reconciliation. “Confirmatory” items retain their original inferential status; “Exploratory” items are presented without confirmatory

p

-values.

Phase	Wall-clock	API cost	GPU hours	Notes
Initial ablations & profiles	72h	$1,580	140h	6 profiles $\times$ 3 runs
Scaling experiments (5K–50K)	36h	$340	24h	4 profiles $\times$ 5 sizes
Retrieval factorial (167 cfg)	52h	$1,150	48h	48 combos, MPS sequential
Generation factorial (122 cfg)	120h	$1,950	35h	21 combos, GPT-5.1 judge
Infrastructure overhead	30h	$180	—	Retries, failed runs, re-ingestion
Total	310h	$5,200	247h	${\sim}$ 13 days wall-clock

Appendix

Table 14: Full experimental timeline with wall-clock hours and costs.

Figure 5: Benchmark composition of the unified multi-domain corpus. Left: the retrieval track aggregates five BEIR benchmarks (NFCorpus, FiQA, SciFact, ArguAna, HotpotQA) into a single 5,132-document pool so every query is retrieved under cross-domain distractor pressure. Right: the E2E generation track embeds QA pairs from three reading-comprehension and multi-hop benchmarks (SQuAD v2, HotpotQA, MuSiQue) in the same pool, augmented with gold-passage documents to guarantee answerability. Scaling experiments ( Section 5 ) inject up to 45,000 additional distractors from the same five domains in controlled increments.

Benchmark	Domain	Eval Queries	Test Queries	Source
SQuAD v2	Reading comprehension	23	10	Rajpurkar et al. (2018)
MuSiQue	Multi-hop reasoning	20	9	Trivedi et al. (2022)
HotpotQA	Multi-hop QA	13	11	Yang et al. (2018)
Total	—	56	30	—

Appendix

Table 16: QA benchmarks used for generation and hallucination evaluation (exploratory).

Dataset	Domain	Eval Queries	Corpus Docs	Relevance
Retrieval Track (5 BEIR benchmarks):
NFCorpus	Biomedical	20	2,730	Expert graded
FiQA	Financial QA	20	719	Crowdsourced
SciFact	Scientific claims	20	583	Expert binary
ArguAna	Argument retrieval	11	580	Counterargument
HotpotQA	Multi-hop reasoning	22	520	Supporting facts

Appendix

Table 17: Benchmark datasets aggregated into our evaluation corpus. The retrieval track draws from five BEIR benchmarks; the end-to-end (E2E) generation track additionally sources QA pairs from three reading-comprehension and multi-hop benchmarks. The pre-registered design included MS MARCO and CUAD; these were not executed because of infrastructure constraints ( Section A.6 ).

Dataset	Description	Samples	Hops	Answer	Used
SQuAD v2	Reading comprehension over Wikipedia paragraphs. Includes unanswerable questions (adversarial examples with plausible but incorrect passages), testing the system’s ability to abstain when evidence is insufficient.	130K	1	Extractive span	✓
MuSiQue	Multi-hop QA requiring 2–4 compositional reasoning steps across multiple Wikipedia articles. Designed to be unsolvable by single-hop shortcuts, with carefully controlled decomposition annotations.	25K	2–4	Extractive span	✓
HotpotQA (QA)	Multi-hop QA with supporting-fact annotations. Requires combining evidence from two Wikipedia paragraphs. Bridge and comparison question types test different reasoning patterns.	113K	2	Extractive span	✓
TriviaQA	Trivia questions with answers sourced from Wikipedia and the web. Tests general knowledge retrieval across diverse topics. Used for the factorial probe experiments ( Section 5 ).	95K	1	Short answer	Probes
Natural Questions	Real Google search queries with long and short answers from Wikipedia. Tests retrieval quality on naturalistic information needs.	307K	1	Short answer	—
BamBoogle	125 multi-hop questions specifically designed to be unanswerable by direct Google search, requiring compositional reasoning. Small but challenging; tests multi-hop capability under adversarial conditions.	125	2–3	Short answer	—

Appendix

Table 19: QA benchmarks supported by lasagna . Benchmarks marked “Used” contribute evaluation queries to this study; others are available through the adapter registry.

	NFCorpus	FiQA	SciFact	ArguAna	HotpotQA	Total
Increment 1	162	3,690	886	1,455	3,807	10,000
Increment 2	167	3,701	854	1,549	3,729	10,000
Increment 3	186	3,684	874	1,548	3,708	10,000
Increment 4	262	5,670	1,284	2,335	5,449	15,000
Total	777	16,745	3,898	6,887	16,693	45,000

Appendix

Table 20: Distractor increment composition. Each increment draws from all five BEIR domains in proportion to the available held-out corpus size.

Label	Count	%
RELEVANT (suspected qrels false negative)	0	0.0
PARTIALLY RELEVANT (adjacent topic)	9	9.0
NOT RELEVANT (correctly a distractor)	91	91.0

Appendix

Table 21: Distractor-relevance audit (

n{=}100

). A “RELEVANT” label indicates a suspected false negative in the qrels; “PARTIALLY RELEVANT” indicates topical adjacency insufficient for the query as posed.

Figure 6: E2E scaling curves. (a) Retrieval quality (NDCG@10) within the generation pipeline. (b) Generation faithfulness (RAGAS). balanced overtakes standard on both axes at 50K. Dashed annotations show total

\Delta

from 5K to 50K.

Profile	Slope/dec	$R^{2}$	$p$	95% CI (slope)	Proj. 500K	Proj. 1M
speed	$-$ 0.115	0.900	0.004	$[-0.168,-0.062]$	0.208 $\pm$ 0.096	0.174 $\pm$ 0.107
standard	$-$ 0.055	0.895	0.004	$[-0.082,-0.029]$	0.412 $\pm$ 0.047	0.395 $\pm$ 0.053
balanced	$-$ 0.015	0.403	0.175	$[-0.039,+0.010]$	0.476 $\pm$ 0.044	0.472 $\pm$ 0.049
max-reliable	$-$ 0.019	0.296	0.265	$[-0.061,+0.022]$	0.431 $\pm$ 0.075	0.425 $\pm$ 0.084

Appendix

Table 23: Log-linear degradation fits (6 data points, 5K–200K).

R^{2}

measures goodness of fit;

p

tests whether the slope differs from zero. Projections use 95% prediction intervals.

Domain	Query
NFCorpus (biomedical)	“pumpkin”
	“Does taking aspirin reduce the risk of colorectal cancer?”
FiQA (financial)	“Does it make any sense to directly contribute to reducing the US national debt?”
	“Opening a Roth IRA account, what is the fee structure for Vanguard vs. Scottrade?”
SciFact (scientific)	“FoxO3a activation in neuronal death is mediated by reactive oxygen species”
	“Basophils counteract disease development in patients with systemic lupus”

Appendix

Table 24: Representative retrieval queries by domain. Queries range from single-term biomedical lookups (NFCorpus) to multi-hop reasoning chains (HotpotQA).

Source	Query
SQuAD v2 (23 queries)	“How much support is there for the UN approach to economic development?”
	“What did Bedau live in to prevent it from being cut down?”
HotpotQA (13 queries)	“The town of Sinclair, WV, is named after the founder of the local oil company, which later merged with…”
	“Cooperative Living Organization is located in a city that is the county seat of…”
MuSiQue (20 queries)	“In what part of Florida is Tom Denney’s birthplace located?”
	“Where does Ulysses Kay’s birth city hold NASCAR races?”

Appendix

Table 25: Representative generation queries by source benchmark. Queries require extractive or compositional answers grounded in retrieved context.

Table / figure	Regime	Sample basis	Query set
Table 2	profile-native (concise-regen for direct-route; verbose-agentic for agentic-route; sensitivity in Section C.2 )	3 seeds $\times$ 43 gen / 56 retrieval	SQuAD v2 / MuSiQue / HotpotQA
Table 22	profile-native answer generation	3 seeds (25K–200K)	TriviaQA test
Fig. 3	concise-regen	122 configs $\times$ 56	TriviaQA eval
Table 65	verbose-agentic	retrospective logs	6-profile deployment
Table 46	verbose-agentic	retrospective logs	5 BEIR subsets

Appendix

Table 26: Which prompt regime feeds which reported faithfulness number. The regen and agentic columns are not directly comparable: verbose answers produce more atomic claims per unit of retrieved evidence, which systematically inflates RAGAS faithfulness by

+0.15

+0.18

( Table 27 ).

Profile	Regen mean	Agentic mean	$\Delta$
sota_standard	0.613	0.774	$+0.161$
sota_balanced	0.660	0.835	$+0.175$
sota_max_reliable	0.653	0.826	$+0.173$

Appendix

Table 27: Same-sample paired gap between the two prompt regimes at 100K. The systematic offset motivates our decision to keep each regime’s results in a dedicated table rather than merging them.

Regime	Mean length (tok)	Raw faithfulness	Within-regime $r$	Residualized faithfulness
Concise (A)	40	0.823	$-0.062$	0.714
Neutral (B)	62	0.862	$-0.064$	0.791
Verbose (C)	249	0.644	$-0.459$	0.824

Appendix

Table 28: Length-controlled faithfulness (

n{=}50

per regime, GPT-5.1 judge). Within-regime Pearson correlation is between answer length and the sample’s faithfulness score. Residualized means are computed after regressing faithfulness on log answer-token count and adding back the grand mean.

Metric	Human 1 vs Human 2	Human 1 vs RAGAS	Human 2 vs RAGAS
Pearson $r$	0.956	0.830	0.771
Kendall $\tau_{b}$	0.674	0.464	0.400
Cohen’s $\kappa$ (at 0.5)	0.929	0.922	—
Agree $\leq 0.1$	80%	54%	—
Agree $\leq 0.2$	96%	86%	—
MAE	0.068	0.108	—

Appendix

Table 29: Inter-rater reliability: two human annotators and GPT-5.1 RAGAS judge on 50 faithfulness evaluations.

Quantity	Value
Items $\times$ annotators (total judgments)	$123\times 2=246$
Tie judgments (out of 246 total)	$195$ ( $79.3\%$ )
Non-tie judgments (out of 246 total)	$51$ ( $20.7\%$ )
Configurations covered by the sample	$12$
Configurations with bootstrap 95% CI $\not\ni 0$	$1$
Inter-annotator Cohen’s $\kappa$ (pairwise)	$0.788$

Appendix

Table 30: Pairwise Bradley–Terry validation: coverage and inter-protocol reliability.

Rank	D5 $\times$ D8 configuration	$\theta_{\text{MLE}}$	95% bootstrap CI
$1$	hyde $\times$ self_rag_quality_gates	$+0.428$	$[-0.13,\ +0.98]$
$2$	none $\times$ speculative_rag	$+0.346$	$[-0.00,\ +0.69]$
$3$	query_decomp $\times$ self_rag_quality_gates	$+0.346$	$[-0.00,\ +0.67]$
$4$	hyde $\times$ speculative_rag	$+0.288$	$[-0.36,\ +0.85]$
$5$	query_decomp $\times$ speculative_rag	$+0.191$	$[-0.00,\ +0.51]$
$6$	query_expansion $\times$ crag	$+0.191$	$[-0.00,\ +0.49]$

Appendix

Table 31: BTL posterior faithfulness ranking of the 12 D5

\times

D8 configurations covered by the sample. Higher

\theta

= more frequently preferred by pairwise LLM annotators; L2-regularized MLE, 1000-bootstrap 95% CIs. Only the bottom-ranked configuration ( query_decomp

\times

crag ) has a CI fully separated from zero; the

-0.00

+0.00

endpoints displayed on ranks 2, 3, 5, 6, and 9 are bootstrap bounds within

10^{-6}

of zero, not exact zeros. Rank 7’s

\theta_{\text{MLE}}=0.000

with degenerate CI

[-0.00,+0.00]

is a structurally exact zero from the L2-regularized fit (not a near-zero bootstrap bound like ranks 2, 3, 5, 6, and 9).

#	Feature	$\Delta$ NDCG@10	Latency (ms)	$\Delta$ Faith.
1	HyDE (hypothetical document embedding)	$+0.026$	$+320$	$+0.01$
2	Disable MMR	$+0.046$	$-10$	$+0.00$
3	Cross-encoder reranker (ms-marco-MiniLM-L6)	$+0.084$	$+180$	$+0.02$
4	Hybrid search ( $\alpha{=}0.5$ )	$+0.037$	$+40$	$+0.01$
5	Context reordering (lost-in-middle)	$-0.012$	$+5$	$+0.01$
6	Query expansion ( $n{=}2$ )	$-0.014$	$+2600$	$-0.01$

Appendix

Table 32: Single-feature ablation results.

\Delta

NDCG@10 is relative to the standard baseline (NDCG@10

=0.412

). Positive values indicate improvement. Features marked with

\dagger

also affect generation quality; their faithfulness delta is noted in the rightmost column. Faithfulness deltas use the concise-regen answer prompt ( Sections C.2 and 26 ); the Agentic RAG row is scored under the same concise-regen regime for comparability, not the verbose-agentic prompt used in Tables 65 and 46 . Full per-configuration results for the underlying reranker/embedding/ chunking/agentic sweeps appear in Tables 51 , 52 , 53 , 54 and 55 .

Interaction pair	$\eta^{2}$	$F$	Raw $p$	Holm $p$
QueryAug $\times$ AgenticGate	0.259 ‡	3.18	0.0006	0.018 ∗
QueryAug $\times$ Reranking	0.125	1.29	0.233	1.000
Enrichment $\times$ Reranking	0.105	2.25	0.044	1.000 †
Chunking $\times$ AgenticGate	0.082	1.12	0.356	1.000
Reranking $\times$ AgenticGate	0.073	0.98	0.464	1.000
Retrieval $\times$ AgenticGate	0.066	1.35	0.239	1.000

Appendix

Table 33: Faithfulness interaction table (122 configs, 56 queries each). Type I SS shown for the full 28-pair scan; the key interaction (QueryAug

\times

AgenticGate) was verified with Type III SS (

\eta^{2}=0.191

F=5.47

p<0.001

). ∗ Holm

p<0.05

; † raw

p<0.05

; ‡ Type I value; Type III gives

\eta^{2}=0.191

p<0.001

. Regime: concise-regen ( Section C.2 ).

Figure 7: Retrieval-track NDCG@10 (93 BEIR queries) vs e2e RAGAS faithfulness (56 QA queries) for the

113

factorial configurations evaluated on both tracks. Pearson

r=-0.01

across the joined set. Shaded regions mark off-diagonal quadrants: red holds

7/113

configurations with strong retrieval but weak faithfulness, yellow holds

6/113

with the inverse pattern, and green holds

17/113

that clear both thresholds simultaneously. The near-zero correlation indicates that retrieval quality and faithfulness are governed by different pipeline stages, consistent with the disjoint factor-pair dominance reported in Fig. 3 .

Profile	Fit	$a$	$b$	$R^{2}$	AIC
sota_standard	log-linear	$0.728$	$-0.055$	$0.891$	$-51.11$
	power-law	$0.799$	$\phantom{-}0.049$	$0.879$	$-50.47$
sota_balanced	log-linear	$0.558$	$-0.015$	$0.364$	$-50.50$
	power-law	$0.564$	$\phantom{-}0.014$	$0.368$	$-50.53$

Appendix

Table 35: Log-linear vs. power-law fits on the same retrieval-NDCG@10 scaling points. Log-linear slope

b

(negative = degradation) and power-law exponent

b

(positive = degradation) agree in sign and in rank ordering: balanced degrades

\sim 3.6\times

slower than standard under both models. Reproducer, available from the authors on request: paper/scripts/refit_power_law.py .

Figure 8: Retrieval NDCG@10 vs. corpus size (log scale) under both fitting protocols. Left: log-linear (

\mathrm{NDCG}=a+b\log_{10}N

). Right: power-law (

\mathrm{NDCG}=a\cdot N^{-b}

). Points are the mean at each corpus size; dashed lines are the fitted curves. The two protocols yield near-identical fitted curves and near-identical goodness of fit ( Table 35 ), so H3’s rank-flip conclusion does not depend on the fitting choice.

Corpus	Size	$n$	Balanced	max-reliable	$\Delta$ (B $-$ MR)	Wilcoxon $p$
faithfulness (RAGAS, GPT-5.1 judge)
HotpotQA	100K	36	$0.741$	$0.745$	$-0.005$	$0.961$
MuSiQue	100K	56	$0.580$	$0.592$	$-0.012$	$0.665$
SQuAD v2	100K	27	$0.691$	$0.710$	$-0.019$	$0.875$
HotpotQA	200K	36	$0.819$	$0.736$	$+0.083$	$0.250$
MuSiQue	200K	54	$0.620$	$0.543$	$+0.077$	$0.299$

Appendix

Table 37: Cross-corpus H2 replication, quality metrics. Per-cell paired Wilcoxon signed-rank (two-sided) against

\Delta{=}0

; Holm correction across the six cells within each metric.

Corpus	Size	$n$	Balanced (s)	max-reliable (s)	Speedup ( $\times$ )	Balanced faster
HotpotQA	100K	39	$6.7$	$66.9$	$10.0$	$39/39$
MuSiQue	100K	60	$7.5$	$76.5$	$10.2$	$60/60$
SQuAD v2	100K	69	$5.1$	$64.4$	$12.7$	$69/69$
HotpotQA	200K	39	$8.9$	$129.5$	$14.6$	$39/39$
MuSiQue	200K	60	$9.6$	$145.0$	$15.1$	$60/60$
SQuAD v2	200K	69	$6.4$	$128.3$	$20.0$	$69/69$

Appendix

Table 38: Cross-corpus H2 replication, latency dominance. Speedup is the ratio of mean latencies; per-sample latency comparisons are paired within (sample_id, seed).

Reference latency	speed	standard	balanced	max-reliable
$10$ s (real-time chat)	0.264	0.299	$0.000$	$0.000$
$30$ s (main reference)	0.277	0.523	0.453	0.389
$45$ s (near-crossover)	0.279	0.561	0.557	0.483
$60$ s (longer reference)	0.280	0.579	0.609	0.562
$120$ s (long reference)	0.282	0.607	0.687	0.650

Appendix

Table 39: Joint hypervolume

\mathrm{HV}(L_{\mathrm{ref}})=\mathrm{NDCG@10}\times\mathrm{Faith}\times\max(0,1-\mathrm{latency}/L_{\mathrm{ref}})

under five reference latencies. All inputs from Table 2 ; higher is better. standard has the highest HV for references (

L_{\mathrm{ref}}{\leq}45

s); the crossover to balanced occurs at

L_{\mathrm{ref}}{\approx}46

s. balanced

>

max-reliable at every reference that admits both.

Query Augmentation (D5)	Agentic Gate (D8)	Mean faithfulness
none	Self-RAG (quality gates)	$0.911$ (grid max)
RAG-Fusion	CRAG	$0.539$ (grid min)
Grid range		$0.372$

Appendix

Table 40: E1 executed cell means: mean faithfulness (GPT-5.1 judge, three-seed read,

n_{\text{runs}}=3

) for the extreme cells of the

5{\times}4

Query Augmentation

\times

Agentic Gate grid. Full grid available from the authors on request ( paper/e1_focused_experiment/results/cell_means.csv ).

D5 level	D8 level	Coefficient	$p$ -value
RAG-Fusion	none	$+0.273$	$0.008$
none	none	$+0.243$	$0.017$
none	Self-RAG (quality gates)	$+0.236$	$0.019$

Appendix

Table 41: E1 surviving interaction contrasts (

p<0.05

, uncorrected) from the full mixed-effects fit. Coefficients are departures from the additive main-effects prediction; positive values indicate the cell exceeds additive expectation.

D5 (Query Augmentation)	D8 (Agentic Gate)	Residual	Direction
Query Decomposition	none	$-0.050$	antagonism
none	Speculative-RAG	$-0.041$	antagonism
Query Expansion	Speculative-RAG	$-0.041$	antagonism
Query Decomposition	Self-RAG (quality gates)	$-0.036$	antagonism
HyDE	CRAG	$-0.027$	antagonism
Query Decomposition	Speculative-RAG	$+0.047$	synergy

Appendix

Table 42: E1 cell-level interaction residuals for the

5{\times}4

Query Augmentation

\times

Agentic Gate grid, computed on the three-seed E1 replication (

n_{\text{runs}}=3

): departure of the observed cell mean from the additive main-effects prediction (observed

-

predicted). Positive

=

synergy (cell exceeds additive expectation); negative

=

antagonism. Top 5 of each direction shown; ten of twenty cells deviate by

{\geq}0.02

faithfulness. Residual range

[-0.050,+0.047]

; observed cell-mean range

0.200

vs. additive-predicted cell-mean range

0.187

(interactions add

{\sim}7\%

extra cell-level spread on top of the additive main-effects surface).

Block	DiD	95% CI	Raw $p$	Holm $p$	Effect
Reranking $\times$ Hybrid Alpha	$-0.110$	$[-0.167,-0.059]$	0.021	0.128	$-38\%$
Reranking $\times$ MMR	$-0.058$	$[-0.087,-0.033]$	0.176	0.880	$-16\%$
HyDE $\times$ Reranking	$-0.066$	$[-0.135,+0.003]$	0.202	0.880	$-23\%$
HyDE $\times$ Hybrid Alpha	$-0.019$	$[-0.075,+0.040]$	0.689	1.000	$-7\%$
HyDE $\times$ MMR	$+0.004$	$[-0.029,+0.035]$	0.868	1.000	$+1\%$
MMR $\times$ Hybrid Alpha	$0.000$	$[0.000,0.000]$	0.999	1.000	$0\%$

Appendix

Table 43: All six

2{\times}2

factorial probes ( descriptive, not confirmatory ). DiD estimates the interaction magnitude; 95% CIs from paired BCa bootstrap over 76 queries. Holm-corrected

p

-values account for six comparisons. None of the six probes survives Holm correction and we therefore report them as descriptive, not confirmatory ; the confirmatory interaction analysis is the 167-config factorial ANOVA in Section 5 .

Feature removed	Standard	Speed	Balanced	Max reliable
HyDE	n/a	n/a	$+0.008$	$-0.031$
MMR (re-enable)	$-0.046$	$-0.038$	$-0.029$	$-0.019$
Reranker	n/a	n/a	$-0.072$	$-0.061$
Hybrid search	$-0.037$	$-0.025$	$-0.028$	$-0.033$
Context reordering	n/a	n/a	$+0.015$	$-0.003$
Query expansion	n/a	n/a	n/a	$+0.018$

Appendix

Table 44: Feature dropout:

\Delta

NDCG@10 when removing each feature from a composed profile. Cells highlighted in red indicate features that help in one profile but hurt in another (interaction signal). All values are on the test split. Pooled across profiles, re-enabling MMR costs NDCG@10 (paired

t=-3.11

p=0.003

; 95% CI on the drop

[-0.062,\,-0.013]

), opposite its typical open-domain benefit.

Profile	nfcorpus	fiqa	scifact	arguana	hotpotqa
Standard (baseline)	0.298	0.385	0.461	0.432	0.394
Speed	0.312	0.397	0.478	0.441	0.418
Balanced	0.351	0.442	0.539	0.478	0.461
Max reliable	0.338	0.431	0.527	0.471	0.482
SOTA balanced	0.362	0.458	0.536	0.491	0.479
SOTA max reliable	0.349	0.447	0.531	0.483	0.475

Appendix

Table 45: NDCG@10 by domain and profile (test split). Best value per domain in bold .

Profile	nfcorpus	fiqa	scifact	arguana	hotpotqa
Standard (baseline)	0.941	0.958	0.982	0.971	0.955
Speed	0.962	0.975	0.991	0.983	0.968
Balanced	0.955	0.969	0.985	0.978	0.962
Max reliable	0.951	0.964	0.988	0.976	0.971
SOTA balanced	0.958	0.972	0.987	0.984	0.973
SOTA max reliable	0.965	0.978	0.989	0.981	0.976

Appendix

Table 46: Faithfulness by domain and profile (test split). Best value per domain in bold . Values come from a separate BEIR-per-subset generation run under the pipeline’s deployment-native answer configuration ( Section C.2 ); regime provenance for other faithfulness tables is documented in Table 26 .

Figure 9: Waterfall chart of individual component gains stacked on the default_standard baseline (NDCG@10

=0.412

). The dashed red line shows the sum of independent gains (0.667); the solid green line shows the actual composed result ( sota_balanced , NDCG@10

=0.483

; 95% CI on the realized lift

[+0.058,\,+0.084]

, paired bootstrap over 93 queries). The gap represents a 72% composition penalty: the composed system captures only 28% of the total independent improvement. This visualizes the non-additivity that motivates factorial evaluation.

Quantity	Faithfulness	Relative to reference
Reference configuration	$0.692$	—
Reference $+$ selected marginal effects	$0.886$	$+28\%$
Best observed complete configuration	$0.911$	$+32\%$

Appendix

Table 47: Exploratory faithfulness comparison. The selected marginal effects and best observed configuration are not the same exact component stack; their proximity motivates a matched composition test but does not establish additivity.

Corpus	Top profile	Second profile	Top mean	Second mean	$p$	$n$
5K	standard	balanced	0.510	0.504	–	1
15K	standard	balanced	0.509	0.496	–	1
25K	balanced	standard	0.508	0.496	0.178	2
50K	balanced	Max-Reliable	0.488	0.474	0.228	2
50K †	balanced	standard	0.505	0.481	0.039	3
100K	balanced	standard	0.471	0.450	0.327	3

Appendix

Table 48: Per-size paired

t

-tests on the top-two profiles from the retrieval scaling track (rows with

n\geq 2

). The 5K and 15K rows are single-seed point estimates (

n=1

, no test), shown for context; standard leads at 5K/15K, balanced leads from 25K onward. † Dedicated 3-seed retrieval multirun; other rows are from the primary cut.

Figure 10: P95 latency vs. corpus size. (a) Retrieval track: standard and balanced grow sub-linearly on a log–log scale, while speed remains below 300 ms at all sizes. (b) E2E generation track at 25K and 50K. Note the log scale on both axes.

Figure 11: Observed NDCG@10 and log-linear extrapolations for all four SOTA profiles. Solid lines (5K–200K) are measured; dotted lines with shaded bands (200K–1M) are 95% prediction intervals from the per-profile log-linear fit. balanced ’s band widens rapidly because its slope is not distinguishable from zero (

p=0.175

), informative about its scale-robustness, but inflating projection uncertainty.

Profile	5K	15K	25K	50K	100K	200K	500K ∗	1M ∗
speed	0.415	0.382	0.396	0.333	0.272	0.245	$0.208\pm 0.096$	$0.174\pm 0.107$
standard	0.510	0.509	0.496	0.456	0.450	0.432	$0.412\pm 0.047$	$0.395\pm 0.053$
balanced	0.504	0.496	0.508	0.488	0.471	0.491	$0.476\pm 0.044$	$0.472\pm 0.049$
max-reliable	0.456	0.455	0.477	0.474	0.431	0.428	$0.431\pm 0.075$	$0.425\pm 0.084$
$\Delta$ (Bal $-$ Std)	$-$ 0.006	$-$ 0.014	$+$ 0.012	$+$ 0.033	$+$ 0.021	$+$ 0.059	$+$ 0.064	$+$ 0.077

Appendix

Table 49: NDCG@10 across corpus sizes: measured values (5K–200K; 3-seed means at 25K–200K, single-seed at 5K/15K) and 500K/1M ∗ projections from log-linear fit with 95% prediction interval.

Reranker configuration	Experiment	NDCG@10	$\Delta$ vs. no_rerank
ms_marco_minilm_l6	Exp. 15	0.4964	$+0.0427$
colbert_v2	ColBERT vs. Cross-Encoder	0.4895	$+0.0358$
ensemble_2model	Ensemble	0.4865	$+0.0328$
cross_encoder_ms_marco	ColBERT vs. Cross-Encoder	0.4829	$+0.0292$
cross_encoder_baseline	ColBERT vs. Cross-Encoder	0.4780	$+0.0243$
ensemble_3model	Ensemble	0.4769	$+0.0232$

Appendix

Table 51: Reranker inventory (retrieval benchmark, 5K corpus except where noted). NDCG@10 reported from the latest clean run per configuration;

\Delta

is the signed difference against no_reranking (0.4537). ⋆ Default in SOTA profiles.

Embedding configuration	Experiment	Dim	NDCG@10
bge_large_late_chunk	Late Chunking	1024	0.4932
dim_768 (BGE-M3 Matryoshka)	Matryoshka	768	0.4885
dim_1024_full (BGE-M3)	Matryoshka	1024	0.4780
bge_m3_1024	Embedding Model	1024	0.4740
dim_512 (BGE-M3 Matryoshka)	Matryoshka	512	0.4729
snowflake_arctic_l_v2	SOTA 2025–2026	1024	0.4705

Appendix

Table 52: Embedding inventory (retrieval benchmark, isolated contribution with hybrid+rerank disabled). “dim” variants are Matryoshka truncations of BGE-M3; “late_chunk” is late-chunking ingestion. The three entries at the bottom suffered model-loading or compatibility issues in our sweep environment (Stella-1.5B OOM on MPS; E5 and BGE-large runs exhibited an embedding-dim mismatch symptom traced to pre-fix session config); we report them for completeness.

Experiment	Configuration	NDCG@10	Notes
Similarity Threshold ( $N=10$ )
	mmr_norm_minmax	0.4203
	mmr_norm_sigmoid_minmax	0.4179
	mmr_norm_none	0.4160
	standard_baseline	0.3954
	threshold_02	0.3954

Appendix

Table 53: Retrieval-side ablation results (NDCG@10, retrieval benchmark). Each sub-block is a separate experiment;

N

is the number of configurations evaluated in that experiment. Configurations are sorted within each experiment by NDCG@10 descending. All runs use the 5K baseline corpus.

Experiment	Configuration	NDCG@10	Faith ${}_{\textsc{{a}}}$	Faith ${}_{\textsc{{r}}}$	EM
Agentic RAG (iterations/depth) ( $N=17$ )
	no_gates	0.7074	0.905	0.607	0.093
	coverage_keyword	0.7074	0.895	0.616	0.116
	both_gates	0.7074	0.894	0.552	0.186
	refine_auto	0.7074	0.891	0.659	0.140
	agentic_5iter	0.7074	0.891	0.597	0.140

Appendix

Table 54: Generation-side ablation results. Faith

{}_{\textsc{{a}}}

is RAGAS faithfulness on the answer produced by the agentic pipeline; Faith

{}_{\textsc{{r}}}

is RAGAS faithfulness on a regenerated answer from the same retrieved context under a common prompt ( Sections C.2 and 26 ); the two columns are not directly comparable . EM is exact-match accuracy. Configurations with Faith

=0.000

indicate RAGAS was not computed for that run (retrieval metric only). Configs sorted by the primary metric of the experiment.

#	Configuration	NDCG@10
1	docling_hier_1600_400	0.4749
2	semantic_threshold_65	0.4699
3	docling_hier_2000_500	0.4685
4	docling_400	0.4651
5	fixed_800	0.4637
6	fixed_400	0.4606

Appendix

Table 55: Chunking Strategy Sweep (Exp. 17): 20 chunker configurations evaluated on the retrieval benchmark. Config names encode the chunker family and its parameters (e.g. docling_hier_1600_400 = Docling pipeline, hierarchical chunker, parent = 1600, child = 400; semantic_threshold_65 = semantic chunker, similarity threshold 0.65). Sorted by NDCG@10 descending.

Configuration	NDCG@10	Faith ${}_{\textsc{{a}}}$	EM
sota_balanced	0.8122	0.855	0.140
standard_baseline	0.6906	0.814	0.047

Appendix

Table 56: Exp. 06 configurations, scored under the GPT-5.1 RAGAS judge. Faith

{}_{\textsc{{a}}}

is the faithfulness value used in the paper’s main-text Exp. 06 references. Faith

{}_{\textsc{{a}}}

here is the verbose-agentic regime ( Sections C.2 and 26 ). Cross-judge (

\pm 0.08

) and cross-model ordinal (

\tau_{b}{=}0.38

) sensitivity for these configurations are reported separately in Section C.1.4 (see also Section F.11 ).

Profile	Generator A	Generator B	Change (paired test)
standard	0.758	0.833	$+0.075$ ( $p=0.007$ )
balanced	0.765	0.792	$+0.027$ (not significant)
max-reliable	0.687	0.771	$+0.083$ ( $p=0.08$ )
speed	0.565	0.749	$+0.184$ ( $p<0.001$ )

Appendix

Table 57: Faithfulness under two generators, with retrieved documents, answer prompt, and judge held fixed. Paired Wilcoxon signed-rank on per-question scores.

Method	NDCG@10	$\Delta$ vs BM25	Latency overhead
BM25 (baseline)	0.3512	—	$+12$ ms
SPLADE v2	0.3389	$-0.012$	$+85$ ms
ColBERT v2	0.4616	$+0.110$	$+210$ ms
mxbai-rerank-large (cross-encoder)	0.4481	$+0.097$	$+195$ ms
ms-marco-MiniLM-L6 (cross-encoder)	0.4568	$+0.106$	$+180$ ms

Appendix

Table 59: Retrieval method comparison on the BEIR evaluation split. Latency overhead is reported per query relative to bge-m3 dense-only retrieval (

{\sim}178

ms per query, not shown as a table row). BM25 serves as the NDCG sparse baseline.

Configuration	NDCG@10	Faithfulness	EM	Token F1	p95 Lat. (ms)
BM25-only ( $\alpha{=}0.0$ )	$0.403\pm 0.000$	0.633	0.116	0.349	—
BGE-M3 recommended	$0.436\pm 0.000$	0.558	0.093	0.299	—
sota_balanced (ours)	$0.840\pm 0.003$	$0.911\pm 0.011$	$0.132\pm 0.011$	—	17,682
sota_standard (ours)	$0.860\pm 0.003$	$0.739\pm 0.013$	$0.163\pm 0.000$	—	6,887

Appendix

Table 60: Executable external baselines on our evaluation corpus. Mean

\pm

std over 3 runs for all cells reporting

\pm

; cells without

\pm

are single-run point estimates (BM25-only and BGE-M3 rows for Faithfulness / EM / Token F1). Prompt regime, generator, and judge for all four rows are recorded in Table 58 (row: tab:external-exec); Faithfulness is scored by RAGAS/GPT-5.1 under a single answer-generation prompt applied uniformly to all rows, so the BM25/BGE-M3 vs sota_* comparison is within-regime.

System	Source	NFCorpus	FiQA	SciFact	ArguAna	HotpotQA
BM25	( Thakur et al., 2021 )	0.325	0.236	0.665	0.315	0.603
ANCE	( Thakur et al., 2021 )	0.237	0.295	0.507	0.415	0.456
TAS-B	( Thakur et al., 2021 )	0.319	0.300	0.643	0.427	0.584
ColBERT v2	( Santhanam et al., 2022 )	0.338	0.356	0.693	—	0.677
BGE-M3	( Chen et al., 2024a )	0.369	0.375	0.725	0.520	0.714

Appendix

Table 61: Published BEIR NDCG@10 baselines (full test sets).

System	EM	Loose-F1	Refusal rate
FlashRAG SequentialPipeline (default prompt)	0.059	0.059	64.7%
LASAGNA analog ( Section C.2 concise prompt)	0.353	0.126	0.0%

Appendix

Table 62: Framework comparison,

17

-query smoke on FlashRAG’s shipped AI-topic corpus with open-domain Natural Questions queries. This measures each framework’s default handling of out-of-corpus queries under identical retriever and generator, not RAG quality on a matched benchmark.

Test	What it verifies	Result
Pydantic type enforcement	Stage boundaries reject invalid types	Pass
Session config isolation	Changing session A’s config leaves session B unchanged	Pass
Embedding dim detection	384-dim vectors into 1024-dim collection raises error	Pass
Chunker statelessness	No cross-document state leakage between calls	Pass
Plugin auto-discovery	New backends register via decorator without code changes	Pass

Appendix

Table 63: Platform isolation verification results. All five tests pass, confirming that toggling one pipeline component does not silently affect another’s behavior.

Figure 12: RAG Configurator, Essential tab. The Essential tab surfaces the highest-impact pipeline toggles ( enable_hybrid_search , enable_reranking , enable_mmr , enable_hyde , use_agentic_rag , enable_hierarchical , docling_enabled , enable_ocr ) alongside the embedding-model selector, a GPU toggle, and three quick-tuning sliders ( default_top_k , similarity_threshold , hybrid_alpha ). Profiles are listed on the left, with the recommended standard profile flagged; user-created profiles appear below and are round-trippable with the YAML contract used by the evaluation SDK.

Figure 13: RAG Configurator, Search & Retrieval tab. The Search & Retrieval tab exposes the retrieval and post-retrieval stack: MMR score normalization (sigmoid vs. min–max), smart result ordering, the RRF fusion constant ( rrf_k ), the keyword-search method ( sparse_method : BM25, SPLADE), context compression with method and ratio controls ( compression_method , compression_ratio ), and reranker-model selection. Every field carries the underlying configuration key so a UI change maps one-to-one to the YAML config consumed by the factorial experiment runner.

Component	Core method	Signature
Embedder	encode()	(texts: List[str]) -> ndarray
Chunker	chunk()	(text: str) -> List[Chunk]
Reranker	rerank()	(query, docs, top_k) -> List[ScoredDoc]
Retriever	retrieve()	(query, collection, top_k) -> List[ScoredDoc]
DocProcessor	extract()	(file_path) -> ProcessedDocument
QueryEnhancer	enhance()	(query) -> List[str]

Appendix

Table 64: Component ABCs and their core methods. Researchers implement one method to integrate a new component.

Figure 14: Pareto positions of the four SOTA profiles in NDCG@10–faithfulness space ( Tables 2 and 5 ). Marker area is proportional to inverse mean latency (larger = faster). sota_standard attains the highest joint retrieval–faithfulness–latency hypervolume at the fixed

30

s response-time reference ( Table 2 ; HV

=0.523

) via the highest NDCG@10 (

0.860

) and the lowest latency among the quality-focused three (

5.3

s). sota_balanced (

\bigstar

) attains the top-band faithfulness (

0.911

, within noise of sota_max_reliable ’s

0.916

) while providing higher NDCG@10 and lower response time; it wins the joint HV once

L_{\mathrm{ref}}

opens past

{\sim}46

s ( Table 39 ).

Figure 15: Profile selection flowchart based on response-time target, faithfulness requirement, and corpus size. Metric values are from the 5K retrieval evaluation ( Table 2 ) except where corpus-size dependent ( balanced at

>

25K uses scaling results from Section 5 ). All profiles use the same generator (GPT-5.1); differences arise from retrieval and post-retrieval components.

Profile	Mean	Median	Std. Dev.	Min
Standard (baseline)	0.963	0.980	0.052	0.714
Speed	0.980	1.000	0.038	0.800
Balanced	0.971	0.990	0.044	0.750
Max reliable	0.968	0.985	0.048	0.667
SOTA balanced	0.975	0.993	0.041	0.778
SOTA max reliable	0.978	1.000	0.036	0.812

Appendix

Table 65: Faithfulness statistics across profiles, aggregated from historical deployment logs (test split, 76 queries per profile). These runs used the pipeline’s deployment-native answer configuration ( Section C.2 ); regime provenance is documented in Table 26 and values are not directly comparable to faithfulness reported in other tables. No statistical tests are applied; values are descriptive summaries only.

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

Feb 25, 2026

Revisiting RAG Retrievers: An Information Theoretic Benchmark

Retrieval-Augmented Generation (RAG) systems rely critically on the retriever module to surface relevant context for large language models. Although numerous retrievers have recently been proposed, each built on different ranking principles such as lexical matching, dense embeddings, or graph citations, there remains a lack of systematic understanding of how these mechanisms differ and overlap. Existing benchmarks primarily compare entire RAG pipelines or introduce new datasets, providing little guidance on selecting or combining retrievers themselves. Those that do compare retrievers directly use a limited set of evaluation tools which fail to capture complementary and overlapping strengths. This work presents MIGRASCOPE, a Mutual Information based RAG Retriever Analysis Scope. We revisit state-of-the-art retrievers and introduce principled metrics grounded in information and statistical estimation theory to quantify retrieval quality, redundancy, synergy, and marginal contribution. We further show that if chosen carefully, an ensemble of retrievers outperforms any single retriever. We leverage the developed tools over major RAG corpora to provide unique insights on contribution levels of the state-of-the-art retrievers. Our findings provide a fresh perspective on the structure of modern retrieval techniques and actionable guidance for designing robust and efficient RAG systems.

Wenqing Zheng, Dmitri Kalaev, Noah Fatsi +5

CapitalOne.

RAG Evaluation IR Evaluation

Ask PDF

May 29, 2026

A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) systems are often evaluated using final answer accuracy, even though their failures can originate from preprocessing, retrieval, context packing, or generation. This paper presents a controlled empirical study of RAG sensitivity, robustness, and stability across 56 experimental runs. We evaluate how chunk size, retrieval depth (top k), embedding-based reranking, probabilistic retrieval noise, and repeated seeded runs affect retrieval, context packing, and generation behavior. Using a fixed 500-question QA subset mapped to 20,958 unique corpus contexts, we analyze both final answer metrics and intermediate failure modes. Across these experiments, retrieval-oriented metrics improved under broader retrieval settings, while downstream exact-match and F1 scores often behaved non-monotonically. We also observe preprocessing-induced answer loss under smaller chunk sizes, progressive degradation under retrieval corruption, and higher observed variance in broader retrieval regimes. These findings suggest that RAG evaluation should include sensitivity, robustness, stability, and multi-stage failure analysis rather than relying only on final answer accuracy.

Bharath Simha Reddy Muthyam

Independent Researcher

Retrieval-Augmented Generation Retrieval Robustness RAG Evaluation RAG Robustness

Ask PDF

Aug 25, 2026

The RAT: A Unified Bayesian Model for RAG Evaluation

Evaluating Retrieval-Augmented Generation (RAG) systems requires assessing not only end-to-end correctness but also how individual components interact and how errors propagate through the pipeline. We introduce a Bayesian evaluation framework that jointly models retrieval success, abstention behavior, and answer correctness, factorized according to the pipeline's information flow. The model distinguishes task success. Whether the user received a correct answer (from generator success) and whether the generator behaved appropriately given the retrieval outcome. We apply the framework to 27 RAG configurations across three datasets, three retrievers, and three generators, and show that the conditional decomposition reveals substantial behavioral differences between systems that appear equivalent under marginal metrics. We further analyze the annotation allocation problem, demonstrating that retrieval-success annotations are more informative than task-success annotations for estimating policy adherence, and provide an information-theoretic explanation for this asymmetry. Finally, we extend the model to incorporate LLM-as-a-judge annotations as calibrated noisy observations, enabling practitioners to combine limited human judgments with cheaper automated assessments within a unified probabilistic model.

Pius von Däniken, Felix Matthias Saaro, Mark Cieliebak +1

Centre for Artificial Intelligence ZHAW School of Engineering

RAG Evaluation

Ask PDF

Abstract

Retrieval-Augmented Generation

Explore similar work

Revisiting RAG Retrievers: An Information Theoretic Benchmark

A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation

The RAT: A Unified Bayesian Model for RAG Evaluation