RAG-NIMBF: A Non-Invasive, Modular Benchmarking Framework for Automated RAG Configuration Analysis

Niclas Hart; Nicolas Weeger

cs.SEOct 20, 2026

RAG-NIMBF: A Non-Invasive, Modular Benchmarking Framework for Automated RAG Configuration Analysis

Authors: Niclas Hart, Nicolas Weeger

Organizations: Ansbach University of Applied Sciences

Abstract

Retrieval-Augmented Generation (RAG) has become the dominant paradigm for grounding the outputs of large language models (LLM) in external knowledge. However, practitioners are faced with a vast configuration space comprising chunking, embedding, retrieval, reranking, prompting and generation models, and there is limited guidance on how these components interact. We present RAG-NIMBF, a modular benchmarking framework that attaches to existing RAG pipelines as a non-invasive evaluation layer. It evaluates the configuration space through controlled ablation with end-to-end experiment tracking. The evaluation combines LLM-based answer quality scores, retrieval correctness measures, and runtime indicators. This study validates the framework across multiple generation models on the SQuAD dataset. This process reveals non-obvious interactions and demonstrates how the experiments are tracked. The findings indicate that the retrieval balance of relevance and diversity substantially enhances the relevance and coverage of retrieved context in comparison to a conventional similarity search, with MMR raising nDCG@5 by +4 to +11 percentage points across the swept models. More detailed prompt instructions meaningfully increase how faithfully generated answers stick to the retrieved evidence, especially for capable and reasoning models (+10 to +13pp on Qwen3.6-27B, GPT-4o-mini and Qwen3.5-think, but negligible on the 20B generator). Overall, RAG-NIMBF transforms the configuration space of RAG into reproducible, comparable experiments. By attaching to existing production pipelines in a non-invasive way, it enables practitioners to systematically evaluate and improve deployed systems without disrupting existing workflows. This replaces ad hoc choices with evidence-based decisions.

Figures & tables

Layer	Metric	Definition
RAGAS	Faithfulness	$\|\{c_{i}\in\mathcal{C}_{\text{ans}}:c_{i}\text{ entailed by }\mathcal{C}_{\text{ctx}}\}\|/\|\mathcal{C}_{\text{ans}}\|$
	Context recall	$\|\text{GT attr.\ to }\mathcal{C}_{\text{ctx}}\|/\|\text{GT}\|$
	Semantic sim.	$\cos(\text{emb}(\text{ans}),\text{emb}(\text{GT}))$
IR	Hit@ $k$	$\mathbb{1}[\|\mathcal{R}_{k}\cap\mathcal{G}\|\geq 1]$
	Recall@ $k$	$\|\mathcal{R}_{k}\cap\mathcal{G}\|/\|\mathcal{G}\|$
	nDCG@ $k$ [ 11 ]	$\text{DCG}@k/\text{IDCG}@k$ , $\text{DCG}@k=\sum_{i=1}^{k}\text{rel}_{i}/\log_{2}(i+1)$

Table 1: Metric definitions per evaluation layer (RAGAS layer from Es et al. [ 8 ] ).

Model	Params	Deployment
Qwen3-32B-AWQ 1	32B dense	Workstation
GPT-OSS-20B 2	20B (3.6B active)	DGX Spark (GB10)
Qwen3.5-35B-A3B 3	35B (3B active)	GPU-Cluster
Qwen3.6-27B-NVFP4 4	27B dense	Workstation
GPT-4o-mini 5	$\sim$ 20-30B (est.)	GPU-Cluster
DeepSeek-V4-Pro 6	862B	API

Table 2: Generation models with parameter counts.

Parameter	Values
LLM Models ( $M=6$ )	See Table 2
Chunking Strategies ( $C=2$ )	Recursive, Semantic
Chunk Sizes ( $S=2$ )	500, 1000
Chunk Overlaps ( $O=2$ )	100, 200
Retrieval Methods ( $R=2$ )	Similarity, MMR
Prompt Templates ( $T=2$ )	Concise, Detailed

Table 3: Configuration space.

Model	Faith. $\uparrow$	BS-F1 $\uparrow$	METEOR $\uparrow$
Qwen3.5-think	0.88 (0.95)	0.86 (0.93)	0.49 (0.60)
GPT-4o-mini	0.89 (0.96)	0.87 (0.93)	0.49 (0.59)
DeepSeek-V4-Pro	0.88 (0.94)	0.91 (0.92)	0.56 (0.66)
Qwen3.6-27B	0.89 (0.95)	0.89 (0.95)	0.51 (0.61)
Qwen3-32B	0.83 (0.92)	0.91 (0.97)	0.48 (0.60)
GPT-OSS-20B	0.77 (0.84)	0.90 (0.96)	0.46 (0.60)

Table 4: Model comparison: mean and (best) across all configurations. Retrieval metrics omitted from per-model rows because generator-invariant by construction; global retrieval quality reported in the final row.

Model	Strategy	Faith.	nDCG@5	BS-F1	METEOR
Qwen3-32B	Sim.	0.80	0.71	0.92	0.51
Qwen3-32B	MMR	0.85	0.81	0.89	0.47
GPT-OSS-20B	Sim.	0.74	0.71	0.92	0.46
GPT-OSS-20B	MMR	0.80	0.75	0.89	0.45
GPT-4o-mini	Sim.	0.90	0.68	0.88	0.50
GPT-4o-mini	MMR	0.86	0.79	0.82	0.47

Table 5: Retrieval strategy comparison: mean over chunk sizes, overlaps, and templates.

Model	Template	Faith.	nDCG@5	BS-F1	METEOR
Qwen3-32B	Concise	0.80	0.74	0.89	0.58
Qwen3-32B	Detailed	0.87	0.74	0.91	0.41
GPT-OSS-20B	Concise	0.76	0.74	0.91	0.56
GPT-OSS-20B	Detailed	0.77	0.74	0.91	0.34
GPT-4o-mini	Concise	0.83	0.75	0.83	0.53
GPT-4o-mini	Detailed	0.94	0.75	0.85	0.42

Table 6: Prompt template comparison: mean over all chunk sizes, overlaps, and retrieval strategies.

Model	Seq. t (s)	Par. t (s)	Seq. TPS	Par. TPS	TTFT (s)
GPT-OSS-20B	77.09	44.56	53.9	22.4	0.47
Qwen3-32B-AWQ	87.13	22.23	58.9	45.4	0.30
GPT-4o-mini	54.93	28.76	79.4	59.1	0.45
Qwen3.5-think	80.19	26.43	78.8	68.5	3.50
Qwen3.6-27B-NVFP4	38.59	24.80	127.5	70.5	1.86

Table 7: Runtime measurements (N=15, max 1024 output tokens). Seq.=sequential, Par.=parallel, TPS=tokens per second, TTFT=time-to-first-token.

Figure 3: Faithfulness vs. median generation throughput.

RAG Evaluation

Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.

See all 352 papers

Mar 6, 2026

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) is a framework in which a Generator, such as a Large Language Model (LLM), produces answers by retrieving documents from an external collection using a Retriever. In practice, Generators must integrate evidence from long contexts, perform multi-step reasoning, interpret tables, and abstain when evidence is missing. However, existing benchmarks for Generators provide limited coverage, with none enabling simultaneous evaluation of multiple capabilities under unified conditions. To bridge the gap between existing evaluations and practical use, we introduce LIT-RAGBench (the Logic, Integration, Table, Reasoning, and Abstention RAG Generator Benchmark), which defines five categories: Integration, Reasoning, Logic, Table, and Abstention, each further divided into practical evaluation aspects. LIT-RAGBench systematically covers patterns combining multiple aspects across categories. By using fictional entities and scenarios, LIT-RAGBench evaluates answers grounded in the provided external documents. The dataset consists of 114 human-constructed Japanese questions and an English version generated by machine translation with human curation. We use LLM-as-a-Judge for scoring and report category-wise and overall accuracy. Across API-based and open-weight models, no model exceeds 90% overall accuracy. By making strengths and weaknesses measurable within each category, LIT-RAGBench serves as a valuable metric for model selection in practical RAG deployments and for building RAG-specialized models. We release LIT-RAGBench, including the dataset and evaluation code, at https://github.com/Koki-Itai/LIT-RAGBench.

Koki Itai, Shunichi Hasegawa, Yuta Yamamoto +2

neoAI Inc., Tokyo, Japan · Tokyo Metropolitan University, Tokyo, Japan · The University of Tokyo, Tokyo, Japan

RAG Evaluation LLM Evaluation LLM Abstention

Ask PDF

Oct 21, 2026

Retrieval Augmented Generation Compositionality

Retrieval Augmented Generation (RAG) grounds large language models in external knowledge. Modern pipelines combine embedding, chunking, retrieval, reranking, query augmentation, context compression, and agentic verification, yet these techniques are typically evaluated one at a time. It remains unclear whether their isolated gains survive composition, whether retrieval quality and answer faithfulness favor the same design choices, and whether conclusions from small corpora hold at large scale. We pre-declare three hypotheses and introduce LASAGNA, an open-source framework for controlled RAG evaluation. We test 450 configurations across eight benchmarks and corpora ranging from 5K to 200K documents. Three findings emerge. First, individually strong techniques from different phases of a RAG pipeline do not necessarily compose: three retrieval techniques predict a +47% combined improvement but deliver only +17% when used together. Second, component gains stack differently for retrieval and faithfulness, and optimizing one can hurt the other. Third, the best composition changes as the corpus grows: the combination that wins on a small corpus may not remain best at large scale. These findings show that gains measured in isolation are not reliable predictors of realistic performance, and that new RAG techniques should be tested with the full surrounding stack. We call on researchers to do so and supply LASAGNA to simplify these experiments.

Ido Levy, Segev Shlomov, Leshem Choshen

IBM · Weizmann Institute of Science

Retrieval-Augmented Generation RAG Evaluation

Ask PDF

Aug 4, 2026

RAG-Stack: Co-Optimizing RAG Serving Performance and Quality

Retrieval-augmented generation (RAG), which augments large language model (LLM) generation with information retrieved from databases, has become a widely used approach for knowledge-intensive applications. Modern RAG systems, however, expose many configuration choices, such as retrieval indexes, model selections, and how models invoke retrieval. Each configuration yields a different trade-off between answer quality and serving performance, making it challenging to choose the optimal setting for a specific application deployment. We present RAG-Stack, a framework for efficiently discovering quality-performance Pareto frontiers across diverse RAG applications and serving systems. RAG-Stack consists of RAG-PE, an iterative design-space exploration algorithm that selects the next RAG configuration to evaluate; RAG-IR, a workload abstraction for diverse RAG algorithms; and RAG-CM, a performance model that predicts the optimal deployment and serving performance on the given hardware. Together, these components allow RAG-Stack to search the joint algorithm-system configuration space without deploying every candidate and to transfer an existing Pareto frontier to a new serving system. Given the same number of optimization iterations across diverse datasets, the Pareto frontiers found by RAG-Stack cover 52.5% to 153.2% more of the normalized quality-performance space than those found by state-of-the-art configuration-search methods evaluated over the same RAG design space.

Haiqiang Zhang, Yuanqing Lei, Wanting Li +2

ETH Zurich · Columbia University · National University of Singapore

Retrieval-Augmented Generation LLM Serving Multi-Objective Optimization

Ask PDF

Abstract

RAG Evaluation

Explore similar work

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

Retrieval Augmented Generation Compositionality

RAG-Stack: Co-Optimizing RAG Serving Performance and Quality