Agentic AutoRAG: RAG Pipeline Optimization through Reasoning-Driven Agents

Lasse B. Strand; Robert Jakob; Kevin O'Sullivan; Markus Kreft

cs.CLOct 6, 2026

Agentic AutoRAG: RAG Pipeline Optimization through Reasoning-Driven Agents

Authors: Lasse B. Strand, Robert Jakob, Kevin O'Sullivan, Markus Kreft

Organizations: ETH Zurich

Abstract

Retrieval-augmented generation (RAG) is a widely used approach for grounding large language models (LLMs) in external knowledge. However, configuring a pipeline is an expensive hyperparameter optimization problem over many interacting choices, from chunking and embedding model to reranking and generation. Existing optimizers, from greedy search to Bayesian optimization, reduce each trial to an aggregate score and search without modeling why a configuration performed as it did, even though the retrieved chunks already provide evidence about whether each failure occurred during retrieval or after it. We introduce Agentic AutoRAG, an LLM-agent optimizer for multi-objective RAG hyperparameter optimization with retrieval-versus-generation failure attribution. It proposes configurations scored on a frozen exam from the corpus: after each trial a Diagnoser attributes each failed question to retrieval or generation, and a Proposer, grounded in a knowledge base of model rankings and pricing, selects the next configuration, weighing accuracy against cost to trace a Pareto frontier. On three multi-hop QA benchmarks it reaches higher LLM-judge accuracy than every baseline we compare, and within its first 10 trials it matches or beats the statistical baselines' full 30-trial judge accuracy. In its cost-aware mode on a real-world healthcare corpus it reaches a median exam accuracy of 77%, above the strongest baseline's 71.5%, at about 58% of that baseline's cost per query, and it matches that 71.5% at about 22% of the cost.

Figures & tables

Panel of Figure 1 — Figure 1: Architecture of Agentic AutoRAG. The exam generation pipeline builds a frozen exam from the corpus once before optimization: TF-IDF neighborhoods seed the Composer, which drafts open-ended questions with verbatim gold spans, validation filters and a sufficiency oracle enforce answerability and multi-hop dependence, and probe-based discrimination admits only questions that differentiate across the search space. The optimization loop alternates between a Diagnoser, which compresses each trial’s per-question failure evidence, and a Proposer, which selects the next configuration grounded in a knowledge base of public model rankings and pricing. The optimizer outputs a Pareto frontier over accuracy and cost, a recommended configuration, and a per-trial reasoning trace.

Corpus	Docs.	Words	Words/doc.
HotpotQA	19,276	1.81M	94
MuSiQue	17,629	1.32M	75
MultiHop-RAG	609	1.07M	1,760
UniDoc	250	1.30M	5,214

Table 1: Corpus size for each benchmark.

Figure 2: Best-so-far validation-exam accuracy over the 30-trial search, one panel per dataset. Each line is a method’s running-best accuracy averaged over the 10 search seeds, and the shaded band spans

\pm 1

standard deviation across seeds. KB Greedy, the no-search reference, is the dashed horizontal line.

	HotpotQA			MuSiQue			MultiHop-RAG
Method	EM	F1	Judge	EM	F1	Judge	EM	F1	Judge
Random	$.506_{\pm.038}$	$.664_{\pm.041}$	$.774_{\pm.046}$	$.165_{\pm.029}$	$.215_{\pm.035}$	$.270_{\pm.036}$	$.633_{\pm.065}$	$.651_{\pm.063}$	$.706_{\pm.048}$
MO-TPE	$.515_{\pm.057}$	$.673_{\pm.056}$	$.784_{\pm.056}$	$.193_{\pm.024}$	$.254_{\pm.032}$	$.301_{\pm.032}$	$.664_{\pm.045}$	$.677_{\pm.042}$	$.724_{\pm.028}$
MO-TPE (warm)	$.521_{\pm.044}$	$.677_{\pm.055}$	$.786_{\pm.057}$	$.183_{\pm.030}$	$.243_{\pm.032}$	$.301_{\pm.039}$	$.666_{\pm.051}$	$.682_{\pm.047}$	$.731_{\pm.045}$
Agentic (no KB/diag)	$.550_{\pm.020}$	$.711_{\pm.024}$	$.819_{\pm.025}$	$.180_{\pm.022}$	$.230_{\pm.028}$	$.298_{\pm.024}$	$\mathbf{.729}_{\pm.037}$	$\mathbf{.748}_{\pm.035}$	$.780_{\pm.021}$
Agentic@10	$.558_{\pm.015}$	$.715_{\pm.017}$	$.817_{\pm.014}$	$.177_{\pm.023}$	$.231_{\pm.028}$	$.303_{\pm.030}$	$.679_{\pm.056}$	$.697_{\pm.055}$	$.773_{\pm.021}$

Table 2: Held-out accuracy on each benchmark’s gold test slice, a 300-question stratified sample disjoint from the validation exam and unseen during search. Exact match, token-level F1, and LLM-judge accuracy are reported as mean

{}_{\pm\text{SD}}

over the 10 search seeds. Rows above the rule are search methods run under a 30-trial budget. Agentic@10 and @20 are the agentic optimizer’s best configuration within its first 10 and 20 trials. KB Greedy, below the rule, is a no-search reference that takes each dimension’s top option from the knowledge base. Best value per column among the search methods is in bold, with ties bold on both rows.

Figure 3: Cost-quality results on the UniDoc healthcare corpus, aggregated over 10 seeds. Mean API cost per query on a log scale against accuracy on the self-generated exam, with each method drawn as the point-wise median across seeds of its cost-accuracy attainment (line) and the interquartile (25th to 75th percentile) spread across seeds (band). We compare our cost-aware optimizer (Agentic) against random search, cold MO-TPE, and warm-started MO-TPE.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Question type	Count
Extraction	16
Definitional	0
Numeric (single-span)	1
Inference	25
Bridge	19
Comparison	25

Appendix

Table 3: Question-type distribution of the 100-question UniDoc exam.

Model	Creator	Intel.	GPQA	IFBench	In $/M	Out $/M	Tok/s	Reasoning
glm-4.7	Z AI	42.1	0.859	0.679	0.60	2.20	86	Yes
gpt-5-mini	OpenAI	38.9	0.803	0.712	0.25	2.00	106	Yes
gpt-5.4-mini	OpenAI	37.7	0.823	0.648	0.75	4.50	172	Yes
gpt-oss-120b	OpenAI	33.3	0.782	0.690	0.15	0.60	289	Yes
gemini-2.5-flash	Google	27.0	0.790	0.503	0.30	2.50	225	Yes
gpt-5-nano	OpenAI	25.9	0.670	0.659	0.05	0.40	151	Yes

Appendix

Table 4: Generator-LLM annotations in the knowledge base, a twelve-model slice across the intelligence range (the full set of 35 models, which render as 49 reasoning and non-reasoning entries, is named in Appendix B ). Intel. is the Artificial Analysis intelligence index. GPQA ( Rein et al., 2024 ) and IFBench ( Pyatkin et al., 2025 ) are accuracies in

[0,1]

. Prices are USD per million input and output tokens. Tok/s is median output throughput. Reasoning marks which variant the listed scores describe. Model names abbreviate the LiteLLM identifiers in Appendix B , and a dash marks a value unpublished in the snapshot.

Model	Dim.	Max tok.	Params (B)	Retrieval	STS	Rerank.	Mem. (MB)
Yuan-embedding-2.0-en	1024	2048	0.596	0.707	0.849	0.533	2272
Qwen3-Embedding-0.6B	1024	32768	0.596	0.618	0.858	0.482	1136
snowflake-arctic-embed-l-v2.0	1024	8192	0.568	0.586	0.770	0.475	2166
granite-embedding-english-r2	768	8192	0.149	0.564	0.781	0.491	284
mxbai-embed-large-v1	1024	512	0.335	0.556	0.772	0.480	639
bge-large-en-v1.5	1024	512	0.335	0.554	0.759	0.483	1242

Appendix

Table 5: The 13 embedding models in the search space with their MTEB(eng, v2) ( Muennighoff et al., 2023 ) scores. Retrieval, STS, and reranking (Rerank.) are the corresponding MTEB task scores in

[0,1]

. Dim. is the embedding dimension, Max tok. the maximum input length, Params (B) the parameter count in billions, and Mem. the memory footprint. The Proposer ranks retrievers by the Retrieval score. Names drop the organization prefix listed in Appendix B , and a dash marks a value unpublished in the snapshot.

Model	Params (B)	MTEB-R	MMTEB-R	FollowIR
bge-reranker-v2-m3	0.6	0.570	0.584	$-0.01$
ms-marco-MiniLM-L-6-v2	0.022	–	–	–
gte-reranker-modernbert-base	–	–	–	–
mxbai-rerank-xsmall-v1	–	–	–	–

Appendix

Table 6: The four reranker models in the search space (a fifth option, none , disables reranking). MTEB-R and MMTEB-R are the MTEB ( Muennighoff et al., 2023 ) and MMTEB ( Enevoldsen et al., 2025 ) reranking-task scores in

[0,1]

, and FollowIR ( Weller et al., 2025 ) is the instruction-following retrieval score. Coverage is sparse, with published scores for only one of the four rerankers in the snapshot, so the Proposer has little benchmark signal here and relies on rerank depth and downstream accuracy. Names abbreviate the identifiers in Appendix B , and a dash marks a value unpublished in the snapshot.

Explore similar work

AutoRAGTuner: A Declarative Framework for Automatic Optimization of RAG Pipelines
May 3, 2026Xintan Zeng, Yongchao Liu, Yice Luo +1Agentic Retrieval-Augmented Generation Systems Retrieval-Augmented Generation Pipelines
Ant Group, China
Agent-Orchestrated Adaptive RAG: A Comparative Study on Structured and Multi-Hop Retrieval
Jun 4, 2026Anuj Maharjan, Devinder Kaur, Richard MolyetAgentic Retrieval-Augmented Generation Systems Decomposition
Dept. of Electrical Engineering and Computer Science University of Toledo Toledo, OH, USA
Retriever Portfolios: A Principled Approach to Adaptive RAG
May 29, 2026Miltiadis Stouras, Vincent Cohen-Addad, Silvio Lattanzi +1Agentic Retrieval-Augmented Generation Systems Portfolio Construction
1EPFL · 2Google Research

May 3, 2026, cs.LGJ/K move · Enter open · S save

AutoRAGTuner: A Declarative Framework for Automatic Optimization of RAG Pipelines

Xintan Zeng, Yongchao Liu, Yice Luo, Jiajun Zhen

Ant Group, China

Agentic Retrieval-Augmented Generation Systems Retrieval-Augmented Generation Pipelines Tuning

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) enhances LLMs, but performance is highly sensitive to complex architecture designs and hyper-parameter configurations, which currently rely on inefficient manual tuning. We present AutoRAGTuner, a declarative, configuration-driven framework that automates the RAG life cycle: construction, execution,evaluation, and optimization. AutoRAGTuner employs a modular architecture to decouple pipeline stages through a component registration mechanism. To unify heterogeneous data, we introduce the Domain-Element Model (DEM), representing objects as atomic elements with bidirectional pointers to support nodes, edges, and hyperedges. Furthermore, AutoRAGTuner integrates an adaptive Bayesian optimization engine for end-to-end hyper-parameter tuning. Experimental results demonstrate AutoRAGTuner's architectural generality: across diverse RAG pipelines, ranging from vanilla to graph-based, the framework consistently outperforms default baselines. Notably, AutoRAGTuner significantly mitigates engineering overhead, where its declarative configuration language enables a up to 95% reduction in code churn for architectural adjustments. Overall, AutoRAGTuner provides a systematically optimizable foundation for building evolvable and reusable RAG systems.

Abstract

Explore similar work

AutoRAGTuner: A Declarative Framework for Automatic Optimization of RAG Pipelines

Agent-Orchestrated Adaptive RAG: A Comparative Study on Structured and Multi-Hop Retrieval

Retriever Portfolios: A Principled Approach to Adaptive RAG