IRAH: Hierarchical Agentic Tree Search for Retrieval Pipeline Optimization

Maria Ana Cardei; Aniket Singh; Amir Moghadam; Srividhya Baikadi; Rashmi Shetty

cs.IROct 16, 2026

IRAH: Hierarchical Agentic Tree Search for Retrieval Pipeline Optimization

Authors: Maria Ana Cardei, Aniket Singh, Amir Moghadam, Srividhya Baikadi, Rashmi Shetty

Organizations: University of Virginia · University of California San Diego · Capital One

Abstract

Optimizing Information Retrieval (IR) pipelines for Retrieval-Augmented Generation (RAG) requires jointly selecting structural components and numerical hyperparameters under limited evaluation budgets. We introduce the Information Retrieval Agentic Harness (IRAH), which combines Language Agent Tree Search for IR (LATS-IR) with offline meta-learned initialization. LATS-IR performs UCT-guided hierarchical search, first exploring macro-structural pipeline choices and then refining numerical parameters, with natural-language reflections to guide subsequent mutations and provide inspectable search traces. We evaluate IRAH on three in-distribution benchmarks and one out of-distribution benchmark to assess generalization under distribution shift. Under the same meta-learned initialization, LATS-IR outperforms TPE on two of three in-distribution benchmarks, with gains of up to 6.4%, and remains competitive on the out-of-distribution benchmark. Across all four benchmarks, warm-started IRAH achieves the highest average held-out nDCG@10 (0.4816) and improves over its cold-start counterpart. These results establish IRAH as a competitive approach to retrieval pipeline optimization under constrained evaluation budgets.

Figures & tables

Figure 1: IRAH : Given a target dataset and task metric, an offline meta-learner proposes promising seed configurations. LATS-IR then performs two-phase hierarchical search over structural and numerical pipeline decisions and returns the best configuration identified under the evaluation budget.

Dataset	Domain	Corpus Size	Opt. Queries	Held-out Queries	$\boldsymbol{d_{\mathrm{meta}}}$
SciFact	Scientific fact verification	5,183	240	60	0.679
FiQA-2018	Financial QA	57,638	519	129	0.354
CQADupStack-STATS	Statistics duplicate questions	42,269	522	130	0.396
CoSQA	Natural-language code search	6,267	400	100	2.092

Table 1: Characteristics of the target retrieval benchmarks. We report dataset domain, corpus size, optimization and held-out query counts, and meta-feature distance

d_{\mathrm{meta}}

Method	SciFact	FiQA	CQADupStack- STATS	CoSQA (OOD)	Average
Grid Search	0.7321	0.4407	0.3602	0.3350	0.4670
TPE (Cold)	$0.7158\pm.0035$	$0.4843\pm.0152$	$0.3630\pm.0053$	$0.3181\pm.0090$	0.4703
TPE (Warm)	$0.7176\pm.0113$	$0.4662\pm.0198$	$\mathbf{0.3675\pm.0036}$	$\mathbf{0.3420\pm.0050}$	0.4733
IRAH (Cold)	$0.7267\pm.0210$	$0.4835\pm.0186$	$0.3542\pm.0111$	$0.3088\pm.0222$	0.4683
IRAH (Warm)	$\mathbf{0.7368\pm.0075}$	$\mathbf{0.4959\pm.0021}$	$0.3601\pm.0013$	$0.3336\pm.0131$	$\mathbf{0.4816}$

Table 2: Held-out retrieval performance across target benchmarks. Results are mean

\pm

standard deviation over three seeds. Grid Search is reported from a single run. Warm variants use meta-learned initialization, whereas Cold variants do not. CoSQA is the out-of-distribution (OOD) target. Average is the unweighted mean held-out

\mathrm{nDCG}@10

across all four benchmarks. Best result per dataset is shown in bold.

Method	SciFact	FiQA	CQADupStack- STATS	CoSQA (OOD)
Grid Search	6.75	56.92	42.83	18.02
TPE (Cold)	$5.73\pm.20$	$50.89\pm 3.19$	$37.11\pm 4.14$	$2.06\pm.47$
TPE (Warm)	$5.62\pm.41$	$\mathbf{44.71\pm 4.72}$	$38.73\pm 2.65$	$\mathbf{1.77\pm.03}$
IRAH (Cold)	$5.99\pm 1.15$	$47.41\pm 6.72$	$59.92\pm 2.75$	$5.69\pm 2.93$
IRAH (Warm)	$\mathbf{3.37\pm.11}$	$74.49\pm 20.06$	$\mathbf{33.28\pm 2.53}$	$2.89\pm.95$

Table 3: Method runtime (hours). Results are mean

\pm

standard deviation over three seeds. Grid Search is a single run. Lower is better. Best per dataset is shown in bold.

Figure 2: Representative IRAH reflection trace on SciFact . An unsuccessful mutation is followed by a reflection-guided corrective mutation that reaches a new trajectory peak.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	CV Spearman $\rho$	Type
RandomForest	0.6352	Pointwise Regressor
XGBRanker (Selected)	0.6294	Pairwise Learning-to-Rank
Ridge	0.3772	Linear Regressor

Appendix

Table 4: Out-of-fold Spearman rank correlation (

\rho

) under dataset-grouped cross-validation for candidate meta-learners.

Target Dataset	Spearman $\rho$	Top-1 Regret	Hit Rate@3	Recall@3	Config. nDCG@3
arguana	0.4432	0.0782	1.0000	0.3333	0.8684
cqadupstack/android	0.9197	0.0000	1.0000	0.6667	0.9977
cqadupstack/gis	0.9794	0.0000	1.0000	0.6667	0.9968
cqadupstack/mathematica	0.9369	0.0000	1.0000	1.0000	1.0000
cqadupstack/physics	0.9799	0.0000	1.0000	1.0000	1.0000
cqadupstack/programmers	0.9669	0.0000	1.0000	0.6667	0.9989

Appendix

Table 5: Leave-one-dataset-out evaluation of XGBRanker across the 11 meta-training datasets. Metrics evaluate predicted configuration rankings; Config. nDCG@3 denotes configuration-ranking nDCG rather than retrieval nDCG.

Feature	Normalized Gain Importance
alpha ( $\alpha$ fusion weight)	0.0988
emb_bge-large-en-v1.5	0.0666
reranker_none	0.0607
emb_bge-m3	0.0604
query_doc_len_ratio	0.0593
type_token_ratio	0.0479

Appendix

Table 6: Top-12 feature importances for the trained XGBRanker , using the default XGBoost sklearn feature_importances values (normalized gain for tree-based boosters).

Feature Group	#	Cumulative Importance	Mean Importance
Continuous Parameters ( $\alpha$ , $k$ , chunk size)	3	0.1636 (16.4%)	0.0545
Embedding Model	3	0.1572 (15.7%)	0.0524
Dataset Structural / Lexical Features	11	0.4359 (43.6%)	0.0396
Reranker Model	4	0.1323 (13.2%)	0.0331
Similarity / Diagnostics	2	0.0530 (5.3%)	0.0265
Chunking Strategy	4	0.0579 (5.8%)	0.0145

Appendix

Table 7: Aggregate feature importance by category for the XGBRanker meta-learner.

Pipeline Parameter	Option Space / Parameter Bounds
Chunking Strategy	none (size: 0, overlap: 0)
	token (256/25, 512/50, 512/100)
	sentence (256/25, 512/50, 512/100)
	recursive_character (1500/150, 3000/300, 3000/600)
Dense Embedding Model	multi-qa-mpnet-base-cos-v1 ( Song et al., 2020 )
	BAAI/bge-large-en-v1.5 ( Xiao et al., 2023 )

Appendix

Table 8: Base pipeline search space used for macro-structural search. Phase 2 performs localized numerical refinement beyond the discrete values listed here, as described below.

Figure 3: Best-so-far optimization-set

\mathrm{nDCG}@10

over the 50-evaluation budget, averaged across three seeds; shading denotes

\pm

one standard deviation. Warm variants use the same meta-learned initialization. Held-out queries are used only for final evaluation in Table 2 .

Dataset	Best seed	Final best	Search gain
SciFact	$0.7577\pm.0000$	$0.7616\pm.0025$	$+0.0039\pm.0025$
FiQA-2018	$0.4725\pm.0000$	$0.4795\pm.0003$	$+0.0070\pm.0003$
CQADupStack-STATS	$0.3795\pm.0003$	$0.3835\pm.0002$	$+0.0040\pm.0003$
CoSQA	$0.3249\pm.0000$	$0.3405\pm.0038$	$+0.0156\pm.0038$

Appendix

Table 9: Optimization-set improvement of warm-started IRAH beyond the meta-learned initialization. “Best seed” is the strongest of the three meta-learned seed configurations, and “Final best” is the strongest configuration identified over the complete search. Values are mean

\pm

standard deviation over three runs; gain is computed within each run before aggregation.

Dataset	IRAH	Best after P1	Final best	P2 gain
SciFact	Cold	$0.7705\pm.0041$	$0.7708\pm.0046$	$+0.0004\pm.0006$
	Warm	$0.7605\pm.0035$	$0.7616\pm.0025$	$+0.0011\pm.0014$
FiQA-2018	Cold	$0.4694\pm.0081$	$0.4762\pm.0028$	$+0.0068\pm.0054$
	Warm	$0.4725\pm.0000$	$0.4795\pm.0003$	$+0.0070\pm.0003$
CQADupStack-STATS	Cold	$0.3757\pm.0078$	$0.3764\pm.0078$	$+0.0007\pm.0008$
	Warm	$0.3795\pm.0003$	$0.3835\pm.0002$	$+0.0040\pm.0003$

Appendix

Table 10: Optimization-set improvement from Phase 2 local numerical refinement. “Best after P1” is the highest

\mathrm{nDCG}@10

observed by the end of Phase 1, and “Final best” is the best value observed over the complete search. Values are mean

\pm

standard deviation over three seeds. Phase 2 gain is computed within each run before aggregation.

Dataset	$d_{\mathrm{meta}}$	Nearest Source	Seed nDCG@10	$\boldsymbol{\Delta}$
SciFact	0.679	NFCorpus	0.7479	$+0.0158$
FiQA-2018	0.354	Webmasters	0.4883	$+0.0476$
CQADupStack-STATS	0.396	Mathematica	0.3610	$+0.0008$
CoSQA	2.092	SciDocs	0.2593	$-0.0757$

Appendix

Table 11: Meta-feature distance (

d_{\mathrm{meta}}

), nearest meta-training benchmark, and held-out performance of the top-ranked XGBRanker configuration.

\Delta

denotes the difference relative to the held-out performance of the configuration selected by Grid Search on the optimization split.

Dataset	Method	Opt. $\mathbf{nDCG@10}$	Held-out $\mathbf{nDCG@10}$	Embedding	Chunker	Chunk Size	Reranker	$\alpha$	$k$
SciFact	Grid Search	0.7629	0.7321	bge-large-en-v1.5	sentence	512	bge-reranker-large	0.33	20
	TPE (Cold)	0.7759	0.7160	bge-large-en-v1.5	sentence	256	bge-reranker-large	0.70	16
	TPE (Warm)	0.7710	0.7182	bge-large-en-v1.5	token	512	bge-reranker-large	0.65	10
	IRAH (Cold)	0.7757	0.7122	bge-large-en-v1.5	sentence	256	bge-reranker-large	0.62	20
	IRAH (Warm)	0.7645	0.7281	bge-large-en-v1.5	sentence	512	bge-reranker-large	0.80	30
FiQA-2018	Grid Search	0.4784	0.4407	multi-qa-mpnet-base-cos-v1	token	256	bge-reranker-v2-m3	1.00	50

Appendix

Table 12: Selected configurations for each optimization method, together with their optimization-set and held-out

\mathrm{nDCG}@10

and pipeline parameters. For stochastic methods, the configuration is taken from the run with the highest optimization-set score.

Figure 4: Trial-level optimization trajectories on FiQA-2018 (Run 0). The shaded regions indicate the Phase 1/Phase 2 schedule for IRAH (Warm) (Trials 0–11 and 12–49, respectively). IRAH (Cold) uses a longer Phase 1 schedule (Trials 0–29), transitioning to Phase 2 at Trial 30. TPE trajectories are shown over the same 50-trial budget for comparison.

Dataset	Method	Mean $\pm$ Std.	Range
SciFact	Grid Search	$18.0$	[18,18]
	TPE (Cold)	$20.0\pm 1.0$	[19,21]
	TPE (Warm)	$19.3\pm 1.2$	[18,20]
	IRAH (Cold)	$17.3\pm 2.1$	[15,19]
	IRAH (Warm)	$\mathbf{11.7\pm 0.6}$	[11,12]
FiQA-2018	Grid Search	$18.0$	[18,18]

Appendix

Table 13: Unique physical index collections required by each optimization method. Stochastic methods report mean

\pm

std. and range across three runs; Grid Search is deterministic. Lowest mean collection count per dataset is shown in bold.

Method	Method Runtime (h)	Pure Search / Eval. (h)	Collections
TPE (Warm)	$44.71$	$3.73$	$18.0$
IRAH (Cold)	$47.41$	$9.71$	$16.7$
IRAH (Warm)	$74.49$	$24.91$	$21.3$

Appendix

Table 14: Runtime diagnostics on FiQA-2018 . Method runtime corresponds to the optimization-loop wall time, while pure search time measures retrieval/evaluation time excluding explicit collection-build cost. Values are means across three runs.

Dataset	IRAH (Cold)	IRAH (Warm)	Reduction
SciFact	$594.1\text{k}\pm 9.8\text{k}$	$\mathbf{530.6\text{k}\pm 6.1\text{k}}$	$10.7\%$
FiQA-2018	$589.3\text{k}\pm 6.1\text{k}$	$\mathbf{568.3\text{k}\pm 7.3\text{k}}$	$3.6\%$
CQADupStack-STATS	$595.9\text{k}\pm 29.9\text{k}$	$\mathbf{559.5\text{k}\pm 2.7\text{k}}$	$6.1\%$
CoSQA	$581.1\text{k}\pm 10.4\text{k}$	$\mathbf{565.9\text{k}\pm 6.2\text{k}}$	$2.6\%$
Average	$590.1\text{k}$	$\mathbf{556.1\text{k}}$	$\mathbf{5.8\%}$

Appendix

Table 15: Agentic LLM token usage for IRAH across three seeds. Values report mean total tokens per run

\pm

standard deviation. Reduction denotes the decrease from Cold to Warm initialization. Grid Search and TPE do not invoke an LLM during optimization and are omitted.

Index	Dense Embedder	Chunking Strategy	Config ID	$\alpha$	$k$	Reranker Model
PART 1: Meta-Learner Training Anchors & Grid Search Baseline ( cfg_01 – cfg_36 )
1	bge-m3	token (512, 50)	cfg_01	1.00	10	none
			cfg_02	0.67	30	none
			cfg_03	0.33	20	bge-reranker-v2-m3
			cfg_04	0.00	50	ms-marco-MiniLM-L-6-v2
			cfg_05	1.00	50	bge-reranker-large

Appendix

Table 16: The 108 pipeline configurations spanning 18 physical indexes used for Grid Search. The first 36 configurations are also used for meta-learner training.

Information Retrieval

Information Retrieval studies how to find, rank, and organize relevant information in large collections of data.

See all 471 papers

RAISE: RAG Design as an Architecture Search Problem
May 28, 2026Zhen Chen, Yibing Liu, Weihao Xie +3RAG Evaluation Neural Architecture Search
City University of Hong Kong, Hong Kong SAR · Baidu Inc.
Revisiting RAG Retrievers: An Information Theoretic Benchmark
Feb 25, 2026Wenqing Zheng, Dmitri Kalaev, Noah Fatsi +5RAG Evaluation IR Evaluation
CapitalOne.
Agentic AutoRAG: RAG Pipeline Optimization through Reasoning-Driven Agents
Oct 6, 2026Lasse B. Strand, Robert Jakob, Kevin O'Sullivan +1Retrieval-Augmented Generation Large Language Model-Guided Optimization
ETH Zurich

May 28, 2026J/K move · Enter open · S save

RAISE: RAG Design as an Architecture Search Problem

Zhen Chen, Yibing Liu, Weihao Xie, Yu Liang+2

City University of Hong Kong, Hong Kong SAR · Baidu Inc.

RAG Evaluation Neural Architecture Search Hyperparameter Optimization

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) systems expose numerous design choices spanning query rewriting, chunking, retrieval depth, reranking, and context compression. In practice, these choices are often configured through heuristics, hindering systematic evaluation and reproducibility across settings. We argue that this challenge is best formulated as RAG architecture search. To support controlled and reproducible study of this problem, we introduce the RAG Intelligence Search Engine (RAISE), a comprehensive framework and benchmark for RAG hyperparameter optimization, which evaluates optimization methods for RAG pipelines under standardized search spaces and budgets. RAISE implements 13 search algorithms and evaluates them across seven public text and multimodal datasets using three random seeds. Our experiments show that optimization performance is highly task-dependent: methods that perform strongly on one dataset may not generalize consistently across others, cautioning against interpreting aggregate rankings as evidence of universally superior strategies. RAISE provides a common experimental substrate for fair, reproducible, and systematic research on RAG hyperparameter optimization.

Abstract

Information Retrieval

Explore similar work

RAISE: RAG Design as an Architecture Search Problem

Revisiting RAG Retrievers: An Information Theoretic Benchmark

Agentic AutoRAG: RAG Pipeline Optimization through Reasoning-Driven Agents