IndexRAG: Index-Time Reasoning for Multi-Hop Retrieval-Augmented Generation

Zhenghua Bao; Yi Shi

cs.CLMar 17, 2026

IndexRAG: Index-Time Reasoning for Multi-Hop Retrieval-Augmented Generation

Authors: Zhenghua Bao, Yi Shi

Organizations: Continuum AI Shanghai, China · Continuum AI San Francisco, CA, USA

Abstract

Multi-hop question answering (QA) requires reasoning across multiple documents, yet existing retrieval-augmented generation (RAG) approaches address this either through graph-based methods requiring additional online processing or iterative multi-step reasoning. We present IndexRAG, a novel approach that shifts cross-document reasoning from online inference to offline indexing. IndexRAG identifies bridge entities shared across documents and generates bridging facts as independently retrievable units, requiring no additional training or fine-tuning. Experiments on three widely-used multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue) show that IndexRAG improves F1 over Naive RAG by 4.6 points on average, while requiring only single-pass retrieval and a single LLM call at inference time. When combined with IRCoT, IndexRAG achieves the best average performance among all evaluated methods, including graph-based baselines such as HippoRAG2 and FastGraphRAG, while relying on a flat vector index. Our code is available at https://github.com/Continuum-AI-Corp/IndexRAG .

Figures & tables

	Single-pass	Cross-doc	Single LLM	Training-	Index-time
	retrieval	reasoning	call	free	reasoning
Naive RAG	✓	×	✓	✓	×
HippoRAG2	×	✓	×	✓	×
IRCoT	×	✓	×	✓	×
IndexRAG (Ours)	✓	✓	✓	✓	✓

Table 1: Qualitative comparison of different RAG approaches. IndexRAG is the only method that achieves cross-document reasoning with single-pass retrieval and a single LLM call at inference time.

Figure 1: (a) Comparison of different RAG approaches during offline indexing. IndexRAG shifts reasoning to index time, generating bridging facts stored alongside AKUs in a unified vector store. (b) Example from 2WikiMultiHopQA Ho et al. (2020) . The query “Where was the director of the film Aylwin born?” requires two-hop reasoning. With naive RAG, Chunk C containing the director’s birthplace is not retrieved, causing the LLM to output the director’s name instead. With IndexRAG, a bridging fact supplying the director’s birthplace is retrieved directly, completing the reasoning chain. The panel is illustrative, with the retrieved units reported in Table 6 and analyzed in Section 5.2.1 .

Figure 2: Overview of IndexRAG. (a) Offline Indexing: In Stage 1, the LLM extracts AKUs and associated entities from each document. In Stage 2, documents sharing the same entity are identified and used to generate bridging facts. The resulting AKUs and bridging facts are encoded and stored in a unified vector store. (b) Online Inference: the user query $q$ is encoded and used to retrieve relevant context. After applying balanced context selection with $k_{b}$ , the selected context (a mix of AKUs and bridging facts) is fed to the LLM to generate the answer $a$ .

Dataset	Questions	Passages
HotpotQA	1,000	9,827
2WikiMultiHopQA	1,000	6,262
MuSiQue	1,000	9,723

Table 2: Dataset statistics for the 1,000 question subsets used in our experiments.

	HotpotQA	2Wiki	MuSiQue
Bridge entities	8,878	4,817	6,471
Bridging facts	12,608	6,821	8,010
Non-empty rate	85%	88%	79%

Table 3: Bridging fact generation statistics.

	HotpotQA			2WikiMultiHopQA			MuSiQue			Average
	EM	Acc	F1	EM	Acc	F1	EM	Acc	F1	EM	Acc	F1
BM25	47.2	51.2	60.3	31.6	32.5	35.9	9.8	10.7	19.2	29.5	31.5	38.5
Naive RAG	50.2	54.0	63.6	42.2	44.5	47.7	19.0	20.4	29.9	37.1	39.6	47.1
FastGraphRAG	50.0	53.8	63.5	49.5	54.8	57.4	17.5	18.8	27.2	39.0	42.5	49.4
RAPTOR	50.2	54.1	63.6	42.3	44.2	47.8	19.3	20.3	29.7	37.3	39.5	47.0
IndexRAG	53.9	59.3	68.9	44.8	50.3	51.7	22.4	24.4	34.4	40.4	44.7	51.7

Table 4: Multi-hop QA performance (%) on three multi-hop QA benchmarks. Best single-call results per dataset are in bold . Best multi-call results are in bold gray . Methods requiring multiple LLM calls at inference time are greyed out.

Method	EM	Time(s)	Calls
Naive RAG	19.0	0.29	1.0
FastGraphRAG	17.5	2.55	1.0
RAPTOR	19.3	0.47	1.0
IndexRAG	22.4	0.30	1.0
HippoRAG2	23.8	3.13	2.0
IRCoT + IndexRAG	24.6	1.08	3.2

Table 5: Quality-efficiency comparison (%) on MuSiQue. Gray rows indicate multi-call methods.

Query: Where was the director of the film Aylwin born?
Gold Answer: Weston-super-Mare
Naive RAG
Retrieved	[1] Aylwin is a 1920 British silent drama film directed by Henry Edwards…
	[2] Jim Wynorski (born August 14, 1950) is an American screenwriter…
	[3] Frank Launder (28 January 1906 – 23 February 1997) was a British writer, film director…
Generated	✗ Henry Edwards

Table 6: Case study from 2WikiMultiHopQA, analyzed in Section 5.2.1 . Only the top-3 results are shown, with the decisive fact highlighted in green .

Method	EM	Acc	F1
Chunking	50.2	54.0	63.6
Summary	51.7	55.4	66.0
QA extraction	53.2	57.4	67.2

Table 7: Stage 1 extraction method comparison (%) on HotpotQA without bridging facts.

Configuration	EM	Acc	F1
Naive RAG	19.0	20.4	29.9
+ Stage 2	22.3	24.4	34.4
QA Extraction	18.1	20.1	30.1
+ Stage 2	22.4	24.4	34.4

Table 8: Stage 2 bridging facts added on top of two different Stage 1 representations (%) on MuSiQue.

Dataset	Datastore	Recall@10	EM
HotpotQA	QA extraction	66.4	53.2
HotpotQA	+ Bridging	57.5	53.9
2Wiki	QA extraction	50.5	41.7
2Wiki	+ Bridging	46.7	44.8
MuSiQue	QA extraction	62.0	18.1
MuSiQue	+ Bridging	52.8	22.4

Table 9: Recall@10 vs EM (%) with and without bridging facts.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

$k_{b}$	HotpotQA	2Wiki	MuSiQue
0 (no bridging)	53.2	41.7	18.1
1	52.7	43.3	21.6
2	54.1	43.2	22.4
3	53.9	44.8	22.4
5	52.9	44.4	22.3

Appendix

Table 10: Balanced retrieval ablation (EM, %).

Dataset	Latency (s)	Index (MB)	Cost ($)
HotpotQA	–	144 / 2657	4.43 / 6.37
2Wiki	0.44 / 2.89	83 / 1426	2.60 / 5.22
MuSiQue	0.30 / 3.13	114 / 2278	3.67 / 6.56

Appendix

Table 11: End-to-end cost. Each cell is IndexRAG / HippoRAG2. Latency is retrieval time per query, and was not measured on HotpotQA. Cost is recorded directly from OpenAI API responses over the full retrieval corpus of each dataset using gpt-4o-mini pricing.

Method	EM	F1
Naive RAG	33.2	47.7
AKU-only (no bridging)	35.4	49.6
IndexRAG (AKU + bridging)	35.9	49.7

Appendix

Table 12: Single-hop sanity check on Natural Questions.

Method	2Wiki		MuSiQue
Method	EM	F1	EM	F1
Naive RAG	42.2	47.7	19.0	29.9
Retrieval-only	41.7	47.2	17.7	29.1
Full IndexRAG	44.8	51.7	22.4	34.4

Appendix

Table 13: Mechanism ablation. Bridging facts remain in the index but are removed from the final prompt in the middle row.

MuSiQue subset	N	$\Delta$ EM	$\Delta$ F1
2-hop	535	+6.2	+5.4
3-hop	318	+3.1	+4.5
4-hop	147	+1.4	$-$ 0.2

Appendix

Table 14: IndexRAG improvement by hop depth on MuSiQue.

Explore similar work

Hierarchical Abstract Tree for Cross-Document Retrieval-Augmented Generation
May 1, 2026Ziwen Zhao, Menglin YangHievi-Rag Agentic Retrieval-Augmented Generation Systems
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.
Retrieval is Cheap, Show Me the Code: Executable Multi-Hop Reasoning for Retrieval-Augmented Generation
May 13, 2026Jiashuo Sun, Jimeng Shi, Yixuan Xie +10Multi-Hop Reasoning Reasoning Trajectory
University of Illinois Urbana-Champaign · Hong Kong University of Science and Technology · Texas A&M University +1
Efficient Retrieval-Augmented Generation via Token Co-occurrence Graphs
Jun 29, 2026Gianluca Bonifazi, Christopher Buratti, Michele Marchetti +5Graph-Based Retrieval Augmented Generation Graph-Llm Integrations
aUniversità Politecnica delle Marche, Ancona, Italy · bUniversità di Modena e Reggio Emilia, Modena, Italy

May 1, 2026, cs.LGJ/K move · Enter open · S save

Hierarchical Abstract Tree for Cross-Document Retrieval-Augmented Generation

Ziwen Zhao, Menglin Yang

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.

Hievi-Rag Agentic Retrieval-Augmented Generation Systems Multi-Hop Reasoning Summarization

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) enhances large language models with external knowledge, and tree-based RAG organizes documents into hierarchical indexes to support queries at multiple granularities. However, existing Tree-RAG methods designed for single-document retrieval face critical challenges in scaling to cross-document multi-hop questions: (1) poor distribution adaptability, where

k

-means clustering introduces noise due to rigid distribution assumptions; (2) structural isolation, as tree indexes lack explicit cross-document connections; and (3) coarse abstraction, which obscures fine-grained details. To address these limitations, we propose

Ψ

-RAG, a tree-RAG framework with two key components. First, a hierarchical abstract tree index built through an iterative "merging and collapse" process that adapts to data distributions without a priori assumption. Second, a multi-granular retrieval agent that intelligently interacts with the knowledge base with reorganized queries and an agent-powered hybrid retriever.

Ψ

-RAG supports diverse tasks from token-level question answering to document-level summarization. On cross-document multi-hop QA benchmarks, it outperforms RAPTOR by 25.9% and HippoRAG 2 by 7.4% in average F1 score. Code is available at https://github.com/Newiz430/Psi-RAG.

Abstract

Explore similar work

Hierarchical Abstract Tree for Cross-Document Retrieval-Augmented Generation

Retrieval is Cheap, Show Me the Code: Executable Multi-Hop Reasoning for Retrieval-Augmented Generation

Efficient Retrieval-Augmented Generation via Token Co-occurrence Graphs