Do We Still Need GraphRAG? Benchmarking RAG and GraphRAG for Agentic Search Systems

Dongzhe Fan; Zheyi Xue; Siyuan Liu; Qiaoyu Tan

cs.IRApr 1, 2026

Do We Still Need GraphRAG? Benchmarking RAG and GraphRAG for Agentic Search Systems

Authors: Dongzhe Fan, Zheyi Xue, Siyuan Liu, Qiaoyu Tan

Organizations: New York University Shanghai, Shanghai, China

Abstract

Retrieval-augmented generation (RAG) and its graph-based extensions (GraphRAG) are effective paradigms for improving large language model (LLM) reasoning by grounding generation in external knowledge. However, most existing RAG and GraphRAG systems operate under static or one-shot retrieval, where a fixed set of documents is provided to the LLM in a single pass. In contrast, recent agentic search systems enable dynamic, multi-round retrieval and sequential decision-making during inference, and have shown strong gains when combined with vanilla RAG by introducing implicit structure through interaction. This progress raises a fundamental question: can agentic search compensate for the absence of explicit graph structure, reducing the need for costly GraphRAG pipelines? To answer this question, we introduce RAGSearch, a unified benchmark that evaluates dense RAG and representative GraphRAG methods as retrieval infrastructures under agentic search. RAGSearch covers both training-free and training-based agentic inference across multiple question answering benchmarks. To ensure fair and reproducible comparison, we standardize the LLM backbone, retrieval budgets, and inference protocols, and report results on full test sets. Beyond answer accuracy, we report offline preprocessing cost, online inference efficiency, and stability. Our results show that agentic search substantially improves dense RAG and narrows the performance gap to GraphRAG, particularly in RL-based settings. Nevertheless, GraphRAG remains advantageous for complex multi-hop reasoning, exhibiting more stable agentic search behavior when its offline cost is amortized. Together, these findings clarify the complementary roles of explicit graph structure and agentic search, and provide practical guidance on retrieval design for modern agentic RAG systems.

Figures & tables

Figure 1. Explicit vs. Implicit Structure in RAG Systems. GraphRAG relies on explicit graph construction, whereas agentic search over dense RAG can induce implicit evidence structure through multi-round retrieval and reasoning.

Figure 2. Overview of the RAGSearch benchmark. RAGSearch models agentic search as an LLM agent interacting with interchangeable retrieval backends (dense RAG or GraphRAG) through a unified interface, and benchmarks both training-free agentic search and RL-based agentic search under standardized protocols.

System	Methods	General QA			Multi-Hop QA
		NQ †	PopQA ⋆	TriviaQA ⋆	HotpotQA †	2Wiki ⋆	Musique ⋆
Single-shot	Qwen-2.5-7B-Dense	46.62	32.14	58.60	19.00	35.53	20.99
	Qwen-2.5-7B-GraphRAG ♠	48.31 ( $\bm{\uparrow}1.69$ )	32.82 ( $\bm{\uparrow}0.68$ )	57.65 ( $\bm{\downarrow}0.95$ )	46.70 ( $\bm{\uparrow}27.70$ )	62.56 ( $\bm{\uparrow}27.03$ )	47.95 ( $\bm{\uparrow}26.96$ )
Training-free	Search-o1-7B-Dense	38.20	25.57	58.74	33.76	29.64	12.62
	Search-o1-7B-GraphRAG ♠	38.34 ( $\bm{\uparrow}0.14$ )	28.01 ( $\bm{\uparrow}2.44$ )	59.50 ( $\bm{\uparrow}0.76$ )	42.75 ( $\bm{\uparrow}8.99$ )	65.56 ( $\bm{\uparrow}35.92$ )	32.44 ( $\bm{\uparrow}19.82$ )
	GraphSearch-7B-Dense	58.27	36.29	68.70	38.22	47.43	13.33

Table 1. Overall Contain Exact Match (EM) for single-shot inference, training-free agentic systems, and RL-based agentic systems.

\spadesuit

denotes the best result among the five GraphRAG variants.

\uparrow

and

\downarrow

indicate, for the same method, the performance difference between graph-based and dense-retrieval-based RAG; † denotes the in-domain dataset, and ⋆ denotes the cross-domain dataset.

Method	Knowledge Base	General QA			Avg Rank	Multi-Hop QA			Avg Rank
		NQ	PopQA	TriviaQA		HotpotQA	2Wiki	Musique
Search-o1	Dense	38.20	25.78	58.74	2.33	33.76	29.64	12.62	5.33
	HypergraphRAG	33.02	25.57	56.72	5.33	33.90	50.58	28.05	3.67
	HippoRAG2	38.34	28.01	59.50	1.00	42.75	65.56	32.44	1.00
	LinearRAG	34.32	25.69	57.03	4.00	35.76	58.94	29.46	2.33
	RAPTOR	34.82	23.20	52.52	5.33	29.51	29.87	29.50	4.33

Table 2. Overall Contain EM results of training-free agentic systems across different retriever backends. and indicate the best and second-best results within each method, respectively. means dense-RAG, means tree-based GraphRAG, means entity graph based GraphRAG, means hypergraph based GraphRAG and means tri-graph based GraphRAG. Avg. Rank is computed within each method (lower is better).

Figure 3. Performance of different retrieval infrastructures on RL-based agentic systems

Method	HotpotQA			PopQA
Method	Search Turn	Recall	Variance	Search Turn	Recall	Variance
Search-o1-Dense	2.20	79.38	33.65 $\pm$ 1.03	1.53	76.33	25.62 $\pm$ 0.61
Search-o1-HippoRAG2	2.03	80.27	42.36 $\pm$ 0.22	1.52	78.12	27.81 $\pm$ 0.36
Search-R1	1.82	81.67	34.82 $\pm$ 0.95	1.36	77.15	33.15 $\pm$ 0.54
Graph-R1-HippoRAG2	1.71	83.50	53.71 $\pm$ 0.18	1.38	78.61	36.13 $\pm$ 0.32

Table 3. Sensitivity Analysis on agentic systems

Panel of Figure 4 — Figure 4. The impact of different RL algorithms

Method	HotpotQA	Musique	NQ	PopQA
Search-R1-3B	23.67	4.96	30.50	25.34
Graph-R1-3B ♠	42.75	31.73	37.72	31.08
Search-R1-7B	35.76	12.35	46.35	32.90
Graph-R1-7B ♠	51.75	36.53	42.12	35.95

Table 4. 3B vs 7B on RL-based agentic systems.

Method	HotpotQA	Musique	NQ	PopQA
Search-o1-Dense-7B	33.76	12.62	38.20	25.78
Search-o1-GraphRAG-7B ♠	42.75	32.44	38.34	28.01
Search-o1-Dense-32B	40.85	18.95	51.80	36.20
Search-o1-GraphRAG-32B ♠	47.01	42.37	50.50	36.69

Table 5. 7B vs 32B on training-free systems.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Task	Knowledge Source	#Train	#Dev	#Test
NQ	General QA	Wiki	79,168	8,757	3,610
PopQA	General QA	Wiki	-	-	14,267
TriviaQA	General QA	Wiki & Web	78,785	8,837	11,313
HotpotQA	Multi-hop QA	Wiki	90,447	7,405	-
Musique	Multi-hop QA	Wiki	19,938	2,417	-
2Wiki	Multi-hop QA	Wiki	15,000	12,576	-

Appendix

Table 6. Dataset Statistics

System	Methods	General QA		Multi-Hop QA
		NQ †	TriviaQA ⋆	HotpotQA †	Musique ⋆
Single-shot	Qwen-2.5-7B-Dense	39.3	61.12	20.12	29.08
	Qwen-2.5-7B-GraphRAG ♠	27.92 ( $\bm{\downarrow}11.38$ )	49.25 ( $\bm{\downarrow}11.87$ )	33.72 ( $\bm{\uparrow}13.60$ )	41.13 ( $\bm{\uparrow}12.05$ )
Training-free	Search-o1-7B-Dense	36.53	59.73	39.08	17.46
	Search-o1-7B-GraphRAG ♠	36.62 ( $\bm{\uparrow}0.09$ )	58.18 ( $\bm{\downarrow}1.55$ )	41.57 ( $\bm{\uparrow}2.49$ )	37.01 ( $\bm{\uparrow}19.55$ )
	GraphSearch-7B-Dense	8.70	13.12	6.02	2.36

Appendix

Table 7. Overall F-1 of different systems

Method	Knowledge Construction		Offline Inference
Method	TM	CM	RT/s	CT/token
HypergraphRAG	1.37h	3.93$	0.77	1680
HippoRAG2	1.19h	2.85$	1.00	3229
LinearRAG	0.68h	0$	1.18	4600
RAPTOR	1.70h	6.38$	8.4	814
GraphRAG	1.72h	13.19$	1.16	22160

Appendix

Table 8. Comparison on cost of GraphRAGs of NQ. TM indicates construction time per 1M token; CM is the cost per 1M token, RT is the average retrieval time and CT is the average context length.

Question	"Are the Laleli Mosque and Esma Sultan Mansion located in the same neighborhood?"
Golden Answer	"No"
qwen2.5-7B-Ins	Search-o1-Dense	Search-R1	Graph-R1-HippoRAG2
Generation	To answer the question of whether the Laleli Mosque and Esma Sultan Mansion are located in the same neighborhood, I need to find the locations of both structures.Assistant: <\|begin_search_query\|> location of Laleli Mosque <\|end_search_query\|> <\|begin_search_result\|> The Laleli Mosque is located in Laleli, Fatih, Istanbul, Turkey… <\|end_search_result\|> Assistant: Based on the information gathered…{No}	<think> To answer the question “Are the Laleli Mosque and Esma Sultan Mansion located in the same neighborhood?”, I will first reason through what I know about these two landmarks. I know that Laleli Mosque is a significant historical site in Istanbul, Turkey. I need more information about the location of Esma Sultan Mansion to determine if it shares a neighborhood with Laleli Mosque. </think> <search> “Esma Sultan Mansion location Istanbul Turkey” </search> <information> >Doc 1(Title: “Esma Sultan Mansion”):… </information> <think> … Based on my search results: Laleli Mosque is in Laleli neighborhood. Esma Sultan Mansion is in Ortaköy neighborhood. Since Laleli Mosque is in Laleli neighborhood and Esma Sultan Mansion is in Ortaköy neighborhood, they are not located in the same neighborhood. </think> <answer> No </answer>	<think> To answer this question, I need to determine the neighborhoods where the Laleli Mosque and Esma Sultan Mansion are located. I will first search for the location of the Laleli Mosque and then for the Esma Sultan Mansion, and finally compare their neighborhoods. </think> <query> {"query": "Laleli Mosque neighborhood"} </query> <knowledge> Wikipedia Title: Laleli Mosque… </knowledge> <think> According to the Wikipedia information retrieved, the Laleli Mosque is located in Laleli, Fatih, Istanbul. Now, I need to find the neighborhood where Esma Sultan Mansion is located. </think> … <answer> No </answer>

Appendix

Table 9. Case study of different agentic systems.

Panel of Figure 5 — Figure 5. Instruction for Search-o1

Panel of Figure 6 — Figure 6. Instruction for Reason-in-Documents

Panel of Figure 7 — Figure 7. Instruction for Open-Domain QA Tasks

Panel of Figure 8 — Figure 8. GraphSearch Query Decomposition

Panel of Figure 9 — Figure 9. GraphSearch Query Decomposition (KG)

Panel of Figure 10 — Figure 10. GraphSearch Evidence Verification

Panel of Figure 11 — Figure 11. GraphSearch Deep Answer Generation

Panel of Figure 12 — Figure 12. GraphSearch Query Expansion

Panel of Figure 13 — Figure 13. Search-R1 Instruction

Panel of Figure 14 — Figure 14. Graph-R1 Instruction

Explore similar work

Is GraphRAG Needed? From Basic RAG to Graph-/Agentic Solutions with Context Optimization
Jun 24, 2026Long Chen, Ryan Razkenari, Yuxuan Zhou +5Retrieval-Augmented Generation Graph Retrieval-Augmented Generation
1Generative AI Innovation Center, Amazon Web Services (AWS) · 2Cisco Systems, Inc.
Towards Robust Retrieval-Augmented Generation Based on Knowledge Graph: A Comparative Analysis
Mar 5, 2026Hazem Amamou, Stéphane Gagnon, Alan Davoust +1RAG Robustness Graph Retrieval-Augmented Generation
Institut national de la recherche scientifique (INRS-EMT), Montréal, Québec, Canada · INRS-UQO Mixed Research Unit on Cybersecurity, Gatineau, Québec, Canada · Université du Québec en Outaouais, Gatineau, Québec, Canada
Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training
Feb 26, 2026Tianle Xia, Ming Xu, Lingxiang Hu +7Agentic RAG LLM Agent Training
Tencent

Jun 24, 2026J/K move · Enter open · S save

Is GraphRAG Needed? From Basic RAG to Graph-/Agentic Solutions with Context Optimization

Long Chen, Ryan Razkenari, Yuxuan Zhou, Yuan Tian+4

1Generative AI Innovation Center, Amazon Web Services (AWS) · 2Cisco Systems, Inc.

Retrieval-Augmented Generation Graph Retrieval-Augmented Generation Agentic RAG RAG Evaluation LLM Context Management

Open paper AskPDF arXiv

As advanced RAG variants like GraphRAG and Agentic RAG emerge, one leading question is when and how to use them. Here, we introduce a framework for different RAG scenarios evaluation and comparison on semi-structured knowledge bases, including regular RAG, GraphRAG, Modular RAG and Agentic RAG. We provide implementation for 9 standardized RAG scenarios, and conduct experiments for a comprehensive comparison. These scenarios are designed for real use cases regarding data and domain restrictions, spanning from simple document-based retrieval to advanced features such as hybrid text-graph retrieval, integration with computed or pre-defined domain knowledge graphs, agentic multi-step planning, and agent-graph integration. Besides, we present a novel context engineering method for GraphRAG and Agentic RAG, addressing the context/memory overflow issues, efficiently managing text and graph retrievals with new representations and agentic loop design, leading to 19%-53% reduction on token usage. Moreover, further analysis identifies a retrieval-generation gap where expanded retrieval does not proportionally improve generation quality, suggesting retrieval-oriented metrics overstate advanced retrieval benefits. This work provides data-driven insights on when and how to use them for building production-ready intelligent RAG systems.

Abstract

Explore similar work

Is GraphRAG Needed? From Basic RAG to Graph-/Agentic Solutions with Context Optimization

Towards Robust Retrieval-Augmented Generation Based on Knowledge Graph: A Comparative Analysis

Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training