SelfGraphRAG: Bridging the Supervision Gap in Graph-Based RAG with Synthetic QA Generation

Ben Lagnese; Manas Gaur

cs.CLAug 25, 2026

SelfGraphRAG: Bridging the Supervision Gap in Graph-Based RAG with Synthetic QA Generation

Authors: Ben Lagnese, Manas Gaur

Organizations: University of Maryland, Baltimore County (UMBC), Baltimore, MD, USA

Abstract

Retrieval-augmented generation (RAG) improves large language models by incorporating external knowledge without retraining, but existing methods often underuse the relational structure encoded in knowledge graphs. Graph-based RAG can capture entity relationships, yet supervised graph retrieval typically requires labeled question-answer data that may not be available for newly constructed graphs. We address this limitation with SelfGraphRAG, a framework that generates question-answer pairs directly from knowledge graph structure and uses them to train a query-conditioned graph retriever. The generated questions capture multi-hop paths and local neighborhoods, providing relational supervision without manual annotation. Experiments on multi-hop question answering and classification benchmarks show that SelfGraphRAG improves retrieval precision and downstream reasoning performance over embedding-based baselines. These results suggest that knowledge graph structure can provide useful supervision for training graph retrievers when labeled data are unavailable.

Figures & tables

Figure 1: The SelfGraphRAG process. Indexing mirrors GraphRAG through knowledge graph summarization. SynthGen then generates a question-answer dataset to train a graph retrieval model, which retrieves information from the graph at inference time.

Figure 2: The SynthGen process. From the knowledge graph, two sets are formed: one of two-hop-connected nodes and another of 3 edges connected by singular nodes. The LLM is prompted to generate question-answer pairs from these sets, forming the SynthGen dataset.

Method	Accuracy	Precision	Recall	F1
RAG	$1.61$	$6.90$	$7.42$	$6.91$
GraphRAG	$3.85$	$9.18$	$10.56$	$9.24$
LightRAG	$3.94$	$9.20$	$11.14$	$9.23$
SelfGraphRAG	4.61	12.49	13.56	12.59

Table 1: Results on MoreHopQA . Answers include yes/no, numerical, date, and named-entity types. Precision and recall are token-level. Best result per metric is bolded .

Method	Accuracy	Precision	Recall	F1
RAG	$1.02$	$1.87$	$26.67$	$2.60$
GraphRAG	$0.00$	$0.00$	$0.16$	$0.01$
LightRAG	$0.00$	$0.50$	$25.53$	$0.98$
SelfGraphRAG	21.02	23.65	31.48	24.62

Table 2: Results on MultiHop-RAG . Answers are constrained to “Yes”, “No”, or a named entity. Token-level precision and recall are reported. Best result per metric is bolded .

Method	Overall				Yes				No				Maybe
	Acc.	Prec.	Rec.	F1	Acc.	Prec.	Rec.	F1	Acc.	Prec.	Rec.	F1	Acc.	Prec.	Rec.	F1
RAG	$51.60$	$51.60$	$51.60$	$51.60$	$51.00$	$54.72$	$92.39$	68.73	$0.00$	$0.00$	$0.00$	$0.00$	$0.60$	$8.82$	$5.46$	$6.74$
GraphRAG	$28.20$	$28.20$	$28.20$	$28.20$	$20.00$	58.82	$36.23$	$44.84$	$0.80$	66.67	$2.37$	$4.57$	$7.40$	$11.42$	67.27	19.53
LightRAG	$25.60$	$25.60$	$25.60$	$25.60$	$18.40$	$58.23$	$33.33$	$42.40$	$0.80$	$57.14$	$2.37$	$4.55$	$7.00$	$10.45$	$63.64$	$17.95$
SelfGraphRAG	55.20	55.20	55.20	55.20	78.80	$58.53$	80.80	$67.88$	29.40	$49.46$	27.22	35.12	36.60	26.92	$12.73$	$17.28$

Table 5

Method	Response
MoreHopQA: What is the date one day after when Prince Nikolai of Denmark’s mother was born? Gold: 1964-07-01
RAG	June 30, 1964
GraphRAG	Prince Nikolai of Denmark’s mother, Alexandra, was born on June 30, 1964. Therefore, the answer is June 30, 1964 .
LightRAG	Prince Nikolai’s mother is Alexandra, Countess of Frederiksborg, who was born on June 30, 1964. Therefore, the requested date is June 30, 1964 .
SelfGraphRAG	1964-07-01 . Prince Nikolai’s mother, Alexandra, was born on June 30, 1964; applying the requested one-day offset gives July 1, 1964.
PubMedQA: Is intrapartum vibroacoustic stimulation an effective predictor of fetal acidosis? Gold: Yes

Table 3: Qualitative comparison on representative MoreHopQA and PubMedQA examples.

Method	MoreHopQA		MultiHopRAG		PubMedQA
	Index (h)	Infer. (s)	Index (h)	Infer. (s)	Index (h)	Infer. (s)
RAG	$1.293$	$5.296$	$9.016$	$7.012$	$2.302$	$4.103$
GraphRAG	$4.307$	$12.571$	$23.206$	$12.524$	$6.108$	$21.600$
LightRAG	$2.014$	$1.987$	$15.849$	$2.545$	$3.493$	$8.119$
SelfGraphRAG	$4.697$	$3.177$	$23.112$	$4.455$	$6.480$	$2.266$

Table 5: Performance comparison across datasets. Index time is reported in hours (h), and inference time is reported in seconds (s) per query.

Graph Retrieval-Augmented Generation

Graph Retrieval-Augmented Generation connects language models to knowledge graphs to retrieve relational evidence for grounded, multi-hop answers.

See all 232 papers

Towards a Learnable Retriever for Textual Graph Question Answering
Oct 24, 2026Kaifeng Jin, Yuchen Yan, Zhichen Zeng +4Graph-Based Retrieval Graph Retrieval-Augmented Generation
University of Illinois Urbana-Champaign · Wayne State University · Meta
Corpus-Guided Dual-Path Propagation for Graph Retrieval-Augmented Generation
Sep 29, 2026Baoxian Liu, Tong WeiMulti-Hop QA Graph Retrieval-Augmented Generation
College of Software Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China · School of Computer Science and Engineering, Southeast University, Nanjing 210096, China
UniGFM-RAG: Unify Graph Foundation Models and Large Language Models for End-to-End Retrieval and Reasoning
Oct 26, 2026Zicheng Zhao, Linhao Luo, Zongji Yu +4Graph Retrieval-Augmented Generation LLM Reasoning with Graphs
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240, China. · Department of Data Science and AI, Monash University, Melbourne, Australia. · Tencent Youtu Lab, Shanghai 200240, China. +2

Oct 24, 2026J/K move · Enter open · S save

Towards a Learnable Retriever for Textual Graph Question Answering

Kaifeng Jin, Yuchen Yan, Zhichen Zeng, Xiao Lin+3

University of Illinois Urbana-Champaign · Wayne State University · Meta

Graph-Based Retrieval Graph Retrieval-Augmented Generation LLMs for Graph Learning

Open paper AskPDF arXiv

Textual graph question answering (GraphQA) and graph-based retrieval-augmented generation (GraphRAG) have gained increasing attention as a way to ground large language models (LLMs) on structured knowledge. Despite extensive efforts, existing methods still face two key gaps. The first is an architectural gap: existing methods typically fine-tune an LLM-based generator, but do not also make the retriever itself a learnable LLM for graph evidence selection within one end-to-end retrieve-then-generate pipeline. The second is a retrieval-supervision gap: existing retrievers rarely use gold answer signals to supervise evidence selection. As a result, they may fail to recall evidence nodes required for reasoning, resulting in insufficient coverage of the gold answer set; meanwhile, naive attempts to boost recall often introduce excessive irrelevant information that compromises compactness and overwhelms the LLM. To address these gaps, we propose TaLenT, an LLM-based end-to-end retriever-generator framework for textual GraphQA. To bridge the architectural gap, TaLenT pairs a learnable LLM-based retriever for evidence selection with a fine-tuned LLM-based generator within one unified retrieve-then-generate pipeline. To bridge the retrieval-supervision gap, we leverage gold answer signals as retriever supervision. Specifically, we stratify graph nodes into gold answers, answer-related nodes, and irrelevant noise. We then adopt a weighted pseudo-label loss to prioritize gold answers and preserve answer-related nodes for coverage, while discarding irrelevant noise for compactness. Experimental results across three GraphQA datasets demonstrate that TaLenT consistently improves downstream performance.

Abstract

Graph Retrieval-Augmented Generation

Explore similar work

Towards a Learnable Retriever for Textual Graph Question Answering

Corpus-Guided Dual-Path Propagation for Graph Retrieval-Augmented Generation

UniGFM-RAG: Unify Graph Foundation Models and Large Language Models for End-to-End Retrieval and Reasoning