Recent literature highlights the potential of graph-based approaches within large language model (LLM) retrieval-augmented generation (RAG) pipelines for answering queries of varying complexity, particularly those that fall outside the LLM's prior knowledge. However, LLMs are prone to hallucination and often face technical limitations in handling contexts large enough to ground complex queries effectively. To address these challenges, we propose a novel iterative, feedback-driven GraphRAG architecture that leverages response quality assessment to iteratively refine outputs until a sound, well-grounded response is produced. Evaluating our approach with queries from the HotPotQA dataset, we demonstrate that this iterative RAG strategy yields responses with higher semantic quality and improved relevance compared to a single-shot baseline.
Figures & tables
Figure 1: System architecture
1. for each question:
2. extract entities using NER
3. embed the question for semantic search
4. initialize context using entity-based semantic search
5. repeat up to 4 times:
6. if first iteration:
7. retrieve initial evidence from graph
Figure 2: KGiRAG processing pseudocode
You are a helpful question-answering assistant. \n
Question: { question } \n
Context: { context } \n
Instructions: \n
- Use the information provided in the context to answer the question. \n
- Pay special attention to the ’Relevant text units’, which contain structured facts from the knowledge graph. \n
- Be direct and concise in your answer. \n
Figure 3: The prompt template supplied to the LLM.
Metric
KGiRAG with NER
KGiRAG without NER
Microsoft GraphRAG
RARR
Faithfulness
0.95 ± 0.01881
0.78 ± 0.03613
0.95 ± 0.01749
0.90 ± 0.02615
Completeness
0.86 ± 0.029802
0.69 ± 0.03663
0.62 ± 0.0396
0.41 ± 0.04145
Relevance
0.64 ± 0.042269
0.76 ± 0.0371
0.23 ± 0.03549
0.59 ± 0.04282
BERTScore
0.79 ± 0.001603
0.81 ± 0.001423
0.77 ± 0.001551
0.797 ± 0.0021
Cosine Similarity
0.325 ± 0.02692
0.52 ± 0.01828
0.075 ± 0.01831
0.33 ± 0.02427
Table 1: Comparison of retrieval architectures across evaluation metrics (mean and margin of error for computing the 95% confidence intervals).
Figure 4: Score density plots of the evaluation metrics for the evaluated architectures
Iteration
KGiRAG plus NER
KGiRAG
1
274
299
2
83
41
3
59
15
4
84
145
Table 2: Number of responses solved in each of the four iterations of the KGiRAG
Figure 5: Metrics evolution during iterations for KGiRAG