Towards Robust Retrieval-Augmented Generation Based on Knowledge Graph: A Comparative Analysis
Authors: Hazem Amamou, Stéphane Gagnon, Alan Davoust, Anderson R. Avila
Organizations: Institut national de la recherche scientifique (INRS-EMT), Montréal, Québec, Canada · INRS-UQO Mixed Research Unit on Cybersecurity, Gatineau, Québec, Canada · Université du Québec en Outaouais, Gatineau, Québec, Canada
Retrieval-Augmented Generation (RAG) was introduced to enhance the capabilities of Large Language Models (LLMs) beyond their encoded prior knowledge. This is achieved by providing LLMs with an external source of knowledge, which helps reduce factual hallucinations and enables access to new information not available during pretraining. However, inconsistent retrieved information can negatively affect LLM responses. The Retrieval-Augmented Generation Benchmark (RGB) was introduced to evaluate the robustness of RAG systems under such conditions. In this work, we use the RGB corpus to evaluate LLMs in four scenarios: noise robustness, information integration, negative rejection, and counterfactual robustness. We perform a comparative analysis between the RGB RAG baseline and GraphRAG, a knowledge graph based retrieval system. We test three GraphRAG customizations to improve robustness. Results show improvements over the RGB baseline and provide insights for designing more reliable RAG systems for real world scenarios.
Figures & tables
Fig. 1: Comparative Framework of RGB Benchmark, GraphRAG, and RobustGraphRAG for Evaluating Robustness in Retrieval-Augmented Generation
Model
Description
RGB
Standard RGB baseline
GR RGB
GraphRAG with RGB prompt
GR def
GraphRAG with its default prompt
GR ext
Customized GraphRAG with only external knowledge
GR comb
Customized GraphRAG with encoded and external knowledge
TABLE I: Descriptions of the structured knowledge based approaches assessed in our experiments.
Fig. 4: Comparative analysis for the noise robustness task measured by accuracy (%) for GPT-3.5 and GPT-4o-mini under varying noise ratios.
ACC
Acc doc
ED*
CR
GPT3.5
RGB
93.00
9.00
7.00
57.14
GR RGB
N/A
1.00
4.00
25.00
GR def
N/A
7.00
60.60
95.00
GR ext
N/A
3.00
15.15
13.33
GR comb
N/A
24.24
94.94
95.74
GPT4omini
RGB
96.00
71.00
77.00
100.00
TABLE II: Comparative analysis for the counterfactual robustness task. ACC represents the accuracy (%) of LLMs without external knowledge. ACC doc is the accuracy (%) of LLMs with counterfactual documents. ED* is the error detection rate and CR is the error correction rate.
Noise Ratio
0.0
0.2
0.4
GPT3.5
RGB
55.00
51.00
34.00
GR RGB
83.83
79.00
71.00
GR def
81.81
77.00
72.00
GR ext
86.86
83.00
78.00
GR comb
86.86
83.00
75.00
TABLE III: Comparative analysis for the information integration task measured in terms of accuracy (%) for different noise ratios..
Fig. 5: Comparative analysis for the negative rejection task measured by rejection rate (%) for GPT-4o-mini and GPT-3.5.
National Engineering Research Center for Software Engineering, Peking University Beijing, China · PX Securities Shenzhen, China · City University of Hong Kong Hong Kong SAR, China +2